Implementation Code of Breakpoint Continued Transmission for C File Download

Note that the breakpoint continued transmission mentioned in this paper refers specifically to the breakpoint continued transmission in HTTP protocol. This article mainly talks about ideas and key codes. Please refer to demo accompanying this article for more details.

Working principle

There are 1 request/response headers defined in the HTTP protocol, and these header information is used in combination. We can request only one part of the data in one file in one HTTP request. In this way, we can save the downloaded data, only request the remaining data next time, and complete the merge after all the data is downloaded locally.

The HTTP protocol indicates that the range of requested data can be specified by the Range header in the HTTP request, and the use of the Range header is also very simple, as long as the following format is specified:

Range: bytes=500-999

It means that only the 500 bytes from the 500th to the 999th of the target file are requested.

For example, I have a 1000 bytes file to download, and I don't need to specify an Range header on the first request, which means downloading the whole file. But after downloading the 499th byte, the download was cancelled. Then, when you request to download the same file next time, you only need to download the data from the 500th byte to the 999th byte. The principle seems simple, but we need to consider the following questions:

1. Do all web servers support Range headers?

2. There may be a long time between multiple requests. What if the files on the server change?

3. How to save some downloaded data and related information?

4. When we spell a file into its original size through byte operation, how can we verify that it is identical to the source file?

Let's take these questions to explore some details of breakpoint continuation.

Check server-side support for breakpoint continued transmission

When the server responds to our request, Accept-Ranges will indicate in the response header whether to accept 1 part of the data of the requested 1 resource. But there seems to be a small trap here, that is, different servers may return different values to indicate that they can accept requests for some resources. It seems that the method of comparing unification 1 is that when the server does not support requesting some data, it will return Accept-Ranges: none, so we only need to judge whether this return value is equal to none. The code is as follows:

private static bool IsAcceptRanges(WebResponse res)
{
  if (res.Headers["Accept-Ranges"] != null)
  {
    string s = res.Headers["Accept-Ranges"];
    if (s == "none")
    {
      return false;
    }
  }
  return true;
}

Check whether the server-side files change

When we download part 1 of a file, it may be downloaded immediately, it may be downloaded after a while, or it may never be downloaded again …
The problem here is how to determine that the file on the server is the same one and a half files that were downloaded at the beginning when the next download is to proceed. If the files on the server have been updated, you need to download them from scratch anyway. Breakpoint continuation is meaningful only if the files on the server have not changed.
For this problem, the HTTP response header provides us with different choices. Both ETag and Last-Modified can accomplish the task.

Look at ETag first:

The ETag response-header field provides the current value of the entity tag the requested variant (quoted from RFC 2616 14.19 ETag)

Simply put, ETag is a string that identifies the content of the current request. When the requested resource changes, the corresponding ETag will also change. Well, the easiest way is to save the ETag in the response header on the first request and compare it on the next request. The code is as follows:

string newEtag = GetEtag(response);
// tempFileName Refers to the contents of some files that have been downloaded locally 
// tempFileInfoName Refers to the preservation of Etag Temporary file of contents 
if (File.Exists(tempFileName) && File.Exists(tempFileInfoName))
{
  string oldEtag = File.ReadAllText(tempFileInfoName);
  if (!string.IsNullOrEmpty(oldEtag) && !string.IsNullOrEmpty(newEtag) && newEtag == oldEtag)
  {
  // Etag There is no change, and it can be continued at breakpoint 
    resumeDowload = true;
  }
}
else
{
  if (!string.IsNullOrEmpty(newEtag))
  {
    File.WriteAllText(tempFileInfoName, newEtag);
  }
}
private static string GetEtag(WebResponse res)
{
  if (res.Headers["ETag"] != null)
  {
    return res.Headers["ETag"];
  }
  return null;
}

Let's look at Last-Modified again:

The Last-Modified entity-header field indicates the and which the the origin HTTP the variant was last modified. (Quoted from RFC 2616 14.29 Last-Modified)

Last-Modified is the last time the requested resource was modified on the server. The usage method is roughly the same as ETag.

Personally, I feel that using either ETag or Last-Modified can achieve our goal. But you can also use both, do double check, who knows web server implementation is strictly followed HTTP protocol!

Save intermediate results

Here is the main use of C # file operations. The general idea is that if there are unfinished downloaded files, the newly downloaded bytes will be added to the end of the file, and no longer wordy. Interested students please look directly at demo code.

Verification file

In the process of breakpoint transmission, we take byte as the unit to download and merge files. If there is a slight exception that is not handled well in the whole process, the final file may not be as good as the source file. Therefore, it is best to be able to check the downloaded files once. But this is also the hardest and least easy to achieve. Because it requires server-side support, such as MD5 hash, which provides a downloadable file at the same time. Of course, if the server side is also created by ourselves, we can implement it. But how can we ask all the existing web servers to provide such functions?

Demo Download