Implementation of Flutter Http Block Download and Breakpoint Continued Transfer

This article comes from "Flutter Practical Combat" written by the author, and readers can also click to view the online electronic version.

Basic knowledge

The Http protocol defines the response header field for block transmission, but whether it is supported depends on the implementation of Server. We can specify the "range" field in the request header to verify whether the server supports block transmission. For example, we can use the curl command to verify:

bogon:~ duwen$ curl -H "Range: bytes=0-10" http://download.dcloud.net.cn/HBuilder.9.0.2.macosx_64.dmg -v
#  Request header 
> GET /HBuilder.9.0.2.macosx_64.dmg HTTP/1.1
> Host: download.dcloud.net.cn
> User-Agent: curl/7.54.0
> Accept: */*
> Range: bytes=0-10
# Response header 
< HTTP/1.1 206 Partial Content
< Content-Type: application/octet-stream
< Content-Length: 11
< Connection: keep-alive
< Date: Thu, 21 Feb 2019 06:25:15 GMT
< Content-Range: bytes 0-10/233295878

The purpose of adding "Range: bytes=0-10" to the request header is to tell the server that we only want to get the file 0-10 (including 10, 11 bytes in total) in this request. If the server supports block transmission, the response status code is 206, which means "partial content", and at the same time, the change in the response header will contain the field "Content-Range", and if it does not support it, it will not be included. Let's look at the contents of "Content-Range" above:

Content-Range: bytes 0-10/233295878

0-10 represents the block returned this time, and 233295878 represents the total length of the file, and the unit is byte, that is, the file is about 1 point more than 233M.

Realization

To sum up, we can design a simple multi-threaded file block downloader, and the idea of implementation is:

First, detect whether block transmission is supported, and if not, download it directly; If it is supported, the remaining content will be downloaded in blocks. Each block is saved to its own temporary file when downloading, and the temporary file is merged after all blocks are downloaded. Delete temporary files.

Here's the overall process:

//  Adopt the 1 Block requests detect whether the server supports block transmission  
Response response = await downloadChunk(url, 0, firstChunkSize, 0);
if (response.statusCode == 206) {  // If supported 
  // Parse the total length of the file, and then calculate the remaining length 
  total = int.parse(
    response.headers.value(HttpHeaders.contentRangeHeader).split("/").last);
  int reserved = total -
    int.parse(response.headers.value(HttpHeaders.contentLengthHeader));
  // Total number of blocks of file ( Including 1 Block )
  int chunk = (reserved / firstChunkSize).ceil() + 1;
  if (chunk > 1) {
    int chunkSize = firstChunkSize;
    if (chunk > maxChunk + 1) {
      chunk = maxChunk + 1;
      chunkSize = (reserved / maxChunk).ceil();
    }
    var futures = <Future>[];
    for (int i = 0; i < maxChunk; ++i) {
      int start = firstChunkSize + i * chunkSize;
      // Download the remaining files in blocks  
      futures.add(downloadChunk(url, start, start + chunkSize, i + 1));
    }
    // Wait for all blocks to be downloaded 
    await Future.wait(futures);
  }
  // Merge file file  
  await mergeTempFiles(chunk);
}

Next we implement downloadChunk using download API of the famous Http library dio under Flutter:

//start  Represents the starting position of the current block, end Represents the end position 
//no  Represents the current block 
Future<Response> downloadChunk(url, start, end, no) async {
 progress.add(0); //progress Record every 1 The length of data received by the block 
 --end;
 return dio.download(
  url,
  savePath + "temp$no", // Temporary files are named according to the serial number of blocks, which is convenient for final merging 
  onReceiveProgress: createCallback(no), //  Create a progress callback, which is implemented later 
  options: Options(
   headers: {"range": "bytes=$start-$end"}, // Specify the requested content interval 
  ),
 );
}

Next, implement mergeTempFiles:

Future mergeTempFiles(chunk) async {
 File f = File(savePath + "temp0");
 IOSink ioSink= f.openWrite(mode: FileMode.writeOnlyAppend);
 // Merge temporary files  
 for (int i = 1; i < chunk; ++i) {
  File _f = File(savePath + "temp$i");
  await ioSink.addStream(_f.openRead());
  await _f.delete(); // Delete temporary files 
 }
 await ioSink.close();
 await f.rename(savePath); // Rename the merged file to its real name 
}

Let's look at 1 complete implementation:

/// Downloading by spiting as file in chunks
Future downloadWithChunks(
 url,
 savePath, {
 ProgressCallback onReceiveProgress,
}) async {
 const firstChunkSize = 102;
 const maxChunk = 3;

 int total = 0;
 var dio = Dio();
 var progress = <int>[];

 createCallback(no) {
  return (int received, _) {
   progress[no] = received;
   if (onReceiveProgress != null && total != 0) {
    onReceiveProgress(progress.reduce((a, b) => a + b), total);
   }
  };
 }

 Future<Response> downloadChunk(url, start, end, no) async {
  progress.add(0);
  --end;
  return dio.download(
   url,
   savePath + "temp$no",
   onReceiveProgress: createCallback(no),
   options: Options(
    headers: {"range": "bytes=$start-$end"},
   ),
  );
 }

 Future mergeTempFiles(chunk) async {
  File f = File(savePath + "temp0");
  IOSink ioSink= f.openWrite(mode: FileMode.writeOnlyAppend);
  for (int i = 1; i < chunk; ++i) {
   File _f = File(savePath + "temp$i");
   await ioSink.addStream(_f.openRead());
   await _f.delete();
  }
  await ioSink.close();
  await f.rename(savePath);
 }

 Response response = await downloadChunk(url, 0, firstChunkSize, 0);
 if (response.statusCode == 206) {
  total = int.parse(
    response.headers.value(HttpHeaders.contentRangeHeader).split("/").last);
  int reserved = total -
    int.parse(response.headers.value(HttpHeaders.contentLengthHeader));
  int chunk = (reserved / firstChunkSize).ceil() + 1;
  if (chunk > 1) {
   int chunkSize = firstChunkSize;
   if (chunk > maxChunk + 1) {
    chunk = maxChunk + 1;
    chunkSize = (reserved / maxChunk).ceil();
   }
   var futures = <Future>[];
   for (int i = 0; i < maxChunk; ++i) {
    int start = firstChunkSize + i * chunkSize;
    futures.add(downloadChunk(url, start, start + chunkSize, i + 1));
   }
   await Future.wait(futures);
  }
  await mergeTempFiles(chunk);
 }
}

Now you can download in blocks:

main() async {
 var url = "http://download.dcloud.net.cn/HBuilder.9.0.2.macosx_64.dmg";
 var savePath = "./example/HBuilder.9.0.2.macosx_64.dmg";
 await downloadWithChunks(url, savePath, onReceiveProgress: (received, total) {
  if (total != -1) {
   print("${(received / total * 100).floor()}%");
  }
 });
}

Thinking

Can block downloading really improve the download speed?

In fact, the main bottleneck of download speed depends on the network speed and the exit speed of the server. If it is the same data source, the significance of block download is not great, because the server is the same, and the export speed is determined, which mainly depends on the network speed. The above example is officially homologous block download, and readers can compare the download speed of block and non-block under 1. If there are multiple download sources, and the exit bandwidth of each download source is limited, then block download may be faster by 1 time. The reason why it is "possible" is that it is not fixed. For example, there are three sources, and the exit bandwidth of all three sources is 1G/s, while the peak value of the network connected to our equipment is assumed to be only 800M/s, so the bottleneck lies in our network. Even if the bandwidth of our device is greater than any one source, If the download speed is still not fixed, it will be faster than single-source single-line download. Imagine 1. Suppose there are two sources A and B, and the speed of A source is 3 times that of B source. If block download is adopted and two sources download 1.5 each, readers can calculate the required download time under 1, and then calculate the time required to download only from A source under 1 to see which is faster.

The final speed of block download is affected by many factors, such as the network bandwidth of the device, the speed of the source and exit, the size of each block, and the number of blocks, etc. It is difficult to ensure the optimal speed in the actual process. In actual development, readers can test and compare before deciding whether to use it.

Is there any practical use in downloading in blocks?

There is also a comparative scenario for block downloading, which is breakpoint continuous transmission. You can divide the file into several blocks, and then maintain a download status file to record the status of each block, so that even after the network interruption, you can restore the state before the interruption. Readers can try it themselves, or there are 1 details that need special attention, such as how much is the appropriate block size? What about downloading blocks to 1 and a half? Do you want to maintain 1 task queue?