Golang USES http Client to download files

When I used beego's http library, I sometimes needed to download files. beego is possible, but it has a few problems: it doesn't support callbacks and doesn't show download speeds, which is intolerable in everyday development.

The implementation of beego mainly USES the io.copy function, so I took a deep look at the implementation principle, found it is quite simple, so I implemented a simple downloader according to the principle of ES9en.copy

// Define the file to download 
var durl = "https://dl.google.com/go/go1.10.3.darwin-amd64.pkg";
// parsing url
uri, err := url.ParseRequestURI(durl)
if err != nil {
 panic(" Site error ")
}

For a normal flow, check for errors in Url

filename := path.Base(uri.Path)
log.Println("[*] Filename " + filename)

Use path. Base to retrieve the file name of url. Here is a place where bug may appear: if the url is 302, you cannot retrieve the file name after the jump.

Get the file name step can send http request when client.CheckRedirect definition 1 function to fetch the file name.

client := http.DefaultClient;
client.Timeout = time.Second * 60 // Set the timeout 
resp, err := client.Get(durl)

One httpClient is created, and the Timeout client sets the timeout for reading data.

Here I prefer to use Do method to pass 1 Reqeust in the past, because some url need to verify the http header, you ask me why not use 1 word lazy.

raw := resp.Body
defer raw.Close()
reader := bufio.NewReaderSize(raw, 1024*32);

To be honest, it's not clear that bufio can actually speed up htpp's reading speed. No comparison has been made with bufio. But it was added for psychological reassurance

file, err := os.Create(filename)
if err != nil {
 panic(err)
}
writer := bufio.NewWriter(file)

It is also not clear how much speed bufio can add to file writes

buff := make([]byte, 32*1024)
written := 0
go func() {
 for {
  nr, er := reader.Read(buff)
  if nr > 0 {
   nw, ew := writer.Write(buff[0:nr])
   if nw > 0 {
    written += nw
   }
   if ew != nil {
    err = ew
    break
   }
   if nr != nw {
    err = io.ErrShortWrite
    break
   }
  }
  if er != nil {
   if er != io.EOF {
    err = er
   }
   break
  }
 }
 if err != nil {
  panic(err)
 }
}()

This source code is I directly copy the ES51en. copyBuffer function just made a few simple changes to understand the general meaning of the line

// Time interval between 
spaceTime := time.Second * 1
// The timer 
ticker := time.NewTicker(spaceTime)
// Last read data size 
lastWtn := 0
stop := false

for {
 select {
 case <-ticker.C:
  // The file size of this read - The size of the last read = speed 
  speed := written - lastWtn
  log.Printf("[*] Speed %s / %s \n", bytesToSize(speed), spaceTime.String())
  if written-lastWtn == 0 {
   ticker.Stop()
   stop = true
   break
  }
  lastWtn = written
 }
 if stop {
  break
 }
}

The code blocks the program, and the timer calculates the speed at intervals. The place where bug might appear here is if it jumps out of the loop at intervals when no data is read. Now this time is very sleepy, write blog is already sleepy to die when you have time to solve this bug

func bytesToSize(length int) string {
 var k = 1024 // or 1024
 var sizes = []string{"Bytes", "KB", "MB", "GB", "TB"}
 if length == 0 {
  return "0 Bytes"
 }
 i := math.Floor(math.Log(float64(length)) / math.Log(float64(k)))
 r := float64(length) / math.Pow(float64(k), i)
 return strconv.FormatFloat(r, 'f', 3, 64) + " " + sizes[int(i)]
}

This function is a conversion from my personal php project

2018/08/17 00:24:50 [*] Filename go1.10.3.darwin-amd64.pkg
2018/08/17 00:24:51 [*] Speed 9.000 MB / 1s 
2018/08/17 00:24:52 [*] Speed 11.125 MB / 1s 
2018/08/17 00:24:53 [*] Speed 11.125 MB / 1s 
2018/08/17 00:24:54 [*] Speed 10.562 MB / 1s 
2018/08/17 00:24:55 [*] Speed 11.187 MB / 1s 
2018/08/17 00:24:56 [*] Speed 11.109 MB / 1s 
2018/08/17 00:24:57 [*] Speed 11.109 MB / 1s 
2018/08/17 00:24:58 [*] Speed 11.141 MB / 1s 
2018/08/17 00:24:59 [*] Speed 11.172 MB / 1s 
2018/08/17 00:25:00 [*] Speed 11.141 MB / 1s 
2018/08/17 00:25:01 [*] Speed 8.453 MB / 1s 
2018/08/17 00:25:02 [*] Speed 6.385 MB / 1s 
2018/08/17 00:25:03 [*] Speed 0 Bytes / 1s 

This is the final result of the run, and Then I put all the source code in the bottom and go to sleep.

package main

import (
 "net/http"
 "log"
 "time"
 "net/url"
 "path"
 "os"
 "io"
 "bufio"
 "math"
 "strconv"
)

var durl = "https://dl.google.com/go/go1.10.3.darwin-amd64.pkg";

func main() {
 uri, err := url.ParseRequestURI(durl)
 if err != nil {
  panic(" Site error ")
 }

 filename := path.Base(uri.Path)
 log.Println("[*] Filename " + filename)

 client := http.DefaultClient;
 client.Timeout = time.Second * 60 // Set the timeout 
 resp, err := client.Get(durl)
 if err != nil {
  panic(err)
 }
 if resp.ContentLength <= 0 {
  log.Println("[*] Destination server does not support breakpoint download.")
 }
 raw := resp.Body
 defer raw.Close()
 reader := bufio.NewReaderSize(raw, 1024*32);


 file, err := os.Create(filename)
 if err != nil {
  panic(err)
 }
 writer := bufio.NewWriter(file)

 buff := make([]byte, 32*1024)
 written := 0
 go func() {
  for {
   nr, er := reader.Read(buff)
   if nr > 0 {
    nw, ew := writer.Write(buff[0:nr])
    if nw > 0 {
     written += nw
    }
    if ew != nil {
     err = ew
     break
    }
    if nr != nw {
     err = io.ErrShortWrite
     break
    }
   }
   if er != nil {
    if er != io.EOF {
     err = er
    }
    break
   }
  }
  if err != nil {
   panic(err)
  }
 }()

 spaceTime := time.Second * 1
 ticker := time.NewTicker(spaceTime)
 lastWtn := 0
 stop := false

 for {
  select {
  case <-ticker.C:
   speed := written - lastWtn
   log.Printf("[*] Speed %s / %s \n", bytesToSize(speed), spaceTime.String())
   if written-lastWtn == 0 {
    ticker.Stop()
    stop = true
    break
   }
   lastWtn = written
  }
  if stop {
   break
  }
 }
}

func bytesToSize(length int) string {
 var k = 1024 // or 1024
 var sizes = []string{"Bytes", "KB", "MB", "GB", "TB"}
 if length == 0 {
  return "0 Bytes"
 }
 i := math.Floor(math.Log(float64(length)) / math.Log(float64(k)))
 r := float64(length) / math.Pow(float64(k), i)
 return strconv.FormatFloat(r, 'f', 3, 64) + " " + sizes[int(i)]
}