golang 40 lines of code implement a common pool

  • 2020-06-15 09:17:38
  • OfStack

Code warehouse

goroutine-pool

Covariance management for golang

The golang coroutine mechanism is very convenient to solve the problem of concurrent programming, but the coroutine is not without overhead, so it is necessary to appropriately limit the number under 1.

Code that does not use a pool (chan for the sample code)


func (p *converter) upload(bytes [][]byte) ([]string, error) {
  ch := make(chan struct{}, 4)
  wg := &sync.WaitGroup{}
  wg.Add(len(bytes))
  ret := make([]string, len(bytes))
  //  upload 
  for index, item := range bytes {
    ch <- struct{}{}
    go func(index int, imageData []byte) {
      defer func() {
        wg.Done()
        <-ch
      }()
      link, err := qiniu.UploadBinary(imageData, fmt.Sprintf("%d.png", time.Now().UnixNano()))
      if err != nil {
        log.Println(" Failed to upload image ", err.Error())
        return
      }
      ret[index] = link
    }(index, item)
  }
  wg.Wait()
  return ret, nil
}

There are two requirements that need to be implemented:

Limit the maximum covariance number, in this case 4

Wait for all coroutines to complete, in this case bytes slice length

Code that USES the pooling


func (p *converter) upload(bytes [][]byte) ([]string, error) {
  ret := make([]string, len(bytes))
  pool := goroutine_pool.New(4, len(bytes))

  for index, item := range bytes {
    index := index
    item := item
    pool.Submit(func() {
      link, err := qiniu.UploadBinary(item, fmt.Sprintf("%d.png", time.Now().UnixNano()))
      if err != nil {
        log.Println(" Failed to upload image ", err.Error())
        return
      }

      ret[index] = link
    })
  }
  pool.Wait()
  return ret, nil
}

The biggest difference you can see is that you only need to focus on the business logic, where concurrency control and waiting have been taken over by the pooling

conclusion


Related articles: