How to Copy Large Files Quickly under linux

Copy data

When copying data remotely, we generally use rsync command, but if a large number of small files are copied, the transmission speed of rsync will be slow. This problem can be solved by using tar pv lz4 package compression transfer. Using this method is equivalent to using scp and rsync to transfer large files.

According to the actual measurement, 1200G is transmitted by rsync, and the size of a single file is several 10KB ~ 2GB. Gigabit network cards need to run six rsync at the same time to run the bandwidth full, and each speed is about 20MB, which fluctuates greatly. It can copy about 4.5 GB per minute.

However, with tar pv lz4, one run is enough, and the speed fluctuation is small. It can copy about 6.8 GB per minute.

rsync Usage Example

rsync Installation: yum install-y rsync

#  Push 
[root@vm5 ~]# rsync -auvzP -e "ssh -p22" mssh.tar.gz root@192.168.176.11:/data/
sending incremental file list
mssh.tar.gz
     1,977 100%  0.00kB/s  0:00:00 (xfr#1, to-chk=0/1)
sent 2,069 bytes received 35 bytes 4,208.00 bytes/sec
total size is 1,977 speedup is 0.94

#  Pull 
[root@vm5 ~]# rm -f mssh.tar.gz
[root@vm5 ~]# rsync -auvzP -e "ssh -p22" root@192.168.176.11:/data/mssh.tar.gz .
receiving incremental file list
mssh.tar.gz
     1,977 100%  1.89MB/s  0:00:00 (xfr#1, to-chk=0/1)
sent 43 bytes received 2,069 bytes 4,224.00 bytes/sec
total size is 1,977 speedup is 0.94

Parameter auvzP Explanation: Parameter a is archive transfer, retain file attributes, u is update transfer, source file modification time is newer, then transfer. v is a display detail process, z is a compressed transfer, and P is a breakpoint transfer.

Note: When rsync transfers folders, folder/with/is the file in the transfer directory, and the folder is transferred without/.

Use compressed transmission

Install pv, lz4 Tools

Note: Install on both sides of the server.

pv is not found in yum source. You can find it in pv official website

#  Go pv Official website, get a rpm Package link, direct rpm Command installation 
[root@vm5 ~]# rpm -ivh http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm
 Get http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm
 Warning: /var/tmp/rpm-tmp.mFbA6u:  Head V3 DSA/SHA1 Signature,  Key  ID 3fc56f51: NOKEY
 In preparation ...             ################################# [100%]
 Upgrading / Installation ...
  1:pv-1.6.6-1            ################################# [100%]
  
# lz4  It can be directly yum Installation   
[root@vm5 ~]# yum install -y lz4

Use

[root@vm5 ~]# time tar -c go |pv |lz4 -B4 |ssh -p22 -c aes128-ctr 192.168.176.11 "lz4 -d |tar -xC /data/"
using blocks of size 64 KB
18.1MiB 0:00:00 [49.5MiB/s] [  <=>                                                                                   ]
real    0m0.376s
user    0m0.080s
sys    0m0.108s
#  Contrast rsync
[root@vm5 ~]# time rsync -auvzP -e "ssh -p22" go 192.168.176.11:/data/
......
sent 11,741,677 bytes received 10,451 bytes 7,834,752.00 bytes/sec
total size is 18,502,481 speedup is 1.57
real 0m1.130s
user    0m0.797s
sys    0m0.160s
[root@vm5 ~]#