Realtime synchronization of files between Linux servers

Usage scenarios

Existing server A and server B, if the specified directory of server A (for example /home/paul/rsync/ Changes (additions, deletions, and property changes) in B are synchronized in real time to the target directory of server B (for example /home/paul/rsync/ ).

Data mirror backup tool Rsync

Rsync is a very fast and flexible file copying tool. It supports file replication either locally or between remote servers. Rsync uses the ES18en-ES19en algorithm, which only needs to transfer the difference between the source side and the target side of the file, greatly reducing the network bandwidth consumption and replication time. Rsync is mostly used for data backup and mirroring.

Rsync uses a quick check algorithm to determine whether a file needs to be synchronized by comparing changes in file size or last modification time.

There are two ways to connect to a remote host: using ssh or rsync daemon. Remote file backup is implemented using Rsync mode.

Installation and operation of Rsync

Install Rsync

terminal on server A and Server B respectively:

sudo yum install rsync

After installation, you will find that the CONFIGURATION file for rsync is located at etc/ rsyncd.conf. This file needs to be used when synchronizing in daemon mode, which is not covered here.

Configure secret free logins between servers A and B

Server A execution:

ssh-keygen
ssh-copy-id  The server B the IP address 

Create the source-side directory and target-side directory

In Server A:

mkdir /home/paul/rsync

In server B:

mkdir /home/paul/rsync

Create the test file on server A

echo "Hello from Server A" >> /home/paul/rsync/demo.txt

Execute file transfer command

Run on server A:

# (1)
rsync -avPz --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync
# (2)
rsync -avPz --delete --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync

You will find in ES86en. txt also appears in server B /home/paul/rsync directory.

Command parsing:

(1) Copy the files in the directory /home/paul/rsync in server A to/home.paul /rsync in server B(192.168.100.130).

(2) Compare the files at the target end and the source end. If the files at the target end do not exist at the source end, delete the files at the target end.

Problems with Rsync

Rsync is only a file replication tool, it can not listen to the source side of the file to add or delete operations. After the changes are made at the source side, the rsync command needs to be executed to synchronize the changes to the target side.

Rsync needs to scan the entire directory before each synchronization. If the number of files in the source directory is large, scanning may take more time.
To meet the requirements of real-time listening, we need to introduce another tool: inotify.

File system event listener inotify

inotify-tools provides a simple interface to inotify. It is a library written in the c language and also contains command-line tools.

inotify - tools detail please click: https: / / github com rvoicilas/inotify - tools/wiki

The installation of inotify - tools

For centos7 system, execute successively:

yum install -y epel-release
yum --enablerepo=epel install inotify-tools

Use the inotifywait command for event listening

The listening script is as follows (ES156en-ES157en.sh):

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e modify,delete,create,attrib /home/paul/rsync/ | while read file
do
rsync -avPz --progress /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
rsync -avPz --delete /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
echo "${file} was synchronized"
done

Argument parsing

-ES163en keeps listening. If you do not write this parameter, inotifywait will exit after listening to 1 event. -r recursively listens for directories. -q quiet mode, printing less content. timefmt output format for specified time. format specifies the format of the event output. -e sets the type of event to listen for. Listen for additions, deletions and metadata changes.

For each listening time triggered, inotifywait executes the code between do and done. Here, we call the rsync command we talked about earlier for file synchronization.

Add crontab to the listening script

crontab -e
* * * * * sh /home/paul/inotifywait-rsync.sh

The resources
https://rsync.samba.org
https://github.com/rvoicilas/inotify-tools/wiki