Implementation details of mongodb Incremental and full backup script

preface

mongodb backup scripts, which can be used for full or incremental backup, were written two years ago, and there are still very few mongodb backup scripts available online. Without further ado, let's take a look at the details

function

Make full or incremental backups of mongodb database data on a regular basis (replica set architecture), and can be compressed and uploaded to Alicloud oss(local assembly is compressed package, oss can be set not to be uploaded).

Script running environment

python, pymongo and mongodb shell clients are installed in python language (python 2.7.6, pymongo 3.0.3 and mongodb shell 2.0.4 are used for testing).

Script deployment steps

1. Put the script on an linux host

2. For incremental backups, create mongodb backup role users or admin library users with higher permissions. (When exporting, switch to admin library first to verify the permissions, which requires the permissions to query local library, mongodump local library and mongodump target library.)

use admin
 db.addUser( { user: "xxxxx",
  pwd: "xxxxx",
  roles: [ "backup" ]
 } )

3. Edit ES45en.properties, modify oss, mongodb connection and other configuration information

##  Ali cloud oss  Connection configuration 
 endpoint= oss.aliyuncs.com
 accessKeyId = xxxxxxx
 accessKeySecret = xxxxxxx
 bucket = db-backup
 ## mongodb  Connection configuration 
 #  The address of the slave library is recommended to reduce the pressure on the master library 
 db_host= localhost
 db_port= 27017
 #  If it is an incremental backup solution, it is a step 2 Created in, or with higher permissions admin Users; In the case of a full-volume backup scenario, you only need to have operation permissions for the target library 
 db_user= testb
 db_passwd= testb
 #  Object library 
 db_name= che
 #  Back up to a local temporary directory  
 db_backup_root_path= /temp/backup/
 #  If you are using mongo Client green, I'll say mongo The absolute path of the client 
 mongo_shell_path= /dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/
 #  Incremental or full backup  1:  Incremental backup  0: Full amount of backup 
 is_inc_backup=1
 #  Every how many days 1 Secondary backup 
 full_backup_period=7
 #  Whether to upload to oss If the  1  , the local backup file will be deleted after uploading successfully; 0: Don't upload oss
 is_upload_to_oss= 0

4. Add ES52en. py to linux timer. crontab task configuration is 0 4 * * * python /xxx/ start.py > > /xxx/xxx.log 2 > & 1

Incremental recovery steps:

1. Create the mongodb role with applyOps permissions and the users who use this role. (User permission to execute mongorestore --oplogReplay is required)

use admin
 db.createRole(
 {
 role: "applyOpsRole",
 privileges: [
 { resource: { anyResource: true }, actions: [ "anyAction"] }
 ],
 roles: []
 }
 )
 db.addUser( { user: "xxxx",
 pwd: "xxxx",
 roles: [ "applyOpsRole" ]
 } )

2. Modify the configuration in restore_ES79en.py

##  Ali cloud oss  configuration 
 endpoint="oss.aliyuncs.com"
 accessKeyId="xxxxxxx"
 accessKeySecret="xxxxxxx"
 bucket="db-backup"

 ## mongodb Configuration of imports 
 db_host="localhost"
 db_port=27017
 #  steps 1 User created 
 db_user="testr"
 db_passwd="testr" 
 db_name="che"

 # recent circle backup direactory on oss  The period name of the most recent backup file, that is, in the backup temporary directory mongodb_inc_backup_info.json the last_circle_backup_dir_name  or  oss Folder name 
 last_circle_backup_dir_name="mongodb_cycle_backup_20151124141133"
 #  from oss Go to the local temporary directory 
 restore_local_temp_path="H:\\pythoncode\\temp\\restore\\"
 #  If you are using mongo Client green, I'll say mongo The absolute path of the client 
 mongo_shell_path= "/alidata1/dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/"
 # backup file has download to local ? if True,will not download backup files from oss
 #  Whether the backup file has been downloaded locally, if true , will not from oss Download and unzip, locally available 
 has_download_to_local=False
 #  Whether to delete the old database when restoring 
 is_drop_old_restore=True

3. Stop mongodb writing during import

4. Perform restore_inc. py

Full time recovery steps:

1. Modify the configuration in restore_ES96en.py

##  Ali cloud oss  configuration 
 endpoint="oss.aliyuncs.com"
 accessKeyId="xxxxxxx"
 accessKeySecret="xxxxxxx"
 bucket="db-backup"

 ## mongodb Configuration of imports 
 db_host="localhost"
 db_port=27017
 #  The user corresponding to the database 
 db_user="test"
 db_passwd="test" 
 db_name="che"

 # recent circle backup direactory on oss  The period name of the most recent backup file,  oss  The name of the file stored on is  last_circle_backup_dir_name+last_full_backup_file_suffix
 last_circle_backup_dir_name="mongodb_cycle_backup_20151124141133"

 last_full_backup_file_suffix=".tar.gz"
 #  Backup directory, the actual full backup path is  restore_local_temp_path+last_circle_backup_dir_name+db_name
 restore_local_temp_path="H:\\pythoncode\\temp\\restore\\"
 #  If you are using mongo Client green, I'll say mongo The absolute path of the client 
 mongo_shell_path="/alidata1/dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/"
 # backup file has download to local ? if True,will not download backup files from oss
 #  Whether the backup file has been downloaded locally, if true , will not from oss Download and unzip, locally available 
 has_download_to_local=False
 #  Whether to delete the old database when restoring 
 is_drop_old_restore=True

2. Perform restore_full. py

Related:

Incremental backup implementation principle

Back up the full database once in a period (e.g., 1 week), and then back up oplog files from the last record point to the latest time each time. Oplog records the change operation information of MongoDB database, which is stored in the ES116en.rs table of local library. It only exists in the cluster architecture, but does not exist on a single machine. Therefore, incremental backup cannot be used on a single machine. The slave library is synchronized with the master library by asynchronously copying the master's Oplog files. oplog has a size limit, exceeding the specified size, the new record overwrites the old operation record.

The flow at full script execution time

Back up the mongodb database locally

Be compressed

Upload oss

Verify that oss is the same size as the local file

Delete the local backup file

The process at which the incremental script is executed

Read the previous cycle execution information to determine if a new cycle needs to be created Obtain the time point of last record of oplog on mongodb current timestamp position oplog point in time when mongodb was read from local once dump exports full data or incremental oplog files locally. The incremental oplog file exports to the oplog file from the last oplog record point to the latest time Save the current timestamp position obtained in step 2 locally as a point in time in the next step 3 execution Be compressed Upload oss Delete the local backup file

The process that the script executes when it is restored

Download the periodic backup file locally from oss Unzip the zip file for full volume and incremental oplog Use mongorestore to import the full amount of files Use mongorestore --oplogReplay to import oplog files for each time period

mongodb Incremental Backup script source address:

https://gitee.com/passer/mongodb_backup_script

conclusion