MongoDB Details two ways to add secondary nodes

preface

During that period, MongoDB was being transferred from op management to db for maintenance. The whole department lacked experience in the operation and maintenance of MongoDB, and the optimization of MongoDB was an unknown challenge. When op approached me and wanted to optimize the MongoDB cluster, the core system's public service platform for SMS messaging, I couldn't afford to take on a task I thought I might not be able to handle.

The developer asked me two questions and asked the operation and maintenance team to solve this problem. However, under the collision of my rational thinking and his emotional thinking, I still won out in the end. I succeeded in persuading him and answering some of his questions. After getting a satisfactory answer, I never asked for me again. Of course, it's not like you can convince a seasoned developer of what to do without some real data in a few words, no matter how rich your theory is or how ambiguous your attitude is. The communication took up most of my day, but his questions were worth discussing.

According to the development logic, the idea is to expand the secondary node horizontally, put other less-demanding businesses on the secondary node, reduce the pressure on the primary node, achieve partial read-write separation, and ensure the priority of major businesses. I think it's a good starting point, but did not make a comment, the one is that he didn't realize that he considered a delay is not the problem of database cluster (there is no detailed screening process, can speak some MongoDB written under article 1 and business logic), the 2 is our hardware is lack of effective resources to expand the nodes.

Different business scenarios application architecture strategy, expand secondary nodes sometimes doesn't solve the problem, especially for those high real-time performance of the business, but sometimes expanded secondary node is really effective, such as hardware upgrades to do service migration online expansion secondary node is needed to meet the business need higher hardware requirements.

There are two ways to expand the secondary node of MongoDB:

1. rs.add() is directly expanded

2. 1 Expansion after backup (personal name)

1, rs. add (" HOST_NAME: PORT ")

The specific implementation method is to log in the machine of the extension node, edit the configuration file, and establish the corresponding directory and permissions, just start the instance of MongoDB.

One point to note is that this expansion method should ensure the data magnitude of the synchronization source, that is, ensure that oplog of MongoDB will not be overwritten before data synchronization is completed. This is similar to the redo log log of MySQL. If overwritten, the synchronized data will not be 1, resulting in synchronization failure.

Another point to be noted is that in the process of synchronizing data, when the cluster data reaches the quantitative level 1, the large size of the synchronized data will cause a certain pressure on the network, which may affect the core switch of the business. Therefore, TC tool is needed to limit the speed of synchronous traffic. This speed limit needs to consider that the synchronization source may not be primary, or secondary node with the same role. Making the external speed limit synchronization is bound to increase the synchronization time, which will increase the probability of oplog being covered. The specific speed limit value can only be grasped after calculation.

2. 1 Quick addition of secondary node in induced snapshot (self-named, welcome to communicate)

a) Take 1 allergenic snapshot backup on primary node

b) 1 induced snapshot recovery is carried out on secondary node, only the data part is restored, and oplog is temporarily not restored

c) initializes the ES72en.rs set and restores the oplog record

d) initializes the other two sets of the local database, db.replset.election, ES82en.system.replset

e) modify the database configuration and restart the database (the instance does not turn on authentication mode, replication set configuration before this 1 step), rs.add ("HOST_NAME:PORT") add secondary to the cluster and observe the synchronization status, verify the integrity of the data, and 1 alignment

The detailed practice process of practice is as follows (for reference only, use with caution in production environment) :

1. Take a snapshot backup on primary

#primary Nodes or whatever secondary Node backup data 
[root@172-16-3-190 mongodb]# /opt/app/mongodb/bin/mongodump -uroot -ppwd4mysql --authenticationDatabase=admin --port=27017 --oplog -o /tmp/dump_mongo/
2018-08-20T15:42:47.028+0800 writing admin.system.users to 
2018-08-20T15:42:47.030+0800 done dumping admin.system.users (1 document)
2018-08-20T15:42:47.030+0800 writing admin.system.version to 
2018-08-20T15:42:47.031+0800 done dumping admin.system.version (2 documents)
2018-08-20T15:42:47.032+0800 writing super_hero.user_address to 
2018-08-20T15:42:47.032+0800 writing super_hero.user_info to 
2018-08-20T15:42:47.033+0800 done dumping super_hero.user_address (1 document)
2018-08-20T15:42:47.033+0800 done dumping super_hero.user_info (1 document)
2018-08-20T15:42:47.034+0800 writing captured oplog to 
2018-08-20T15:42:47.036+0800 dumped 1 oplog entry

# View the backup files 
[root@172-16-3-190 mongodb]# ls -lh /tmp/dump_mongo/
total 12K
drwxr-xr-x 2 root root 4.0K Aug 20 15:42 admin
-rw-r--r-- 1 root root 110 Aug 20 15:42 oplog.bson
drwxr-xr-x 2 root root 4.0K Aug 20 15:42 super_hero

# Pass the backup to be ready to add as secondary On the node 
[root@172-16-3-190 tmp]# scp -r -P22222 /tmp/dump_mongo/ liyingxiao@172.16.3.189:/tmp

2. secondary node 1 induced snapshot recovery

#auth=true
#replSet = repl_mongo
#clusterAuthMode=keyFile
#keyFile=/opt/app/mongodb/keyfile/mongodb.key

## Restore data 
[root@172-16-3-189 we_ops_admin]# /opt/app/mongodb/bin/mongorestore --oplogReplay --port=27017 /tmp/dump_mongo/      
2018-08-20T15:56:32.161+0800 preparing collections to restore from
2018-08-20T15:56:32.193+0800 reading metadata for super_hero.user_info from /tmp/dump_mongo/super_hero/user_info.metadata.json
2018-08-20T15:56:32.194+0800 reading metadata for super_hero.user_address from /tmp/dump_mongo/super_hero/user_address.metadata.json
2018-08-20T15:56:32.222+0800 restoring super_hero.user_address from /tmp/dump_mongo/super_hero/user_address.bson
2018-08-20T15:56:32.300+0800 restoring super_hero.user_info from /tmp/dump_mongo/super_hero/user_info.bson
2018-08-20T15:56:32.867+0800 no indexes to restore
2018-08-20T15:56:32.867+0800 finished restoring super_hero.user_address (1 document)
2018-08-20T15:56:32.881+0800 no indexes to restore
2018-08-20T15:56:32.881+0800 finished restoring super_hero.user_info (1 document)
2018-08-20T15:56:32.881+0800 restoring users from /tmp/dump_mongo/admin/system.users.bson
2018-08-20T15:56:32.993+0800 replaying oplog
2018-08-20T15:56:32.997+0800 done

3. Initialize the ES106en. rs set and restore the oplog record

Create the ES111en. rs collection and initialize the size

 use local
 db.createCollection("oplog.rs",{"capped":true,"size":100000000})

Restore the data from the oplog. rs collection to the secondary node of a sexual backup

[root@172-16-3-189 we_ops_admin]# /opt/app/mongodb/bin/mongorestore -d local -c oplog.rs --port=27017 /tmp/dump_mongo/oplog.bson 
2018-08-20T16:12:49.848+0800 checking for collection data in /tmp/dump_mongo/oplog.bson
2018-08-20T16:12:49.852+0800 restoring local.oplog.rs from /tmp/dump_mongo/oplog.bson
2018-08-20T16:12:49.925+0800 no indexes to restore
2018-08-20T16:12:49.925+0800 finished restoring local.oplog.rs (1 document)
2018-08-20T16:12:49.925+0800 done

4. Initialize the db.replset.election, db.system.replset set, in which ES128en.election needs to query the master node data and store these data to secondary node, or two combined save to secondary node. Another set system. replset can automatically recognize primary node content after adding the copy set (Here I take self-synchronization data)

 #primary node 
 repl_mongo:PRIMARY> db.replset.election.find()
 { "_id" : ObjectId("5b7a6ee5de7a24b82a686139"), "term" : NumberLong(1), "candidateIndex" : NumberLong(0) }
 #secondary node 
 db.replset.election.save({ "_id" : ObjectId("5b7a6ee5de7a24b82a686139"), "term" : NumberLong(1), "candidateIndex" : NumberLong(0) })

5. Modify the database configuration and restart, and add secondary nodes to the replication cluster

#auth=true
#replSet = repl_mongo
#clusterAuthMode=keyFile
#keyFile=/opt/app/mongodb/keyfile/mongodb.key

[root@172-16-3-189 we_ops_admin]# /opt/app/mongodb/bin/mongod --shutdown -f /opt/app/mongodb/mongo.conf 
killing process with pid: 5331
[root@172-16-3-189 we_ops_admin]# vim /opt/app/mongodb/mongo.conf # Comments are removed and restarted 
[root@172-16-3-189 we_ops_admin]# /opt/app/mongodb/bin/mongod -f /opt/app/mongodb/mongo.conf   
about to fork child process, waiting until server is ready for connections.
forked process: 5722
child process started successfully, parent exiting

# add secondary node 
repl_mongo:PRIMARY> rs.add({"_id":1,"host":"172.16.3.189:27017"})
{
  "ok" : 1,
  "operationTime" : Timestamp(1534752953, 1),
  "$clusterTime" : {
    "clusterTime" : Timestamp(1534752953, 1),
    "signature" : {
      "hash" : BinData(0,"Tt9nzhoVYdUtGFZnc1Kg1exl0Hc="),
      "keyId" : NumberLong("6591702943026642945")
    }
  }
}

6. Login the added secondary node to verify the status, data integrity and 1 consistency of the replication set.

 [root@172-16-3-189 we_ops_admin]# /opt/app/mongodb/bin/mongo -uroot -ppwd4mysql --authenticationDatabase=admin --port=27017

The second method of adding secondary node, which is time-saving but labor-intensive, is mainly introduced. In practice, the parameters of authentication and replication set are removed in the early stage of the database instance, which is convenient for the following 1 operation requiring user authority, avoiding the establishment of administrator account, and synchronizing the account of primary node by itself after joining the cluster. When logging in to the secondary node after restart to verify the availability of the service and data 1, use the cluster's administrative account to enter, otherwise an authentication error will be reported.

To sum up the above two expansion methods, it is necessary to ensure that oplog is not overwritten and evaluate the impact of synchronous traffic for simple and convenient expansion of approach 1, which is our usual method to add secondary nodes in horizontal replication set. For the second method, the operation is tedious but there is no need to worry about the coverage of oplog, and the network traffic will not be much concerned during the operation, only the traffic impact of network transmission will be considered. In the first mode, the operation time cycle is long, and the uncontrollable influence range is large, which takes time and energy; in the second mode, the operation time is short, and the operation steps are many, which is prone to other problems.

MongoDB secondary node appears recovering state

After MongoDB did replica sets,secondary node appeared recovering state

After the mongo cluster was suspended for one time, it was restarted and found that the mongo node 1 of one server was in the STATE of recovering and could not be changed to secondary or primary.

After querying the official documents, find the solution and log here.

A reason for

The operation of a backup node can be roughly described as follows: the backup node periodically polls data operations on the master node and then performs these operations on its own copy of the data, thus ensuring that the data is synchronized with the master node.

All database state changes on the master node are stored in a specific system table. The backup node updates its own data based on this data.

The database state change operation mentioned above is called oplog (operation log, master node operation record). oplog is stored in the "ES191en.rs" table in the local database. The backup node in the replica sets asynchronously synchronizes oplog from the primary node, and then re-performs its recorded operations, thus achieving data synchronization.

A few notes about oplog:

oplog only records operations that change the state of the database The operations stored in oplog are not exactly the same as those performed by the master node; for example, "$inc" operations are converted to "$set" operations oplog is stored in a fixed collection (capped collection), and when the number of oplog exceeds oplogSize, the new operation overrides the old one

Data synchronization

In a replica set, there are two ways to synchronize data:

initial sync (initialization) : this process happens when the focus to create a new database or copy one node has just recovered from a downtime, or to concentrate in copy to add new members, by default, the copy of the concentration of nodes will copy from the nearest node oplog to synchronize data, the nearest node can be primary can also is to have a copy of the latest oplog secondary node. This operation usually reinitializes the backup node and is expensive replication (replication) : This operation is performed continuously after initialization to keep data synchronization between secondary nodes.

initial sync

When you encounter the problem of unsynchronization in the above example, you can only do initial sync in one of two ways

The first option is to stop the node, delete the files in the directory, and restart the node. In this case, the node executes initial sync
Note: In this way, the sync time is based on the amount of data, and if the amount of data is too large, the sync time will be very long
There will be a lot of network traffic at the same time, which may affect the work of other nodes Second, stop the node, then delete the files in the directory, find a new node, and copy the files in the node directory to the node directory where you want sync

conclusion