mongodb a solution for a process to hang while setting up an T level database

  • 2020-06-15 10:27:36
  • OfStack

When the mongod process exits, about 120 million records are recorded in the database. The database size is 600G

The error message is as follows:


Sat May 12 08:37:18 Invalid access at address: 0x7ef6036a9000

Sat May 12 08:37:19 Got signal: 7 (Bus error).

Sat May 12 08:37:19 [conn4878] moveChunk data transfer progress: { active: true, ns: "genbank.main", from: "shard1/192.168.136.72:29022,192.168.136.73:29023,192.168.136.79:29021", min: { _id: ObjectId('4fabda86be960438ed0099d2') }, max: { _id: ObjectId('4fabdaa1be960439500040b1') }, state: "clone", counts: { cloned: 8138, clonedBytes: 50270960, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0
Sat May 12 08:37:19 Backtrace:
0xa90999 0xa90f70 0x7f3c7ffcc060 0x7f3c7f5b3873 0xa48342 0xa48c76 0x97cd04 0x97e2ff 0x940ca5 0x944574 0x8866b7 0x88dc29 0xaa33f6 0x637407 0x7f3c7ffc3efc 0x7f3c7f55d89d
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90999]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa90f70]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7f3c7ffcc060]
 /lib/x86_64-linux-gnu/libc.so.6(+0x137873) [0x7f3c7f5b3873]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo17MigrateFromStatus5cloneERSsRNS_14BSONObjBuilderE+0x3d2) [0xa48342]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo19InitialCloneCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x16) [0xa48c76]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a4) [0x97cd04]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97e2ff]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x940ca5]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1724) [0x944574]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod() [0x8866b7]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dc29]
/usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa33f6]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637407]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f3c7ffc3efc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3c7f55d89d]

Logstream::get called in uninitialized state
Sat May 12 08:37:19 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Sat May 12 08:37:19 ERROR: Client::shutdown not called: conn

The problem has been solved. The useful machine is the numa architecture, so when you start the mongod process add: numactl --interleave=all
The mongodb website has the following instructions for your reference:
Linux, NUMA and MongoDB tend not to work well together. If you are running MongoDB on numa hardware, we recommend turning it off (running with an interleave memory policy). Problems will manifest in strange ways, such as massive slow downs for periods of time or high system cpu time.

numactl
To turn off NUMA, start mongod with


numactl --interleave=all ${MONGODB_HOME}/bin/mongod --config conf/mongodb.conf

proc settings


echo 0 > /proc/sys/vm/zone_reclaim_mode

You can change zone_reclaim_mode without restarting mongod. For more information on this setting seehttp://www.kernel.org/doc/Documentation/sysctl/vm.txt.


Related articles: