Use MongoDB as an Redis in memory database

The basic idea

The use of MongoDB as an in-memory database (in-memory database), that is, not having MongoDB save data to disk at all, is of growing interest. This usage is extremely useful for the following applications:

Write-intensive caching before the slow RDBMS system Embedded system An PCI compatible system that does not require persistent data Unit tests that require a lightweight database with data that can be easily erased (unit testing)

It would be elegant if this one slice could be implemented: we would be able to take advantage of MongoDB's query/retrieval capabilities smartly without involving disk operations. As you probably know, disk IO (especially random IO) is the bottleneck 99% of the time, and if you want to write data, disk operations are unavoidable.

One of MongoDB's cool design decisions is that she can handle read and write requests to data in disk files using in-memory mapped files (memory-mapped file). That said, MongoDB doesn't treat RAM and disk any differently, but instead treats the file as a huge array, accesses the data in bytes, and leaves the rest to the operating system (OS)! It was this design decision that enabled MongoDB to run on RAM without any modifications.

Implementation method

This is done by using a special type of file system called tmpfs. In Linux it looks like the regular file system (FS) 1, except that it is completely in RAM (unless it is larger than RAM, it can also swap, which is very useful!). . I have 32GB RAM on my server, so let's create a 16GB tmpfs:

# mkdir /ramdata

# mount -t tmpfs -o size=16000M tmpfs /ramdata/

# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/xvde1             5905712   4973924    871792  86% /

none                  15344936         0  15344936   0% /dev/shm

tmpfs                 16384000         0  16384000   0% /ramdata

Next, start MongoDB with the appropriate Settings. To reduce the amount of wasted RAM, smallfiles and noprealloc should be set to true. Since it is now based on RAM, doing so will not degrade performance at all. It makes no sense to use journal again, so you should set nojournal to true.

dbpath=/ramdata

nojournal = true

smallFiles = true

noprealloc = true

Once you start MongoDB, you'll find that it works very well, and the files in the file system appear as expected:

# mongo

MongoDB shell version: 2.3.2

connecting to: test

> db.test.insert({a:1})

> db.test.find()

{ "_id" : ObjectId("51802115eafa5d80b5d2c145"), "a" : 1 }
# ls -l /ramdata/

total 65684

-rw-------. 1 root root 16777216 Apr 30 15:52 local.0

-rw-------. 1 root root 16777216 Apr 30 15:52 local.ns

-rwxr-xr-x. 1 root root        5 Apr 30 15:52 mongod.lock

-rw-------. 1 root root 16777216 Apr 30 15:52 test.0

-rw-------. 1 root root 16777216 Apr 30 15:52 test.ns

drwxr-xr-x. 2 root root       40 Apr 30 15:52 _tmp

Now let's add some data and verify that 1 works perfectly. Let's create an document of 1KB and add it to MongoDB 4 million times:

> str = ""
> aaa = "aaaaaaaaaa"

aaaaaaaaaa

> for (var i = 0; i < 100; ++i) { str += aaa; }
> for (var i = 0; i < 4000000; ++i) { db.foo.insert({a: Math.random(), s: str});}

> db.foo.stats()

{

        "ns" : "test.foo",

        "count" : 4000000,

        "size" : 4544000160,

        "avgObjSize" : 1136.00004,

        "storageSize" : 5030768544,

        "numExtents" : 26,

        "nindexes" : 1,

        "lastExtentSize" : 536600560,

        "paddingFactor" : 1,

        "systemFlags" : 1,

        "userFlags" : 0,

        "totalIndexSize" : 129794000,

        "indexSizes" : {

                "_id_" : 129794000

        },

        "ok" : 1

}

It can be seen that the average size of document is 1136 bytes, and the data takes up a total space of 5GB. The index size above _id is 130MB. Now we need to verify one very important thing: is there any duplication of data in RAM? Is there one copy in MongoDB and one in the file system? Remember that MongoDB does not cache any data in her own process, her data is only cached in the file system cache. So let's clear 1 file system cache and see what else is in RAM:

# echo 3 > /proc/sys/vm/drop_caches 

# free

             total       used       free     shared    buffers     cached

Mem:      30689876    6292780   24397096          0       1044    5817368

-/+ buffers/cache:     474368   30215508

Swap:            0          0          0

As you can see, of the 6.3GB RAM that has been used, 5.8GB is used for file system caching (buffers, buffer). Why y is there still a 5.8GB file system cache on the system even after all caches have been cleared? The reason is that Linux is so smart that she doesn't store duplicate data in tmpfs and the cache. That's great! That means you only have one copy of data at RAM. Now let's visit all document under 1 and verify that the usage of RAM does not change under 1:

> db.foo.find().itcount()

4000000
# free

             total       used       free     shared    buffers     cached

Mem:      30689876    6327988   24361888          0       1324    5818012

-/+ buffers/cache:     508652   30181224

Swap:            0          0          0

# ls -l /ramdata/

total 5808780

-rw-------. 1 root root  16777216 Apr 30 15:52 local.0

-rw-------. 1 root root  16777216 Apr 30 15:52 local.ns

-rwxr-xr-x. 1 root root         5 Apr 30 15:52 mongod.lock

-rw-------. 1 root root  16777216 Apr 30 16:00 test.0

-rw-------. 1 root root  33554432 Apr 30 16:00 test.1

-rw-------. 1 root root 536608768 Apr 30 16:02 test.10

-rw-------. 1 root root 536608768 Apr 30 16:03 test.11

-rw-------. 1 root root 536608768 Apr 30 16:03 test.12

-rw-------. 1 root root 536608768 Apr 30 16:04 test.13

-rw-------. 1 root root 536608768 Apr 30 16:04 test.14

-rw-------. 1 root root  67108864 Apr 30 16:00 test.2

-rw-------. 1 root root 134217728 Apr 30 16:00 test.3

-rw-------. 1 root root 268435456 Apr 30 16:00 test.4

-rw-------. 1 root root 536608768 Apr 30 16:01 test.5

-rw-------. 1 root root 536608768 Apr 30 16:01 test.6

-rw-------. 1 root root 536608768 Apr 30 16:04 test.7

-rw-------. 1 root root 536608768 Apr 30 16:03 test.8

-rw-------. 1 root root 536608768 Apr 30 16:02 test.9

-rw-------. 1 root root  16777216 Apr 30 15:52 test.ns

drwxr-xr-x. 2 root root        40 Apr 30 16:04 _tmp

# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/xvde1             5905712   4973960    871756  86% /

none                  15344936         0  15344936   0% /dev/shm

tmpfs                 16384000   5808780  10575220  36% /ramdata

Sure enough! :)

What about copy (replication)?

Since the data in RAM is lost when the server is restarted, you may want to use replication. Automatic failover (failover) can be obtained with a standard copy set (replica set) and data reading capability can be improved (read capacity). If a server is restarted, it can read from another server in the same replica set to rebuild its own data (resynchronization, resync). This is fast enough even with large amounts of data and indexes, as the indexing operations are all done in RAM :)

One important point is that the write operation writes to a special collection called oplog, which is located in the local database. By default, its size is 5% of the total data volume. In my case, oplog would have 5% of 16GB, which is 800MB. In case of doubt, it is safe to use the option oplogSize to select a fixed size for oplog. If the alternate server goes down for more than oplog, it will have to be resynchronized. To set its size to 1GB, go like this:

oplogSize = 1000

What about sharding (sharding)?

Now that you have all the query capabilities of MongoDB, how do you use it to implement a large service? You can use sharding as much as you like to implement a large scalable in-memory database. Configuring the servers (which hold block allocations) is still a disk-based solution, because the number of activities on these servers is small and it is not fun to rebuild the cluster from scratch.
Matters needing attention

RAM is a scarce resource, and in this case you want the entire data set to fit into RAM. Although tmpfs has the ability to use disk switching (swapping), the performance degradation will be significant. To take full advantage of RAM, you should consider:

The storage bucket is normalized using the usePowerOf2Sizes option Periodically run the compact command or resynchronize the nodes (resync) The design of schema should be fairly standardized (to avoid large Numbers of larger document)

conclusion

Baby, you can now use MongoDB as an in-memory database and use all of its features! Well, the performance should be pretty amazing: I was able to get 20K writes per second in tests with a single thread/core, and add as many cores as you want.