MongoDB's quick page turning method

Flipping through data is one of the most common operations in MongoDB. A typical scenario is to display your results in your user interface. If you're processing data in bulk, it's also important to get your paging strategy right so your data processing can scale.

Next, let's take a look at the different ways of flipping through data in MongoDB with an example. In this example, we have user data for an CRM database that we need to browse through and display 10 users at the same time. So actually, our page size is 10. Below is the structure of our user document:

{

  _id,

  name,

  company,

  state

}

Method 1: Using skip() and limit()

MongoDB itself supports paging operations using the skip() and limit() directives. The skip(n) directive tells MongoDB that it should skip the "n" result and the limit(n) directive indicating MongoDB, which should limit the result length to "n" result. Normally, you would use the skip() and limit() directives, but to illustrate the case, we provide console commands that achieve the same results. At the same time, restricted check code is excluded for the sake of code brevity.

//Page 1

db.users.find().limit (10)

//Page 2

db.users.find().skip(10).limit(10)

//Page 3

db.users.find().skip(20).limit(10)

........

1 in general, the code for retrieving page n looks like this:

db.users.find().skip(pagesize*(n-1)).limit(pagesize)
However, as the size of the data increases, this approach presents serious performance problems. The reason for this is that every time a query is executed, a complete result set is established, and the server must walk from the beginning of the collection to the specified offset. As the offset increases, the process becomes slower and slower. At the same time, the process does not use the index effectively. So, when you have a small data set, the typical "skip()" and "limit()" methods are useful. If you are working with large data sets, you need to consider other approaches.

Method 2: Using find() and limit()

The reason the previous method did not extend well was the skip() command. Therefore, the goal of this section is to achieve pagination without the skip() command. To do this, we will take advantage of the natural order in which data is stored, such as timestamps or identifiers stored in documents.

In this example, we will use "_id" to store each document. "_id" is an ObjectID structure of MongoDB, that is, a 12-byte structure containing timestamps, machining, process identifiers, counters, and so on. The general idea is as follows:

Retrieves the last document _id on the current page
On the next page retrieve the file greater than "_id"

//Page 1

db.users.find().limit(pageSize);

//Find the id of the last document in this page

last_id = ...

//Page 2

users = db.users.find({'_id'> last_id}). limit(10);

//Update the last id with the id of the last document in this page

last_id = ...

This method takes advantage of the inherent rules that exist in the _id field. Also because the "_id" field is the default lookup operation, it is a very good performance indicator. If you're using a field that's not indexed, you're going to get stuck, so it's important to make sure that the field is indexed.

Also, if you want the data to be sorted and paged in a particular order, you can use the sort() clause and the above technique. It is important to ensure that the sorting process is using indexes for optimal performance. You can query using the.explain () suffix.

users = db.users.find({'_id'> last_id}). sort(..).limit(10);

//Update the last id with the id of the last document in this page

last_id = ...