Redis tutorial of: introduction to virtual memory

1. Introduction:

Like most NoSQL databases, Redis also follows the Key/Value data storage model. In some cases, Redis will save Keys/Values in memory to improve the efficiency of data queries and data modifications, but this is not always a good choice. In view of this, we can further optimize it by keeping only Keys data in memory to ensure the efficiency of data retrieval, while Values data can be swapped out to disk when rarely used.
In practice, only about 10% of       is a relatively common key, so Redis can swap out the rest of Keys and Values to disk through virtual memory, and Redis reads back into main memory once the swapped Keys or Values needs to be read.

ii. Application scenario:

      for most databases, the ideal way to run is to load all the data into memory, and the subsequent queries can be done entirely based on the in-memory data. In reality, however, such scenarios are not common, and more often than not, only part of the data can be loaded into memory.
      has a very important concept in Redis that keys is not generally swapped, so if you have a large number of keys in your database, each key only has a very small value correlation, then this scenario is not very suitable for using virtual memory. If, on the contrary, the database contains only a small number of keys, while the value associated with each key is very large, this scenario is perfect for using virtual storage.
In the practical application of      , we can combine Keys with a lot of small values into Keys with a few large values in order to make virtual memory play its role more fully and help us to improve the operating efficiency of the system. The main approach is to change the Key/Value mode to Hash based mode, which makes many of the original Keys properties in Hash.

3. Configuration:

    1). Add the following configuration items to the configuration file so that the current Redis server starts with virtual memory enabled.
      vm-enabled yes
     
      2). Set the maximum number of bytes available for Redis in the configuration file. If the data in memory is greater than the value, some objects are swapped out to disk, where the memory occupied by the swapped out object is freed until the used memory is less than the value.
 

    vm-max-memory (bytes)

 

      Redis's swap rule is to try to take into account the "oldest" data, that is, the data that has not been used for the longest time will be swapped out. If the age of two objects is the same, the larger Value data will be swapped out first. It is important to note that Redis does not swap Keys to disk, so if only keys's data is already full of vram, this data model is not suitable for using vram, or the value is set larger to hold the entire Keys data. In practice, if we consider using Redis virtual memory, we should allocate as much memory as possible to Redis to avoid frequent swapping in and out.
     
3). Set the number of pages and the number of bytes per page in the configuration file. To transfer data from memory to disk, we need to use swap files. These files have nothing to do with data persistence, and Redis will delete them all before exiting. Since most of the access to the swap file is random, it is recommended to store the swap file on a solid-state disk, which can greatly improve the system operation efficiency.
 

    vm-pages 134217728

    vm-page-size 32    

 

In the above configuration, Redis divides the swap file into vm-pages pages, each page consuming vm-page-size bytes. The final swap file size available for Redis is vm-pages * vm-page-size. Since one value can be stored on one or more pages, but one page cannot hold more than one value, we need to take full account of this feature of Redis when setting vm-page-size.
 
      4). In the configuration file of Redis there is a very important configuration parameter,
 

    vm-max-threads 4

 

      this parameter represents the maximum number of threads that Redis applies when performing an IO operation on the swap file. In general, we recommend that this value be equal to CPU cores of the host. If this value is set to 0, Redis will do this synchronously when interacting with IO in the swap file.
      for Redis, if the operation swap file is performed synchronously, then when one client accesses the data in the swap file and another client attempts to access the data in the swap file, the client's request is suspended until the previous operation is completed. The effects of this blocking are more obtuse, especially when large data values are read on relatively slow or busy disks. Synchronous operations are not all bad, however. In fact, from the perspective of global execution efficiency, synchronous is better than asynchronous because it saves extra overhead for thread switching, inter-thread synchronization, and thread pull-up operations. This is especially true when most frequently used data can be read directly from main memory.
If you have a real-world application that does the opposite, where you have a lot of swapping in and out, and your system has a lot of cores, then you don't want the client to have to block for a short period of time before accessing the swap file, and if that's the case, I think an asynchronous approach might be more appropriate for your system.
The best answer to the final configuration will come from constant experimentation and tuning.