Simple database migration method from MySQL to Redis
- 2020-05-13 03:48:42
- OfStack
Moving a large table from mysql to redis, you will find that extracting, converting, or loading 1 row of data is unbearably slow. Here's a tip to get you out of trouble. The content generated by the mysql command line is passed directly to redis-cli in a "piped out" manner, bypassing the "middleware" so that both can achieve optimal speed when performing data operations.
An mysql table with about 8 million rows of data, which would have taken 90 minutes to import into redis, takes only two minutes with this method. Believe it or not, I do.
Mysql to Redis data protocols
The redis-cli command line tool has a batch insert mode, designed specifically for batch command execution. The first step is to format the content of the Mysql query into the data format available to redis-cli. here we go!
My statistics:
CREATE TABLE events_all_time (
id int(11) unsigned NOT NULL AUTO_INCREMENT,
action varchar(255) NOT NULL,
count int(11) NOT NULL DEFAULT 0,
PRIMARY KEY (id),
UNIQUE KEY uniq_action (action)
);
The redis command to be executed in each line of data is as follows:
HSET events_all_time [action] [count]
Create an events_to_redis.sql file in accordance with the above redis command rules. The content of SQL is used to generate the redis data protocol format:
-- events_to_redis.sql
SELECT CONCAT(
"*4\r\n",
'$', LENGTH(redis_cmd), '\r\n',
redis_cmd, '\r\n',
'$', LENGTH(redis_key), '\r\n',
redis_key, '\r\n',
'$', LENGTH(hkey), '\r\n',
hkey, '\r\n',
'$', LENGTH(hval), '\r\n',
hval, '\r'
)
FROM (
SELECT
'HSET' as redis_cmd,
'events_all_time' AS redis_key,
action AS hkey,
count AS hval
FROM events_all_time
) AS t
ok, execute with the following command:
mysql stats_db --skip-column-names --raw < events_to_redis.sql | redis-cli --pipe
Important mysql parameter description:
--raw: causes mysql not to convert newlines in field values. -- skip-column-names: causes each row of mysql output to contain no column names.