- Published on
How to Migrate Redis database with +100M keys
- Authors
- Name
- Nico Bistolfi
- @nicobistolfi
Dealing with large Redis databases isn't a walk in the park, mainly when you have to step aside from the commonly used SET and GET. Recently we had to perform a migration of our database and here comes the story...
We manage a Redis database in certain parts of our architecture. Initially, everything was working smoothly as the database size was not an issue. Whenever we have to count keys based on a pattern we could just run redis-cli KEYS "abc:*" | wc -l and we were ready to go, but...
What happens when the database grows to over 170 million keys? Well, now the story begins to get a bit messy. Running that KEYS operation was not an option anymore as it blocks the entire database. We had to find out a way to perform operations based on SCAN. There are many ways of doing this, we decided to go on a Nodejs based solution as it's frequently used by our team and performs very well.
The Migration
At first glance, this migration is pretty straightforward, we just had to copy the dump to the new server and point the new Redis installation to it. However, it was a good opportunity to do some cleaning.
We figure out that there were lots of keys that were not being used. Therefore we decided to migrate certain keys only. This is why we built a tool to make this not as painful as it seems.
Generating Redis Dump File
First, we had to download the desired keys. Using ioredis we performed a SCAN with the match argument to obtain the keys and on each iteration an MGET to get its values.
const stream = redis.scanStream({ match: pattern, count: 10000 }); stream.on('data', (resultKeys) => { // Check if we have something to get if (resultKeys.length > 0) { const hashes = {}; redis.mget(resultKeys) .then((resultValues) => { for (let i = 0; i < resultKeys.length; i++) { hashes[resultKeys[i]] = resultValues[i]; } // Store key value }) } });
As the database was quite big, storing the whole set of key-values on an array was not good practice, so we went for a file approach. On each SCAN iteration, these sets of key-values were appended to a file, this way we prevented having unmanageable arrays.
fs.appendFileSync(fd, JSON.stringify(hashes), 'utf8');
The above operation generated a .json file with the desired keys and its values, now it's time to upload them. Although the file could be quite big, reading it and parsing it directly to a variable is not possible. Luckily, stream-json comes to help us. We only had to wait for a key-value pair to come up and SET it on the new Redis database.
const pipeline = fs.createReadStream(filename).pipe(parser());
Set a TTL to multiple KEYs
Once everything was working, we had to do some maintenance since some of the keys we found didn't have an expiry date. Part of that changes involved setting a TTL to some keys on the database. We needed to accomplish that without locking our entire database. The following code will scan all the keys in your Redis, optionally matching a pattern, and setting an EXPIRE to them.
const pipeline = fs.createReadStream(filename).pipe(parser()); pipeline.on('data', (data) => { switch (data.name) { case 'keyValue': key = data.value; break; case 'stringValue': ({ value } = data); redis.set(key, value); break; } });
Since this was a time-consuming process, we decided to pack everything together in one single package. You can download and use Redis-utils at:
Feel free to create any issue you may find and to contribute to it.
Check these articles about How ‘display none’ works for images, What is first contentful paint, A guide for Image Optimization and How to use Optimized Responsive Images.