Storing millions of data to redis

This week I was experimenting with storing 2 million entries that could be searched efficiently. I started to explore different solutions:

1) A simple C program to store data in tries
2) Rest service in node.js with C binding for efficiency
3) No SQL Database solution

Finally I decided to try using Redis. After doing some experiment with data set, I was able to store few entries into Redis in the format I needed, a redis hash. Now the big question came was of performance and time complexity?

I decided to try plain vanilla node-js code with redis client. Having finished coding, I decided to step out assuming it will take couple of minutes for it to run. To my surprise, even after 2 hours the task didn't complete. At this point I started researching better techniques.

Some google queries lead me to redis pipe line. However the standard redis client doesn't explain its usage. More research took me to the redis article about mass insertion in database.

At this point the task was carved out. I needed to implement redis protocol and pipe it to redis using redis-cli. After some trials, I came up with following function:

function gen_redis_proto(cmd_ar){

  var out_cmd = '*' + cmd_ar.length.toString() + '\r\n';

  _.each(cmd_ar,function(cmd_arg){
    var cmd_arg_str = cmd_arg;

    out_cmd += '$'+cmd_arg_str.length.toString()+'\r\n'+cmd_arg_str+'\r\n';
  });
  return out_cmd;
}

Above code takes an array of Redis commands and then spit out redis protocol. I was ready to run the code with some instrumentation. The code would crash redis-cli in windows. I switched to my MAC and still errors persisted. At that point I felt my data has some problem and began doing Divide & Concur on my data set. Finally after hours of debugging, I found out that my data has spaces which breaks redis protocol. Now I turned my eyes to simple encodeURIComponent function of JS.

After implementing the code I tried to experiment with 10,000 entries. Everything worked fine, so I decided to try 2 million entries. When I ran my code, I thought there is light at the end of the tunnel and it crashed again.

The problem was that redis-cli has default timeout of 30 sec. Again, I started looking around and found redis-cli code. The parameter to pass was --pipe-time with argument 0. This makes it wait for ever.

I ran my code with following command:

node app.js | redis-cli --pipe --pipe-timeout 0

It worked and I could populate redis with 2 million entries stored in less than 2 minutes with O(1) search time on a standard mac book pro.

PS: redis-cli still crashes on windows, that's something to look at another day or simply avoid windows and stick with linux/mac.

Fun reading:

http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value-pairs

http://blog.pivotal.io/pivotal/case-studies-2/using-redis-at-pinterest-for-billions-of-relationships

http://labs.opendns.com/2014/10/01/redesigning-dns-database-low-latency/