antirez weblog

With this blog post I'll try to keep you posted on the progresses of Redis, the recent changes to the INFO output, and the ETA for diskstore and cluster.

New INFO output

During the week end I managed to improve the INFO output that was rapidly becoming a mess ;) INFO is a very interesting command for Redis users, but there are a few issues that need to be addressed:

• Too much output., not separated by section. This has many drawbacks: complex to read for humans. Time consuming to generate as it can take more than a millisecond sometime, for some Redis uses this is not ideal.
• Collisions between the roles of INFO and CONFIG GET.
• More information needed, sometimes even super verbose. Things that we don't like to show in the default INFO output.

To address these problems what I did was:

• Now the INFO output is divided into sections, even from the point of view of human eyes.
• It is possible to ask for a specific section, passing one argument to INFO. Example: INFO memory will only show information about a single section of the output.
• For default INFO will output only default sections, like if you called INFO default. Instead INFO all will output everything. So now it is possible to add a few very verbose sections that simply are not printed by default, but are printed if the INFO argument is all or when the specific question is requested explicitly. On crashes or failed asserts all the sections are included in the stack trace.

All this will NOT be merged into 2.2, but is already part of Redis unstable.

An example of the new output:

redis> info
# Server
redis_version:2.3.0
redis_git_sha1:9b45592c
redis_git_dirty:1
arch_bits:64
multiplexing_api:kqueue
process_id:17790
tcp_port:6379
uptime_in_seconds:1012
uptime_in_days:0
lru_clock:1661879

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:931360
used_memory_human:909.12K
used_memory_rss:1085440
mem_fragmentation_ratio:1.17
use_tcmalloc:0

... more sections ...

It is possible to ask for specific sections. The following output shows one of the most interesting new sections:

redis> info commandstats
# Commandstats
cmdstat_get:calls=10019,usec=62234,usec_per_call=6.21
cmdstat_set:calls=10049,usec=83995,usec_per_call=8.36
cmdstat_incr:calls=10029,usec=80895,usec_per_call=8.07
cmdstat_lpush:calls=20091,usec=138814,usec_per_call=6.91
cmdstat_lpop:calls=10023,usec=69270,usec_per_call=6.91
cmdstat_lrange:calls=40126,usec=2607061,usec_per_call=64.97
cmdstat_sadd:calls=10048,usec=68385,usec_per_call=6.81
cmdstat_spop:calls=10009,usec=59072,usec_per_call=5.90
cmdstat_mset:calls=10049,usec=128621,usec_per_call=12.80
cmdstat_ping:calls=20053,usec=98911,usec_per_call=4.93
cmdstat_info:calls=3,usec=598,usec_per_call=199.33

(against my MBA 11", so typical servers will have smaller values)

I hope this will help us debugging better user problems, as for instance big set intersections or calls to KEYS will be easily detected. Note that in fast computers the 1 microsecond resolution provided by gettimeofday() may not be enough, but it's not a big problem, this is our equivalent of the MySQL "slow log". As long as slow operations are logged, this is fine. For all the rest there is the count of calls that is good to understand what fast operations are stressing the server.

As you can see the new format is almost backward compatible with the old one. There is just to handle empty lines and to filter lines starting with "#", that are now considered comments.

Diskstore and Cluster ETA

At the end of 2010 I was actively hacking again on Redis Cluster almost full time. Now the project is a bit delayed to be more aggressive developing diskstore brach.

Why, you may ask?

Because our logic was as follow:

• Great implementation of in memory single instance
• Decent option for datasets bigger than RAM in single instance
• Good cluster support, now that the single node works well

Then we discovered that the Virtual Memory implementation is not as good as we want it to be. So I started working at diskstore, that is a much simpler project, already in alpha stage and working, in a public repository.

In a few weeks I'll start against with Redis Cluster, that will be Redis 3.0, while Redis 2.4 will provide a stable implementation of Diskstore.

In the meamtime Pieter Noordhuis is doing a awesome work in the area of client libraries. We were focusing too much on the server side, but clients are also a very important part, and a few clients can really benefit from some serious optimization Pieter is working on.

A final note. It's almost one year at this point that I and Pieter are working at Redis thanks to VMware. I want to say thank you to this awesome company that made all you saw about Redis in the latest year possible.

Edit: so what's the ETA? More or less 2 months for beta quality (RC) diskstore, and at least six months for beta quality cluster. This are a bit optimistic forecasts, but after all in a few weeks we'll not have to work too much at 2.2 as it will ship stable, so there will be more time for the unstable branches.