<antirez>

Redis children can now report amount of copy-on-write

antirez 4148 days ago.
This is one of this trivial changes in Redis that can make a big difference for users. Basically in the unstable branch I added some code that has the following effect, when running Redis on Linux systems:

[32741] 19 Nov 12:00:55.019 * Background saving started by pid 391
[391] 19 Nov 12:01:00.663 * DB saved on disk
[391] 19 Nov 12:01:00.673 * RDB: 462 MB of memory used by copy-on-write

As you can see now the amount of additional memory used by the saving child is reported (it is also reported for AOF rewrite operations).

On Twitter, at Twitter

antirez 4154 days ago.
On Twitter:

@War3zRub1 "Hahaha it's silly how people use Redis when they need a reverse proxy"
@C4ntc0de "ZOMG! Use a real message queue, Redis is not a queue!"
@L4m3tr00l "My face when Redis BLABLABLA..."

Meanwhile *at* Twitter:

OP1: "Hey guys, there is a spike in the number of lame messages today, load is increasing..."
OP2: "Yep noticed, it's the usual troll fiesta trowing shit at Redis, 59482 messages per second right now."
OP1: "Ok, no prob, let's spawn two additional Redis nodes to serve their timelines as smooth as usually".

Eventual consistency: when, and how?

antirez 4157 days ago.
This post by Peter Bailis is a must read. "Safety and Liveness: Eventual consistency is not safe" [1].

[1] http://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is-not-safe/

An extreme TL;DR of this is.

1) In an eventually consistent system, when all the nodes will agree again after a partition?
2) In an eventually consistent system, HOW the nodes will agree about inconsistencies?
3) In immediately consistent systems, when I'm no longer able to write? When I'm no longer able to read?

Welcome to RethinkDB

antirez 4158 days ago.
There is a new DB option out there, I know it took a long time to be developed. While I don't know very well how it works I hope it will be an interesting player in the database landscape.

My initial feeling is that it will compete closely with Riak and MongoDB (the system seems more similar to MongoDB itself, but if it can scale well multi-nodes people that don't need high write availability may pick an immediate consistent database such as RethinkDB instead of Riak for certain applications).

Redis data model and eventual consistency

antirez 4158 days ago.
While I consider the Amazon Dynamo design, and its original paper, one of the most interesting things produced in the field of databases in recent times, in the Redis world eventual consistency was never particularly discussed.

Redis Cluster for instance is a system biased towards consistency than availability. Redis Sentinel itself is an HA solution with the dogma of consistency and master slave setups.

This bias for consistent systems over more available but eventual consistent systems has some good reasons indeed, that I can probably reduce to three main points:

Client side highly available Redis Cluster, Dynamo-style.

antirez 4165 days ago.
I'm pretty surprised no one tried to write a wrapper for redis-rb or other clients implementing a Dynamo-style system on top of Redis primitives.

Basically something like that:

1) You have a list of N Redis nodes.
2) On write, use consistent hashing and write the same thing to M nodes (M configurable).
3) On reads, read from M nodes and pick the most common reply to return to the client. For all the non-matching replies, use DUMP / RESTORE in Redis 2.6 to update the value of nodes that are in the minority.

Designing Redis replication partial resync

antirez 4166 days ago.
In this busy days I had the idea to focus on a single, non-huge, self contained project for some time, that could allow me to work focused as much as hours as possible, and could provide a significant advantage to the Redis community.

It turns out, the best bet was partial replication resync. An always wanted feature that consists in the ability to a slave to resynchronize to a master without the need of a full resync (and RDB dump creation on the master side) if the replication link was interrupted for a short time, because of a timeout, or a network issue, or similar transient issue.

Why Github pull requests lack support for labels?

antirez 4170 days ago.
I love Github issues, it is one of the awesome things at Github IMHO: as simple as possible but actually under the hood pretty full featured.

However one of the things I love more is labels. It is a truly powerful thing to organize issues in a project-specific way. Unfortunately if an issue is a pull request, no labels can be attached. I wonder why.

Also I would love the ability to merge against multiple branches instead of the taget one, directly from the web UI.

On complexity and failure

antirez 4172 days ago.
From a comment on Hacker News:

(link: http://news.ycombinator.com/item?id=4705387)

--- quoted comment ---
Full disclosure: I work for an AWS competitor.
While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have made this mistake with respect to network storage in the cloud; I, too, was fooled.[1]

Redis 2.6.1 is out

antirez 4173 days ago.
Achievement unlocked: releasing a Redis version the same day your daughter was born ;-)

But that was a bad issue as there was a bug preventing compilation on pretty old Linux systems that are still pretty widespread (RHLE5 & similar).

Redis 2.6.1 fixes just that issue and is available as usually at http://redis.io as a tar.gz or at github/antirez/redis as a "2.6.1" tag.

Back to technology

antirez 4175 days ago.
It's a more quite time now. Redis 2.6 released, the sexism issue almost forgotten. Time to relax, be wise, and focus on work. Right, but, that's not me. I've a few more things to say about what happened, and to reply to the many people that asked me why I felt "obligated" to stop using my Twitter account as before, with a mix of work, thoughts on technology, and personal stuff.

I can change idea easily if it is the case, but this time it was not the case. As much as people that criticised me for my blog post may think that I've a problem, I also think they have huge limits. Oh well, different opinions, I don't like you, you don't like me, I don't freaking care after all. I don't think on the same line as most people alive if that's the matter.

HN comment about Linus

antirez 4175 days ago.
h2s writes about Linus:

"I love this guy's balanced approach to steering the kernel. Somebody asked whether a bunch of security-related patches would be getting into Linus' tree, and his response was great.
Basically, he spent a few minutes explaining how security people tend to think that problems are either security problems or not worth thinking about. They see things in black and white and only care about increasing security at any cost. He said performance fanatics can be the same in their approach to improving performance, and he tries not to treat security or performance patches as being too massively different from any other types of patches such as ones for correctness.

About the recent EC2 issues.

antirez 4175 days ago.
I don't like people that are using recent EC2 problems to get an easy advantage / marketing. Stuff go down and cloud services are not magical, it is better to adjust the expectations.

But there are other reasons why people IMHO should consider going bare metal.

* EC2 (and similar services) are extremely costly. With 100 euros per month you can rent a beast of a dedicated server with 64 GB of RAM and fast RAID disks.

* As you can see you are not down-time safe, and to be down together with a zillion of other sites may be a good excuse with your boss maybe, but does not change your uptime percentage, so it's a poor shield.

Redis 2.6 is out!

antirez 4176 days ago.
Redis 2.6 is finally out and I think that now that we reached this point we'll start to see how the advantages of a release that was already exploited in production by few, will turn into a big advantage for the rest of the community.

Scripting, bitops, and all the big features are good additions but my feeling is that Redis 2.6 is especially significative as a step forward in the maturity of the Redis implementation. This does not mean that's bug free, it's new code and we'll likely discover bugs especially in the early days as with every new release that starts to be adopted more and more.

Github: where you see how cool humanity can be.

antirez 4177 days ago.
You are there in the morning with your coffee in front of you, scanning pull requests and bug reports, then you see a conversation around a commit among a few guys that modified the code to make it better, than there is another one suggesting to improve it in another way. You click in the account names and you see this people with their transparent eyes and your trust in humanity is restored.

Mission accomplished: videos talks for Redis Conf...

antirez 4180 days ago.
Takeaways:

1) Making videos is in some way harder than doing a talk live.
2) Screen Flow is awesome, but could be improved with more video editing capabilities, apparently you can't "cut" the video.
3) The problem is to upload files when they are big and you have normal ADSL connection :-)

But it feels good to be able to send the video talks a few days in advance, so the conf organizers will be able to perform editing, filter audio if needed, whatever.

Today is the day...

antirez 4181 days ago.
of the final recording of the videos I'll send to the Redis Conf. That was hard! The timing of the conf was not excellent for my attending, but producing the video was also less trivial than I thought, but finally I've the slides, an idea about what to say, and the ScreenFlow skills ;) Maybe after this experience I'll produce some video tutorial of Redis new features as I introduce it, in order to accelerate the adoption of new things in our community.

Now back to work...

Almost 1000 followers for @redisfeed in a couple of days

antirez 4181 days ago.
On twitter I read a few concerns about inability to read what I think about tech non-redis topics. First of all, thanks to everybody interested in my thoughts :) Second, this blog is exactly the place where I'll post everything like that.

So:

@redisfeed -> Redis news, mostly low traffic, high signal.
@antirez -> Will be converted into my personal account, mostly italian language, non work related.
@zeritna -> Will be simply dismissed.
This blog -> Everything about day by day Redis development, personal opinion about sexism, sky driving, shit eating and japanese food.

Next days...

antirez 4182 days ago.
I'm going to be a bit away from Redis code and this blog as I need to freaking focus on finishing the video talks for the Redis Conference that is very near at this point...

I'm also tuning and testing the final bits into 2.6 to make sure to release it ASAP :-)

See you in a couple of days.

p.s. also my wife feels contraction since a few hours, so maybe Greta is going to birth in a few... (!!!)

New 2.6 MONITOR behaviour with transactions

antirez 4182 days ago.
The commit message says it all:

    Fix MULTI / EXEC rendering in MONITOR output.
    
    Before of this commit it used to be like this:
    
    MULTI
    EXEC
    ... actual commands of the transaction ...
    
    Because after all that is the natural order of things. Transaction
    commands are queued and executed *only after* EXEC is called.
    
    However this makes debugging with MONITOR a mess, so the code was
    modified to provide a coherent output.
    
    What happens is that MULTI is rendered in the MONITOR output as far as

Random victim on Twitter

antirez 4182 days ago.
These are the real victims of flamewars:

"After the recent affair with @antirez's blogpost, I'm seriously considering forgoing further usage of #Redis in all my projects." (@nathell)

Obvious considerations:

1) Somebody will be happy about this as there was definitely a force trying to resonate as much as possible what was happening in the worst way.
2) The guy should read my blog post seriously.

But this is how it goes, you write a blog post that is a point of view about how to handle sexism (focusing on its effects, not the cause that anyway at the most subtle level, for instance a wrong promotion, can simply be denied by the author of the sexism), and the net result thanks to a number of people in part evil, in part simply bigot as hell, is that I'm sexist.

Ask your questions on Stack Overflow if you wish

antirez 4182 days ago.
Hey, since I'm no longer active on Twitter, that was a channel where from time to time I replied to requests from users, I'm now making sure I reply to a few questions on Stack Overflow every day, lurking on the "Redis" tag:

http://stackoverflow.com/questions/tagged/redis

I can't ensure you that I'll reply to your question, but I can ensure you I'll be there often to reply to a few questions a day, and together we can create a Redis ecosystem in one of the best places on the internet where to give or receive help, that is Stack Overflow.

New site look improved a bit.

antirez 4183 days ago.
Now it's like a mix between a twitter timeline and a blog. I fixed the RSS feed, but still could be generated better than that. Well the point is that as far as I've the min needed to improve it in the future will be easy and fun.

p.s. yes the layout and fonts are a bit of a mess, but it's not going to be too hard to fix it. For now I focused more on what it should display.

Warning: I forgot to increment the js version counter so it takes a few hard reloads of the page to get the right CSS.

Welcome to the new site!

antirez 4183 days ago.
Hi visitor! This blog was conceived for low traffic blogging. Now that I plan to don't use my Twitter accounts the old blog engine was not good enough.

The simplest thing to do was to take Lamer News and create a quick modified version that could be used as a blog engine... that's the result, for now. I hope to evolve it, but the point here is, I can write both long posts and very small ones that can be read directly from the home page.

It is also possible to write just short titles linking to external site, that is a feature I plan to use as well to link to interesting Google Groups comments and stuff like that.
: