Gradual Epiphany

Further Thoughts on Dynamo's "Flawed Architecture"

Mr. Sarma revisits his claims that Dynamo is a universally “flawed architecture”. I certainly concur that Dynamo has its flaws, but making sweeping claims about something being universally so is to under-value the contribution to production thinking that Dynamo contributes. So, once again, I’m going to take a few choice quotes from Mr. Sarma and respond to them.

However, i remain convinced that one should not force clients to deal with stale reads in environments where they can be avoided. As i have mentioned in the updated initial post - there are simple examples where stale reads cause havoc. One may not be able to do conflict resolution or the reads can affect other keys in ways that are hard to fix later.

Arguing applications “may not be able to do conflict resolution” is non-sensical — by definition, Dynamo requires that the application be cognizant of conflict resolution! This isn’t an arbitrary decision to make clients aware of conflicts. It’s a part of a measured approach to building a robust system. One may not agree with it, but to claim that Dynamo is universally flawed just because it does not conform with one’s personal feeling about layering is dis-ingenous at best.

Please understand me, I make no claim that Dynamo is the end-all-be-all for data stores. It is a terrible, terrible choice for some problem spaces. However, if you want a low-latency, highly-robust key/value store it works quite well.

About Vector Clocks and multiple versions - it’s not a surprise that they were not implemented in Cassandra. In Cassandra - the cost of having to retrieve many versions of a key increases the disk seek costs reads multi-fold. Due to the usage of LSM trees, a disk seek may be required for each file that has a version of the key. Even though the versions may not require reconciliation, one still has to read them.

This is an argument about implementation details of Cassandra and has nothing to do with whether or not Dynamo is a universally flawed architecture. I can say from experience that vector clocks do not have to be slow — as with anything, careful implementation can yield surprisingly fast results. I would also note that in the production systems where I’ve deployed Dynamo-clones, the actual occurrence of multiple versions (or conflicts, in Dynamo terms) is quite rare. The original Dynamo paper (sect 6.3, para 3) notes that 99.94% of all requests return a single version; this matches closely with what I’ve observed in my own production deployments today (99.91%).

Also, implementation-wise, one doesn’t typically keep resolved versions lying around — the only time there are multiple versions present on disk is when a conflict has not been resolved. One could keep old versions around, I suppose, and in that situation I agree that you would want to carefully design your store so as to avoid unnecessary seeks when reading the “current” version.

So, unfortunately, i am repeating this yet again - Dynamo’s quorum consensus protocol seems fundamentally broken. How can one write outside the quorum group and claim a write quorum? And when one does so - how can one get consistent reads without reading every freaking replica all the time? (well - the answer is - one doesn’t - which is why Dynamo is eventually consistent. I just hope that users/developers of Dynamo clones realize this now).

As Mr. Sarma astutely points out, the reason Dynamo works is because it makes no guarantees about instantaneous consistency. Assuming (again) that the client can tolerate conflicts and that the cluster will attempt to resync at the earliest possible opportunity, writing to non-authoritative nodes is perfectly fine. The system will eventually come back into consensus.

Unfortunately, I’m pretty sure that my arguments will be insufficient to convince Mr. Sarma of the utility of Dynamo. I hope, however, that anyone reading this discussion will consider that reviewing the concepts of a paper is a very different task from executing on those concepts. As someone who has successfully executed ideas from that paper, I can assure Mr. Sarma that the concepts not only work, but they work surprisingly well.

Finally, the real contribution of the Dynamo paper is the balance that was struck between performance, reliability and pragmatism in the design of a production DHT. It underscores the importance of taking nothing for granted and being willing to consider counter-intutitive solutions to hard problems.