Ten minutes doesn’t seem like much of a commitment on the surface. But in my world of 2 kids, always-connected IM and other distractions, I actually found the first week of the regime quite difficult. I’m used to hoping topics quickly and doing a lot of skimming. Writing emails and IMs is often very close to stream of consciousness and only occasionally requires a lot of focused thought. It seems like I’ve lost touch with the ability to put words to paper/screen efficiently and effectively. What’s worse – I’ve lost touch with the ability to absorb other topics, so ruthless is my drive to be expert in my field.

So, I’m trying this 10 minute regime to try and recover the skills that I’ve lost. Before I picked up programming, I loved to write; poems and short stories were great creative outlets for me. Perhaps I can recover these delights, or at least, take back the ability to communicate in the ambiguous but graceful grammar of humans.

I was diagnosed with follicular lymphoma three weeks ago now.

I’m blessed in a lot of ways. The cancer is slow moving, non aggressive – or so it appears at this point. I might not even require treatment in the near future. Even if I do require treatment, survival rates have jumped from 60% to 90% in the past five years – the treatment for this cancer is progressing quickly. My company, Basho, has been wonderful to me in terms of helping me sort out a variety of insurance issues and arranging access to very good doctors.

All of these things are probably the reason I’ve not had any trouble sleeping until tonight.

It’s still scary though. Cancer – just the word inspires fear when you first hear it. You are struck, relatively quickly, with the fragility and preciousness of life. You suddenly have a deep desire to grow old. The prospect of death is a powerful incentive to live.

I cried more the first few days and weeks than I ever have in my 32 years. I cried because I was scared. I cried because I was worried about my wife, our 2 year old and the new baby on the way. I cried because it felt unfair, unwarranted! I cried because I realized that there were some areas of my life that I had wasted – and I wondered if I would have the chance to rectify them.

As I’ve gotten further into this process, emotions have settled out a bit. I realize now just how good I have it with this cancer. What I’m facing is absolutely nothing compared to other people I know with chronic medical conditions. It’s a smudge on the screen; a minor distraction. There might be some tough times ahead, but my overall probability for immediate mortality is relatively stable and low.

That said, I’m determined to make the most of this challenge. If I must go through this valley, I’m going to extract every bit of growth from it that I can. I choose to grow, to push my boundaries in every dimension: physically, spiritually, mentally, emotionally. I choose to spend more time with my family and less time with wandering the mental spaces of coding. I choose to listen more and speak less. I choose to be grateful that all of these realizations have been granted to me at 32 instead of 64.

It’s now 2.21 am. I think it was just the Chinese food from dinner that woke me up.

Thus, rebar was born. As a strictly Erlang implementation, it’s possible for Erlang developers to dig into it and improve/modify with minimal effort. It’s also wickedly fast, since it starts the VM up only once and has direct access to all the tools one needs to build and validate Erlang code. It has the added advantage of being able to take advantage of Erlang’s inherent parallelism, so where possible, it runs commands concurrently. Finally, it’s designed to be a self-contained escript, so using rebar doesn’t introduce any build dependencies other than a stock Erlang install. You simply drop the rebar script into your code tree and go!

You can see a demonstration of converting an existing app to rebar here.

Create and compile a simple OTP application by doing the following steps on a terminal:

1
2
3
4
5
6

 
$ mkdir myapp; cd myapp
$ wget http://bitbucket.org/basho/rebar/downloads/rebar; chmod u+x rebar
$ ./rebar create-app appid=myapp
$ ./rebar compile
 

Documentation is still scarce – that’s something I’m going to be working on over the next few weeks. The core pieces of rebar are mostly at a point that I’m happy with; now it’s time to polish. :)

If you have questions about rebar, or especially feedback after using it IRL, please ping me on Freenode IRC – I’m typically in the #riak room.

1
2
3

 
 exec $BINDIR/erlexec ${1+"$@"}
 

to:

1
2
3
4
5
6
7

 
if [ ! -z "$USE_GDB" ]; then
        gdb $BINDIR/erlexec --args $BINDIR/erlexec ${1+"$@"}    
else
        exec $BINDIR/erlexec ${1+"$@"}
fi
 

Now all you have to do to get Erlang running in gdb is:

1
2
3
4

 
$ export USE_GDB=1
$ erl
 

If all goes well, you should see something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

 
(dizzyd@sigr).(~)% export USE_GDB=1
(dizzyd@sigr).(~)% erl
GNU gdb 6.3.50-20050815 (Apple version gdb-1344) (Fri Jul  3 01:19:56 UTC 2009)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries .. done

(gdb) r
Starting program: /Users/dizzyd/Applications/erlang-r13b03/lib/erlang/erts-5.7.4/bin/erlexec 
Reading symbols for shared libraries +. done

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007fff5fc01028 in __dyld__dyld_start ()
(gdb) c
Continuing.
Reading symbols for shared libraries ... done
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> 
 

However, i remain convinced that one should not force clients to deal with stale reads in environments where they can be avoided. As i have mentioned in the updated initial post - there are simple examples where stale reads cause havoc. One may not be able to do conflict resolution or the reads can affect other keys in ways that are hard to fix later.

Arguing applications “may not be able to do conflict resolution” is non-sensical – by definition, Dynamo requires that the application be cognizant of conflict resolution! This isn’t an arbitrary decision to make clients aware of conflicts. It’s a part of a measured approach to building a robust system. One may not agree with it, but to claim that Dynamo is universally flawed just because it does not conform with one’s personal feeling about layering is dis-ingenous at best.

Please understand me, I make no claim that Dynamo is the end-all-be-all for data stores. It is a terrible, terrible choice for some problem spaces. However, if you want a low-latency, highly-robust key/value store it works quite well.

About Vector Clocks and multiple versions - it’s not a surprise that they were not implemented in Cassandra. In Cassandra - the cost of having to retrieve many versions of a key increases the disk seek costs reads multi-fold. Due to the usage of LSM trees, a disk seek may be required for each file that has a version of the key. Even though the versions may not require reconciliation, one still has to read them.

This is an argument about implementation details of Cassandra and has nothing to do with whether or not Dynamo is a universally flawed architecture. I can say from experience that vector clocks do not have to be slow – as with anything, careful implementation can yield surprisingly fast results. I would also note that in the production systems where I’ve deployed Dynamo-clones, the actual occurrence of multiple versions (or conflicts, in Dynamo terms) is quite rare. The original Dynamo paper (sect 6.3, para 3) notes that 99.94% of all requests return a single version; this matches closely with what I’ve observed in my own production deployments today (99.91%).

Also, implementation-wise, one doesn’t typically keep resolved versions lying around – the only time there are multiple versions present on disk is when a conflict has not been resolved. One could keep old versions around, I suppose, and in that situation I agree that you would want to carefully design your store so as to avoid unnecessary seeks when reading the “current” version.

So, unfortunately, i am repeating this yet again - Dynamo’s quorum consensus protocol seems fundamentally broken. How can one write outside the quorum group and claim a write quorum? And when one does so - how can one get consistent reads without reading every freaking replica all the time? (well - the answer is - one doesn’t - which is why Dynamo is eventually consistent. I just hope that users/developers of Dynamo clones realize this now).

As Mr. Sarma astutely points out, the reason Dynamo works is because it makes no guarantees about instantaneous consistency. Assuming (again) that the client can tolerate conflicts and that the cluster will attempt to resync at the earliest possible opportunity, writing to non-authoritative nodes is perfectly fine. The system will eventually come back into consensus.

Unfortunately, I’m pretty sure that my arguments will be insufficient to convince Mr. Sarma of the utility of Dynamo. I hope, however, that anyone reading this discussion will consider that reviewing the concepts of a paper is a very different task from executing on those concepts. As someone who has successfully executed ideas from that paper, I can assure Mr. Sarma that the concepts not only work, but they work surprisingly well.

Finally, the real contribution of the Dynamo paper is the balance that was struck between performance, reliability and pragmatism in the design of a production DHT. It underscores the importance of taking nothing for granted and being willing to consider counter-intutitive solutions to hard problems.

Joydeep Sarma posted an entry claiming that Dynamo is a “flawed architecture”. I’m not really qualified to prove or disprove Mr. Sarma’s claim, but having implemented a Dynamo clone, I think that he may be a little confused about how things work in these systems. What follows are a few quotes from his write-up followed by my own responses.

Let’s say that one is storing key-value pairs in Dynamo - where the value encodes a ‘list’. If Dynamo returns a stale read for a key and claims the key is missing, the application will create a new empty list and store it back in Dynamo. This will cause the existing key to be wiped out. Depending on how ’stale’ the read was - the data loss (due to truncation of the list) can be catastrophic. This is clearly unacceptable. No application can accept unbounded data loss - not even in the case of a Disaster.

Dynamo implementations protect against this scenario by using vector clocks. If we define a “stale read” as one which returns the key (or absence thereof) and an older vector clock, then any writes which use this older/non-existent vector clock will generate a conflict and the server will store two versions of the same key. The application then has the opportunity to resolve this conflict on the next read. When used in conjuction with quoroms for reads and writes, this approach proves to be exceedingly robust.

Dynamo starts by saying it’s eventually consistent - but then in Section 4.5. it claims a quorum consensus scheme for ensuring some degree of consistency. It is hinted that by setting the number of reads (R) and number of writes (W) to be more than the total number of replicas (N) (ie. R+W>N) - one gets consistent data back on reads. This is flat out misleading. On close analysis one observes that there are no barriers to joining a quorum group (for a set of keys). Nodes may fail, miss out on many many updates and then rejoin the cluster - but are admitted back to the quorum group without any resynchronization barrier. As a result, reading from R copies is not sufficient to give up-to-date data.

One of the foundational assumptions in the Dynamo system is that you define as many replicas as necessary to achieve your desired level of reliability. As with any replication based system, if you lose all of your replicas, there is no meaningful recovery. However, if we assume that you will always have some number of replicas functional, and we introduce an appropriate quorum on operations, we can identify those nodes which return stale data and repair them appropriately. In other words, it’s perfectly possible not to have resync barrier on joining, yet still ensure consistency in the answers provided to the client.

It might be helpful to recall that there are three levels of repair: read-repairs, hinted handoffs and replica synchronization. Two of these three are done in near-real time, thus minimizing the actual drift between nodes. Read repair deals with stale data on a per key/operation basis; the coordinator for a request can identify nodes responding with stale data and update them accordingly, using responses from other less stale nodes. Hinted handoffs are a bulk operation that is done when a node rejoins the cluster – the keys updated while the node was down are replayed (in essence) to the rejoining node. Replica sync is something that is typically done once a day and does require a traversal of all the data for a given partition. Tricks like Merkel trees, however, permit only the changed portion of the data to be exchanged, so in practice it’s not nearly as expensive as one might imagine in the abstract.

Lack of point in time consistency at the surviving replica (that is evident in this scenario) is very problematic for most applications. In cases where one transaction (B) populates entites that refer to entities populated in previous transactions (A), the effect of B being applied to the remote replica without A being applied leads to inconsistencies that applications are typically ill equipped to handle (and doing so would make most applications complicated).

The Dynamo paper makes it very clear that applications do require more logic to deal with these situations. Yes, it’s more work for the application, but in practice, it’s not that bad. It’s also important to point out that data dependencies are handled differently in these key/value stores than they are in a typical ACID environment. Usually apps will store the data in a denormalized form, so dependencies amongst key versions are minimal (if they exist at all). This makes it much easier to deal with conflicts as all the relevant data is on hand during the resolution phase.

I’ll leave it to someone else to do a more exhaustive analysis of Mr. Samra’s arguments. It’s been my experience over the past 2 years that Dynamo is one of those systems that you really have to see in action (or implement it) to appreciate the wonderful elegance and resiliency of the design. It’s certainly not a one-size-fits-all solution, but works very well in the appropriate problem space.

Please note that erlbox (and this blog entry) isn’t necessarily where you want to start when you’re first learning Erlang – see any of the excellent books for a good introduction to Erlang.

To get started with erlbox, you first need to install Ruby and RubyGems. One you have RubyGems installed, you can then do:

1

 $ gem install erlbox 

This should pull down erlbox as well as Rake and any other dependencies. Note that the Debian version of RubyGems is a little weird – my experience is that using RubyGems from source on Debian yields the best results.

Once you have erlbox installed, you’re ready to put together your an Erlang application. In keeping with OTP guidelines, we’ll start by creating a standard OTP directory structure:

1

 $ mkdir -p testapp/ebin testapp/src 

The next step is to create an application descriptor (ebin/testapp.app) so that Erlang/OTP knows how to start our application up. Drop the following text into testapp/ebin/testapp.app:

1
2
3
4
5
6
7
8
9
10
11
12

 {application, testapp,
 [{description, "Test Application"},
  {vsn, "1"},
  {modules, [ testapp,
              testapp_sup ]},
  {registered, []},
  {applications, [kernel, 
                  stdlib]},
  {mod, {testapp, []}},
  {env, [
        ]}
 ]}. 

The hows/whys of OTP .app files are beyond the scope of this entry – see the Erlang docs, specifically “Working with OTP/Design Principles” for more details.

With the .app file and the basic directory structure in place, we are now ready to create the Rakefile that will be used by Rake to kick off the erlbox tasks. For this simple build, it’s a one-liner:

1

 $ echo "require 'erlbox'" > testapp/Rakefile 

We can now use Rake to build our Erlang app:

1
2
3
4
5
6
7
8
9
10

 $ cd testapp; rake
(in /home/dizzyd/src/testapp)
validating testapp.app...
rake aborted!
One or more modules listed in testapp.app do not exist as .beam:
 * testapp
 * testapp_sup

(See full trace by running task with --trace)
 

As you can see, we still have some work to do. One of the important things that erlbox does is validate the application descriptor ebin/testapp.app and ensure that all the modules it lists are present in compiled form in the ebin/ directory. In this case, the .app file claimed that a module named “testapp” would be present as ebin/testapp.beam, and erlbox generated an error when the module was not found.

So, let’s create the source for the testapp module. Drop the following text into testapp/src/testapp.erl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

 -module(testapp).

-behaviour(application).

%% Application callbacks
-export([start/2, stop/1]).

%% ===============================================================
%% Application callbacks
%% ===============================================================

start(_StartType, _StartArgs) ->
    testapp_sup:start_link().

stop(_State) ->
    ok.
 

We also need to define the application supervisor in testapp/src/testapp.erl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

 -module(testapp_sup).

-behaviour(supervisor).

%% API
-export([start_link/0]).

%% Supervisor callbacks
-export([init/1]).

%% ===================================================================
%% API functions
%% ===================================================================

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

%% ===================================================================
%% Supervisor callbacks
%% ===================================================================

init([]) ->
    {ok,{{one_for_one,5,10}, []}}.
 

The erlbox tasks know to look for all source files in testapp/src/.erl and compile them to testapp/ebin/.beam. With these files in place let’s try rake again:

1
2
3
4
5
6

 $ cd testapp; rake
(in /home/dizzyd/src/testapp)
compiling src/testapp.erl...
compiling src/testapp_sup.erl...
validating testapp.app...
$ 

The build completed cleanly. Congratulations, you’ve just built a basic Erlang/OTP application using erlbox.

There are a lot more features in erlbox other than what I’ve covered here. It also has the ability to build/compile OTP embedded nodes, SNMP MIBs, port drivers and other everyday components that make up the OTP platform.

I am slowly recovering. Now in this slow, happy time of Christmas and New Year’s I find myself without my normal drive. I feel empty and light; it’s disturbing after the harried pace of the past 4 months. There are so many side projects that I want to work on, but simply can’t find the energy or desire to focus on them.

Over the years I’ve come to realize that the creative expenditure of creating software comes at a price. I have a fixed capacity for creating software – if I expend that capacity it requires time to refuel. In the interim, I can still create but at a much diminished pace, and typically with a much lower quality than what I am accustomed to. The best thing to do, typically, is NOT create. Wait, pause and be patient. Permit focus to drift until it’s ready to snap back to laser precision for the next Push.

This post probably sounds like nonsense – perhaps it is.

I’ve always loved the feeling that I get when I lose myself to a particularly challenging or fun piece of coding. It’s that state of mind where you lose track of the passage of time and focus all your energies on turning ephemeral ideas into billions of electronic pulses. There is a clarity of thought in that state, and I would love to experience it more often.

The problem is, there is always clutter and noise. So, the logical question is, how does one eliminate these things and encourage a more constant state of clarity?

For myself, I’ve found that GTD is at least a starting point. It provides a framework on which to capture actions and ideas in a way that shunts the responsibility for tracking stuff from my brain to a more reliable store. As I’ve been consistently doing this for the past week, my list of actions/projects has grown far more rapidly than I would have ever thought. The amount of stuff that we juggle in our heads is truly prodigious – no wonder the average attention span in our society is under 3 minutes.

At any rate, I’m now using emacs on a regular basis for editing C/C++ code and got tired of switching buffers manually between header (.h/.hpp) and implementation (.c/.cpp) files. So I hacked a little lisp for my .emacs to make life better. Maybe someone else will find this useful too..

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

 
;; Association list of extension -> inverse extension
(setq exts '(("cpp" . ("hpp" "h"))
             ("hpp" . ("cpp" "c"))
             ("h"   . ("cpp" "c"))))

;; Process the association list of extensions and find the last file
;; that exists
(defun find-other-file (fname fext)
  (dolist (value (cdr (assoc fext exts)) result)
    (if (file-exists-p (concat fname "." value))
        (setq result (concat fname "." value)))))

;; Toggle function that uses the current buffer name to open/find the 
;; other file
(defun toggle-header-buffer()
  (interactive)
  (let ((ext (file-name-extension buffer-file-name))
        (fname (file-name-sans-extension buffer-file-name)))
    (find-file (find-other-file fname ext))))

;; Bind the toggle function to a global key
(global-set-key "\M-t" 'toggle-header-buffer)
 

When you build the new SSH Assistant, the output is a single pref pane bundle. Install that bundle by double-clicking on it. Then go into the pref pane and click the “enable” checkbox. You may have to logout so that it can setup your environment.plist properly, but that’s the extent of the install process.

There are two tricks to get this to work. First, use the mkhybrid from MacPorts. Secondly, make sure that the location of the boot image is inside the hierarchy of files being burned to disk. I’m not sure why mkhybrid requires this, but it means the difference between working or not. Ultimately the command line should look something like:

mkhybrid -joliet-long -iso-level 4 -r -b 4.1/i386/cdrom41.fs -c boot.catalog -o obsd.iso OpenBSD

One thing I have been doing is continuing to work on my Erlang skills. The tricks you can pull with that platform is outright amazing.

I suppose that these characteristics that I miss are really the most superficial ones; I know that family was far more important than any amount of code or elegant design to him. That’s how it should — code lasts only long enough to get re-written, typically.

I guess all of this is to say that I remember him and wish things had worked out differently.

One week, $25 in shipping (both ways) and 10 minutes on the phone later, I have a functioning motherboard on its way back to me. Obviously, I haven’t tested it yet, but my interactions with J&N left me feeling that it’s a skilled outfit that knows how to keep customers happy. More importantly, I find them to have a sense of class – they went out of their way to avoid flaming Monarch in my blog comments. It’s rare to find this sort of respect (if that’s the right word) for fallen competitors in today’s business world.

This the type of company I like doing business with – technically competent, user friendly and well-mannered. I wasn’t planning on ever building another system, but with a company like J&N providing good service…why not?

I want to live less cheaply in 2007.

For me, this means reading, thinking and writing more – working and procrastinating less. It means more time listening to things that people say and less time focusing on how to respond. More yogurt, less coffee. I want to spend more time thinking about what I believe and why.

These are my resolutions. This is the person I want to be.