Stuff you need:

sudo apt-get install build-essential
sudo apt-get install libexpat1-dev zlib1g-dev libssl-dev
sudo apt-get install m4 libncurses5-dev

Erlang:

wget http://www.erlang.org/download/otp_src_R12B-3.tar.gz
tar -xzf otp_src_R12B-3.tar.gz
cd otp_src_R12B-3
./configure
make
sudo make install

Ejabberd:

wget http://www.process-one.net/downloads/ejabberd/2.0.1/ejabberd-2.0.1_2.tar.gz
tar -xzf ejabberd-2.0.1_2.tar.gz
cd ejabberd-2.0.1/src
./configure
make
sudo make install

With that done just edit the config in /etc/ejabberd as required, generate or otherwise obtain an SSL certificate, and you’re ready to go.

sudo apt-get install libicu36 libicu36-dev libreadline5-dev
sudo apt-get install subversion-tools xsltproc automake libtool
svn -r 305 co http://couchdb.googlecode.com/svn/trunk/ couchdb
cd couchdb
./bootstrap
./configure --with-erlang=/usr/local/lib/erlang
make
sudo make install

Starting up CouchDB is now more straightforward:

couchdb

Let’s start with indexing. Last time we defined a table and queried it as follows:

-record(animal, {name,legs,noise}).
 
mnesia:create_table(animal,
                         [{disc_copies, [node()]},
                          {attributes, record_info(fields,animal)}]),
 
Fun fun() ->
  mnesia:match_object({animal,'_',4,'_'})
  end,
{atomic,Results}=mnesia:transaction(Fun).

Our query is returning us all animals with 4 legs, but if we’ve got any sizable amount of records in that table it’s going to run like a dog. The solution is simple:

mnesia:create_table_index(animal,legs).

The legs field is now indexed, and the match_object function will automatically pick up on that and give us more sensible performance that doesn’t involve full table scanning. Remember, as I said last time, that the first field is always considered to be the key, and is always indexed, so you only need to explicitly create an index for additional fields you intend matching against.

Another way of speeding things up is using ‘dirty’ access methods. These involve bypassing the transaction layer altogether, losing the Atomicity and Isolation from Mnesia’s ACID properties, but gaining a major reduction in overhead. Most of the normal access functions have a dirty equivalent – for example, dirty_match_object is the direct equivalent of the match_object function we’re using above. Methods specific to dirty operations are also available, including the wonderfully specific dirty_update_counter.

Using replicated copies of a table across multiple nodes, as well as providing redundancy, allows parallel access to the data. I can see a case for using this to improve performance by distributing the workload across multiple nodes, although in this case it needs to be a table which is read often and written occasionally, since there is a transactional penalty to writing to replicated tables.

Table fragmentation (i.e. the ability to distribute a table across multiple nodes) has performance possibilities too, since with different parts of the table on different nodes, operations can be conducted in parallel between them. I haven’t got fragmentation up and running yet though, so I don’t know how well this will work in practice.

When you consider that in addition to the above, Mnesia already has a head start due to the fact that it’s running in the same address space as the code, and with no mismatch between the Mnesia and Erlang to slow things down, the potential for high performance looks very good indeed. I’ve already got an Mnesia/Erlang back-end up and running which seems to be outperforming the MySQL/Erlang equivalent, although my tests so far are rather unscientific. More to come on this when time allows.

% On entry, A#arg.querydata is a string of the form "inc=n,n,n&exc=n,n,n
QueryArgs=string:tokens(A#arg.querydata,"&"),
QueryArgs2=lists:map(fun(T) -> string:tokens(T,"=") end,QueryArgs),
[[_|IncP]|_]=lists:filter(fun(T) -> [N|_Rest]=T,"inc"==N end,QueryArgs2),
[[_|ExcP]|_]=lists:filter(fun(T) -> [N|_Rest]=T,"exc"==N end,QueryArgs2),
IncS=string:tokens(lists:nth(1,IncP),","),
ExcS=string:tokens(lists:nth(1,ExcP),","),
Includes=lists:map(fun(T)->list_to_integer(T) end,IncS),
Excludes=lists:map(fun(T)->list_to_integer(T) end,ExcS).
% On exit, Includes=[n1,n2,...] and Excludes=[n1,n2,...]

Just as shocking as the code are the facts that a) it took me half an hour to make it work, and b) it actually works at all. Worse still, I don’t even need the code, since ultimately it will all be handled via JSON-RPC and not some nasty ad-hoc HTTP query processing – I just wanted to do a quick test in the meantime. The mission was accomplished, but surely there’s a far more elegant way to do the above. Unfortunately, I’m not going to find it today. I will be compelled to come back to it of course, and when I do I hope I can keep the lovely [[_|IncP]|_] part.

Needless to say, Erlang makes it easy to implement this kind of thing anyway but I just expected to find it there. You can imagine my excitement then when I came across plists, a module that’s compatible on a function-by-function basis with Erlang’s lists module, adding a range of powerful but simple to use parallel processing options. In addition, mapreduce and several other operations are added.

I haven’t had a chance to play yet, and it’s far too late in the evening to start now, so for the full low-down have a read here and then move on to the docs here. On top of all the nice looking functionality, you’ll soon be able to discuss ‘malts’ with those in the know.

Mnesia works best in conjunction with Erlang’s records. A simple record might look something like this:

-record(animal, {name,legs,noise}).

Here we’ve defined a record called “animal” with three fields. We can use this record definition to create an Mnesia table like this:

  mnesia:create_schema([node()]),
  mnesia:start(),
  mnesia:create_table(animal,
                         [{disc_copies, [node()]},
                          {attributes, record_info(fields,animal)}]).

When running this code, you will most likely want to use an Erlang command-line option like -mnesia dir ‘”db”‘ in order to specify the directory where the database will be created. It’s also important to pay attention to your node name and Erlang cookies if you intend to run a distributed Mnesia database.

Once we have a table, we can easily add records. The first point to note here is that ALL access to Mnesia goes within a transaction, which is defined in the form of a function. Here’s some simple code to add an item to the table we just defined:

  Fun = fun() ->
    mnesia:write(#animal{name="cow",legs=4,noise="moo"})
    end,
  mnesia:transaction(Fun).

Getting the data out again is also straightfoward, although there are many options. Firstly, note that by default Mnesia considers the first field in the record to be the key. With that in mind, you can use mnesia:read/2 to retreive a row by key:

  Fun = fun() ->
    mnesia:read({animal,"cow"})
    end,
  {atomic,[Row]}=mnesia:transaction(Fun),
  io:format("~p~n",[Row#animal.noise]).

All being well, the above should output “moo”. Another useful method for retreiving records is mnesia:match_object, which works as follows:

  Fun = fun() ->
    mnesia:match_object({animal,'_',4,'_'})
    end,
  {atomic,Results}=mnesia:transaction(Fun).

Here we end up with Results being a list of all the animals with four legs. Each entry in the list is a complete ‘animal’ record. Yet another option, and a very powerful one, is the Query List Comprehension. This uses the same concept as normal List Comprehensions, but mapped onto Mnesia, with some clever tricks behind the scenes to make it efficient. A quick example which simply retrieves all the animal records:

  mnesia:transaction(fun() -> qlc:eval(qlc:q([X || X <- mnesia:table(animal)])) end).

Finally, there is a very useful tool hidden away within the YAWS source that allows you to peruse your database from a browser. The code in question is ymnesia.erl, a YAWS appmod which can be found in the main src directory although it isn’t compiled by default. I found the easiest thing to do was to pull the file out into my project and run the appmod with YAWS in embedded mode as part of my application, which ensures it’s accessing the same database.

Any corrections to the above are welcome, since I’ve mostly typed it up from memory. I’ll no doubt cover more aspects of Mnesia at a later date, but in the meantime you can look at my earlier XML import code for a complete working example.

• I didn’t want to spend any time on it, and for various reasons I can do this kind of crap in C# with my eyes shut and no RTFMing.
• I needed a C# wrapper class to patch into an existing application, and this started out as a testbed for that.

If you haven’t been scared off already, here’s a brief summary of what you can do:

• Browse, create and delete databases
• Execute ‘queries’ (temporary views)
• Browse the documents in a database
• Create and delete documents

Here are some screenshots of the imaginitively titled CouchBrowse application:

It’s worth pointing out that it’s still fairly rudimentary, but working. In particular, it needs some proper JSON parsing rather than the ad-hoc horrors I threw together, and some decent formatting and syntax highlighting would be good too, but it suits my purposes for now.

If anyone is actually interested in getting hold of it, shout and I’ll stick the code and/or binaries somewhere. It was built with SharpDevelop 2.2 on Windows – I don’t know how hard it would be to get it running under Mono.

The first was due to xmerl returning unexpected data when an ampersand was present in an attribute value. Say, for example, the attribute has a value of “A&ampB”, I would expect Element#xmlAttributeValue to return the list [65,38,66]. What it actually returns is [65,[38],66]. That’s not just my expectation either – ejson, on which erlcouch is based, goes into a terminal sulk when asked to encode that. That’s surely a bug, it should at least fail rather than sitting there indefinitely. I’m inclined to think the xmerl output is wrong in the first place, as I can’t find anything in the documentation that says it would return anything other than a flat list.

In any case, the simple solution to that problem was to lists:flatten(Element#xmlAttribute).

The second problem was a bit more predictable – UTF-8 encoding. (Damn that Björk). I thought this must have been the same bug reported by Sam Ruby the other day, and I can see some UTF-8 related stuff in SVN since I last took at copy, but updating to latest revision (184) didn’t solve my problem. To get it working I had to do this:

cp -r /usr/local/lib/erlang/lib/xmerl-1.1.4/ $HOME/couchdb/lib

For some reason, CouchDB couldn’t find the xmerl_ucs library until I did. Presumably something wrong with my setup, although it works fine from the Erlang shell. (Update: see comments for a better fix.)

After that, it transpired that I had to UTF-8 encode the data myself BEFORE handing it to erlcouch, so with all the above in mind (and a bit of de-windification of the filename field) the new import code looks like this:

-module(djimportc).
-export([start/0]).
 
-include_lib("/usr/local/lib/erlang/lib/xmerl-1.1.4/include/xmerl.hrl").
 
start() ->
 
  % Connect to CouchDB and create database...
  Host = couch:get_host([{host, "localhost"}, {port, 8888}]),
  DB = couch:create_db(Host, "dj"),
 
  % Parse the document and add record records to the table...
  { Xml, _Rest } = xmerl_scan:file("../media/LwDJState.xml"),
  Records = xmerl_xpath:string("/lwdj/record",Xml),
  process_record(Records,DB).
 
% Process a list of records...
process_record([],DB) -> ok;
process_record([Head|Rest],DB) ->
  process_record_attrs(Head#xmlElement.attributes,[],DB),
  process_record(Rest,DB).
 
% Process attributes for a record...
process_record_attrs([],New_rec,DB) ->
  io:format("Writing document: ~p~n",[New_rec]),
  {{_,_},_}=couch:save_doc(DB, New_rec);
 
process_record_attrs([Head|Rest],New_rec,DB) ->
  Fieldname=Head#xmlAttribute.name,
  V=lists:flatten(Head#xmlAttribute.value),
  if
    Fieldname==filename ->
      {_,Value2,_}=regexp:gsub(V,"\\\\","/"),
      {_,Value,_}=regexp:gsub(Value2,"M:","");
    true ->
      Value=V
  end,
  process_record_attrs(Rest,New_rec++[{Fieldname,xmerl_ucs:to_utf8(Value)}],DB).

With that done, I have some data to play with, and I must say I was surprised (pleasantly) to see it all come back to me with the non-ASCII stuff intact when I tested it from the Javascript console.

This is how I built it from source, using my previously documented Debian test setup as a base system:

sudo apt-get install libicu36 libicu36-dev
mkdir src
cd src
svn co -r 180 http://couchdb.googlecode.com/svn/trunk/ couchdb
cd couchdb
export ERLANG_INCLUDE_DIR=/usr/local/lib/erlang/usr/include
./build.sh
./build.sh --install=$HOME

For whatever reason, I had to use a different libicu to Sam Ruby, but it worked. Starting it up:

~/couchdb/bin/startCouchDb.sh

Working with CouchDB is easy from any language or platform, since it’s based around a straightforward HTTP/REST interface. For the purposes of my initial experiments, I’m using erlcouch, which is a wrapper library for Erlang. I did this to set it up:

cd ~
svn co -r 4 http://erlcouch.googlecode.com/svn/trunk/ erlcouch

The SVN version includes the compiled BEAM files, so I didn’t bother building anything, and went straight to importing the same dataset as I did into Mnesia previously, with an adapted version of that Erlang program as follows:

-module(djimportc).
-export([start/0]).
 
-include_lib("/usr/local/lib/erlang/lib/xmerl-1.1.4/include/xmerl.hrl").
 
start() ->
 
  % Connect to CouchDB and create database...
  Host = couch:get_host([{host, "localhost"}, {port, 8888}]),
  DB = couch:create_db(Host, "dj"),
 
  % Parse the document and add record records to the table...
  { Xml, _Rest } = xmerl_scan:file("../media/LwDJState.xml"),
  Records = xmerl_xpath:string("/lwdj/record",Xml),
  process_record(Records,DB).
 
% Process a list of records...
process_record([],DB) -> ok;
process_record([Head|Rest],DB) ->
  io:format("Processing record~n",[]),
  process_record_attrs(Head#xmlElement.attributes,[],DB),
  process_record(Rest,DB).
 
% Process attributes for a record...
process_record_attrs([],New_rec,DB) ->
  couch:save_doc(DB, New_rec);
 
process_record_attrs([Head|Rest],New_rec,DB) ->
  Fieldname=Head#xmlAttribute.name,
  Value=Head#xmlAttribute.value,
  process_record_attrs(Rest,New_rec++[{Fieldname,Value}],DB).

The good news is that it sort of worked, in that 44 of the 6714 records were uploaded before the HTTP sending process seemed to stall. I could see that the data was in the database using the Javascript console (http://localhost:8888/_utils/shell.html), so I suspect that either something is wrong in erlcouch, or I have done something stupid in the above code – CouchDB itself seemed to be quite happy and responsive. The bad news is that I’m determined to get enough sleep tonight, what with having real work to do tomorrow, so no answers will be forthcoming until tomorrow night at the earliest.

I also can’t help viewing those VMware ‘appliance’ things with suspicion, and I’m starting to put real data on these things now, so I went for a clean install from debian-40r1-i386-netinst.iso. Doing the initial setup via the VMware Server console, viewed via VNC, is unbelievably painful but all that was needed after the actual install was to reconfigure the networking and install sshd before getting back to a nice clean SSH login.

I’m slightly cross with myself for not having scripted the rest of the setup yet, but on the other hand I’m pleased I was anal enough to document it all in the first place, because all I had to do was copy the commands out of my previous posts here and paste them into the SSH window to get back to where I was. I also set up Debian to use sudo, because I prefer that to using su.

The only differences in all the setup steps from the previous ‘doing X on Ubuntu’ posts due to the change of distribution were:

• The m4 package was already installed.
• Instead of libssl-dev, I needed openssl.
• I left out the detail of how to fix ejabberdctl. (added as a comment to that post)

I didn’t go as far as the Mnesia data import stage, stopping instead after installing and testing Erlyweb, for reasons that will become apparent later. For the record, the steps were:

• Erlang
• Ejabberd
• Jabberlang
• YAWS
• ErlyWeb