Archive for the ‘Erlang’ Category

h1

Changes in Chatterl

July 23, 2009

I haven’t worked on Chatterl for a few months but there are some fundemental changes that had been made back in March, apologies for the slow update but honestly I haven’t had much time with learning Rails & freelance work.

Summary

Well the main changes in chatterl are the following:

  • Message storage
  • Registration
  • Introduction of unit tests
  • CWIGA now RESTFUL
  • API calls

I’ll go over each section, highlighting the main changes & who it affects Chatterl as a whole.

Meet CWIGA

After implementing Chatterl‘s basic functionality (see chatterl post). The next step was to create to implement CWIGA, Chatterl Web Interface Gateway API. With this new module Chatterl is able to be interacted with purely via HTTP. The first implementation was rough, the original source can be found here, though it allowed me to get a decent understanding of mochiweb at the time helped me produce this article. After implementing unit tests to cover the project as a whole I felt comfortable to make a few fundamental changes to CWIGA’s API and requests to Chatterl.

Registration

I’ve added very basic registration functionality to Chatterl, allowing users to register to the system, giving them the ability to retrieve archived messages & later creation of their own chat groups. In hindsight I’d like to move this functionality to the front-end (which I’ll more than likely be in Rails).

Message Storage

Again this was more experimental on my part and something I would like to move over to couchDB when I get the chance, for the moment, Chatterluses mnesia to store users messages when they are not logged into the system.

Added test cases

One of the main issues I’ve been having with Sinan more than anything else, was seperating my tests from the source code. I really hated the idea of having test & implementation code mixed in, so I stayed away from any kind of real testing until I got my head around it. Thanks to Kev Smith‘s own code, I finally got my head around EUnit, which I posted a short post on here.

Before I was comfortable with implementing any more functionality I had set out to cover as much of Chatterl‘s code as possible. In doing so I uncovered a number of silly little bugs (oversights) in the coding logics & made the necessary changes.

General cleanup

The way chatterl is called has also been improved, instead of having to go into erl you can simple run ./start-server.sh within the cd _build/development/apps/chatterl-0.1.2.0/ directory. This will start up all the processes need to use chatterl.

CWIGA is setup to listen on port 9000, so you can call it using curl or via a browser of your choice. It accepts XML & JSON requests & has been made more RESTful.

Hopefully in the near future I’ll have time to rewrite the code base and improve on the basic functionality but for the meantime I hope this helps others as much as its been an experience for me.

h1

Lighting up the tunnErl Pt. 9 – The Gotchaz

March 13, 2009

Over the years I’ve become pretty good at picking up on errors, learning a new paradigm as well as a new language helps to introduce a lot more of these than usual so I’ve put some of the more common errors I have come accross to help others pick up on the errrors that had me baphled from one extent to another.

  • function_clause – generally means that the result was not matched for one reason or another, make sure you are retrieving the result you expected.
  • noproc – either the process is not started or the node cannot communicate with it, first try net_adm:ping(hostname). then check to see if the process is alive erlang:is_process_alive(Pid).
  • undef – Happen when a function in module does not exist, the module is not loaded or not exist at all. Maybe the function name has been misspelt.
  • ‘*.app not found’ – this typically means that the application can not be found or it is not included, check that it is part of you dependancies and that has been started properly.
  • error:{badfun,ok} – EUnit based error missing comma, usually because assert is returned in wrong place.
  • syntax error before: ‘end’ – have added an “,/;’ proceeding the end
  • none:error:internal error in beam_asm – Sinan based error, happens when code is out of sync, best solution is to kill sinserv and run sinan again.

As time goes by I’ll continue to add more, what this space. If anyone has come across a gotcha that i’ve missed drop me a line, i’ll be more than happy to add it to this list.

h1

Lighting the tunnErl Pt 6 – Using Mochiweb for Chatterl’s RESTful API

January 15, 2009

This post is the next in the ‘Lighting the tunnErl‘ series, documenting my experiences with Erlang as I learn it and its tools, as the previous post focused on Faxien & Sinan, I decided to concentrate on working with yaws or Mochiweb. As I’m still actively working on Chatterl, which as mentioned here, is a multi-node chat system, and it needed a web interface, I did some research and decided to use Mochiweb, though the part six was originally going to be ‘Erlang Gotchaz’, I thought I should post this first whilst the information is still fresh in my mind.

General overview

I recently started to play around with Mochiweb & BeepBeep to see what functionality they could provide Chatterl, after a couple of days working around the BeepBeep’s code, I decided to focus on the actual gateway API (CWIGA) and use Mochiweb along with a little manipulation to create the gateway service that would in turn serve the frontend which will use BeepBeep to render the data. A bit convoluted maybe but I want other to be able to use the backend separately and plug in their own frontends ;), not to mention it’s an excellent chance to play with new toys.

The foundations weren’t that hard to put together, think it took all of 2 minutes to get Mochiweb running, setting up the basics are just that & simple interactions with other systems is straight forward. Though saying that I quickly found that one can encounter some serious head banging moments with Mochiweb especially with the lack of documentation at present.

I also feel Mochiweb lacks fundamental features like destroying cookies, manipulating headers/query string & the such like a lot of this is left down to the developer, which in my opinion isn’t ideal, I’d rather spend time on the problem at hand rather than deal with mini side effects.

I really really hate pros/cons, loves/hates, but hell I really do have a love/hate relationship with Mochiweb at the moment, I’m hoping that this is just part of the curve but I have to admit it certainly is an interesting beast.

Loves

  • Quick & easy to initialise Mochiweb.
  • Initialising cookies is simple.
  • Fully customisable
  • Setting up basic handling is easy.
  • Is fast and lightweight.

Hates

  • No encryption methods.
  • Fully customisable.
  • No real work examples.
  • Converting content body has to be done manually.
  • Isn’t well documented.

The past week has had a lot of ups & downs in regards to Mochiweb, after a week of tinkering, I have finally finished the crux of CWIGA (Chatterl’s Web Interface Gateway API) & generally enjoyed working with Mochiweb & will utilise it for the front-end of Chatterl integrated with BeepBeep, which will deal with cookie management.

Ok, with the mini review over with, here’s some info on how I’m using Mochiweb at the moment, the source code and be found here.

Initialising Mochiweb

init([Port]) ->
    io:format("Initialising Chatterl Web Interface~n"),
    process_flag(trap_exit, true),
    mochiweb_http:start([{port, Port}, {loop, fun dispatch_requests/1}]),
    erlang:monitor(process,mochiweb_http),
    {ok, #state{}}.

Excerpt taken from a init method, here we initialise the gen_server, as the gen_server has a supervisor we use trap_exit, just below is our call to initialise Mochiweb, where we pass it the port {port, Port} & tell it what function to pass the request to (dispatch_requests).

Dispatching requests

Now we need to be able to handle all our request, to do this we use the following method:

dispatch_requests(Req) ->
    [Path|Ext] = string:tokens(Req:get(path),"."),
    Action = clean_path(Path),
    handle(Action,Ext,Req).

As I want to know what the extension type was requested (so that we can respond with the appropriate format) we split the request path using tokens and supplying ‘.’ as the parameter, this way URL like http://127.0.0.1:9000/controller/action.xml our Ext will store ‘xml’.

We then clean the path using clean_path, which will be discussed next to determine the Action we want to carry out. We then have all we need to handle our requests.

Cleaning paths

clean_path(Path) ->
    case string:str(Path, "?") of
	0 ->
	    Path;
	N ->
	    string:substr(Path, 1, string:len(Path) - (N + 1))
    end.

Here all we want is the actual path to the quest ‘/some/action/here’, so we check to see if there is a ‘?’, if so return everything preceding it, otherwise we return the path, that simple really. Though having said that, I’m sure this type of functionality should be in Mochiweb (one day eh, it’s still not production ready).

Handling requests

Now this is the fun bit, I’m going to delve into building JSON structures to use as response, purely as this one as serious pain in the ass to implement & hardly documented online.

handle("/connect/" ++ Client,ContentType,Req) ->
    {Type,Record} =
    case gen_server:call({global,chatterl_serv},{connect,Client}) of
	{ok,_} -> {"success",Client++" now connected"};
	{error,Error} -> {"failure",Error}
    end,
    send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)});

Now this method took a little while to structure properly and can look a little daunting, lets walk through it’s purpose.

So our handle method needs to be able to make calls to Chatterl and build the appropriate response using records (carriers). each response has a type which determine the type of response to return (success/failure) here we can respond with the appropriate HTTP response code (200/401,etc). We also need the actual response data (Record) which is packaged and returned in a Tuple ({Type,Record}) along with the response type. These pieces of data are then sent off to send_response (send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)}); which will be explained shortly. Before that there are two methods that need to be explained, get_content_type & build_carrier. The former is a simple wrapper which returns a record with the record type (“success”) & message (“user joined”). Where as the former passes the desired content type (Ext) and receives the appropriate content type contents:

get_content_type(Type) ->
    case Type of
	["json"] ->
	    "text/plain";
	["xml"] ->
	    "text/xml";
	_ -> "text/plain"
    end.

As mentioned earlier I want to be able to respond in either JSON or XML, with this method we are able to pass the type (remember Ext?) and return the correct content type. As I prefer JSON I’ve implemented that as the default.

Sending responses

Now here is where it starts to get interesting, we have our content type along with our results & response type, we now need to find out what type of response we have and build the appropriate format (JSON/XML).

send_response(Req, {ContentType,Record}) when is_list(ContentType) ->
    Response = get_response_body(ContentType,Record),
    Code = get_response_code(Record),
    Req:respond({Code, [{"Content-Type", ContentType}], list_to_binary(Response)}).

Now this method is pretty simplistic, though a lot of work is being done by get_response_body, We basically retrieve our formatted response, then get the appropriate response code, then we shuttle the data off to the client. Job done.

Building response bodies

Here we want to know what type of response we should contruct, once doing so, build & return the response.

get_response_body(ContentType,Record) ->
    case ContentType of
	"text/plain" ->
	    json_message(Record);
	"text/xml" ->
	    xml_message(Record);
	_ -> json_message(build_carrier("error","Illegal content type!"))
    end.

Next we need to correct response code:

get_response_code(Record) ->
    case Record of
	{carrier,Type,_Message} ->
	    case Type of
		"failure" -> 200;
		"success" -> 200;
		"error" -> 500;
		_ -> 500
	    end
    end.

As all errors and unknown responses shouldn’t happen we want to our response code to return 500 in those cases, otherwise return with a 200.

Now this is where it gets fun & where I still need to iron some things out, ideally I have a message_body method passing an atom (JSON/XML & the data) and returning the appropriate data. Below I will go into my current solution.

Building JSON

I have to say construction response can be quite a pain in Erlang, I’d suggest doing the thing I haven’t, test your patterns. One day I’ll starting testing my Erlang code as vigously as the rest of my code.

Ok back on track, so we need to one generate three types of carrier (groups,clients & messages), if the type passed doesn’t exist we print a message on the node (ideally the client should get some kind of error, I have yet to implement this yet though). Otherwise the helper method (handle_messages_json) is called and the results are injected into the outter structure ({struct,[{chatterl,{struct,[{response,Struct}]}}]})which in turn is passed to mochijson2 for conversion.

json_message(CarrierRecord) ->
    {carrier, CarrierType, Message} = CarrierRecord,
    Struct =
	case Message of
	    {carrier,Type,MessagesCarrier} ->
		case Type =:= "groups" orelse Type =:= "clients" orelse Type =:= "messages" of
		    true ->
			handle_messages_json(Type,MessagesCarrier,CarrierType);
		    false ->
			io:format("dont know ~s~n",[Type])
		end;
	    _ ->
		{struct,[{CarrierType,list_to_binary(Message)}]}
	end,
    mochijson2:encode({struct,[{chatterl,{struct,[{response,Struct}]}}]}).

Now we have differing types of carriers so here we need to check what we have and deal with it appropriately, for this post we will just focus on handling our JSON responses. Just past the case statement which checks that our carrier type is either a groups, client or messages we send the Type, along with the Carrier (holding the actual data) & the carrier type (storing which type of carrier we are handling). Once the results are back from handle_responses_json, they are inserted into a wrapper structure which is in turn used to create the whole JSON body:

{"chatterl":
	{"response":
		{"success":
			{"groups":[
				{"group":"nu_group"},
				{"group":"anuva_group"},
				{"group":"one_more"}
			]}
		}
	}
}

Handling JSON responses

handle_messages_json(Type,MessagesCarrier,CarrierType) ->
    case Type =:= "messages" of
	true ->
	    case MessagesCarrier of
		[] -> %Empty list.
		    {struct,[{Type,[]}]};
		[{carrier,_MessageType,MessageData}] ->	% A Single message.
		    {struct,[{CarrierType,
			      {struct,[{Type,loop_json_carrier(MessageData)}]}}]};
		Messages -> % Multiple messages
		    {struct,[{CarrierType,
			      {struct,[{Type,inner_loop_json_carrier(Messages)}]}}]}
	    end;
	false ->
	    {struct,[{CarrierType,{struct,[{Type,loop_json_carrier(MessagesCarrier)}]}}]}
    end.

As mentioned earlier we have three type of responses, empty, singular & multidimensional. In this method we work out what type of response structure we have and construct the appropriate JSON structure for it. We need another helper here mainly for building our structured responses, which is where loop_json_carrier & inner_loop_json_carrier comes in handy (code shown below). They basically iterate over the response carrier. Once complete it wraps this structure up in another which stores the carrier & response types.

loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(DataType),clean_message(Data)}]} || {carrier,DataType,Data} <- CarrierRecord].
inner_loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(MsgType),loop_json_carrier(Msg)}]} || {carrier,MsgType,Msg} <- CarrierRecord].

Now this methood isn’t ideal I’m sure that I can improve these methods but to be honest I’ve been working tirelessly on this & decided to leave it as something else to work out. The latter loops over each carrier in turn alling loop_json_carrier & wrapping the result in a struct tuple. The former method does more aless the same functionality apart from it cleans the message body using the clean_message/1 method.

clean_message(Data) when is_tuple(Data) ->
    {A,B,C} = Data,
    [A,B,C];
clean_message(Data) ->
    list_to_binary(Data).

We’ll I use this as a wrapper to clean up responses, ideally you could convert data to UTF-8 list_to_atom, what ever you fancy. I’m presently using it to clean Erlang dates (as I’ve not worked out how to convert them properly yet) & lists into binary.

Summary

Well thats basically it for now. I’ve left out XML parsing partly as I haven’t have the time to implemented parsing multidimensional structures yet & the code can be found here for those who are really curious ;).

I will say this though working with XML in Erlang is probably one of the most unpleasant experiences for me at the moment. All in all Mochiweb is a nice powerful tool, it still has a lot of growing to do but it is definitely a shining example of what can be done with Erlang, hopefully more documentation will enter the wild & we will see it hosting a number of services in the future.

h1

Lighting up the TunnErl Pt. 5 – The LightErls

January 9, 2009

Well what was initially one past has turned into a good few, they seem to be popular & I’ve had this in the drafts for the past week, this one focuses more on handling builds and the issues I’ve come across doing so, with a focus on what Faxien & Sinan have to offer.

Getting off the stumbling blocks
After a couple week of tinkering (ages by my standards), I had a simple Twitter interface that could run from the shell.

Though Twitterl is far from perfect, it was a good starting point, there were still things missing from my skill set:

  • A simple way to manage the application
  • How to make OTP compatible
  • How to create an application that can run from a bash shell
  • Handle documentation for projects.

How kill a few birds with one stone
Well for this I had to do a little reading and knew how to make a basic gen_server, what I needed for Chatterl was a number of gen_servers controlled by supervisors, which are encapsulated by an outer supervisor which handles the main application.

I decided for this it would be a good idea to start a new project, something I’ve wanted to create for some time, hence Chatterl, when learning I feel it makes sense to learn whilst working (I’ve worked on a number of chat systems so making a multi-node version is definitely compelling).

This project was not only something that I wanted to build for practical use but also to help me delve into Faxien & Sinan, which will help to kill a few birds at the same time.

I hate hackish code, its fine for playing around but if you want to upgrade/remove or search for applications, its a pain in the arse. So after some reading, I installed Faxien.

Now I’m not going to rewrite Sinan documentation it is pretty decent as is & there is already a decent post here, but I will give you a run down of the basics.

Now I decided to use Faxien & Sinan for a number of reasons, it saves when it comes to generating the files nessary for creating our OTP stucture. Faxien will help us install our project into our include path ‘/usr/local/erlware’, whilst Sinan will help generate our project, handle the builds and run tests and analytics on our project.

The following commands are what I seem to use the most at the moment:

  • sinan gen
  • sinan build
  • sinan dist
  • sinan test
  • faxien install-release
sinan gen

Allows us to generate the base code needed to create an OTP based application, I use this to generate a project skeleton, saving time in messing around with make files and the such like. There is no need to end the command with the project name as this and other pertaining data will be requested within the followed prompt.

sinan build

Allows us to build the application when we have made changes, simply run this from the root of the project directory and Sinan will handle building the application for you (note this does not build the documentation).

sinan doc

This will build all the documentation for our project, see edocs for more information, one thing I will mention, seeing as it took me a while to work out, the edocs overview.edoc is located with project/lib/src/PROJECT/doc. Simply write up our application documentation within that file and it will be generated with the rest of your source documentation.

sinan dist

Builds our application and creates the release tar for us, which is stored in _build/development/tar.

sudo faxien install-release _build/development/tar/project-x.x.x.tar

Which will install our project into our erlware directory, this will allow us to run our application from the bash shell:

erl -s project -s reloader

See few birds, one stone ;), we have a way to manage, generate our docs, install them using Faxien & run our project from the bash shell.

The last step was simply a pain in the ass up until this afternoon, when I was playing around with BeepBeep, there is an erl application called reloader which helps to hot swap modules, that is what ‘-s reloader’ was in the above code example ;), this had me for ages, without it applications need to be restarted, which is a pain & waste of a bloody good feature.

Still to work out

Well I’ve pretty much got the basic tools I need to develop confortably with Erlang, there a just a few more things left to work out:

  • TDD (mainly on applications & gen_servers
  • Automate the generation of docs & project build
  • Error logging/reporting

TDD
I still need to get my head around the TDD process within Erlang, thought conceptionally I think I have it down I have yet to implement it in any of my current projects, I saw some post earlier which I mailed to myself, hopefully I’ll have the chance to read over them again finally get back to my testing (feel naked without it).

Automate the generation of docs & project build
After a while it gets tiring running the same few commands over and over, at some point I really should find a way to integrate sinan/faxien into emacs, so I can can within emacs.

Error logging/handling

To be honest I’ve been kind of lazy with this, if you’ve noticed I’ve neglected to talk about sasl or anything like that, hopefully over the coming days I’ll have time to brush up on that & my Mochiweb/BeepBeep skills.

Finally

Well that yet another post finished, hopefully in the up and coming days I’ll have time to document using BeepBeep & integrating it into a Sinan application.

h1

Lighting the tunnErl Pt 4 – Erlang from a OOPer

January 6, 2009

Well this is a little deviation from the previous posts but I’ve ponder writing this post and held back until going back and working with PHP for the first time in a number of weeks.

I’ll focus purely on the productivity of Erlang vs C clones purely from an observation stand point and will leave code comparisons for another blog.

Productivity

To be honest from my perspective people are as productive as the knowledge and tools they have at their disposal (tried coding a web site in assembly or with buttons?). Initially learning any language or paradigm can appear to slow things down, I feel this is down to the fact the time is need to understand the basics (things that you already know how to do with other tools). At the same time if one is observant of the things that slow them down and work on improving their skills (or addopt new ones) in that area, they soon find their selves just as efficient in those area (sometimes more) as those areas you are traditionally more productive in.

PHP

I picked up the language a while after learning and not liking Perl for a web language (for the same reason you dont usea fork to eat soup). I’d been coding C/C++ for a while years prior so the transition was not an issue at all. Initially there wasn’t any decent libraries out there so I had to build things myself. Now 6/7 years later we have frameworks coming out the wood works and all I have to do really is do a little molding, today was a perfect example, being bored I remember I needed a exception handler. The mailer had already been built & for the logging I simple extended Zend_Log, after an 20 mins I had a basic exception handler and was off to the next task.

Now saying this I’m pro TDD and use tools to improve the way I work within PHP, as with most I use frameworks, test units & read a crap load, so naturally that helps.

Fence sitting

It has been said that Erlang lacks the tools to compete against PHP/Ruby & the such like, it certainly has a path to walk but  saying that it definitely has some decent tools & frameworks around as I’ve highlighted in past posts, which path the way for Erlang taking over some tasks we use PHP & Ruby for these days. There are already a few decent projects out there for the erlanger yaws, mochiweb, beepbeep, erlyweb, etap, not to forget faxien & sinan.

On the other side of the coin, most OOPers wont like the language and I’ve heard/read a few comment on how the syntax makes them feel uneasy, admittedly when I reflect back to the first time I saw Erlang syntax it threw me for six but honestly I thrive on change and quickly found myself wanting to know how the language worked.

Once delving in it was a trial of swings and round abouts, if it wasn’t down to the fact that I practically sleep with ‘Programming Erlang‘ & watch ‘Erlang in Practice‘ I would of given up past Erlang’s docs title page. The resources in my post helped me no bound, along with my stubborness to work things out I don’t think I would of held out to get my head around it. Oh and I forgot to mention, if you don’t know about faxien, then 9/10 one has to resign to compile source, which is all when and good up until upgrade or removal time :/.

Erlang

Well like i’ve said before I am no way near as proficient in Erlang as with PHP/C and its clones, simply down to the amount of time I have spent using it (4 months study, 3 1/2 practical). Having said that, there are a few things I have observed.

Things that were hard to do in C or PHP (created/manage multiple tasks) are pretty damn easy in Erlang implement a web server, 2mins… can’t do better than that in any other language I know, not even with apt-get. Saying that it works both ways, creating test cases/suites is not well documented leading to not testing and generally hacking, which is pretty crap in my eyes & the analyse tools could be abit more friendly (I’m still trying to get my head ronud dialyze).

The documentation for Erlang isn’t the best (I’ve yet to find decent TDD examples out in the wild on on Erlang & edocs takes some getting used to). Though the community is filling the void with alot of examples and articles, not to forget the community based sites (TrapExit, PlanetErlang, etc). The modules could be more concise and could be organised better but hell, I’ve seen all this before in PHP (remember pre OOP?) & C, they started of patchy and then become stronger as the community recognised them and bridge those gapst.

Erlang makes communication between nodes childs play, as I’ve mentioned in previous posts, a lot of my stumbling blocks with erl come down to over complicating things (an OOP trait). I think once I have gotten out of that habit with Erlang then my productivity increase that bit more. Spawning processes are made as simple as it could ever be & interacting with processes are just the same once you get over the pitfalls (from stores the pid & its ref, forgetting to catch a message, etc).

Things like creating servers is childs play also, making it easy to not only create a number of daemons but hotswap them as and when you feel.

Tools of the trade

Now I’m a firm believer in using the right tool for the right job (just because you just got a shiney new hammer, dont mean you would paint windows with it), the main reason I took up Erlang was because it seems like the right tool for scaleable, system & reliable systems.

Erlang does an excellent job and connecting with numerous nodes and spawning processes to deals with pieces of work. PHP does and goes a good job and helping to create dynamic websites (though to be honest this is a quickly changing trend) Ruby seems to be the bread winner at the moment whilst PHP tries to play catch up. C/C++ do except jobs of allowing us to work closely with assembly code & easily add assembly directly to its sources. Saying this it can take a good few days to get a C application to work ok with threading (barring all the checks & locks) not to mention the time taken to do the real work.

The one thing I love about Erlang is that it hasn’t shunned off the rest of the languages and makes it quite easy to manipulate data even if its at binary level (Joe Armstrongs Icecast example being and shining display of manipulating ID3 tags without the use of external libraries & the such like).

From my current perspective Erlang is excellent for prototyping systems and for creating those reliable highly configurable backend system, we all dream about, that said, it doesnt mean that you should use it for any & everything, sure tinker with it & having a play with the possiblity but hell taking Erlang on board and disregarding the rest would just be plan silly (imo its just something else to add to the mix). I generally don’t pick up things unless it has a practical use, interests me or I enjoy working with it. At the moment Erlang is all of that but Im still not going to push it on all my new projects.

Final words

In the last week or two I have become real comfortable with Erlang and have been working on another chat-engine (Chatterl) more and more, I’ve done a few in other languages mainly PHP & Perl, so once I have some of the functionality reflected in my older versions, I’ll dig out the most recent & do some code comparisons. For now I invite you to check out Chatterl’s source over the coming weeks I’ll be focusing on the web interface so I should be able to make some comparisons that that area also.

I’ll say this though, it took a day to whip up the base of my PHP chat system, with basic chat functionality. It took me a few more days to do the same with Erlang and create a system that can spawn groups & users over a number of nodes & shut down them nicely. Doing something like that with pure PHP is just not possible & in C++ a pain in the ass.

h1

Lighting up the tunnErl Pt. 3 – AftErl the basics

January 5, 2009

As promised this post follows on from the previous post, documenting my experience with Erlang over the past few months.

This one focuses on cock-ups and general mistakes I’ve made over the past few months, putting them here helps you to not make the same mistake & helps me to remember what to I did at that time, as I’ve said before killing a few birds with one stone is my thing.

AftErl the basics

By now I had decided to play around with Twitter’s API seeing as I use it a hell of a lot and web based applications are my thing (there is no better way to learn something that to use it). This provided me with quite a bit of insight when it came to developing in Erlang. I found these to be common over sights & issues:

  • Forgetting to end a line properly.
  • State weirdness.
  • Not catching a process message.
  • Spawning processes within processes.
  • Handling stopping child processes.
  • Module definition not the same as file name.
  • Creating result structures.
  • Parsing XML.
  • Working processes die once they are linked to a supervisor.

Forgetting to end a line properly

This was one of my most common mistakes, something that I still miss from time to time, the easiest thing to do is to remember the following:

  1. Each pattern match within a case ends with a semi-colon, unless its the last which is not terminated.
  2. If a case encapsulates multiple cases each end clause should finish with a semi-colon, with the last encapsulated case not being terminated.
  3. When refactoring a case, be sure to take note of the end of the case, moving the last case to the anywhere else means that that and the now last case needs to be refactored.

I assure you you will no doubt get into this issue as a newbie, as I’ve read it is one of the few issues others have with it as beginners. As long as you remember those points and if really stuck check against a piece of code that you know works & you should be fine, just a couple of examples taken from a system I am working on at the moment:

case gb_trees:is_defined(User, State#foobarz.users) of
    false->
        io:format("~p connected to ~p~n", [User,?MODULE]),
        {{ok, "connected"},
          gb_trees:insert(User, {User,From}, State#foobarz.users)};
    true -> {{error, "Unable to connect."},
                       State#foobarz.users}
end.

This is an example of a basic case clause, as we don’t want to process the results, we simply terminate with a full stop. to Tell Erlang to return case’s results.

determine_user_action(GroupName,{Action,PayLoad},UsersList) ->
    case Action of
        drop_group ->
            GroupMsg = "Sending disconnect message to ~s~n",
            send_msg_to_users({drop_group,GroupName},UsersList,GroupMsg);
        receive_msg ->
            case PayLoad of
               {CreatedOn,Sender,Message} ->
                   GroupMsg = "Sending to users ~s~n",
                   send_msg_to_users({receive_msg, CreatedOn,Sender,Message},UsersList,GroupMsg);
               _ ->
                  {error, "Illegal payload format"}
            end;
        _ -> {error, "Illegal action!"}
    end.

This method is a little more involve and displays and example of a case encapsulated method, first we do the basic pattern matching, if we get the atom recieve_msg we want to use another case to check that the payload is what we expect it to be, you main thing to notice here is that the inner case statement terminates with a semi-colon which tells Erlang that this is the end of the expression.

Not catching a process message.

This is another common mistake on my behalf, either passing the wrong pattern or forgetting a trap_exit, this is something to watch out for. to save a lot of unnecessary system crashes, I’d suggest testing out pattern matches via a separate virtual node, allowing you to determine the best patterns for the task. If the process is supervised make sure that you use process_monitor(trap_exit, true) with the init of the gen_server.

State weirdness

Remember when using gb_trees:insert and the such like the new data is returned as the result & not updated internally, it is very easy to forget this, especially coming from an OOP background. So if you notice that your state is not being updated, check that the result of such methods is being passed and not left in the ether.

Another similar issue I have recently come across is mangled states within gen_server, I was basically trying to update a state if something changed, though instead of return the state property if the was no update I passed the whole state, leading in warped data, this can be quite serious & very hard to find. Here is the culprit:

handle_call({disconnect, User}, _From, State) ->
    {Reply,NewTree} =
        case gb_trees:is_defined(User, State#foobarz.users) of
            true ->
                {{ok, "User dropped"},
                 gb_trees:delete(User, State#foobarz.users)};
            false ->
                 {{error, "Unable to drop group."},
                  State}
        end,
{reply, Reply, State#foobarz{ users = NewTree }};

Well it seems safe enough right?, after noticing a bug in part of my system and realising that the state looked stranged, it wasn’t until I took a step back that I noticed the actual bug & to my amazement the bug was present in a couple of methods (I hate copy & paste). To be honest this to date is probably one of my biggest cock-ups & potentially one of the damaging bugs I’ve introduced into a system in a long time, I’d hate to see how many other system have fallen prey to this oversight.

Just so that there is an example out there for future reference here is the corrected version of the above code.

handle_call({disconnect, User}, _From, State) ->
{Reply,NewTree} =
case gb_trees:is_defined(User, State#foobarz.users) of
true ->
{{ok, "User dropped"},
gb_trees:delete(User, State#foobarz.users)};
false ->
{{error, "Unable to drop group."},
State#foobarz.users}
end,
{reply, Reply, State#foobarz{ users = NewTree }};

Spawning processes within processes

I was stuck on this for a few days and still think there is a better way I about this, spawning a new process is so simple it didnt occur to me that when creating a gen_server the gen_server was actually spawning a new process (duh, read the source). I subconciously took this in whilst reading up on it but when it came down to doing it myself it had me stumped, here is what I was trying to do:

spawn_link(mod_gen_server,start_link,arg).

The process I was trying to spawn was a gen_server and already spawning a process within itself, so what I was effectively trying do was spawn_link something that was trying to spawn a another seperate process, not quite what I wanted & Erlang definately didn’t like it, all I needed was to call the process and to link/monitor it. That simple.

Multiple processes

Well this was another one, once I worked out how to link a process I soon realised I needed that child process to have a unique module name, this is another one that may seem hard but in practicality it is yet again very simple.

start(Name,Desc) ->
    gen_server:start_link({global, Name}, ?MODULE, start, [Name,Desc]).

This way, the process is given the name passed the user, there are probably safer secure ways of doing this but for what I need at the moment, it works at treat.

Module definition not the same as file name

Is such a basic thing to oversee especially coming from a language where file names aren’t so important, its a simple one to fix but yet a common gotcha. Emacs has a nice lil feature that which basics alerts you to this fact and asks you if you want to changed it.

Creating result structures

Yet again another simple practice but if you come from a purely OO/C based language environment & just finished working on a OO based project or something, watch out for this, getting used to it takes some time. Simple rule:

  • Manipulate Term, giving unique names to needed terms
  • Store resulting terms into new pattern

Lists can simply be manipulated by postfixing a new list element to an existing on:

{{FName,_MName,LName},Address,Number,_OtherStuff} = UserDetails,
[[FName,SName,Address,Number]|Results].

Parsing XML

Whilst messing around with my Twitterl project, I quickly realised one thing, how was I to parse XML (second nature to me in PHP/C++) in Erlang though it did take me a while to get my head around, thanks to some related code by Nick Gerakines, reading his code definately shed some light on the area.

This is still a doozy, hopefully I’ll find a better way of doing this, basically we use xmerl_xpath:string/2 to retrieve an XML element, this result is a tuple, so we need to work on it further to retrieve the actual value, this could simply be done by:

[#xmlText{value = Value}] = xmerl_xpath:string(XML, "/item/title/text()").

I had implemented a variation of this code within my module but found it slow, so after reading a comment from Nick Gerakine’s in relation to my project, I had a look at his code for his Twitter API & noticed a bit of code that did they type of functionality I wanted, I’d like to find a way to improve this some more, mainly by looping over a record (still to learn) but it definately does the trick & imspires me to write more elegant code:

parse_item(Node) ->
    Item = #tweet {
      title = format_text(Node, ["/item/title/text()"],""),
      pubDate = format_text(Node, ["/item/pubDate/text()"],""),
      link = format_text(Node, ["/item/link/text()"], "")
    },
    Item.

This method also takes two Terms, the XML node & a list of XPath strings, it passes the node off to format_text, which does most of the work:

format_text(_, [], Result) -> Result;
format_text(Xml, [Xpath | Tail], Result) ->
    Results = lists:foldr(
        fun(#xmlText{value = Value}, Acc) ->
            lists:append(Value, Acc);
        (_, Acc) -> Acc
        end,
        Result,
        xmerl_xpath:string(Xpath, Xml)
    ),
    format_text(Xml, Tail, Results).

Here we basically loop over the XPath list retrieving the results and storing them in a list called Results, when the list of XPaths is empty we return the result, as most of the time I only need the one XPath it this method wasn’t introduced until i needed to play around with Twitters Users & Statuses XML.

Well that generally covers it, there is still a few more things a I want to cover & no doubt I’ll add more to this post as

Stuff I have yet to learn

  • Still not got round to getting getting auto-complete work under emacs.
  • Error and log handling principles, how to recover from a termination.
  • erl_crash.dump, is totally alien to me.
  • RESTful applications (have done some R&D but have yet to implement the actual functionality).
  • TDD & the whole process is still not well documented, ideally I want to be able to develop using the same TDD process I use in OOP.
  • Find out how to inplement a continous integration system to make development smoother.

Admittedly alot of this is more best practices and handling production code but I’m never happy until I have these things ironed out ;).

Well its been near four months since I started coding Erlang & I’ve got to say I’m enjoying the hell out of it, I thought the curve would be steep & admittedly initially it was, I still have moments of total confusion but I’m sure the more I play with Erlang the less this will happen, hell I still have those moments in C & PHP so nothing really different there.

Next

The next post will focus on how to put all this together to make an OTP system & how to main builds using Erlware’s Faxien & Sinan.

h1

Lighting up the tunnErl Pt 2 – Discovering a new wErld

January 5, 2009

This post continues on from Part 1, this post focuses primarily with the concepts of Erlang, throwing in a few tips as I go along.

The Basics

There are heaps of places to pick up the basics as mentioned before (Erlang, TrapExit, PlanetErlang & Lighting up the tunnErl Part 1 are good starts) but I’ll highlight some functional fundermentals to help understand the basics of Erlang code.

Term definitions

  • Atoms – depicted by a lower case string some_atom
  • Tuples – depicted as {Term}, can have a number of atoms, tuples & lists.
  • Lists – depicted as ["a","list"]
  • Term (a mixture of the above).{a_simple_atom,Term,["a","list"]}

A few tips

Well things are as easy as you make them, so here are some tips that may help whilst programming with Erlang.

  • Terms are pretty much immutable (though state & records can be changed).
  • Make sure that Terms that are different have different names, coming from OOP it is all too easy to over look the fact that two actual Terms can not share the same name. The following will cause an error
    Term = 'hello". Term = {term}.

    & even harder to debug

    Term = 4. Term = 2 + 2.

    just means that 4 is identical to 2+2.

  • If you change a match pattern, make sure all references of it are refactored.
  • Terms that you’re not bothered with should start with a ‘_’ ie. _DontWant, could just use _ but when coming back to the code, it makes it harder to understand.
  • Start off small, get familiar with erl & passing around Terms.
  • Test out the modules & commands your not familiar with in erl, it will save you time when it comes to actual coding.
  • Have a browse of other Erlang projects, read through the source, see what modules are commonly used & how things are done.
  • Get used to the Erlang documentation & the commonly used methods (lists,ets.dets.gb_trees).
  • OOP natives beware: the sooner you get used to using the word ‘term’ instead of ‘variable’, when thinking in Erlang, the quicker the transition.

A few examples

Now I’m not going to go into great detail here, though I will show some basic examples of the various terms, I wont go into details as usual, I’ll leave that for you guys to read up on if you already don’t know, curiously is king:

a_simple_atom
ATerm
{a_simple_atom,ATerm}
{}
{{ATerm,[SumList]},_DontWantThis}
[]
["foo","bar"]
Term = {{{ATerm,[SumList]},_DontWantThis},["foo","bar"]}

Are funs fun?

It didn’t take long to get to grips with the basics, understanding funs was a different ball game though. I think it comes down to the same issue I initially had with XML, it is so flexible you don’t quite know where to start with it, so here’s something I use in one of my projects:

    lists:foreach(
      fun(GroupName) ->
      io:format("Dropping group ~s...~n",[GroupName]),
      gen_server:call({global,GroupName},stop,infinity)
      end,
      GroupNames).

Here the following steps are taken:

  1. Loop over each of the GroupNames them (using method lists:foreach)
  2. Each result is stored and passed as the term Groupname
  3. Prints a message using GroupName for a dynamic value.
  4. A call is made to the process to shutdown.

Steps 2 & 3 are where the fun begins (pardon the pun), we tell it to take an argument (GroupName) & then to execute everything within the fun, this is an extremely powerful feature, one which I still have yet to manipulate properly. I’d suggest messing around with them abit (using others code) and getting used to their power and flexiblity, you’ll soon find funs are fun.

That does it for this post, the next will focus on after the basic, hopefully giving you a chance to delve into the fundermentals of Erlang.

Next
I will focus on the actual problems & issues I’ve faces whilst working with Erlang, this can be found here.