Posts Tagged ‘Erlang’

h1

Changes in Chatterl

July 23, 2009

I haven’t worked on Chatterl for a few months but there are some fundemental changes that had been made back in March, apologies for the slow update but honestly I haven’t had much time with learning Rails & freelance work.

Summary

Well the main changes in chatterl are the following:

  • Message storage
  • Registration
  • Introduction of unit tests
  • CWIGA now RESTFUL
  • API calls

I’ll go over each section, highlighting the main changes & who it affects Chatterl as a whole.

Meet CWIGA

After implementing Chatterl‘s basic functionality (see chatterl post). The next step was to create to implement CWIGA, Chatterl Web Interface Gateway API. With this new module Chatterl is able to be interacted with purely via HTTP. The first implementation was rough, the original source can be found here, though it allowed me to get a decent understanding of mochiweb at the time helped me produce this article. After implementing unit tests to cover the project as a whole I felt comfortable to make a few fundamental changes to CWIGA’s API and requests to Chatterl.

Registration

I’ve added very basic registration functionality to Chatterl, allowing users to register to the system, giving them the ability to retrieve archived messages & later creation of their own chat groups. In hindsight I’d like to move this functionality to the front-end (which I’ll more than likely be in Rails).

Message Storage

Again this was more experimental on my part and something I would like to move over to couchDB when I get the chance, for the moment, Chatterluses mnesia to store users messages when they are not logged into the system.

Added test cases

One of the main issues I’ve been having with Sinan more than anything else, was seperating my tests from the source code. I really hated the idea of having test & implementation code mixed in, so I stayed away from any kind of real testing until I got my head around it. Thanks to Kev Smith‘s own code, I finally got my head around EUnit, which I posted a short post on here.

Before I was comfortable with implementing any more functionality I had set out to cover as much of Chatterl‘s code as possible. In doing so I uncovered a number of silly little bugs (oversights) in the coding logics & made the necessary changes.

General cleanup

The way chatterl is called has also been improved, instead of having to go into erl you can simple run ./start-server.sh within the cd _build/development/apps/chatterl-0.1.2.0/ directory. This will start up all the processes need to use chatterl.

CWIGA is setup to listen on port 9000, so you can call it using curl or via a browser of your choice. It accepts XML & JSON requests & has been made more RESTful.

Hopefully in the near future I’ll have time to rewrite the code base and improve on the basic functionality but for the meantime I hope this helps others as much as its been an experience for me.

h1

Lighting up the tunnErl Pt. 9 – The Gotchaz

March 13, 2009

Over the years I’ve become pretty good at picking up on errors, learning a new paradigm as well as a new language helps to introduce a lot more of these than usual so I’ve put some of the more common errors I have come accross to help others pick up on the errrors that had me baphled from one extent to another.

  • function_clause – generally means that the result was not matched for one reason or another, make sure you are retrieving the result you expected.
  • noproc – either the process is not started or the node cannot communicate with it, first try net_adm:ping(hostname). then check to see if the process is alive erlang:is_process_alive(Pid).
  • undef – Happen when a function in module does not exist, the module is not loaded or not exist at all. Maybe the function name has been misspelt.
  • ‘*.app not found’ – this typically means that the application can not be found or it is not included, check that it is part of you dependancies and that has been started properly.
  • error:{badfun,ok} – EUnit based error missing comma, usually because assert is returned in wrong place.
  • syntax error before: ‘end’ – have added an “,/;’ proceeding the end
  • none:error:internal error in beam_asm – Sinan based error, happens when code is out of sync, best solution is to kill sinserv and run sinan again.

As time goes by I’ll continue to add more, what this space. If anyone has come across a gotcha that i’ve missed drop me a line, i’ll be more than happy to add it to this list.

h1

Lighting up the tunnErl Pt.8 – Testing processes

March 13, 2009

Well after months of trying to get my head around running units around a process I think I have it sussed.

Simple unit tests
Well I’m sure you have seen these type of erlang tests before

-module(test_foo).
-include_lib("eunit/include/eunit.hrl").
foo_test_() ->
[?_assertEqual("foo",say("foo")),
?_assertEqual("bar",say("bar"))].

The above code basically asserts that the method say returns what we expect. Okay so now what if we want to setup a process, lets say chatterl:start() and test its functionality, how would we do that? well it took me a while to get my head around this part.

I have noticed that if any processes are left alive during a unit test, the process will still be active whilst the other tests run, so if for some reason you find previous test cases failing where they were initially passing, first check that you have killed all the necessary processes in all units.

Creating simple units was a walk in the park and nothing really different from the standard units I’ve created in various C clones. The doozy was creating units that focus on a specific process rather than a method or function.

Testing processes

chatterl_mid_man_basics_test_() ->
[{setup, fun() ->
chatterl:start() end,
fun(_) ->
chatterl:stop() end,
[{timeout, 5000,
fun() ->
?assertEqual(<<"Illegal content type!">>,
check_json(mochijson2:decode(chatterl_mid_man:user_list("text/json")))),
?assertEqual([],chatterl_serv:list_users()),
?assertEqual({struct,[{<<"clients">>,[]}]},
check_json(mochijson2:decode(chatterl_mid_man:user_list(["text/json"]))))
end}]}].

The above test is from chatterl’s test cases, the test starts chatterl on setup, waits 5000 msecs using a timeout & runs the units within the final fun/0. Once our tests have run we need to stop chatterl (which will drop all its connected processes). I’ve used a timeout because it seems the processes I am working with need some time to be dropped before the next test is started, I’ve noticed that if this wasn’t done I’d get all kinds of unrelated errors or complaining that the process is still alive.

Though this process seems cumbersome, especially as I’m used to a single setup/teardown method which does the same thing each time a unit is run, the above handles our tests pretty well.

For the extra curious, you can find test cases used for chatterl/libs/chatter/src/chatterl_test.erl @ github.

You can also checkout Kevin Smith’s code as he seems pretty up to scratch with his tests membox is a good place to start.

h1

Lighting up the tunnErl part 7 – Chatterl meets Nitrogen

March 13, 2009

Well it’s been a has been a while since I have posted a blog, mainly down to the amount of time I’ve been spending playing around with Chatterl & Nitrogen. I have come a way since first playing around with Erlang I decided a few months ago to make Chatterl my main personal project. I played around with BeepBeep around the same time as Nitrogen, for one reason or another I picked up Nitrogen quite quickly and decided to run with that as Chatterl‘s prototype frontend.

I have to admit though I have never really been a fan of event driven frameworks, especially those that mask mark up language but after a day or two of messing around with Nitrogen I found it quite a pleasant experience.

The learning curve can be somewhat steep as the documentation is not as good as it could be, I also have issues with having the root directory starting with /web/, so the admin directory would be http://blah.com/web/admin, hopefully this will be changed in the near future. Besides those minor gripes Nitrogen is starting off as a nice framework.

Well I wish I had some time to run through a little tutorial, for the moment the best I can do is show you how I implemented some of the functionality need for Chatterl‘s front-end. Below is a list of features I’ll be concentration on:

  • Main body
  • Validate Message
  • Client Panel
  • Message Panel
  • Groups Body
  • Users Panel

We’re working with Chatterl‘s Web Interface API (CWIGA) which works on 127.0.0.1:9000, I prefer JSON so I’ve only implemented functionality to handle this. The implementation code is very similar to that I’ve written about previous so I wont waste bytes (see Twitterl), I’ll just say that we need to connect to CWIGA via HTTP.

For the curious, the code can be found the functionality needed to interact with Chatterl can be found in the file chatterl_cwiga_handler.erl. The actual calls are defined within Chatterl interface.

Now we have the basics for interacting with Chatterl over HTTP, lets have a look at how this all ties in with working with Nitrogen. I won’t go over how Nitrogen works as that part is pretty well documented in a few placed (Nitrogen, Jón Grétar) but I will explain the functionality I’ve implemented for chatterl_nitrogen.

Main body
body() ->
ChatBody = [ #panel { id=messageBodyPanel} ],
Left = [ #panel
{ id=leftPanel,body=[client_panel(),message_panel(),
[#panel { id=groupsPanel, body=[groups_body()]}],users_panel()] } ],
Right = [ #panel { id=rightPanel, body=[ChatBody] } ],
Container = #panel { id=containerPanel, body=[Left,Right] },
validate_message(),
wf:render(Container).

Ok now what the hell is all that??? Well it doesn’t look like it but this is how Nitrogen renders HTML, we structure our HTML using Nitrogen‘s elements & actions which in turns renders our pages for us.

Firstly we define the containers needed to organise the chat window, Allowing me to have the chat box on the right, whilst having all the other features (message box, groups list, etc) on the left. As you may of already guessed the element #panel represents a div with XHTML, id’s act as you would expect & can be manipulated with CSS as usual.

Validate Message
validate_message() ->
wf:wire(messageTextBox, messageTextBox, #validate
{ validators=[
#is_required { text="You forgot to submit a message." },
#min_length { length=2, text="Title must be at least 2 characters long." }]}).

Here we use the method wf:wire which basically connects an element to another, so a input box could be ‘wired’ to a submit button. In this case we want to make sure that we validation only when the messageTextBox has data, validate it. The next two lines do exactly that, firstly #is_required makes sure that we actually have a value, whilst #min_length gives a validation error if it has not been adhered to. The response are triggered by AJAX using JQuery, so as soon as an event is caught it is handled asynchronously.

Client Panel
client_panel() ->
[#panel { id=clientPanel, body=[get_client_panel()]}].

This panel is pretty straight forward, it simply defines a DIV elements which calls get_client_panel/0 which in turn determine what actions are needed. This method will be explained better shortly.

Message Panel
message_panel() ->
[#panel { id=messagePanel, body=[
#label { id=messageLabel, text="Message:" },
#textbox { id=messageTextBox, text="", style="width: 100px;", postback=group_message}
]}].

This is the the DIV element that is used by the user to send messages to chatterl. We have a couple of things we have encountered yet, our postback, which will be explained shortly & two elements (#label & #textbox). Pretty straight forward right. What about this postback thingie? Well postbacks are in fact our AJAX calls, allowing the user to interact with the systems backend seemlessly. We’ll see how these are handled in the Events section.

Groups Body
groups_body() ->
SelectedGroup = wf:q(groupsDropDown),
#panel { class=groupsPanel, body=[
#h1 { text="Groups" },
#label { text="Select a group to join:" },
#dropdown { id=groupsDropDown, value=SelectedGroup,
options = [ #option { value=Group, text=Group }
|| Group <- chatterl_interface:get_groups()],
postback=join_group} ] }.

Here we have an panel which allows users to select a group to join, we again use a postback to handle our AJAX call & use the chatterl_interface module to retrieve a list of groups from Chatterl.

Users Panel
users_panel() ->
SelectedUser = wf:q(usersDropDown),
[#panel { id=usersPanel, body=[
#label { id=usersLabel, text="Select a user:" },
#dropdown { id=usersDropDown, value=SelectedUser,
options = [ #option { value=User, text=User }
|| User <- chatterl_interface:get_users()] } ]}].

This should be pretty self explanatory as it is more aless the same functionality as groups_body/0.

Get client Panel
get_client_panel() ->
case wf:session(client) of
undefined ->
wf:wire(groupsPanel, #hide {}),
wf:wire(messagePanel, #hide {}),
client_login_panel();
Client ->
wf:comet(fun() -> update_status("messages",1000) end),
wf:wire(groupsPanel, #show {}),
wf:wire(messagePanel, #show {}),
client_logout_panel(Client)
end.

Well this method basically checks to see if a user is connected if they are then we display the messagePanel, groupsPanel along with groupsJoinedPanel. We allow the messages panel to update, giving the user the most up to date messages from Chatterl. We lastly set the client_panel to client_logout_panel/1, which we’ll have a look at shortly, this method basically allows the user to disconnect once they have connected successfully to Chatterl.

Client login Panel
client_login_panel() ->
#panel { id=loginPanel,
body=[
#label { text="Client name:" },
#textbox { id=userNameTextBox, text="", style="width: 100px;", html_encode=true, postback=connect }]}.

Client logout panel
client_logout_panel(Client) ->
#panel { id=logoutPanel,
body=[
#inplace_textbox { id=userNameTextBox, text=Client, html_encode=true, start_mode=view },
#button { id=theButton, text="Disconnect", postback=disconnect }]}.

This method is pretty much the same the the client_login_panel/0, with 1 small change, instead of connecting to Chatterl we want to disconnect. The principle is still the same, create an event, passing the postback (disconnect).
which in turn disconnects the user from Chatterl.

Update status
update_status(Type,Timeout) ->
process_flag(trap_exit, true),
case Type of
undefined -> ok;
"messages" -> loop_messages(Timeout);
{"general",StatusType} -> loop_status(Timeout,StatusType);
_ -> io:format("Unknown loop type: ~s",[Type])
end,
wf:comet_flush(),
update_status(Type,Timeout).

Events

Connect Event
event(connect) ->
[Client] = wf:q(userNameTextBox),
case chatterl_interface:connect(Client) of
{ok,Message} ->
wf:flash(Message),
wf:session(client,Client),
wf:update(clientPanel, #panel { body=[get_client_panel()] });
{error,Error} -> wf:wire(#alert { text=Error })
end.

Above is the actually postback method call event(connect) which utilises chatterl_interface:connect/1 allowing us to make a connection to Chatterl directly. If the request is successful we create a new session wf:session(client,Client) & update our clientPanel so that they user can logout as you would expect.We do this by using the wf:update/2 method that dynamically changes XHTML elements on the fly.

Disconnect Event
event(disconnect) ->
case chatterl_interface:disconnect(wf:session(client)) of
{ok,Message} ->
wf:flash(Message),
wf:session(client,undefined),
wf:session(selectedGroup,undefined),
wf:update(messageBodyPanel, #panel { body=[] }),
wf:update(clientPanel, #panel { body=[get_client_panel()] }); % Should redirect here.
{error,Error} ->
wf:wire(#alert { text=Error })
end.

Well there’s the disconnect event, nothing to dramatic here, simply unregister the client & selectedGroup cookies& update the page so that the client_panel is set back to its default.

h1

Lighting the tunnErl Pt 6 – Using Mochiweb for Chatterl’s RESTful API

January 15, 2009

This post is the next in the ‘Lighting the tunnErl‘ series, documenting my experiences with Erlang as I learn it and its tools, as the previous post focused on Faxien & Sinan, I decided to concentrate on working with yaws or Mochiweb. As I’m still actively working on Chatterl, which as mentioned here, is a multi-node chat system, and it needed a web interface, I did some research and decided to use Mochiweb, though the part six was originally going to be ‘Erlang Gotchaz’, I thought I should post this first whilst the information is still fresh in my mind.

General overview

I recently started to play around with Mochiweb & BeepBeep to see what functionality they could provide Chatterl, after a couple of days working around the BeepBeep’s code, I decided to focus on the actual gateway API (CWIGA) and use Mochiweb along with a little manipulation to create the gateway service that would in turn serve the frontend which will use BeepBeep to render the data. A bit convoluted maybe but I want other to be able to use the backend separately and plug in their own frontends ;), not to mention it’s an excellent chance to play with new toys.

The foundations weren’t that hard to put together, think it took all of 2 minutes to get Mochiweb running, setting up the basics are just that & simple interactions with other systems is straight forward. Though saying that I quickly found that one can encounter some serious head banging moments with Mochiweb especially with the lack of documentation at present.

I also feel Mochiweb lacks fundamental features like destroying cookies, manipulating headers/query string & the such like a lot of this is left down to the developer, which in my opinion isn’t ideal, I’d rather spend time on the problem at hand rather than deal with mini side effects.

I really really hate pros/cons, loves/hates, but hell I really do have a love/hate relationship with Mochiweb at the moment, I’m hoping that this is just part of the curve but I have to admit it certainly is an interesting beast.

Loves

  • Quick & easy to initialise Mochiweb.
  • Initialising cookies is simple.
  • Fully customisable
  • Setting up basic handling is easy.
  • Is fast and lightweight.

Hates

  • No encryption methods.
  • Fully customisable.
  • No real work examples.
  • Converting content body has to be done manually.
  • Isn’t well documented.

The past week has had a lot of ups & downs in regards to Mochiweb, after a week of tinkering, I have finally finished the crux of CWIGA (Chatterl’s Web Interface Gateway API) & generally enjoyed working with Mochiweb & will utilise it for the front-end of Chatterl integrated with BeepBeep, which will deal with cookie management.

Ok, with the mini review over with, here’s some info on how I’m using Mochiweb at the moment, the source code and be found here.

Initialising Mochiweb

init([Port]) ->
    io:format("Initialising Chatterl Web Interface~n"),
    process_flag(trap_exit, true),
    mochiweb_http:start([{port, Port}, {loop, fun dispatch_requests/1}]),
    erlang:monitor(process,mochiweb_http),
    {ok, #state{}}.

Excerpt taken from a init method, here we initialise the gen_server, as the gen_server has a supervisor we use trap_exit, just below is our call to initialise Mochiweb, where we pass it the port {port, Port} & tell it what function to pass the request to (dispatch_requests).

Dispatching requests

Now we need to be able to handle all our request, to do this we use the following method:

dispatch_requests(Req) ->
    [Path|Ext] = string:tokens(Req:get(path),"."),
    Action = clean_path(Path),
    handle(Action,Ext,Req).

As I want to know what the extension type was requested (so that we can respond with the appropriate format) we split the request path using tokens and supplying ‘.’ as the parameter, this way URL like http://127.0.0.1:9000/controller/action.xml our Ext will store ‘xml’.

We then clean the path using clean_path, which will be discussed next to determine the Action we want to carry out. We then have all we need to handle our requests.

Cleaning paths

clean_path(Path) ->
    case string:str(Path, "?") of
	0 ->
	    Path;
	N ->
	    string:substr(Path, 1, string:len(Path) - (N + 1))
    end.

Here all we want is the actual path to the quest ‘/some/action/here’, so we check to see if there is a ‘?’, if so return everything preceding it, otherwise we return the path, that simple really. Though having said that, I’m sure this type of functionality should be in Mochiweb (one day eh, it’s still not production ready).

Handling requests

Now this is the fun bit, I’m going to delve into building JSON structures to use as response, purely as this one as serious pain in the ass to implement & hardly documented online.

handle("/connect/" ++ Client,ContentType,Req) ->
    {Type,Record} =
    case gen_server:call({global,chatterl_serv},{connect,Client}) of
	{ok,_} -> {"success",Client++" now connected"};
	{error,Error} -> {"failure",Error}
    end,
    send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)});

Now this method took a little while to structure properly and can look a little daunting, lets walk through it’s purpose.

So our handle method needs to be able to make calls to Chatterl and build the appropriate response using records (carriers). each response has a type which determine the type of response to return (success/failure) here we can respond with the appropriate HTTP response code (200/401,etc). We also need the actual response data (Record) which is packaged and returned in a Tuple ({Type,Record}) along with the response type. These pieces of data are then sent off to send_response (send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)}); which will be explained shortly. Before that there are two methods that need to be explained, get_content_type & build_carrier. The former is a simple wrapper which returns a record with the record type (“success”) & message (“user joined”). Where as the former passes the desired content type (Ext) and receives the appropriate content type contents:

get_content_type(Type) ->
    case Type of
	["json"] ->
	    "text/plain";
	["xml"] ->
	    "text/xml";
	_ -> "text/plain"
    end.

As mentioned earlier I want to be able to respond in either JSON or XML, with this method we are able to pass the type (remember Ext?) and return the correct content type. As I prefer JSON I’ve implemented that as the default.

Sending responses

Now here is where it starts to get interesting, we have our content type along with our results & response type, we now need to find out what type of response we have and build the appropriate format (JSON/XML).

send_response(Req, {ContentType,Record}) when is_list(ContentType) ->
    Response = get_response_body(ContentType,Record),
    Code = get_response_code(Record),
    Req:respond({Code, [{"Content-Type", ContentType}], list_to_binary(Response)}).

Now this method is pretty simplistic, though a lot of work is being done by get_response_body, We basically retrieve our formatted response, then get the appropriate response code, then we shuttle the data off to the client. Job done.

Building response bodies

Here we want to know what type of response we should contruct, once doing so, build & return the response.

get_response_body(ContentType,Record) ->
    case ContentType of
	"text/plain" ->
	    json_message(Record);
	"text/xml" ->
	    xml_message(Record);
	_ -> json_message(build_carrier("error","Illegal content type!"))
    end.

Next we need to correct response code:

get_response_code(Record) ->
    case Record of
	{carrier,Type,_Message} ->
	    case Type of
		"failure" -> 200;
		"success" -> 200;
		"error" -> 500;
		_ -> 500
	    end
    end.

As all errors and unknown responses shouldn’t happen we want to our response code to return 500 in those cases, otherwise return with a 200.

Now this is where it gets fun & where I still need to iron some things out, ideally I have a message_body method passing an atom (JSON/XML & the data) and returning the appropriate data. Below I will go into my current solution.

Building JSON

I have to say construction response can be quite a pain in Erlang, I’d suggest doing the thing I haven’t, test your patterns. One day I’ll starting testing my Erlang code as vigously as the rest of my code.

Ok back on track, so we need to one generate three types of carrier (groups,clients & messages), if the type passed doesn’t exist we print a message on the node (ideally the client should get some kind of error, I have yet to implement this yet though). Otherwise the helper method (handle_messages_json) is called and the results are injected into the outter structure ({struct,[{chatterl,{struct,[{response,Struct}]}}]})which in turn is passed to mochijson2 for conversion.

json_message(CarrierRecord) ->
    {carrier, CarrierType, Message} = CarrierRecord,
    Struct =
	case Message of
	    {carrier,Type,MessagesCarrier} ->
		case Type =:= "groups" orelse Type =:= "clients" orelse Type =:= "messages" of
		    true ->
			handle_messages_json(Type,MessagesCarrier,CarrierType);
		    false ->
			io:format("dont know ~s~n",[Type])
		end;
	    _ ->
		{struct,[{CarrierType,list_to_binary(Message)}]}
	end,
    mochijson2:encode({struct,[{chatterl,{struct,[{response,Struct}]}}]}).

Now we have differing types of carriers so here we need to check what we have and deal with it appropriately, for this post we will just focus on handling our JSON responses. Just past the case statement which checks that our carrier type is either a groups, client or messages we send the Type, along with the Carrier (holding the actual data) & the carrier type (storing which type of carrier we are handling). Once the results are back from handle_responses_json, they are inserted into a wrapper structure which is in turn used to create the whole JSON body:

{"chatterl":
	{"response":
		{"success":
			{"groups":[
				{"group":"nu_group"},
				{"group":"anuva_group"},
				{"group":"one_more"}
			]}
		}
	}
}

Handling JSON responses

handle_messages_json(Type,MessagesCarrier,CarrierType) ->
    case Type =:= "messages" of
	true ->
	    case MessagesCarrier of
		[] -> %Empty list.
		    {struct,[{Type,[]}]};
		[{carrier,_MessageType,MessageData}] ->	% A Single message.
		    {struct,[{CarrierType,
			      {struct,[{Type,loop_json_carrier(MessageData)}]}}]};
		Messages -> % Multiple messages
		    {struct,[{CarrierType,
			      {struct,[{Type,inner_loop_json_carrier(Messages)}]}}]}
	    end;
	false ->
	    {struct,[{CarrierType,{struct,[{Type,loop_json_carrier(MessagesCarrier)}]}}]}
    end.

As mentioned earlier we have three type of responses, empty, singular & multidimensional. In this method we work out what type of response structure we have and construct the appropriate JSON structure for it. We need another helper here mainly for building our structured responses, which is where loop_json_carrier & inner_loop_json_carrier comes in handy (code shown below). They basically iterate over the response carrier. Once complete it wraps this structure up in another which stores the carrier & response types.

loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(DataType),clean_message(Data)}]} || {carrier,DataType,Data} <- CarrierRecord].
inner_loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(MsgType),loop_json_carrier(Msg)}]} || {carrier,MsgType,Msg} <- CarrierRecord].

Now this methood isn’t ideal I’m sure that I can improve these methods but to be honest I’ve been working tirelessly on this & decided to leave it as something else to work out. The latter loops over each carrier in turn alling loop_json_carrier & wrapping the result in a struct tuple. The former method does more aless the same functionality apart from it cleans the message body using the clean_message/1 method.

clean_message(Data) when is_tuple(Data) ->
    {A,B,C} = Data,
    [A,B,C];
clean_message(Data) ->
    list_to_binary(Data).

We’ll I use this as a wrapper to clean up responses, ideally you could convert data to UTF-8 list_to_atom, what ever you fancy. I’m presently using it to clean Erlang dates (as I’ve not worked out how to convert them properly yet) & lists into binary.

Summary

Well thats basically it for now. I’ve left out XML parsing partly as I haven’t have the time to implemented parsing multidimensional structures yet & the code can be found here for those who are really curious ;).

I will say this though working with XML in Erlang is probably one of the most unpleasant experiences for me at the moment. All in all Mochiweb is a nice powerful tool, it still has a lot of growing to do but it is definitely a shining example of what can be done with Erlang, hopefully more documentation will enter the wild & we will see it hosting a number of services in the future.

h1

Lighting up the TunnErl Pt. 5 – The LightErls

January 9, 2009

Well what was initially one past has turned into a good few, they seem to be popular & I’ve had this in the drafts for the past week, this one focuses more on handling builds and the issues I’ve come across doing so, with a focus on what Faxien & Sinan have to offer.

Getting off the stumbling blocks
After a couple week of tinkering (ages by my standards), I had a simple Twitter interface that could run from the shell.

Though Twitterl is far from perfect, it was a good starting point, there were still things missing from my skill set:

  • A simple way to manage the application
  • How to make OTP compatible
  • How to create an application that can run from a bash shell
  • Handle documentation for projects.

How kill a few birds with one stone
Well for this I had to do a little reading and knew how to make a basic gen_server, what I needed for Chatterl was a number of gen_servers controlled by supervisors, which are encapsulated by an outer supervisor which handles the main application.

I decided for this it would be a good idea to start a new project, something I’ve wanted to create for some time, hence Chatterl, when learning I feel it makes sense to learn whilst working (I’ve worked on a number of chat systems so making a multi-node version is definitely compelling).

This project was not only something that I wanted to build for practical use but also to help me delve into Faxien & Sinan, which will help to kill a few birds at the same time.

I hate hackish code, its fine for playing around but if you want to upgrade/remove or search for applications, its a pain in the arse. So after some reading, I installed Faxien.

Now I’m not going to rewrite Sinan documentation it is pretty decent as is & there is already a decent post here, but I will give you a run down of the basics.

Now I decided to use Faxien & Sinan for a number of reasons, it saves when it comes to generating the files nessary for creating our OTP stucture. Faxien will help us install our project into our include path ‘/usr/local/erlware’, whilst Sinan will help generate our project, handle the builds and run tests and analytics on our project.

The following commands are what I seem to use the most at the moment:

  • sinan gen
  • sinan build
  • sinan dist
  • sinan test
  • faxien install-release
sinan gen

Allows us to generate the base code needed to create an OTP based application, I use this to generate a project skeleton, saving time in messing around with make files and the such like. There is no need to end the command with the project name as this and other pertaining data will be requested within the followed prompt.

sinan build

Allows us to build the application when we have made changes, simply run this from the root of the project directory and Sinan will handle building the application for you (note this does not build the documentation).

sinan doc

This will build all the documentation for our project, see edocs for more information, one thing I will mention, seeing as it took me a while to work out, the edocs overview.edoc is located with project/lib/src/PROJECT/doc. Simply write up our application documentation within that file and it will be generated with the rest of your source documentation.

sinan dist

Builds our application and creates the release tar for us, which is stored in _build/development/tar.

sudo faxien install-release _build/development/tar/project-x.x.x.tar

Which will install our project into our erlware directory, this will allow us to run our application from the bash shell:

erl -s project -s reloader

See few birds, one stone ;), we have a way to manage, generate our docs, install them using Faxien & run our project from the bash shell.

The last step was simply a pain in the ass up until this afternoon, when I was playing around with BeepBeep, there is an erl application called reloader which helps to hot swap modules, that is what ‘-s reloader’ was in the above code example ;), this had me for ages, without it applications need to be restarted, which is a pain & waste of a bloody good feature.

Still to work out

Well I’ve pretty much got the basic tools I need to develop confortably with Erlang, there a just a few more things left to work out:

  • TDD (mainly on applications & gen_servers
  • Automate the generation of docs & project build
  • Error logging/reporting

TDD
I still need to get my head around the TDD process within Erlang, thought conceptionally I think I have it down I have yet to implement it in any of my current projects, I saw some post earlier which I mailed to myself, hopefully I’ll have the chance to read over them again finally get back to my testing (feel naked without it).

Automate the generation of docs & project build
After a while it gets tiring running the same few commands over and over, at some point I really should find a way to integrate sinan/faxien into emacs, so I can can within emacs.

Error logging/handling

To be honest I’ve been kind of lazy with this, if you’ve noticed I’ve neglected to talk about sasl or anything like that, hopefully over the coming days I’ll have time to brush up on that & my Mochiweb/BeepBeep skills.

Finally

Well that yet another post finished, hopefully in the up and coming days I’ll have time to document using BeepBeep & integrating it into a Sinan application.

h1

Lighting the tunnErl Pt 4 – Erlang from a OOPer

January 6, 2009

Well this is a little deviation from the previous posts but I’ve ponder writing this post and held back until going back and working with PHP for the first time in a number of weeks.

I’ll focus purely on the productivity of Erlang vs C clones purely from an observation stand point and will leave code comparisons for another blog.

Productivity

To be honest from my perspective people are as productive as the knowledge and tools they have at their disposal (tried coding a web site in assembly or with buttons?). Initially learning any language or paradigm can appear to slow things down, I feel this is down to the fact the time is need to understand the basics (things that you already know how to do with other tools). At the same time if one is observant of the things that slow them down and work on improving their skills (or addopt new ones) in that area, they soon find their selves just as efficient in those area (sometimes more) as those areas you are traditionally more productive in.

PHP

I picked up the language a while after learning and not liking Perl for a web language (for the same reason you dont usea fork to eat soup). I’d been coding C/C++ for a while years prior so the transition was not an issue at all. Initially there wasn’t any decent libraries out there so I had to build things myself. Now 6/7 years later we have frameworks coming out the wood works and all I have to do really is do a little molding, today was a perfect example, being bored I remember I needed a exception handler. The mailer had already been built & for the logging I simple extended Zend_Log, after an 20 mins I had a basic exception handler and was off to the next task.

Now saying this I’m pro TDD and use tools to improve the way I work within PHP, as with most I use frameworks, test units & read a crap load, so naturally that helps.

Fence sitting

It has been said that Erlang lacks the tools to compete against PHP/Ruby & the such like, it certainly has a path to walk but  saying that it definitely has some decent tools & frameworks around as I’ve highlighted in past posts, which path the way for Erlang taking over some tasks we use PHP & Ruby for these days. There are already a few decent projects out there for the erlanger yaws, mochiweb, beepbeep, erlyweb, etap, not to forget faxien & sinan.

On the other side of the coin, most OOPers wont like the language and I’ve heard/read a few comment on how the syntax makes them feel uneasy, admittedly when I reflect back to the first time I saw Erlang syntax it threw me for six but honestly I thrive on change and quickly found myself wanting to know how the language worked.

Once delving in it was a trial of swings and round abouts, if it wasn’t down to the fact that I practically sleep with ‘Programming Erlang‘ & watch ‘Erlang in Practice‘ I would of given up past Erlang’s docs title page. The resources in my post helped me no bound, along with my stubborness to work things out I don’t think I would of held out to get my head around it. Oh and I forgot to mention, if you don’t know about faxien, then 9/10 one has to resign to compile source, which is all when and good up until upgrade or removal time :/.

Erlang

Well like i’ve said before I am no way near as proficient in Erlang as with PHP/C and its clones, simply down to the amount of time I have spent using it (4 months study, 3 1/2 practical). Having said that, there are a few things I have observed.

Things that were hard to do in C or PHP (created/manage multiple tasks) are pretty damn easy in Erlang implement a web server, 2mins… can’t do better than that in any other language I know, not even with apt-get. Saying that it works both ways, creating test cases/suites is not well documented leading to not testing and generally hacking, which is pretty crap in my eyes & the analyse tools could be abit more friendly (I’m still trying to get my head ronud dialyze).

The documentation for Erlang isn’t the best (I’ve yet to find decent TDD examples out in the wild on on Erlang & edocs takes some getting used to). Though the community is filling the void with alot of examples and articles, not to forget the community based sites (TrapExit, PlanetErlang, etc). The modules could be more concise and could be organised better but hell, I’ve seen all this before in PHP (remember pre OOP?) & C, they started of patchy and then become stronger as the community recognised them and bridge those gapst.

Erlang makes communication between nodes childs play, as I’ve mentioned in previous posts, a lot of my stumbling blocks with erl come down to over complicating things (an OOP trait). I think once I have gotten out of that habit with Erlang then my productivity increase that bit more. Spawning processes are made as simple as it could ever be & interacting with processes are just the same once you get over the pitfalls (from stores the pid & its ref, forgetting to catch a message, etc).

Things like creating servers is childs play also, making it easy to not only create a number of daemons but hotswap them as and when you feel.

Tools of the trade

Now I’m a firm believer in using the right tool for the right job (just because you just got a shiney new hammer, dont mean you would paint windows with it), the main reason I took up Erlang was because it seems like the right tool for scaleable, system & reliable systems.

Erlang does an excellent job and connecting with numerous nodes and spawning processes to deals with pieces of work. PHP does and goes a good job and helping to create dynamic websites (though to be honest this is a quickly changing trend) Ruby seems to be the bread winner at the moment whilst PHP tries to play catch up. C/C++ do except jobs of allowing us to work closely with assembly code & easily add assembly directly to its sources. Saying this it can take a good few days to get a C application to work ok with threading (barring all the checks & locks) not to mention the time taken to do the real work.

The one thing I love about Erlang is that it hasn’t shunned off the rest of the languages and makes it quite easy to manipulate data even if its at binary level (Joe Armstrongs Icecast example being and shining display of manipulating ID3 tags without the use of external libraries & the such like).

From my current perspective Erlang is excellent for prototyping systems and for creating those reliable highly configurable backend system, we all dream about, that said, it doesnt mean that you should use it for any & everything, sure tinker with it & having a play with the possiblity but hell taking Erlang on board and disregarding the rest would just be plan silly (imo its just something else to add to the mix). I generally don’t pick up things unless it has a practical use, interests me or I enjoy working with it. At the moment Erlang is all of that but Im still not going to push it on all my new projects.

Final words

In the last week or two I have become real comfortable with Erlang and have been working on another chat-engine (Chatterl) more and more, I’ve done a few in other languages mainly PHP & Perl, so once I have some of the functionality reflected in my older versions, I’ll dig out the most recent & do some code comparisons. For now I invite you to check out Chatterl’s source over the coming weeks I’ll be focusing on the web interface so I should be able to make some comparisons that that area also.

I’ll say this though, it took a day to whip up the base of my PHP chat system, with basic chat functionality. It took me a few more days to do the same with Erlang and create a system that can spawn groups & users over a number of nodes & shut down them nicely. Doing something like that with pure PHP is just not possible & in C++ a pain in the ass.