Posts Tagged ‘Erlang basics’

h1

Lighting up the tunnErl Pt. 9 – The Gotchaz

March 13, 2009

Over the years I’ve become pretty good at picking up on errors, learning a new paradigm as well as a new language helps to introduce a lot more of these than usual so I’ve put some of the more common errors I have come accross to help others pick up on the errrors that had me baphled from one extent to another.

  • function_clause – generally means that the result was not matched for one reason or another, make sure you are retrieving the result you expected.
  • noproc – either the process is not started or the node cannot communicate with it, first try net_adm:ping(hostname). then check to see if the process is alive erlang:is_process_alive(Pid).
  • undef – Happen when a function in module does not exist, the module is not loaded or not exist at all. Maybe the function name has been misspelt.
  • ‘*.app not found’ – this typically means that the application can not be found or it is not included, check that it is part of you dependancies and that has been started properly.
  • error:{badfun,ok} – EUnit based error missing comma, usually because assert is returned in wrong place.
  • syntax error before: ‘end’ – have added an “,/;’ proceeding the end
  • none:error:internal error in beam_asm – Sinan based error, happens when code is out of sync, best solution is to kill sinserv and run sinan again.

As time goes by I’ll continue to add more, what this space. If anyone has come across a gotcha that i’ve missed drop me a line, i’ll be more than happy to add it to this list.

Advertisements
h1

Lighting up the tunnErl part 7 – Chatterl meets Nitrogen

March 13, 2009

Well it’s been a has been a while since I have posted a blog, mainly down to the amount of time I’ve been spending playing around with Chatterl & Nitrogen. I have come a way since first playing around with Erlang I decided a few months ago to make Chatterl my main personal project. I played around with BeepBeep around the same time as Nitrogen, for one reason or another I picked up Nitrogen quite quickly and decided to run with that as Chatterl‘s prototype frontend.

I have to admit though I have never really been a fan of event driven frameworks, especially those that mask mark up language but after a day or two of messing around with Nitrogen I found it quite a pleasant experience.

The learning curve can be somewhat steep as the documentation is not as good as it could be, I also have issues with having the root directory starting with /web/, so the admin directory would be http://blah.com/web/admin, hopefully this will be changed in the near future. Besides those minor gripes Nitrogen is starting off as a nice framework.

Well I wish I had some time to run through a little tutorial, for the moment the best I can do is show you how I implemented some of the functionality need for Chatterl‘s front-end. Below is a list of features I’ll be concentration on:

  • Main body
  • Validate Message
  • Client Panel
  • Message Panel
  • Groups Body
  • Users Panel

We’re working with Chatterl‘s Web Interface API (CWIGA) which works on 127.0.0.1:9000, I prefer JSON so I’ve only implemented functionality to handle this. The implementation code is very similar to that I’ve written about previous so I wont waste bytes (see Twitterl), I’ll just say that we need to connect to CWIGA via HTTP.

For the curious, the code can be found the functionality needed to interact with Chatterl can be found in the file chatterl_cwiga_handler.erl. The actual calls are defined within Chatterl interface.

Now we have the basics for interacting with Chatterl over HTTP, lets have a look at how this all ties in with working with Nitrogen. I won’t go over how Nitrogen works as that part is pretty well documented in a few placed (Nitrogen, Jón Grétar) but I will explain the functionality I’ve implemented for chatterl_nitrogen.

Main body
body() ->
ChatBody = [ #panel { id=messageBodyPanel} ],
Left = [ #panel
{ id=leftPanel,body=[client_panel(),message_panel(),
[#panel { id=groupsPanel, body=[groups_body()]}],users_panel()] } ],
Right = [ #panel { id=rightPanel, body=[ChatBody] } ],
Container = #panel { id=containerPanel, body=[Left,Right] },
validate_message(),
wf:render(Container).

Ok now what the hell is all that??? Well it doesn’t look like it but this is how Nitrogen renders HTML, we structure our HTML using Nitrogen‘s elements & actions which in turns renders our pages for us.

Firstly we define the containers needed to organise the chat window, Allowing me to have the chat box on the right, whilst having all the other features (message box, groups list, etc) on the left. As you may of already guessed the element #panel represents a div with XHTML, id’s act as you would expect & can be manipulated with CSS as usual.

Validate Message
validate_message() ->
wf:wire(messageTextBox, messageTextBox, #validate
{ validators=[
#is_required { text="You forgot to submit a message." },
#min_length { length=2, text="Title must be at least 2 characters long." }]}).

Here we use the method wf:wire which basically connects an element to another, so a input box could be ‘wired’ to a submit button. In this case we want to make sure that we validation only when the messageTextBox has data, validate it. The next two lines do exactly that, firstly #is_required makes sure that we actually have a value, whilst #min_length gives a validation error if it has not been adhered to. The response are triggered by AJAX using JQuery, so as soon as an event is caught it is handled asynchronously.

Client Panel
client_panel() ->
[#panel { id=clientPanel, body=[get_client_panel()]}].

This panel is pretty straight forward, it simply defines a DIV elements which calls get_client_panel/0 which in turn determine what actions are needed. This method will be explained better shortly.

Message Panel
message_panel() ->
[#panel { id=messagePanel, body=[
#label { id=messageLabel, text="Message:" },
#textbox { id=messageTextBox, text="", style="width: 100px;", postback=group_message}
]}].

This is the the DIV element that is used by the user to send messages to chatterl. We have a couple of things we have encountered yet, our postback, which will be explained shortly & two elements (#label & #textbox). Pretty straight forward right. What about this postback thingie? Well postbacks are in fact our AJAX calls, allowing the user to interact with the systems backend seemlessly. We’ll see how these are handled in the Events section.

Groups Body
groups_body() ->
SelectedGroup = wf:q(groupsDropDown),
#panel { class=groupsPanel, body=[
#h1 { text="Groups" },
#label { text="Select a group to join:" },
#dropdown { id=groupsDropDown, value=SelectedGroup,
options = [ #option { value=Group, text=Group }
|| Group <- chatterl_interface:get_groups()],
postback=join_group} ] }.

Here we have an panel which allows users to select a group to join, we again use a postback to handle our AJAX call & use the chatterl_interface module to retrieve a list of groups from Chatterl.

Users Panel
users_panel() ->
SelectedUser = wf:q(usersDropDown),
[#panel { id=usersPanel, body=[
#label { id=usersLabel, text="Select a user:" },
#dropdown { id=usersDropDown, value=SelectedUser,
options = [ #option { value=User, text=User }
|| User <- chatterl_interface:get_users()] } ]}].

This should be pretty self explanatory as it is more aless the same functionality as groups_body/0.

Get client Panel
get_client_panel() ->
case wf:session(client) of
undefined ->
wf:wire(groupsPanel, #hide {}),
wf:wire(messagePanel, #hide {}),
client_login_panel();
Client ->
wf:comet(fun() -> update_status("messages",1000) end),
wf:wire(groupsPanel, #show {}),
wf:wire(messagePanel, #show {}),
client_logout_panel(Client)
end.

Well this method basically checks to see if a user is connected if they are then we display the messagePanel, groupsPanel along with groupsJoinedPanel. We allow the messages panel to update, giving the user the most up to date messages from Chatterl. We lastly set the client_panel to client_logout_panel/1, which we’ll have a look at shortly, this method basically allows the user to disconnect once they have connected successfully to Chatterl.

Client login Panel
client_login_panel() ->
#panel { id=loginPanel,
body=[
#label { text="Client name:" },
#textbox { id=userNameTextBox, text="", style="width: 100px;", html_encode=true, postback=connect }]}.

Client logout panel
client_logout_panel(Client) ->
#panel { id=logoutPanel,
body=[
#inplace_textbox { id=userNameTextBox, text=Client, html_encode=true, start_mode=view },
#button { id=theButton, text="Disconnect", postback=disconnect }]}.

This method is pretty much the same the the client_login_panel/0, with 1 small change, instead of connecting to Chatterl we want to disconnect. The principle is still the same, create an event, passing the postback (disconnect).
which in turn disconnects the user from Chatterl.

Update status
update_status(Type,Timeout) ->
process_flag(trap_exit, true),
case Type of
undefined -> ok;
"messages" -> loop_messages(Timeout);
{"general",StatusType} -> loop_status(Timeout,StatusType);
_ -> io:format("Unknown loop type: ~s",[Type])
end,
wf:comet_flush(),
update_status(Type,Timeout).

Events

Connect Event
event(connect) ->
[Client] = wf:q(userNameTextBox),
case chatterl_interface:connect(Client) of
{ok,Message} ->
wf:flash(Message),
wf:session(client,Client),
wf:update(clientPanel, #panel { body=[get_client_panel()] });
{error,Error} -> wf:wire(#alert { text=Error })
end.

Above is the actually postback method call event(connect) which utilises chatterl_interface:connect/1 allowing us to make a connection to Chatterl directly. If the request is successful we create a new session wf:session(client,Client) & update our clientPanel so that they user can logout as you would expect.We do this by using the wf:update/2 method that dynamically changes XHTML elements on the fly.

Disconnect Event
event(disconnect) ->
case chatterl_interface:disconnect(wf:session(client)) of
{ok,Message} ->
wf:flash(Message),
wf:session(client,undefined),
wf:session(selectedGroup,undefined),
wf:update(messageBodyPanel, #panel { body=[] }),
wf:update(clientPanel, #panel { body=[get_client_panel()] }); % Should redirect here.
{error,Error} ->
wf:wire(#alert { text=Error })
end.

Well there’s the disconnect event, nothing to dramatic here, simply unregister the client & selectedGroup cookies& update the page so that the client_panel is set back to its default.

h1

Lighting the tunnErl Pt 6 – Using Mochiweb for Chatterl’s RESTful API

January 15, 2009

This post is the next in the ‘Lighting the tunnErl‘ series, documenting my experiences with Erlang as I learn it and its tools, as the previous post focused on Faxien & Sinan, I decided to concentrate on working with yaws or Mochiweb. As I’m still actively working on Chatterl, which as mentioned here, is a multi-node chat system, and it needed a web interface, I did some research and decided to use Mochiweb, though the part six was originally going to be ‘Erlang Gotchaz’, I thought I should post this first whilst the information is still fresh in my mind.

General overview

I recently started to play around with Mochiweb & BeepBeep to see what functionality they could provide Chatterl, after a couple of days working around the BeepBeep’s code, I decided to focus on the actual gateway API (CWIGA) and use Mochiweb along with a little manipulation to create the gateway service that would in turn serve the frontend which will use BeepBeep to render the data. A bit convoluted maybe but I want other to be able to use the backend separately and plug in their own frontends ;), not to mention it’s an excellent chance to play with new toys.

The foundations weren’t that hard to put together, think it took all of 2 minutes to get Mochiweb running, setting up the basics are just that & simple interactions with other systems is straight forward. Though saying that I quickly found that one can encounter some serious head banging moments with Mochiweb especially with the lack of documentation at present.

I also feel Mochiweb lacks fundamental features like destroying cookies, manipulating headers/query string & the such like a lot of this is left down to the developer, which in my opinion isn’t ideal, I’d rather spend time on the problem at hand rather than deal with mini side effects.

I really really hate pros/cons, loves/hates, but hell I really do have a love/hate relationship with Mochiweb at the moment, I’m hoping that this is just part of the curve but I have to admit it certainly is an interesting beast.

Loves

  • Quick & easy to initialise Mochiweb.
  • Initialising cookies is simple.
  • Fully customisable
  • Setting up basic handling is easy.
  • Is fast and lightweight.

Hates

  • No encryption methods.
  • Fully customisable.
  • No real work examples.
  • Converting content body has to be done manually.
  • Isn’t well documented.

The past week has had a lot of ups & downs in regards to Mochiweb, after a week of tinkering, I have finally finished the crux of CWIGA (Chatterl’s Web Interface Gateway API) & generally enjoyed working with Mochiweb & will utilise it for the front-end of Chatterl integrated with BeepBeep, which will deal with cookie management.

Ok, with the mini review over with, here’s some info on how I’m using Mochiweb at the moment, the source code and be found here.

Initialising Mochiweb

init([Port]) ->
    io:format("Initialising Chatterl Web Interface~n"),
    process_flag(trap_exit, true),
    mochiweb_http:start([{port, Port}, {loop, fun dispatch_requests/1}]),
    erlang:monitor(process,mochiweb_http),
    {ok, #state{}}.

Excerpt taken from a init method, here we initialise the gen_server, as the gen_server has a supervisor we use trap_exit, just below is our call to initialise Mochiweb, where we pass it the port {port, Port} & tell it what function to pass the request to (dispatch_requests).

Dispatching requests

Now we need to be able to handle all our request, to do this we use the following method:

dispatch_requests(Req) ->
    [Path|Ext] = string:tokens(Req:get(path),"."),
    Action = clean_path(Path),
    handle(Action,Ext,Req).

As I want to know what the extension type was requested (so that we can respond with the appropriate format) we split the request path using tokens and supplying ‘.’ as the parameter, this way URL like http://127.0.0.1:9000/controller/action.xml our Ext will store ‘xml’.

We then clean the path using clean_path, which will be discussed next to determine the Action we want to carry out. We then have all we need to handle our requests.

Cleaning paths

clean_path(Path) ->
    case string:str(Path, "?") of
	0 ->
	    Path;
	N ->
	    string:substr(Path, 1, string:len(Path) - (N + 1))
    end.

Here all we want is the actual path to the quest ‘/some/action/here’, so we check to see if there is a ‘?’, if so return everything preceding it, otherwise we return the path, that simple really. Though having said that, I’m sure this type of functionality should be in Mochiweb (one day eh, it’s still not production ready).

Handling requests

Now this is the fun bit, I’m going to delve into building JSON structures to use as response, purely as this one as serious pain in the ass to implement & hardly documented online.

handle("/connect/" ++ Client,ContentType,Req) ->
    {Type,Record} =
    case gen_server:call({global,chatterl_serv},{connect,Client}) of
	{ok,_} -> {"success",Client++" now connected"};
	{error,Error} -> {"failure",Error}
    end,
    send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)});

Now this method took a little while to structure properly and can look a little daunting, lets walk through it’s purpose.

So our handle method needs to be able to make calls to Chatterl and build the appropriate response using records (carriers). each response has a type which determine the type of response to return (success/failure) here we can respond with the appropriate HTTP response code (200/401,etc). We also need the actual response data (Record) which is packaged and returned in a Tuple ({Type,Record}) along with the response type. These pieces of data are then sent off to send_response (send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)}); which will be explained shortly. Before that there are two methods that need to be explained, get_content_type & build_carrier. The former is a simple wrapper which returns a record with the record type (“success”) & message (“user joined”). Where as the former passes the desired content type (Ext) and receives the appropriate content type contents:

get_content_type(Type) ->
    case Type of
	["json"] ->
	    "text/plain";
	["xml"] ->
	    "text/xml";
	_ -> "text/plain"
    end.

As mentioned earlier I want to be able to respond in either JSON or XML, with this method we are able to pass the type (remember Ext?) and return the correct content type. As I prefer JSON I’ve implemented that as the default.

Sending responses

Now here is where it starts to get interesting, we have our content type along with our results & response type, we now need to find out what type of response we have and build the appropriate format (JSON/XML).

send_response(Req, {ContentType,Record}) when is_list(ContentType) ->
    Response = get_response_body(ContentType,Record),
    Code = get_response_code(Record),
    Req:respond({Code, [{"Content-Type", ContentType}], list_to_binary(Response)}).

Now this method is pretty simplistic, though a lot of work is being done by get_response_body, We basically retrieve our formatted response, then get the appropriate response code, then we shuttle the data off to the client. Job done.

Building response bodies

Here we want to know what type of response we should contruct, once doing so, build & return the response.

get_response_body(ContentType,Record) ->
    case ContentType of
	"text/plain" ->
	    json_message(Record);
	"text/xml" ->
	    xml_message(Record);
	_ -> json_message(build_carrier("error","Illegal content type!"))
    end.

Next we need to correct response code:

get_response_code(Record) ->
    case Record of
	{carrier,Type,_Message} ->
	    case Type of
		"failure" -> 200;
		"success" -> 200;
		"error" -> 500;
		_ -> 500
	    end
    end.

As all errors and unknown responses shouldn’t happen we want to our response code to return 500 in those cases, otherwise return with a 200.

Now this is where it gets fun & where I still need to iron some things out, ideally I have a message_body method passing an atom (JSON/XML & the data) and returning the appropriate data. Below I will go into my current solution.

Building JSON

I have to say construction response can be quite a pain in Erlang, I’d suggest doing the thing I haven’t, test your patterns. One day I’ll starting testing my Erlang code as vigously as the rest of my code.

Ok back on track, so we need to one generate three types of carrier (groups,clients & messages), if the type passed doesn’t exist we print a message on the node (ideally the client should get some kind of error, I have yet to implement this yet though). Otherwise the helper method (handle_messages_json) is called and the results are injected into the outter structure ({struct,[{chatterl,{struct,[{response,Struct}]}}]})which in turn is passed to mochijson2 for conversion.

json_message(CarrierRecord) ->
    {carrier, CarrierType, Message} = CarrierRecord,
    Struct =
	case Message of
	    {carrier,Type,MessagesCarrier} ->
		case Type =:= "groups" orelse Type =:= "clients" orelse Type =:= "messages" of
		    true ->
			handle_messages_json(Type,MessagesCarrier,CarrierType);
		    false ->
			io:format("dont know ~s~n",[Type])
		end;
	    _ ->
		{struct,[{CarrierType,list_to_binary(Message)}]}
	end,
    mochijson2:encode({struct,[{chatterl,{struct,[{response,Struct}]}}]}).

Now we have differing types of carriers so here we need to check what we have and deal with it appropriately, for this post we will just focus on handling our JSON responses. Just past the case statement which checks that our carrier type is either a groups, client or messages we send the Type, along with the Carrier (holding the actual data) & the carrier type (storing which type of carrier we are handling). Once the results are back from handle_responses_json, they are inserted into a wrapper structure which is in turn used to create the whole JSON body:

{"chatterl":
	{"response":
		{"success":
			{"groups":[
				{"group":"nu_group"},
				{"group":"anuva_group"},
				{"group":"one_more"}
			]}
		}
	}
}

Handling JSON responses

handle_messages_json(Type,MessagesCarrier,CarrierType) ->
    case Type =:= "messages" of
	true ->
	    case MessagesCarrier of
		[] -> %Empty list.
		    {struct,[{Type,[]}]};
		[{carrier,_MessageType,MessageData}] ->	% A Single message.
		    {struct,[{CarrierType,
			      {struct,[{Type,loop_json_carrier(MessageData)}]}}]};
		Messages -> % Multiple messages
		    {struct,[{CarrierType,
			      {struct,[{Type,inner_loop_json_carrier(Messages)}]}}]}
	    end;
	false ->
	    {struct,[{CarrierType,{struct,[{Type,loop_json_carrier(MessagesCarrier)}]}}]}
    end.

As mentioned earlier we have three type of responses, empty, singular & multidimensional. In this method we work out what type of response structure we have and construct the appropriate JSON structure for it. We need another helper here mainly for building our structured responses, which is where loop_json_carrier & inner_loop_json_carrier comes in handy (code shown below). They basically iterate over the response carrier. Once complete it wraps this structure up in another which stores the carrier & response types.

loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(DataType),clean_message(Data)}]} || {carrier,DataType,Data} <- CarrierRecord].
inner_loop_json_carrier(CarrierRecord) ->
    [{struct,[{list_to_binary(MsgType),loop_json_carrier(Msg)}]} || {carrier,MsgType,Msg} <- CarrierRecord].

Now this methood isn’t ideal I’m sure that I can improve these methods but to be honest I’ve been working tirelessly on this & decided to leave it as something else to work out. The latter loops over each carrier in turn alling loop_json_carrier & wrapping the result in a struct tuple. The former method does more aless the same functionality apart from it cleans the message body using the clean_message/1 method.

clean_message(Data) when is_tuple(Data) ->
    {A,B,C} = Data,
    [A,B,C];
clean_message(Data) ->
    list_to_binary(Data).

We’ll I use this as a wrapper to clean up responses, ideally you could convert data to UTF-8 list_to_atom, what ever you fancy. I’m presently using it to clean Erlang dates (as I’ve not worked out how to convert them properly yet) & lists into binary.

Summary

Well thats basically it for now. I’ve left out XML parsing partly as I haven’t have the time to implemented parsing multidimensional structures yet & the code can be found here for those who are really curious ;).

I will say this though working with XML in Erlang is probably one of the most unpleasant experiences for me at the moment. All in all Mochiweb is a nice powerful tool, it still has a lot of growing to do but it is definitely a shining example of what can be done with Erlang, hopefully more documentation will enter the wild & we will see it hosting a number of services in the future.

h1

Lighting up the TunnErl Pt. 5 – The LightErls

January 9, 2009

Well what was initially one past has turned into a good few, they seem to be popular & I’ve had this in the drafts for the past week, this one focuses more on handling builds and the issues I’ve come across doing so, with a focus on what Faxien & Sinan have to offer.

Getting off the stumbling blocks
After a couple week of tinkering (ages by my standards), I had a simple Twitter interface that could run from the shell.

Though Twitterl is far from perfect, it was a good starting point, there were still things missing from my skill set:

  • A simple way to manage the application
  • How to make OTP compatible
  • How to create an application that can run from a bash shell
  • Handle documentation for projects.

How kill a few birds with one stone
Well for this I had to do a little reading and knew how to make a basic gen_server, what I needed for Chatterl was a number of gen_servers controlled by supervisors, which are encapsulated by an outer supervisor which handles the main application.

I decided for this it would be a good idea to start a new project, something I’ve wanted to create for some time, hence Chatterl, when learning I feel it makes sense to learn whilst working (I’ve worked on a number of chat systems so making a multi-node version is definitely compelling).

This project was not only something that I wanted to build for practical use but also to help me delve into Faxien & Sinan, which will help to kill a few birds at the same time.

I hate hackish code, its fine for playing around but if you want to upgrade/remove or search for applications, its a pain in the arse. So after some reading, I installed Faxien.

Now I’m not going to rewrite Sinan documentation it is pretty decent as is & there is already a decent post here, but I will give you a run down of the basics.

Now I decided to use Faxien & Sinan for a number of reasons, it saves when it comes to generating the files nessary for creating our OTP stucture. Faxien will help us install our project into our include path ‘/usr/local/erlware’, whilst Sinan will help generate our project, handle the builds and run tests and analytics on our project.

The following commands are what I seem to use the most at the moment:

  • sinan gen
  • sinan build
  • sinan dist
  • sinan test
  • faxien install-release
sinan gen

Allows us to generate the base code needed to create an OTP based application, I use this to generate a project skeleton, saving time in messing around with make files and the such like. There is no need to end the command with the project name as this and other pertaining data will be requested within the followed prompt.

sinan build

Allows us to build the application when we have made changes, simply run this from the root of the project directory and Sinan will handle building the application for you (note this does not build the documentation).

sinan doc

This will build all the documentation for our project, see edocs for more information, one thing I will mention, seeing as it took me a while to work out, the edocs overview.edoc is located with project/lib/src/PROJECT/doc. Simply write up our application documentation within that file and it will be generated with the rest of your source documentation.

sinan dist

Builds our application and creates the release tar for us, which is stored in _build/development/tar.

sudo faxien install-release _build/development/tar/project-x.x.x.tar

Which will install our project into our erlware directory, this will allow us to run our application from the bash shell:

erl -s project -s reloader

See few birds, one stone ;), we have a way to manage, generate our docs, install them using Faxien & run our project from the bash shell.

The last step was simply a pain in the ass up until this afternoon, when I was playing around with BeepBeep, there is an erl application called reloader which helps to hot swap modules, that is what ‘-s reloader’ was in the above code example ;), this had me for ages, without it applications need to be restarted, which is a pain & waste of a bloody good feature.

Still to work out

Well I’ve pretty much got the basic tools I need to develop confortably with Erlang, there a just a few more things left to work out:

  • TDD (mainly on applications & gen_servers
  • Automate the generation of docs & project build
  • Error logging/reporting

TDD
I still need to get my head around the TDD process within Erlang, thought conceptionally I think I have it down I have yet to implement it in any of my current projects, I saw some post earlier which I mailed to myself, hopefully I’ll have the chance to read over them again finally get back to my testing (feel naked without it).

Automate the generation of docs & project build
After a while it gets tiring running the same few commands over and over, at some point I really should find a way to integrate sinan/faxien into emacs, so I can can within emacs.

Error logging/handling

To be honest I’ve been kind of lazy with this, if you’ve noticed I’ve neglected to talk about sasl or anything like that, hopefully over the coming days I’ll have time to brush up on that & my Mochiweb/BeepBeep skills.

Finally

Well that yet another post finished, hopefully in the up and coming days I’ll have time to document using BeepBeep & integrating it into a Sinan application.

h1

Lighting up the tunnErl Pt. 3 – AftErl the basics

January 5, 2009

As promised this post follows on from the previous post, documenting my experience with Erlang over the past few months.

This one focuses on cock-ups and general mistakes I’ve made over the past few months, putting them here helps you to not make the same mistake & helps me to remember what to I did at that time, as I’ve said before killing a few birds with one stone is my thing.

AftErl the basics

By now I had decided to play around with Twitter’s API seeing as I use it a hell of a lot and web based applications are my thing (there is no better way to learn something that to use it). This provided me with quite a bit of insight when it came to developing in Erlang. I found these to be common over sights & issues:

  • Forgetting to end a line properly.
  • State weirdness.
  • Not catching a process message.
  • Spawning processes within processes.
  • Handling stopping child processes.
  • Module definition not the same as file name.
  • Creating result structures.
  • Parsing XML.
  • Working processes die once they are linked to a supervisor.

Forgetting to end a line properly

This was one of my most common mistakes, something that I still miss from time to time, the easiest thing to do is to remember the following:

  1. Each pattern match within a case ends with a semi-colon, unless its the last which is not terminated.
  2. If a case encapsulates multiple cases each end clause should finish with a semi-colon, with the last encapsulated case not being terminated.
  3. When refactoring a case, be sure to take note of the end of the case, moving the last case to the anywhere else means that that and the now last case needs to be refactored.

I assure you you will no doubt get into this issue as a newbie, as I’ve read it is one of the few issues others have with it as beginners. As long as you remember those points and if really stuck check against a piece of code that you know works & you should be fine, just a couple of examples taken from a system I am working on at the moment:

case gb_trees:is_defined(User, State#foobarz.users) of
    false->
        io:format("~p connected to ~p~n", [User,?MODULE]),
        {{ok, "connected"},
          gb_trees:insert(User, {User,From}, State#foobarz.users)};
    true -> {{error, "Unable to connect."},
                       State#foobarz.users}
end.

This is an example of a basic case clause, as we don’t want to process the results, we simply terminate with a full stop. to Tell Erlang to return case’s results.

determine_user_action(GroupName,{Action,PayLoad},UsersList) ->
    case Action of
        drop_group ->
            GroupMsg = "Sending disconnect message to ~s~n",
            send_msg_to_users({drop_group,GroupName},UsersList,GroupMsg);
        receive_msg ->
            case PayLoad of
               {CreatedOn,Sender,Message} ->
                   GroupMsg = "Sending to users ~s~n",
                   send_msg_to_users({receive_msg, CreatedOn,Sender,Message},UsersList,GroupMsg);
               _ ->
                  {error, "Illegal payload format"}
            end;
        _ -> {error, "Illegal action!"}
    end.

This method is a little more involve and displays and example of a case encapsulated method, first we do the basic pattern matching, if we get the atom recieve_msg we want to use another case to check that the payload is what we expect it to be, you main thing to notice here is that the inner case statement terminates with a semi-colon which tells Erlang that this is the end of the expression.

Not catching a process message.

This is another common mistake on my behalf, either passing the wrong pattern or forgetting a trap_exit, this is something to watch out for. to save a lot of unnecessary system crashes, I’d suggest testing out pattern matches via a separate virtual node, allowing you to determine the best patterns for the task. If the process is supervised make sure that you use process_monitor(trap_exit, true) with the init of the gen_server.

State weirdness

Remember when using gb_trees:insert and the such like the new data is returned as the result & not updated internally, it is very easy to forget this, especially coming from an OOP background. So if you notice that your state is not being updated, check that the result of such methods is being passed and not left in the ether.

Another similar issue I have recently come across is mangled states within gen_server, I was basically trying to update a state if something changed, though instead of return the state property if the was no update I passed the whole state, leading in warped data, this can be quite serious & very hard to find. Here is the culprit:

handle_call({disconnect, User}, _From, State) ->
    {Reply,NewTree} =
        case gb_trees:is_defined(User, State#foobarz.users) of
            true ->
                {{ok, "User dropped"},
                 gb_trees:delete(User, State#foobarz.users)};
            false ->
                 {{error, "Unable to drop group."},
                  State}
        end,
{reply, Reply, State#foobarz{ users = NewTree }};

Well it seems safe enough right?, after noticing a bug in part of my system and realising that the state looked stranged, it wasn’t until I took a step back that I noticed the actual bug & to my amazement the bug was present in a couple of methods (I hate copy & paste). To be honest this to date is probably one of my biggest cock-ups & potentially one of the damaging bugs I’ve introduced into a system in a long time, I’d hate to see how many other system have fallen prey to this oversight.

Just so that there is an example out there for future reference here is the corrected version of the above code.

handle_call({disconnect, User}, _From, State) ->
{Reply,NewTree} =
case gb_trees:is_defined(User, State#foobarz.users) of
true ->
{{ok, "User dropped"},
gb_trees:delete(User, State#foobarz.users)};
false ->
{{error, "Unable to drop group."},
State#foobarz.users}
end,
{reply, Reply, State#foobarz{ users = NewTree }};

Spawning processes within processes

I was stuck on this for a few days and still think there is a better way I about this, spawning a new process is so simple it didnt occur to me that when creating a gen_server the gen_server was actually spawning a new process (duh, read the source). I subconciously took this in whilst reading up on it but when it came down to doing it myself it had me stumped, here is what I was trying to do:

spawn_link(mod_gen_server,start_link,arg).

The process I was trying to spawn was a gen_server and already spawning a process within itself, so what I was effectively trying do was spawn_link something that was trying to spawn a another seperate process, not quite what I wanted & Erlang definately didn’t like it, all I needed was to call the process and to link/monitor it. That simple.

Multiple processes

Well this was another one, once I worked out how to link a process I soon realised I needed that child process to have a unique module name, this is another one that may seem hard but in practicality it is yet again very simple.

start(Name,Desc) ->
    gen_server:start_link({global, Name}, ?MODULE, start, [Name,Desc]).

This way, the process is given the name passed the user, there are probably safer secure ways of doing this but for what I need at the moment, it works at treat.

Module definition not the same as file name

Is such a basic thing to oversee especially coming from a language where file names aren’t so important, its a simple one to fix but yet a common gotcha. Emacs has a nice lil feature that which basics alerts you to this fact and asks you if you want to changed it.

Creating result structures

Yet again another simple practice but if you come from a purely OO/C based language environment & just finished working on a OO based project or something, watch out for this, getting used to it takes some time. Simple rule:

  • Manipulate Term, giving unique names to needed terms
  • Store resulting terms into new pattern

Lists can simply be manipulated by postfixing a new list element to an existing on:

{{FName,_MName,LName},Address,Number,_OtherStuff} = UserDetails,
[[FName,SName,Address,Number]|Results].

Parsing XML

Whilst messing around with my Twitterl project, I quickly realised one thing, how was I to parse XML (second nature to me in PHP/C++) in Erlang though it did take me a while to get my head around, thanks to some related code by Nick Gerakines, reading his code definately shed some light on the area.

This is still a doozy, hopefully I’ll find a better way of doing this, basically we use xmerl_xpath:string/2 to retrieve an XML element, this result is a tuple, so we need to work on it further to retrieve the actual value, this could simply be done by:

[#xmlText{value = Value}] = xmerl_xpath:string(XML, "/item/title/text()").

I had implemented a variation of this code within my module but found it slow, so after reading a comment from Nick Gerakine’s in relation to my project, I had a look at his code for his Twitter API & noticed a bit of code that did they type of functionality I wanted, I’d like to find a way to improve this some more, mainly by looping over a record (still to learn) but it definately does the trick & imspires me to write more elegant code:

parse_item(Node) ->
    Item = #tweet {
      title = format_text(Node, ["/item/title/text()"],""),
      pubDate = format_text(Node, ["/item/pubDate/text()"],""),
      link = format_text(Node, ["/item/link/text()"], "")
    },
    Item.

This method also takes two Terms, the XML node & a list of XPath strings, it passes the node off to format_text, which does most of the work:

format_text(_, [], Result) -> Result;
format_text(Xml, [Xpath | Tail], Result) ->
    Results = lists:foldr(
        fun(#xmlText{value = Value}, Acc) ->
            lists:append(Value, Acc);
        (_, Acc) -> Acc
        end,
        Result,
        xmerl_xpath:string(Xpath, Xml)
    ),
    format_text(Xml, Tail, Results).

Here we basically loop over the XPath list retrieving the results and storing them in a list called Results, when the list of XPaths is empty we return the result, as most of the time I only need the one XPath it this method wasn’t introduced until i needed to play around with Twitters Users & Statuses XML.

Well that generally covers it, there is still a few more things a I want to cover & no doubt I’ll add more to this post as

Stuff I have yet to learn

  • Still not got round to getting getting auto-complete work under emacs.
  • Error and log handling principles, how to recover from a termination.
  • erl_crash.dump, is totally alien to me.
  • RESTful applications (have done some R&D but have yet to implement the actual functionality).
  • TDD & the whole process is still not well documented, ideally I want to be able to develop using the same TDD process I use in OOP.
  • Find out how to inplement a continous integration system to make development smoother.

Admittedly alot of this is more best practices and handling production code but I’m never happy until I have these things ironed out ;).

Well its been near four months since I started coding Erlang & I’ve got to say I’m enjoying the hell out of it, I thought the curve would be steep & admittedly initially it was, I still have moments of total confusion but I’m sure the more I play with Erlang the less this will happen, hell I still have those moments in C & PHP so nothing really different there.

Next

The next post will focus on how to put all this together to make an OTP system & how to main builds using Erlware’s Faxien & Sinan.

h1

Lighting up the tunnErl Pt 2 – Discovering a new wErld

January 5, 2009

This post continues on from Part 1, this post focuses primarily with the concepts of Erlang, throwing in a few tips as I go along.

The Basics

There are heaps of places to pick up the basics as mentioned before (Erlang, TrapExit, PlanetErlang & Lighting up the tunnErl Part 1 are good starts) but I’ll highlight some functional fundermentals to help understand the basics of Erlang code.

Term definitions

  • Atoms – depicted by a lower case string some_atom
  • Tuples – depicted as {Term}, can have a number of atoms, tuples & lists.
  • Lists – depicted as ["a","list"]
  • Term (a mixture of the above).{a_simple_atom,Term,["a","list"]}

A few tips

Well things are as easy as you make them, so here are some tips that may help whilst programming with Erlang.

  • Terms are pretty much immutable (though state & records can be changed).
  • Make sure that Terms that are different have different names, coming from OOP it is all too easy to over look the fact that two actual Terms can not share the same name. The following will cause an error
    Term = 'hello". Term = {term}.

    & even harder to debug

    Term = 4. Term = 2 + 2.

    just means that 4 is identical to 2+2.

  • If you change a match pattern, make sure all references of it are refactored.
  • Terms that you’re not bothered with should start with a ‘_’ ie. _DontWant, could just use _ but when coming back to the code, it makes it harder to understand.
  • Start off small, get familiar with erl & passing around Terms.
  • Test out the modules & commands your not familiar with in erl, it will save you time when it comes to actual coding.
  • Have a browse of other Erlang projects, read through the source, see what modules are commonly used & how things are done.
  • Get used to the Erlang documentation & the commonly used methods (lists,ets.dets.gb_trees).
  • OOP natives beware: the sooner you get used to using the word ‘term’ instead of ‘variable’, when thinking in Erlang, the quicker the transition.

A few examples

Now I’m not going to go into great detail here, though I will show some basic examples of the various terms, I wont go into details as usual, I’ll leave that for you guys to read up on if you already don’t know, curiously is king:

a_simple_atom
ATerm
{a_simple_atom,ATerm}
{}
{{ATerm,[SumList]},_DontWantThis}
[]
["foo","bar"]
Term = {{{ATerm,[SumList]},_DontWantThis},["foo","bar"]}

Are funs fun?

It didn’t take long to get to grips with the basics, understanding funs was a different ball game though. I think it comes down to the same issue I initially had with XML, it is so flexible you don’t quite know where to start with it, so here’s something I use in one of my projects:

    lists:foreach(
      fun(GroupName) ->
      io:format("Dropping group ~s...~n",[GroupName]),
      gen_server:call({global,GroupName},stop,infinity)
      end,
      GroupNames).

Here the following steps are taken:

  1. Loop over each of the GroupNames them (using method lists:foreach)
  2. Each result is stored and passed as the term Groupname
  3. Prints a message using GroupName for a dynamic value.
  4. A call is made to the process to shutdown.

Steps 2 & 3 are where the fun begins (pardon the pun), we tell it to take an argument (GroupName) & then to execute everything within the fun, this is an extremely powerful feature, one which I still have yet to manipulate properly. I’d suggest messing around with them abit (using others code) and getting used to their power and flexiblity, you’ll soon find funs are fun.

That does it for this post, the next will focus on after the basic, hopefully giving you a chance to delve into the fundermentals of Erlang.

Next
I will focus on the actual problems & issues I’ve faces whilst working with Erlang, this can be found here.

h1

Lighting up the tunnErl Pt 1.

January 4, 2009

Well after a few months of playing around with Erlang, I thought I’d put my findings in one place for myself and others.

Well for those that aren’t familiar with Erlang or never heard of it, in a nut shell Erlang is a functional programming language create multiple processes across multiple nodes, allowing for a reliable & fault tolerant system.

It allows us as developers to develop applications over a number of nodes, these nodes can be either local to the machine they are running on or on a totally different machine running the same software, this allows for concurrent reliable systems.

I wont go into too much detail on how all this works because it’s done so well in other places (Erlang).

This post will cover the learning resources I’ve found useful & invaluable to my exploration into Erlang, the 2nd will highlight mistakes & general cock-ups made over the past few months. Followed by the 3rd, dealing with project management and how we can achieve this & more with Faxien & Sinan.

There is a decent number of sources out there for Erlang from code examples to general tutorials & articles a few of them which you can find at the bottom of this post. Not to mention a hell of alot of useful and interesting projects created with Erlang (CouchDB, ErlyWeb, Mochiweb to name a few).

Myself I’m focusing on using Mochiweb, Beepbeep & CouchDB for web applications. Mochiweb to deal with RESTful requests/responses, Beepbeep to deal with the MVC  of the application & CouchDB to store data (documents). If you haven’t already read up on CouchDB I suggest you do, it may well give you some nice ideas ;).

Well I as mentioned in a previous post I’ve had the itch to learn a new language. I started off by reading Programming Erlang by Joe Armstrong (a must have for anyone making the jump into the world of Erl. It has helped me no end when my brain goes blank & I’ve forgotten how something in Erlang works. Over the months I’ve practically slept with my copy, the examples are elegant and Joe has an excellent job of creating a good introduction to Erlang and its practices.

That is where ‘Erlang In Practice‘ (Screencasts by Kevin Smith) this set of screencasts are definately worth the £20 odd I spent on it. The most useful being episode 6 & 8 which focus on e-unit & supervisors respectively. Though admittedly the episode on e-unit could of focused a bit more on testing OTP based modules (I guess I’ll have to pick this up later).

Still there is more to learn, personally the best way to learn something is to do it, so I’ve played around with the shell whilst working through examples, got myself familar with the OTP documentation and stifted the net for code examples, githubs a nice place to look (ngerakines, davebryson & few others have some decent code you can get some ideas/insipration from)

Resources
There are alot of new tools coming out and it is always interesting to see what others are doing, heres some sites/blogs that I find myself practically attached to:

Well I think that pretty much wraps it up for part 1, part 2 is in the drafts so it should be published in the near future. If you have come across any other useful resource, feel free to drop me a comment, the more information the better imo 😉