This post is the next in the ‘Lighting the tunnErl‘ series, documenting my experiences with Erlang as I learn it and its tools, as the previous post focused on Faxien & Sinan, I decided to concentrate on working with yaws or Mochiweb. As I’m still actively working on Chatterl, which as mentioned here, is a multi-node chat system, and it needed a web interface, I did some research and decided to use Mochiweb, though the part six was originally going to be ‘Erlang Gotchaz’, I thought I should post this first whilst the information is still fresh in my mind.
General overview
I recently started to play around with Mochiweb & BeepBeep to see what functionality they could provide Chatterl, after a couple of days working around the BeepBeep’s code, I decided to focus on the actual gateway API (CWIGA) and use Mochiweb along with a little manipulation to create the gateway service that would in turn serve the frontend which will use BeepBeep to render the data. A bit convoluted maybe but I want other to be able to use the backend separately and plug in their own frontends ;), not to mention it’s an excellent chance to play with new toys.
The foundations weren’t that hard to put together, think it took all of 2 minutes to get Mochiweb running, setting up the basics are just that & simple interactions with other systems is straight forward. Though saying that I quickly found that one can encounter some serious head banging moments with Mochiweb especially with the lack of documentation at present.
I also feel Mochiweb lacks fundamental features like destroying cookies, manipulating headers/query string & the such like a lot of this is left down to the developer, which in my opinion isn’t ideal, I’d rather spend time on the problem at hand rather than deal with mini side effects.
I really really hate pros/cons, loves/hates, but hell I really do have a love/hate relationship with Mochiweb at the moment, I’m hoping that this is just part of the curve but I have to admit it certainly is an interesting beast.
Loves
- Quick & easy to initialise Mochiweb.
- Initialising cookies is simple.
- Fully customisable
- Setting up basic handling is easy.
- Is fast and lightweight.
Hates
- No encryption methods.
- Fully customisable.
- No real work examples.
- Converting content body has to be done manually.
- Isn’t well documented.
The past week has had a lot of ups & downs in regards to Mochiweb, after a week of tinkering, I have finally finished the crux of CWIGA (Chatterl’s Web Interface Gateway API) & generally enjoyed working with Mochiweb & will utilise it for the front-end of Chatterl integrated with BeepBeep, which will deal with cookie management.
Ok, with the mini review over with, here’s some info on how I’m using Mochiweb at the moment, the source code and be found here.
Initialising Mochiweb
init([Port]) ->
io:format("Initialising Chatterl Web Interface~n"),
process_flag(trap_exit, true),
mochiweb_http:start([{port, Port}, {loop, fun dispatch_requests/1}]),
erlang:monitor(process,mochiweb_http),
{ok, #state{}}.
Excerpt taken from a init method, here we initialise the gen_server, as the gen_server has a supervisor we use trap_exit, just below is our call to initialise Mochiweb, where we pass it the port {port, Port} & tell it what function to pass the request to (dispatch_requests).
Dispatching requests
Now we need to be able to handle all our request, to do this we use the following method:
dispatch_requests(Req) ->
[Path|Ext] = string:tokens(Req:get(path),"."),
Action = clean_path(Path),
handle(Action,Ext,Req).
As I want to know what the extension type was requested (so that we can respond with the appropriate format) we split the request path using tokens and supplying ‘.’ as the parameter, this way URL like http://127.0.0.1:9000/controller/action.xml our Ext will store ‘xml’.
We then clean the path using clean_path, which will be discussed next to determine the Action we want to carry out. We then have all we need to handle our requests.
Cleaning paths
clean_path(Path) ->
case string:str(Path, "?") of
0 ->
Path;
N ->
string:substr(Path, 1, string:len(Path) - (N + 1))
end.
Here all we want is the actual path to the quest ‘/some/action/here’, so we check to see if there is a ‘?’, if so return everything preceding it, otherwise we return the path, that simple really. Though having said that, I’m sure this type of functionality should be in Mochiweb (one day eh, it’s still not production ready).
Handling requests
Now this is the fun bit, I’m going to delve into building JSON structures to use as response, purely as this one as serious pain in the ass to implement & hardly documented online.
handle("/connect/" ++ Client,ContentType,Req) ->
{Type,Record} =
case gen_server:call({global,chatterl_serv},{connect,Client}) of
{ok,_} -> {"success",Client++" now connected"};
{error,Error} -> {"failure",Error}
end,
send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)});
Now this method took a little while to structure properly and can look a little daunting, lets walk through it’s purpose.
So our handle method needs to be able to make calls to Chatterl and build the appropriate response using records (carriers). each response has a type which determine the type of response to return (success/failure) here we can respond with the appropriate HTTP response code (200/401,etc). We also need the actual response data (Record) which is packaged and returned in a Tuple ({Type,Record}) along with the response type. These pieces of data are then sent off to send_response (send_response(Req,{get_content_type(ContentType),build_carrier(Type,Record)}); which will be explained shortly. Before that there are two methods that need to be explained, get_content_type & build_carrier. The former is a simple wrapper which returns a record with the record type (“success”) & message (“user joined”). Where as the former passes the desired content type (Ext) and receives the appropriate content type contents:
get_content_type(Type) ->
case Type of
["json"] ->
"text/plain";
["xml"] ->
"text/xml";
_ -> "text/plain"
end.
As mentioned earlier I want to be able to respond in either JSON or XML, with this method we are able to pass the type (remember Ext?) and return the correct content type. As I prefer JSON I’ve implemented that as the default.
Sending responses
Now here is where it starts to get interesting, we have our content type along with our results & response type, we now need to find out what type of response we have and build the appropriate format (JSON/XML).
send_response(Req, {ContentType,Record}) when is_list(ContentType) ->
Response = get_response_body(ContentType,Record),
Code = get_response_code(Record),
Req:respond({Code, [{"Content-Type", ContentType}], list_to_binary(Response)}).
Now this method is pretty simplistic, though a lot of work is being done by get_response_body, We basically retrieve our formatted response, then get the appropriate response code, then we shuttle the data off to the client. Job done.
Building response bodies
Here we want to know what type of response we should contruct, once doing so, build & return the response.
get_response_body(ContentType,Record) ->
case ContentType of
"text/plain" ->
json_message(Record);
"text/xml" ->
xml_message(Record);
_ -> json_message(build_carrier("error","Illegal content type!"))
end.
Next we need to correct response code:
get_response_code(Record) ->
case Record of
{carrier,Type,_Message} ->
case Type of
"failure" -> 200;
"success" -> 200;
"error" -> 500;
_ -> 500
end
end.
As all errors and unknown responses shouldn’t happen we want to our response code to return 500 in those cases, otherwise return with a 200.
Now this is where it gets fun & where I still need to iron some things out, ideally I have a message_body method passing an atom (JSON/XML & the data) and returning the appropriate data. Below I will go into my current solution.
Building JSON
I have to say construction response can be quite a pain in Erlang, I’d suggest doing the thing I haven’t, test your patterns. One day I’ll starting testing my Erlang code as vigously as the rest of my code.
Ok back on track, so we need to one generate three types of carrier (groups,clients & messages), if the type passed doesn’t exist we print a message on the node (ideally the client should get some kind of error, I have yet to implement this yet though). Otherwise the helper method (handle_messages_json) is called and the results are injected into the outter structure ({struct,[{chatterl,{struct,[{response,Struct}]}}]})which in turn is passed to mochijson2 for conversion.
json_message(CarrierRecord) ->
{carrier, CarrierType, Message} = CarrierRecord,
Struct =
case Message of
{carrier,Type,MessagesCarrier} ->
case Type =:= "groups" orelse Type =:= "clients" orelse Type =:= "messages" of
true ->
handle_messages_json(Type,MessagesCarrier,CarrierType);
false ->
io:format("dont know ~s~n",[Type])
end;
_ ->
{struct,[{CarrierType,list_to_binary(Message)}]}
end,
mochijson2:encode({struct,[{chatterl,{struct,[{response,Struct}]}}]}).
Now we have differing types of carriers so here we need to check what we have and deal with it appropriately, for this post we will just focus on handling our JSON responses. Just past the case statement which checks that our carrier type is either a groups, client or messages we send the Type, along with the Carrier (holding the actual data) & the carrier type (storing which type of carrier we are handling). Once the results are back from handle_responses_json, they are inserted into a wrapper structure which is in turn used to create the whole JSON body:
{"chatterl":
{"response":
{"success":
{"groups":[
{"group":"nu_group"},
{"group":"anuva_group"},
{"group":"one_more"}
]}
}
}
}
Handling JSON responses
handle_messages_json(Type,MessagesCarrier,CarrierType) ->
case Type =:= "messages" of
true ->
case MessagesCarrier of
[] -> %Empty list.
{struct,[{Type,[]}]};
[{carrier,_MessageType,MessageData}] -> % A Single message.
{struct,[{CarrierType,
{struct,[{Type,loop_json_carrier(MessageData)}]}}]};
Messages -> % Multiple messages
{struct,[{CarrierType,
{struct,[{Type,inner_loop_json_carrier(Messages)}]}}]}
end;
false ->
{struct,[{CarrierType,{struct,[{Type,loop_json_carrier(MessagesCarrier)}]}}]}
end.
As mentioned earlier we have three type of responses, empty, singular & multidimensional. In this method we work out what type of response structure we have and construct the appropriate JSON structure for it. We need another helper here mainly for building our structured responses, which is where loop_json_carrier & inner_loop_json_carrier comes in handy (code shown below). They basically iterate over the response carrier. Once complete it wraps this structure up in another which stores the carrier & response types.
loop_json_carrier(CarrierRecord) ->
[{struct,[{list_to_binary(DataType),clean_message(Data)}]} || {carrier,DataType,Data} <- CarrierRecord].
inner_loop_json_carrier(CarrierRecord) ->
[{struct,[{list_to_binary(MsgType),loop_json_carrier(Msg)}]} || {carrier,MsgType,Msg} <- CarrierRecord].
Now this methood isn’t ideal I’m sure that I can improve these methods but to be honest I’ve been working tirelessly on this & decided to leave it as something else to work out. The latter loops over each carrier in turn alling loop_json_carrier & wrapping the result in a struct tuple. The former method does more aless the same functionality apart from it cleans the message body using the clean_message/1 method.
clean_message(Data) when is_tuple(Data) ->
{A,B,C} = Data,
[A,B,C];
clean_message(Data) ->
list_to_binary(Data).
We’ll I use this as a wrapper to clean up responses, ideally you could convert data to UTF-8 list_to_atom, what ever you fancy. I’m presently using it to clean Erlang dates (as I’ve not worked out how to convert them properly yet) & lists into binary.
Summary
Well thats basically it for now. I’ve left out XML parsing partly as I haven’t have the time to implemented parsing multidimensional structures yet & the code can be found here for those who are really curious ;).
I will say this though working with XML in Erlang is probably one of the most unpleasant experiences for me at the moment. All in all Mochiweb is a nice powerful tool, it still has a lot of growing to do but it is definitely a shining example of what can be done with Erlang, hopefully more documentation will enter the wild & we will see it hosting a number of services in the future.
Baphled's Weblog 