Chatterl – Erlang based chat engine

Well I’ve been hacking around for the past few months trying to get my head around Erlang. After playing around with Twitterl I decided I should start a new project (OTP based) which I have a decent understanding of & have wanted to work on for some time. A chat system was the first thing that came to mind, I’ve worked on and built a few in my time and thought Erlang would be a good language to build a chat engine from scratch.

For those not interested in the long description and prefer to dive in source, you can find a copy of the project at github.

Description

A chat system that can be housed over a number of nodes and track clients over varying devices, at the time of the writing the system works over multiple nodes and is able to do the basics (connect to a group, send message to other clients and connected groups).

The main focus of this project is to create a chat system that is highly reliable as well as scaleable. Other developers will be able to create add-on modules that are able to interact with chatterl and further enhance the functionality of chatterl and the chatterl clients experience.

As mentioned the system is OTP based, of which it uses Sinan to maintain its builds.

Installing Chatterl

At the moment of this writing Chatterl is still in alpha so their is no real release at the moment, to get it running you will need to do the following:

git-clone git://github.com/baphled/chatterl.git &&
cd chatterl &&
sinan doc && 
sinan dist &&
cd _/build/development/tar &&
sudo faxien install-release chatterl-0.1.0.2.tar.gz

The above presumes that you have Sinan configured & installed, if you haven’t refer to erlware.
You will need to change to cookie & name values to something else, doing so should drop you into the erlang shell & ready to run the Chatterl application. To run a client on a different machine you will need to do the above if (making sure that the -sname is not the same as any other connected nodes & that the cookie is the same) if connecting on the same box simply cd to the ebin directory and run the following command:

erl -name bar -setcookie abc

From here you will need to make sure that they nodes can connect via:

net_adm:ping(foo@bar.net).

Where foo is the node name & bar.net is the tld of the node box (indicated within yout hosts file or dns server).
Once you receive the infamous pong response you are ready to roll.

Features

• Client login/logout to Chatterl.
• List Chatterl groups.
• List Chatterl users.
• Login/logout of a chatterl group.
• Send message to a group and other clients.

Future Features

• Centralised Error logging and data storage.
• Client customisable routines (able to poll RSS feeds, twitter, FB and the such like).
• Better handling of errors.
• User registration.
• FB Connect.
• Chat bots (AIML based).
• Web interface/API.
• Chat modules handler(banning, censorship, chatbots).

Useage

Starting the server
Chatterl server runs as an OTP application and uses a supervisor to manage it (in later versions there will be options to spawn multiple servers, allowing for a more fault tolerant chat system). To start up the server you simply need to run the following command:

application:start(chatterl).

Which will initialise the server allowing clients to connect and groups to be created. Groups can be created on differing nodes as long as the node can communicate with the chatterl_serv.

Starting a group
Chatterl groups can be started on any node that can communicate with the server, this allows the user to create a number of groups on varying nodes, helping with general organisation as well a performance and reliablity.

A group can be initialised by calling the command:

chatterl_serv:create("room","description").

which will spawn a group process which users can connect to.

Connection to chatterl
At the time of this writing chatterl_clients can only spawn a client per node, this will later be changed once the web interface has been fully implemented, possibly to a refactoring the client to a parameterised module.
For the moment node users must follow the basic OTP configurations (same cookie, valid DNS name, etc). Creating a connection to the server is done by using the following command.

chatter_client:start(UserName).

This will initialise a user and connect them to chatterl_serv (must be done before users can join a group or communicate with other chatterl users).

Disconnecting from chatterl
Chatterl clients can simply disconnect from chatterl by issuing the following command:

chatterl_client:stop().

This will disconnect the user from all the groups they are currently connected to as well as the actual Chatterl server.

Joining a group
This can be done by using the following command:

chatterl_client:join(GroupName).

If the group exists the user is able to join the group allowing them to send message to the room.

Dropping from a group
This is as simple as connection, simply supply the following command:

chatterl_client:drop(GroupName).

This will send a message to the group, which will handle the termination.

Sending group message

chatterl_client:send_msg(GroupName,Message).

GroupName being the name of the group the client is connected to, Message being the message that you want to send to the receiving client. If the message is sent successfully all users connected to the group will receive the message.

Sending a private message
This allows a Chatterl client to send a private message to another client, by executing the following:

chatterl_client:private_msg(RecipientName,Message).

If the message is sent successfully then the sender will receive follow message:

{ok,msg_sent}

in turn sending the message to the receipients node.

Zend_PHPUnit_Fixtures uploaded to GitHub

Well I’ve been working on this project (Zend PHPUnit Fixtures) along with a few collegues over the past few months, It was my intension to submit it to Zend some point in the new year but thought I’d put it out in the wild for others to pick at.

Well I use TDD (PHPUnit is my friend) on a daily basis within ZendFramework and soon found it cumbersome to create test data (fixtures) ending up in huge test cases with more test data setup than actual assertions, this matched up with the fact that I hate integrating DB into my tests (mainly because the tables would still contain old test data & fail depending on whether the data was added in previous tests or not) So I decided to do some digging, the best solution seemed to be to create a fixture handling system that plugs into ZendFramework, leaving myself and others with more time actually testing and less time building up test data and removing fixtures from a database (which can be near impossible when it comes to automated testing).

With this realisation I decided to create my own which I’ve reversed engineered from CakePHP. When I have the time (hopefully over the holidays) I’ll go into how the mini framework can be used. For now I’ll just describe the features or each class and what they are for:-

PHPUnit_Fixtures

Basic fixture handler, used for creating test data that does not interact with a DB. With this object we are able to create basic fixtures that we can use for dummy data with our test cases. Each piece of test data can have an alias (‘ALIAS’) with the aliases name as the value, doing so will allow us to use the PHPUnit_Fixtures::find($aliasName) which will retrieve the desired fixture.

PHPUnit_Fixtures_DB

Has the same functionality as PHPUnit_Fixtures but used specifically for DB centric tests, DB test data will be added to our ‘_test’ DB, and cleaned up (truncated) on each test case, to make sure that we have the expected data.

PHPUnit_Fixtures_DynamicDB

Has the same functionality as PHPUnit_Fixture_DB, with the added functionality of being able to create tables setup my MySQL Workbench. With an child object of this class we are able to specify retrieving all schema or a specific on (denoted by the schema table name).

DevelopmentHandler

Used to handle our development environments, there are times when we want to quickly place test data on our staging DB for functionality testing and the such like, this class along with one of our PHPUnit_Fixtures, will easily allow us to populate this environment with the data we have been using for our unit tests, making it quicker to migrate test data from one place to another.

You can find the project at GitHub, If anyones interested in adding to this project or have any comments/questions drop me a line.

Finally got myself on Github

Well after a while of staying away from learning yet another versioning system, I’ve fallen in line, I decided last night that it would be a good idea to put all my Erlang based work up on github.  I was hoping to use TDD with this project but that will have to be looked at later (TDD & Erlang is a bit much for me ATM).

I’ve submitted my second attempt at an Twitter client (Twitterl), which I’ll be using with other projects, for now feel free to have a look & possibly take part. Over time I’ll gradually add more functionality to the system an blog my experiences as I go along.

Creating a Twitter client in Erlang

So I’ve been playing around with Erlang for a little while now and decided to create a small client for one of my favourite mini blogging services (Twitter), which I’ve decided to share with all, both to help other noobs and for myself a personal reference.

So we want to be able to retrieve a REST like response from twitter, where do we start? First thing was to start up erl from the command lineerl -pa ~/erlware/packages/5.6.3/lib/*/ebin "$@"

I had installed erlware to my local directory & linked it into erl by running the above command. Now that we are in erl, I started inetsinets:start().

Now we are ready to start playing with http:response, I want to be able to retrieve the HTTP body response and do some XML parsing with it http:request("http://twitter.com/statuses/public_timeline.rss")
which returned me something simular to
{ok,{{"HTTP/1.1",200,"OK"},
[{"cache-control","max-age=1800"},
{"connection","close"},
{"date","Mon, 01 Dec 2008 10:46:06 GMT"},
{"server","hi"},
{"content-encoding","UTF-8"},
{"content-length","9251"},
{"content-type","application/xml"},
{"expires","Mon, 01 Dec 2008 11:16:07 GMT"},
{"set-cookie",
"JSESSIONID=2C9CFD77A4EF365B46884168D089FD4A; Path=/"}],
[60,63,120,109,108,32,118,101,114,115,105,111,110,61,34,49,
46,48,34,32,101,110,99,111|...]}}
With a little digging, we come to find out that the following pattern matches pretty nicely{ok,{_Status,_Head,Body}} = http:request("http://twitter.com/statuses/public_timeline.rss")
Typing Body. in the erlang shell returns a list simular to the following:[60,63,120,109,108,32,118,101,114,115,105,111,110,61,34,49,
46,48,34,32,101,110,99,111,100,105,110,103,61|...]
Now this is our HTTP body response, now we can fire up emacs or your favourite text editor and create our first method. -modules(twitterl).
-compile(export_all).

retrieve_response(Url) ->
{ok,{_Status,_Head,Body}} = http:request(Url),
Body.
Make sure you save this file with the same name as the module (or Erlang will complain) & make sure the file is in the same directory that erl is running in. c(twitterl).
Will compile our piece of code. Now we should have the following command availabletwitterl:retrieve_response("http://twitter.com/statuses/public_timeline.rss").
Which will give u the same result as the previous commands passed to erl. Now the Body term still has our HTTP body so well play with this for the time being, if not then just do the following:Body = twitterl:retrieve_response("http://twitter.com/statuses/public_timeline.rss").
Now for the next piece, we actualy want to parse this body and retrieve all its descriptions (which stores our actual twitters).xmerl_scan:string(Body) Give us a result simular to the following:xmerl_scan:string(Body).
{{xmlElement,rss,rss,[],
{xmlNamespace,[],[]},
[],1,
[{xmlAttribute,version,[],[],[],[],1,[],"2.0",false}],
[{xmlText,[{rss,1}],1,[],"\n ",text},
{xmlElement,channel,channel,[],
{xmlNamespace,[],[]},
[{rss,1}],
2,[],
[{xmlText,[{channel,2},{rss,1}],1,[],"\n ",text},
{xmlElement,title,title,[],{xmlNamespace,...},[...],...},
{xmlText,[{channel,2},{rss,...}],3,[],[...],...},
{xmlElement,link,link,[],...},
{xmlText,[{...}|...],5,...},
{xmlElement,description,...},
{xmlText,...},
{...}|...],
[],"/home/baphled/projects/erlang/status_managerl",
undeclared},
{xmlText,[{rss,1}],3,[],"\n ",text}],
[],"/home/baphled/projects/erlang/status_managerl",
undeclared},
[]} This returns our XML and the rest of the HTTP body, we only want the XML so{Xml, _Rest} = xmerl_scan:string(Body) Will store our XML within the Xml term. Now we have the data we want to parse, we can use the following command to retrieve a list of tuples, containing our descriptions in XML format simular to the following.[{xmlText,[{description,4},{item,12},{channel,2},{rss,1}],
1,[],
[101,120,116,101,101,110,114,101,99,101,110,116,58,32,224,
184,161,224,184,178,224,185,129|...],
text},
{xmlText,[{description,4},{item,13},{channel,2},{rss,1}],
1,[],
"ston3monk: wild place where everyday brings somethnig new ",
text},
{xmlText,[{description,4},{item,13},{channel,2},{rss,1}],
2,[],
"& unepected. do i continue on the unknown, or do i go hang out on koh samui ",
text},
{xmlText,[{description,4},{item,13},{channel,2},{rss,1}],
3,[],"& work 4 a while?",text},
{xmlText,[{description,4},{item,14},{channel,2},{rss,1}],
1,[],
"juffrouwjannie: @Hoof is het al latenweoveralweerzoutopgaanleggendag?",
text},
{xmlText,[{description,4},{item,15},{channel,2},{rss,1}],
1,[],
[116,104,97,105,50,104,97,110,100,58,32,91,84,50,72,93,32,
224|...],
text},
{xmlText,[{description,4},{item,16},{channel,2},{rss,1}],
1,[],
[65,111,121,97,89,117,121,58,32,227,129,138,232,143,147,229,
173|...],
text},
{xmlText,[{description,4},{item,17},{channel,2},{rss,1}],
1,[],
[101,99,108,117,99,105,102,101,114,58,32,64,105,99,104,105|...],
text},
{xmlText,[{description,4},{item,18},{channel,2},{rss,1}],
1,[],
"simonandre: @fabienthomas envoie leur un mail ils sont super r\303\251actifs.. je l'avais fait y'a un an, dans les 48h \303\247a avait \303\251t\303\251 corrig\303\251",
text},
{xmlText,[{description,4},{item,19},{channel,2},{rss,1}],
1,[],
"debugz: @draconiams I saw some male chicks from here today,they are hott too =P",
text},
{xmlText,[{description,4},{item,20},{channel,2},{rss,1}],
1,[],
"csinctw1: [JCSAGE] About to build project=mobility status=Success",
text},
{xmlText,[{description,4},{item,21},{channel,2},{rss,1}],
1,[],
[101,108,101,99,116,114,105,99,97,108,80,101|...],
text},
{xmlText,[{description,4},{item,22},{channel,2},{rss,1}],
1,[],
[108,117,110,97,114,121,117,101,58,32,231|...],
text},
{xmlText,[{description,4},{item,23},{channel,2},{rss,1}],
1,[],
[109,49,48,56,58,32,230,172,161,227|...],
text},
{xmlText,[{description,4},{item,24},{channel,2},{rss,1}],
1,[],"DerAlbert: @adventskalender open 1",text},
{xmlText,[{description,4},{item,25},{channel,2},{rss,1}],
1,[],
[100,97,105,95,97,105,114,58|...],
text},
{xmlText,[{description,4},{item,26},{channel,2},{rss,1}],
1,[],
"backfunshop: backfunshop.de : backfun.de ** Bratentopf 28cm 6,7l auch f\303\274uktionsherde http://tinyurl.com/69ezcc",
text},
{xmlText,[{description,4},{item,27},{channel,2},{rss,1}],
1,[],
"drillhalllib: Want the chance to win \302\243100 in Amazon vouchers? Take part in this years Student Satisfaction Survey now! http://moourl.com/dhl08",
text},
{xmlText,[{description,4},{item,28},{channel,2},{rss,1}],
1,[],
"techdigest: Considering moving to a Dvorak keyboard - anyone ever tried it? http://tinyurl.com/9x84j",
text},
{xmlText,[{description,4},{item,29},{channel,2},{rss,1}],
1,[],
"franksting: @stilgherrian huh, whats that about throat infections? Both ms3 (i can say that now) and I have something and we both went swimming on S ...",
text},
{xmlText,[{description,4},{item,30},{channel,2},{rss,1}],
1,[],
[65,108,108|...],
text},
{xmlText,[{description,4},{item,31},{channel,2},{rss,...}],
1,[],
"abcnewsVic: Great Ocean Road reopens after fatal smash: A stretch of the Great Ocean Road that was the scene of a .. http://is.gd/9HAP",
text}]
Now we go back to our source file and create a method for our discovery:latest_twitters() ->
Body = retrieve_response(?PubTimeUrl),
{Xml, _Rest} = xmerl_scan:string(Body),
Twitters = xmerl_xpath:string("//item/description/text()", Xml),
Twitters.
This method will use our previously created method to retrieve our response and then we parse XML searching for the description element within the item element, this then stores each of the enteries in Twitters, which is then returned. Save and recompile the file. We can now use our latest_twitters method to returl a list of our results.Twitters = twitterl:latest_twitters() will store our list to our Twitters term. Now we now we have a list so we will manipulate it as follows to see how we can deal with our results.[H|T] = Twitters. gives us the first term in the list, something simular to the following:{xmlText,[{description,4},{item,12},{channel,2},{rss,1}],
1,[],
[107,105,114,105,108,108,95,121,117,114,105,58,32,228,187,
138,229,185,180,227,129,174,231,180|...],
text}
Lets break this down a little so we can deal with the specific data we are interested in, now our term is a tuple containing an atom, a tuple,another atom,an empty list, followed by our text, proceeded by an atom. We only need the text string so the following pattern will do us.{_,_,_,_,Title,_}Lets give it a try.{_,_,_,_,Title,_} = H.
io:format(Title). We get a result simular to:kirill_yuri: \344\273\212\345\271\264\343\201\256\347\264\205\347\231\275\343\201\257\347\264\205\343\202\222\345\213\235\343\201\237\343\201\233\343\201\237\343\201\204\351\233\260\345\233\262\346\260\227\343\201\214\345\205\205\346\272\200\343\201\227\343\201\246\343\202\213\346\260\227\343\201\214\343\201\231\343\202\213\343\202\223\343\201\240\343\201\214\343\201\251\343\201\206\343\201\213\343\201\255\343\200\202 The above is our actual twitter, now armed with this piece of information & a little tinkering we can add another method to our source file.loop_twitters([Twit|Twitters]) ->
case is_tuple(Twit) of
true ->
{_,_,_,_,Title,_} = Twit,
io:format("~s~n",[[Title]]),
loop_twitters(lists:reverse(Twitters));
false ->
{error, "Unable to read twitter"}
end;
loop_twitters([]) ->
okWe’ll need to do a little refactoring to make everything play nice. Namely refactor our latest_twitters method to pass our twitters on to loop twitters. I guess we’ll want to only export one method now also, which in this case will only be the latest_twitters method. This is done by removing the line-compile(export_all). and replacing it with-export([latest_twitters/0]).

Our source file should look like this:
-module(status_managerl).
-compile(export_all).
-export([init/0,stop/0,latest_twitters/0]).
-include_lib("xmerl/include/xmerl.hrl").
-define(PubTimeUrl, "http://twitter.com/statuses/public_timeline.rss").

init() ->
inets:start().

stop() ->
inets:stop().

retrieve_response(Url) ->
{ok,{_Status,_Head,Body}} = http:request(Url),
Body.

latest_twitters() ->
Body = retrieve_response(?PubTimeUrl),
{Xml, _Rest} = xmerl_scan:string(Body),
Twitters = xmerl_xpath:string("//item/description/text()", Xml),
loop_twitters(Twitters).

loop_twitters([Twit|Twitters]) ->
case is_tuple(Twit) of
true ->
{_,_,_,_,Title,_} = Twit,
io:format("~s~n",[[Title]]),
loop_twitters(lists:reverse(Twitters));
false ->
{error, "Unable to read twitter"}
end;
loop_twitters([]) ->
ok.
Now finally we'll compile our source file and run using the following commandsc(twitterl).
twitterl:latest_twitters()
Will produce our list of latest twitters simular to the following:timetoleadfr: @otromundo merci d'avoir relay\303\251 le message
bernardoruas: @rejaneayres I hope so.
osa03: \343\202\246\343\203\274\343\200\202\343\203\255\343\202\270\343\203\203\343\202\257\343\201\214\343\203\257\343\202\253\343\203\203\343\202\277\343\203\274\343\200\202
olemi: @DavidFeng: \345\260\215\344\270\215\350\265\267\357\274\214\346\210\221\344\270\215\347\237\245\351\201\223Catch-22\346\230\257\347\224\232\351\272\274\346\204\217\346\200\235.... \350\203\275\350\267\237\346\210\221\350\247\243\351\207\213\344\270\200\344\270\213\345\227\216\357\274\237
ijournal: RenRe's WeatherPredict Acquires Assets of Commodity Hedgers http://twurl.nl/pa5urw
sydsvenskan: Man knivr\303\245nad i Malm\303\266:
En man blev under knivhot r\303\245nad strax efter klockan elva i en g\303\245ngtunnel vid .. http://tinyurl.com/6zktzc
prensa: NOUVELOBS: Venise : la mont\303\251e des eaux atteint 1,56 m\303\250tre: La mer commence
à se retirer l.. http://tinyurl.com/5nb6dd
Erik_Visser: Blijft toch een leuk verhaal van die belgische minister op het blog van Nathalie Lubbebakker. http://www.nathalielubbebakker.com/
petulak: @punio miedo me dais, cinematograficamente hablando!!
...
Well that's it for the moment, the code is far from perfect & i'll already started to refactor the code to do more, currently, Im able to search for a user, retrieve their latest tweets (from & to) as well as query terms used ie. #Erlang. Hopefully when I get the time I'll post some code to help others do simular things.

Learning a new Er lang…

I’ve had the itch to learn a different language for a good while now, I thought about picking up Ruby or Python & saw no personal practical use for them, plus the fact that I’d still be stuck in the same paradigm just with different a syntax.

Whilst surfing a three weeks ago I came across a piece of code of which I had never encountered (I can’t recall where I stumpled upon this right now).

Being naturally curious my fingers sent me to Google, quickly followed by Amazon, the day after I had Joe Armstong’s “Erlang Programming” book. I have to say I cant remember the last time reading a techie book threw me into boughts of laughter chapter after chapter, not because it a helarious book but purely down to how the language solved problems.

list([Item|Items]) -> io:format("~p~n", [Item]), list(Items);
list([]) -> [].

As apposed to a ‘C type language’ which would look like:-

foeach($items as $item) {
print($item ."\n");
}

I’ve been playing with Erlang for three weeks now and am impressed, at the moment I’m playing with a twitter API, basically retrieving public & user specific twitters. Once I get my head around HTTP authentication with Erlang i’ll extend it further utilising a few other systems tying them together to track and send messages over multiple networks.

For now I’ll show you a little bit of code that will store the body of a HTTP response:
{ok,{_Status,_Head,Body}} = http:request("http://www.google.co.uk").


Firstly this is a piece of pattern matching one of the main princples of Erlang, on the right we retrieve a Tuple which we match to the left hand pattern, if the pattern is matched the response body is stored in Body.The pattern reads as:

“Our http:request from Google must match a Tuple containing the atom ‘ok’ preceeded by a Tuple containing 2 items we don’t care (_Status,_Head) and 1 which we’ll store in the item body.”

The other items (Status,_Head) are dismissed indicated by the prefixed underscore, removing the prefix would mead Status & Head would also be assign values.

It’s this pattern matching that makes Erlang fun to play with, you can quite easily find yourself getting losting in tuples, lists & atoms, at the moment I tend to find myself playing around with matching patterns within erl (Erlangs shell) storing the needed values.

In my opinion reading Joe Armstrong’s “Software for a Concurrent World” is a must, I’m still going through it now & find the reference in the appendix an constant help, I also found Kevin Smith’s screencast’s “Erlang in practice” an excellent help too, especially in regards to the TDD using eunit, I still have a great deal to learn in regards to TDD with Erlang, mainly down to me hacking around & trying to get my self familiar with the syntax.

There are also a number of cool tool, some of which I have yet to play with, mnesia (Erlang DBMS) is definately on my to play with list, along with Yaws (erlang based webserver) & Erlyweb (web ramework).

Why test anyway?

Before I start I’d like to point out that this is purely from my perspective and pretty much a subjective point of view. There are so many articles out there from an objective stand point, that I thought I do this from my own, seeing as personal accounts are pretty much one sided & usually only highlight the good points. Here, though I’m obviously a TDD advocate I will try to address the issues I have come across whilst working with the methodology.

I know I’m not the only one who hates spending hours trying to track a bug & have phrases like: “where did that bug come from?” and “i didn’t change that” swish around my brain.

Over the years I’ve learnt that there is no such thing as a perfect solution but have adopted TDD purely because I have always felt that software projects lacked the flexibility of car designers (who test our concepts before making a final decision). Being security conscious and taking pride in my work I thought it would be of benefit to explore the practice first hand.

I’ve always hated pros & cons, especially as some peoples pros are others cons, so I’ll just bullet each methodologies attributes that I am aware of, make of it what you will.

No tests

• Quick to initially code.
• No need for test suite.
• Cab be hard to trace errors & debug.
• Can leads to misconceptions in design.
• Changes could break existing code, without knowing and/or introducing new bugs.
• Developers feel uneasy making changes to the system.
• Debugging gets left to last or even out entirely due to time restraints.
• Functionality can get left out or reproduced.

Tests

• Need a test suite at hand.
• Able to test each module as its built.
• Easy to created dependencies within test cases.
• Ideas can be played with without actually implement production code.
• Need to adopt a different way of approaching problems.
• Easy to track when the application is deliverable.
• Errors/bugs can be picked up quicker.
• There is always something to test.
• Easy to recreate rare scenario’s/exception.
• Can take time to learn how to create useful tests.

With those points out of the way, I’ll explain my reasons for taking the testing route.

Though software development is maturing at a pace we still have an urgent need to track change & check our assumptions. The quicker we can check our assumptions are correct the quicker we’ll find bugs in our software, I’m the first to admit I get bored quick, when my attention wains so does the software I’m coding, whilst testing I find that I become more productive & my focus is more on problem at hand, rather than tracking errors or debugging as most of the errors are expected and handled quickly.

Below is a list of my loves & hates in regards to the testing process, like I mentioned earlier I don’t believe in perfect solutions so I think it prudent not to gloss over the fact that just because something is tested doesn’t mean it does what it is supposed to or that a project will end up better.

Loves

• Sped up my understanding of the problem at hand.
• Alerted me to error I’ve over seen.
• Easy to replicate hard to detect errors & exceptions.
• Allows me to test ideas before implementing them.
• Very handy when doing exploratory development (not sure how something works).

Hates

• Creating useful tests can take time to learn.
• Just because a test passes, doesn’t mean it is bug free.
• Is easy to create useless tests.
• If not careful planned, can find yourself testing more than creating production code.

From personal experience I found that once I grasped the concepts of testing (more so the TDD process, which I’ll go over in later blogs), the quicker I was able to implement functional code, whilst allowing me to test out assumptions I may have had. This along with a few other techniques (mainly design patterns & refactoring) has evolved my programming allowing me to create flexible, reusable code which I end up using in a range of projects.

Setting up PHPUnderControl on Centos5.1

So for the past few days I’ve been tinkering around with PHPUnderControl, for all kinds of reasons I decided to install it on centos5.1 & thought I’d docuement it for myself & others, I know I’ll definately be installing it again, some time soon.

Needed files
JDK
CruiseControl
PHPUnderControl
XDebug

Make sure that there are no JDK alternatives already install, remove if there is
chmod 755 &&
run ./jdk-6u10-linux-i586-rpm.bin

which will uncompress the RPM for your, now run the below to install it.
rpm -Uvh jdk-6u10-linux-i586.rpm

We’ll also need to modify php.ini so that the memory limit is increased the below should do.
memory_limit = 256M

Still having php.ini open, we will also need to modify the date.timezone property, as not doing so, will produce errors. Set it to your locale, the below works for me.
date.timezone = Europe/London

Once this is done, we will need to setup our JAVA_HOME environment, to initialise this at login, place in .bashrc as java is installed in /usr/bin as default.
export JAVA_HOME=/usr

we then unpack cruisecontrol & run in the background
unzip cruisecontrol-bin-2.8-dev.zip -d /opt &&
cd /opt &&
ln -s /opt/cruisecontrol-bin-2.4.1 /opt/cruisecontrol

We’ll need to configure ant so that it knows where to find our projects & setup build times etc, the below config file will do for now:
<?xml version="1.0"?>
<cruisecontrol>
<project name="PROJECTNAME" buildafterfailed="false">
<listeners>
<currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
</listeners>
<modificationset quietperiod="30">
<filesystem folder="projects/${project.name}"/>
</modificationset>
<schedule interval="300">
<ant anthome="apache-ant-1.7.0" buildfile="projects/${project.name}/build.xml"/>
</schedule>
<log dir="logs/${project.name}">
<merge dir="projects/${project.name}/build/logs/"/>
</log>
<publishers>
<currentbuildstatuspublisher file="logs/${project.name}/buildstatus.txt"/>
<artifactspublisher dir="projects/${project.name}/build/api" dest="logs/${project.name}" subdirectory="api"/>
<artifactspublisher dir="projects/${project.name}/build/coverage" dest="logs/${project.name}" subdirectory="coverage"/>
<execute command="phpuc graph logs/${project.name}"/>
<execute command="phpuc graph logs/${project.name} artifacts/${project.name}" />
</publishers>
</project>
</cruisecontrol>


We’ll have to retrieve the project from SVN manually, so do the follow:
cd /opt/cruisecontrol &&
mkdir -p projects/PROJECTNAME/build/logs &&
cd projects/PROJECTNAME &&
svn co pathto.svn.repo/svn source

We’ll need PEAR 1.6.1 to be able to install phpundercontrol, so we’ll do an update. which I found here
wget http://www.smudge-it.co.uk/pub/yum/centos/4/i386/php-pear-1.6.1-2.noarch.rpm
rpm -Uvh php-pear-1.6.1-2.noarch.rpm

we now need to install phpundercontrol
pear config-set preferred_state beta
pear channel-discover components.ez.no
pear channel-discover pear.phpunit.de
pear install --alldeps phpunit/phpUnderControl

The above command will install PHPUnderControl along with all its dependancies. All we need to do is the follow:
phpuc install /opt/cruisecontrol

Which will install PHPUnderControl into our CruiseControl file. Now we’ll start adding the bells & whistles.

So lets get PHPDocumentor to automatically build  or API docs for us.

<project name='PROJECTNAME' default='build' basedir='.'>
....
<target name="php-documentor">
<exec executable="phpdoc" dir="${basedir}/source" logerror="on">
<arg line="--title '${ant.project.name}' -ue on -t ${basedir}/build/api -d source -tb '/usr/share/pear/data/phpUnderControl/data/phpdoc' -o HTML:Phpuc:phpuc"/>
</exec>
</target>
.....
</project>

Now we want to create the documents directory & run ant (from within our project directory) to test that our docs get created.

mkdir -p build/api &&
../../apache-ant-1.7.0/bin/ant php-documentor

Once everything has run without a hitch, we’ll need XDebug so that PHPUnit will work under phpundercontrol (note that XDebug conflicts with ZendDebugger), type the command below if you don’t already have gcc & php-devel they’ll be needed to compile XDebug.
yum install php-devel gcc

Now we can compile XDebug.
pecl install xdebug

All we need now is to do add XDebug to PHP is
echo zend_extension=/usr/lib/php/modules/xdebug.so >>/etc/php.ini

Now lets get PHPUnit to do our testing for us, we’ll need to configure build.xml file for this.
<project name='PROJECTNAME' default='build' basedir='.'>
....
<target name="phpunit">
<exec executable="phpunit" dir="${basedir}/source" failonerror="on">
<arg line=" --log-xml ${basedir}/build/logs/phpunit.xml --log-pmd ${basedir}/build/logs/phpunit.pmd.xml  --log-metrics ${basedir}/build/logs/phpunit.metrics.xml --coverage-xml  ${basedir}/build/logs/phpunit.coverage.xml --coverage-html ${basedir}/build/coverage ProjectTests test/AllTests.php"/>
</exec>
</target>
.....
</project>

The above will setup PHPUnit to run our tests AllTests.php from the test directory and call our tests ProjectTests (all of which can be changed). AllTests.php contains a list of all the test suites within the project. The log related flags builds our logs, which PHPUnderControl will use for statistics, the coverage flags are used for code coverage & are useful to determine how much of the project has been tested.

We now create our logs directory & make sure our setup is correct.
mkdir -p build/logs &&
../../apache-ant-1.7.0/bin/ant phpunit

Again, once all is well, we’ll move on to the next step, CodeSniffer.
<project name='PROJECTNAME' default='build' basedir='.'>
....
<target name="php-codesniffer">
<exec executable="phpcs" dir="${basedir}/source" output="${basedir}/build/logs/checkstyle.xml" error="/tmp/checkstyle.error.log">
<arg line="--report=checkstyle --standard=PEAR source"/>
</exec>
</target>
.....
</project>

The above configuration will check the code against the PEAR (please refer to CodeSniffer for other standards),  Once we have saved the changes, we run ant again to make sure everything is working as intended.
../../apache-ant-1.7.0/bin/ant php-codesniffer

We have now setup PHPUnderControl & our build.xml file should look similar to below:
<?xml version="1.0" encoding="UTF-8"?>
<project name="PROJECTNAME" default="build" basedir=".">
<target name="build" depends="php-documentor,php-codesniffer,phpunit"/>
<target name="php-documentor">
<exec executable="phpdoc" dir="${basedir}/source" logerror="on">
<arg line="--title '${ant.project.name}' -ue on -t ${basedir}/build/api -d src -tb '/usr/share/pear/data/phpUnderControl/data/phpdoc' -o HTML:Phpuc:phpuc"/>
</exec>
</target>
<target name="php-codesniffer">
<exec executable="phpcs" dir="${basedir}/source" output="${basedir}/build/logs/checkstyle.xml" error="/tmp/checkstyle.error.log">
<arg line="--report=checkstyle --standard=Zend src"/>
</exec>
</target>
<target name="phpunit">
<exec executable="phpunit" dir="${basedir}/source" failonerror="on">
<arg line=" --log-xml ${basedir}/build/logs/phpunit.xml --log-pmd ${basedir}/build/logs/phpunit.pmd.xml  --log-metrics ${basedir}/build/logs/phpunit.metrics.xml --coverage-xml  ${basedir}/build/logs/phpunit.coverage.xml --coverage-html ${basedir}/build/coverage ProjectTest test/AllTests.php"/>
</exec>
</target>
</project>

As one last addage, we’ll setup cruisecontrol to send out emails on success, failure & always (not the spam potential here).

<email mailhost="smtp.emailserver.com"
username="username"
password="password"
returnaddress="y@me.com"
buildresultsurl="http://localhost:8080/buildresults/${project.name}"
skipusers="true"
spamwhilebroken="true">
<always address="always@me.com"/>
<failure address="fail@me.com"/>
<success address="success@me.com"/>
</email>

Note that the SMTP server, login credentials & the address attribute are always valid, otherwise there will be no easy way to determine the mails are getting sent out,

And the config.xml file should look simular to the below:
<?xml version="1.0"?>
<cruisecontrol>
<project name="PROJECTNAME" buildafterfailed="false">
<listeners>
<currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
</listeners>
<modificationset quietperiod="30">
<!-- touch any file in project to trigger a build -->
<filesystem folder="projects/${project.name}"/>
</modificationset>
<schedule interval="300">
<ant anthome="apache-ant-1.7.0" buildfile="projects/${project.name}/build.xml"/>
</schedule>
<log dir="logs/${project.name}">
<merge dir="projects/${project.name}/build/logs/"/>
</log>
<publishers>
<currentbuildstatuspublisher file="logs/${project.name}/buildstatus.txt"/>
<artifactspublisher dir="projects/${project.name}/build/api" dest="logs/${project.name}" subdirectory="api"/>
<artifactspublisher dir="projects/${project.name}/build/coverage" dest="logs/${project.name}" subdirectory="coverage"/>
<execute command="phpuc graph logs/${project.name}"/>
<execute command="phpuc graph logs/${project.name} artifacts/${project.name}" />
<email mailhost="smtp.emailserver.com"
username="username"
password="password"
returnaddress="y@me.com"
buildresultsurl="http://localhost:8080/buildresults/${project.name}"
skipusers="true"
spamwhilebroken="true">
<always address="always@me.com"/>
<failure address="fail@me.com"/>
<success address="success@me.com"/>
</email>
</publishers>
</project>
</cruisecontrol>

Finally we tie everything together by editing PHPUnderControls build.xml file
<target name="build" depends="phpunit,php-documentor,php-codesniffer" />

We now have PHPUnderControl working with our project, we can further customise CruiseControl so that it will run on bootup & listen on a specific host & port. To do this we need to run the following
./cruisecontrol -host testing.myhostname.com -webport 8000

Now if we want to automatically start CruiseControl we could place it in rc.local to do so we do the following:
vim /etc/rc.local
Appending the below command to the file.
/opt/cruisecontrol/cruisecontrol.sh -host testing.myhostname.com -webport 8000&

We should now have PHPUnderControl fully functional & running ready to retrieve our project from SVN and test all of its test cases periodically.

Getting ZendDebugger to work on Centos5.1

Well I had to setup a Zend Studio for Eclipse on Centos today, me thinking that it would be a blissful journey, abruptly followed by the realization that Centos wasn’t going to play nice. First thing after a clean install I ran yum update to make sure the system was up to date.

I tried to install ZendFramework 1.7, ZendStudio6.1 & PHPUnit3 and installed as I have before but came against a brick wall, PHPUnit would not install, due to PHP-XML being missing.
yum install php-xml
Which installs php-xml5.1.6, okay, I know I need 5.2 at least but for now I want to make sure PHPUnit will install.
pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit

Once PHPUnit was installed I ran the test cases, ZS complains with the follow:
Error launching 'AllTests.php
The session counld not be startd.
In order to generate debug inormation, please make sure that the debugger is properly configured as a php.ini directive

I run phpunit from shell to get:
Failed loading /usr/lib/php/modules/ZendDebugger.so: /usr/lib/php/modules/ZendDebugger.so: undefined symbol: zend_memory_usage


Okay so ZendDebugger isn’t working, php -v shows me:
PHP 5.1.6 (cli) (built: Sep 20 2007 10:16:10)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies

So I need a new version of PHP, PHP5.2.6 will do, yum update php tells me that 5.1.6 is that latest version, so I need to find a repository that has one.

cd /etc/yum.repos.d
wget http://dev.centos.org/centos/5/CentOS-Testing.repo
vim CentOS-Testing.repo

Edit the line below.
enabled=0

to
enabled=1


your file should now look like this:
[c5-testing]
name=CentOS-5 Testing
baseurl=http://dev.centos.org/centos/$releasever/testing/$basearch/
enabled=1
gpgcheck=1
gpgkey=http://dev.centos.org/centos/RPM-GPG-KEY-CentOS-testing
# CentOS-Testing:
# !!!! CAUTION !!!!
# This repository is a proving grounds for packages on their way to CentOSPlus and CentOS Extras.
# They may or may not replace core CentOS packages, and are not guaranteed to function properly.
# These packages build and install, but are waiting for feedback from testers as to
# functionality and stability. Packages in this repository will come and go during the
# development period, so it should not be left enabled or used on production systems without due
# consideration.

Now run the following

yum install php

You should get something simular to this….
=============================================================================
Package Arch Version Repository Size
=============================================================================
Updating:
php i386 5.2.6-2.el5s2 c5-testing 1.2 M
php-cli i386 5.2.6-2.el5s2 c5-testing 2.4 M
php-common i386 5.2.6-2.el5s2 c5-testing 230 k
Updating for dependencies:
php-ldap i386 5.2.6-2.el5s2 c5-testing 31 k
php-xml i386 5.2.6-2.el5s2 c5-testing 97 k
Transaction Summary


Now I ran the test and got the following error.
Failed loading /usr/lib/php/modules/ZendDebugger.so: /usr/lib/php/modules/ZendDebugger.so: cannot restore segment prot after reloc: Permission denied

Okay, that was totally unexpected, so after a little searching with my best friend (google) I found the following command
/usr/sbin/setenforce 0
Ran the tests again….
phpunit AllTests.php
PHPUnit 3.3.2 by Sebastian Bergmann.
.......

Time: 0 seconds

OK (7 tests, 0 assertions)

Perfect & running within ZS gives me the below results, job done.

One thing to make note of; when I restarted the box, SELinux had been reactivated so I disabled SELinux by doing the following (this is not ideal, i’ll have to look allowing doing this a better way).

vim  /etc/sysconfig/selinux
and changing
SELINUXTYPE=enforcing
to
SELINUXTYPE=disabled
Doing this disabled SELinux so that we I wouldn’t get the permissions error. Alternatively I could of placed the setenforce command in a startup script (bashrc or something simular).

Up and coming blogs

Well for a while I’ve been affected by the TDD bug, initially as I hate debugging, I also liked the notion of being able to catch errors before they arise.

I quickly noticed a lack of information on the practice within PHP (as that’s the main language I code with at work). So I decided a while ago to put my notes together, that soon turned into a couple of documents which will now be transformed into articles focused primarily on the subject.

Im far from an expert on anything but feel the need to share this information with all, as it matures. Hopefully after a few months there should be a nice little collection of articles concentrating on the subject.

I’m trying my best to as objective as possible but also understand that I’m a pragmatist some of the time and a passionate hypocrite the rest of the time, so some of what I write may be biases (one always knows best), I’ll rectify anything I later find to be false or am enlighten to a better way.

Installing ZendStudio 6.1

Okay, I’ve done this so many times & helped a few people do the same that I thought It would be a good idea to actually document the process. I’ll explain the basic steps to getting ZendStudio working with PHP5.2.6 & PHPUnit3, this setup will allow you to use the same library files as the actual system instead of the environment prepacked with ZS6.1 (I prefer this way as running tests in ZS6.1 should produce the same results when run via CLI.

Needed:
PHP5.2.6
ZF latest version
ZendStudio for Eclipse 6.1
PEAR

Make sure that php is atleast at version 5.2 (php -v will tell you the version), update if needed. Download Zend Framework and place in your php include path (typically /usr/share/php or /usr/share/pear).

If pear is not installed we’ll need install it using the following for Debian based systems (remember you’ll need to be root to install via apt or yum).
apt-get install php-pear
and
yum install php-pear
for RHEL based systems.

cd /opt &&
wget http://framework.zend.com/releases/ZendFramework-1.7.0PR/ZendFramework-1.7.0PR.zip &&
unzip ZendFramework-1.7.0PR.zip &&
cd ZendFramework-1.7.0PR/library/ &&
mv Zend INCLUDE_PATH

Now we have ZF within our include path we need to setup PHPUnit & ZendStudio.
pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit
We need to download ZendStudio from here & running the file
./ZendStudioForEclipse-6_1_0.bin Follow the instructions & wait for the installation to complete.

Now we’ll need to customise ZS so that it will work with using the systems PHP & PHPUnit instead of the ones packed with ZS.

We’ll need to setup ZendDebugger to work with our systems php.ini file, so we will need to copy ZendDebugger from ZS and place in with the other modules.
cp /PATH/TO/ZEND/plugins/org.zend.php.debug.debugger.linux.x86_5.2.14.v20080602/resources/php5/ZendDebugger.so /usr/lib/php5/

Here we need to add the below text into our php.ini file, ideally at the bottom, so it is easy to find.
[Zend]
zend_extension=/usr/lib/php5/ZendDebugger.so
zend_debugger.allow_hosts=127.0.0.1
zend_debugger.expose_remotely=always
zend_debugger.connector_port=10000
Once installed and started, select Window, Preferences & you’ll be given a window simular to below.

Click Add, which will bring up the dialog box below.

Enter the following, the path to the systems php executable & your php.ini file as entered in the above screenshot and click Finish.

The dialog box will close and leave you with Figure 1.1, select the newly added settings and click Default. We have now setup ZS to work with our systems PHP version instead of the one packed with ZS.

We now need to add a our PHP include path to ZS so that we have all our library files available for ZS.

Click New, which will display the below dialog box, this is where we need to set our current PHP path, the screenshot below displays an example.

Once the include path has been added we click OK and OK again, now when ever we create a new project, we will just remove the predefined paths and just include our one.

Well that’s it, now we should be able to run tests within ZendStudio & the shell, knowing that our results come from the same environment..