h1

SimpleTest Vs PHPUnit

January 28, 2009

Ideology

We want to be creating tests for every piece of functionality being developed. This will help us to keep our project scalable as well as alerting us to any state or behavioural errors/smells that may arise over the projects life time.

Tests are typically used as a way of making a project as stable as possible with the view of spending as little time as possible on debugging and error finding.

As you may know from personal experience, it can be more than a pain to figure out where a particular error is coming from, not to mention, what fired it off in the first place. Test cases are there to relieve this, firstly by way of testing each expected and unexpected response to the test case. The other way is allowing us to create test situations that would rarely take place in untested situations, allowing us to deal with those issues before they come up as opposed to waiting for them to appear later on in development or worst yet, when in production after a compromise.

Test suites are not just used as a safety net, they can be used to give the developer a better understanding of the implementation of the system, as well as documentation for other developers, describing what the developer does and doesn’t expect from the case.

The idea is to incrementally create test cases & then accompany them with the actual implementation of the functionality in questions. The PHPUnit tutorials explain this procedure pretty well so I will not reiterate (see http://www.phpunit.de/pocket_guide/3.2/en/test-first-programming.html for more info). Developing this way not only helps find bugs as soon as they appear, it also helps to find them later down the line it. TDD also helps to realise over sights in the design and implementation of the system allowing us to deal with them as soon as they appear.

Ideals

  • Test incrementally, creating test first, then the implementation.
  • Have a testing suite that allows us to run test via a web browser and command line (needs to be possible with no change to code).
  • Tests are integrated into phing, so tests are run before system is deployed or updated.
  • Tests are able to run separately, a group & as a whole.
  • Able to customise results front end so we can view pass & fail results ( useful to ascertain that we actually have the data we expect.
  • Use Reflection API to test a classes structure (properties, access type, etc.).
  • Test for the unexpected as well as expected results & errors.
  • Test of exceptions & exception handling.

Findings

I’ve been looking into both PHPUnit3 & SimpleTest to determine the best test suite for us to use. Both are pretty good suites at a glance but there a few fundamental differences to be noted.

PHPUnit3

Iis the most widely used and the most popular to date though it does present a few problems. Since version 3 mock objects have been introduced but still lacks the power that SimpleTest possesses. It can also only be run via a command prompt so view-ability can be an issue, especially when the suite grows. This can be eleviated with the use of reports which can be generated once a test is run, allowing for testers to view the results without needing to know the actual command to run the suite by itself. As of ZFW 1.6 Zend_Test_PHPUnit is now integrated allowing us to test our zend application explicitly with PHPUnit. This is an obvious attraction as Zend_Test_PHPUnit will have functionality specific to the framework, allowing us to spend time on the actual tests and not creating the functionality for them.

pros

  • Widely used, part of ZFW.
  • Loads of example on-line.
  • Extended by Zend_Test_PHPUnit as of 1.6RC1
  • Able to test controllers with no further extending.
  • Can create various type of reports.
  • Customisable tests results.

cons

  • Mock objects not as fluent as SimpleTest
  • Can not run directly via a web browser.
  • Less functional than SimpleTest.

SimpleTest is the not as widely used as the above but has some fundamental differences. It allows us to not just test an objects validity but also test our application in varying ways (check its state, behaviour). With SimpleTest we are able to not just test the back end integrity but we can also test that the front end also deals with situations as we expect it to.

pros

  • Can be used with PHPUnit.
  • Can custom output.
  • Can be run via both command line and browser.
  • Can test both states & behaviour.
  • Customisable tests results.
  • Can test both state, behaviour & front end functionality.

cons

  • Not as well documented as PHPUnit
  • Will need to extend to use with ZFW
  • Not naturally a part of ZFW.

Over the past few years I’ve used both suites quite extensively and found that SimpleTest is by far the most flexible. First off we’ll be able to customise the display of our results so we can properly determine whether a test is correctly passed or not, I’ve found that sometimes, though a test passes, it can sometimes be a false positive. SimpleTest allows us to not just display the test result, but also display the actual result data. Mock objects are also exceptionally powerful in SimpleTest, as mentioned before mock objects allow us to create instances of an object and set its return values. Once this is done, we can then test to make sure a method is only run ‘x’ amount of times, as well as being able to test for results, behaviour & states as well as property types. On top of all that it lessen the dependency issues that can arise from having to use real objects to test other objects (see http://simpletest.sourceforge.net/en/mock_objects_documentation.html for more info).

Conclusion

Both suites can be used with Zend framework (SimpleTest needing some extending), as well both having an Eclipse plugin (PHPUnit with ZFE out of the box) which has a feature allowing developers to run unit tests within the IDE. Both need to be downloaded and placed somewhere PHP can see it (include_path/webroot). As well as both frameworks will allow us to test a systems state plus its behaviours.

After initially going for SimpleTest, ZFW released 1.6RC1 (19/07/08), which now includes testing framework that allows us to test our controllers easier. This is a large factor in the decision making as it now means that by using SimpleTest, we will have to create a simular wrapper to which is already implemented within ZFW already using PHPUnit3. For this reason I prefer to work with PHPUnit, along with ZFW 1.7 giving me functionality needed to test ZF based applications.

I’ve added a couple of links to better explain the concept of stub and mock objects.

Resource Links

http://martinfowler.com/articles/mocksArentStubs.html – An excellent article explaining the difference between stubs & mocks

http://simpletest.org/api/- SimpleTest

http://phpunit.de/pocket_guide/3.3/en/ - PHPUnit3

About these ads

8 comments

  1. [...] http://baphled.wordpress.com/2009/01/28/simpletest-vs-phpunit/ Posted in Uncategorized. Tags: LAMP, php, PHPUnit, SimpleTest, test tools. [...]


  2. [...] SimpleTest versuss phpunit 2. [...]


  3. Thanks for this article.

    Could you please explain your sentence, when you speak about SimpleTest:
    “Can be used with PHPUnit”?

    Does it mean you can integrate existing PHPUnit tests with SimpleTest? If so, how could we handle that?


    • Apologies for the delay in the response.

      It’s been a while since I have worked with both tools but if I remember correctly it is possible to run both.

      Having said that, its been a while since I’ve had to do something like that so, things may of changed since.

      When I get some spare time I’ll test this out and update this post.


  4. Simple Test does not support database testing.


    • It’s been a while since I’ve worked with SimpleTest but I’ll do some digging when I get a chance and update this article accordingly.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: