Solar: Modified Testing Strategy

By | January 31, 2006

In the current release of Solar (version 0.10.0) I use the .phpt test format, which is supported very well by Greg BeaverTomas V.V. Cox‘s “pear run-tests” code. The .phpt testing strategy compares the output of a test script to an expected output string, and passes if the outputs match.

I noticed some time ago that Greg has adopted a testing strategy that combines .phpt with techniques found in PHPUnit and SimpleTest. These testing packages use assertions to compare method returns, rather than comparing output at the end of a test script. Greg’s strategy uses assertions within .phpt tests: if there is no output, the test succeeds, but if an assertion fails, it generates unexpected output, which causes the .phpt test to fail.

(Aside: While true testing gurus may not think of this as a robust solution [hi Jason! hi Marcus! hi Sebastian!], I have thus far found it to be a “good-enough” approach that combines simplicity with … well, more simplicity. :-)

Inspired by Greg’s PEAR_PHPTest class, I’ve developed a bare-bones Solar_Test_Assert class that tests assertions and throws PHP5 exceptions on assertion failures. Add to that a quick hack to run all tests at once recursively through subdirectories (courtesy of the “pear run-tests -r” switch), and you get a convenient all-at-once testing tool for Solar. Thanks, Greg, for producing such useful tools.

I spent one day over the weekend rewriting all the existing Solar tests to use this new strategy, and met with great success. I did run into one problem, though. The class-method tests in the subdirectories use a common prepend file. Normally, you would think that “include ‘../prepend.php’” would work, but because the tests are running at the command line, PHP doesn’t include “..” with respect to the subdirectory; instead, it tries to include “..” with respect to the current directory. This is the proper behavior, but it was causing every test to fail.

Mike Naberezny pointed out a cool trick to work around that behavior; instead of using “..”, you can use “dirname(dirname(__FILE__))” to get to the next-higher directory. After applying this fix, all my tests ran from the main directory using “pear run-tests -r”. Thanks, Mike.

Update: (2006-02-01) Greg pointed out that the “pear run-tests” is from Tomas V.V. Cox, so I update the above link. Thanks, Tomas, for this great testing tool.

9 thoughts on “Solar: Modified Testing Strategy

  1. Jason Sweat

    Something I have used in the past is:

    /**
    * modify php include path to include parent directory
    *
    * this is required becuase the tests are run from the test subdirectory
    * and the application is run from (and coded for) the parrent directory
    * @ignore
    */
    if (!defined(‘TEST_PATH_MODIFIED’)) {
    ini_set(‘include_path’, ‘..:’.ini_get(‘include_path’));
    define(‘TEST_PATH_MODIFIED’, true);
    }

    Reply
  2. Travis Swicegood

    Some testing is better than no testing… :-) And to top it off, you’re following the KISS principle which should be a foundation of any good testing strategy.

    Actually, in some cases introducing a more robust testing framework can be a hindrence as you have to be familiar with at least the xUnit methodology in order to understand what’s happening. The one limitation that I see (and I might be wrong here) with the current testing is the lack of an ability to do any randomized testing if you’re checking output. Can you expect variable output from a phpt file?

    That is definitely an interesting idea, though.

    Reply
  3. Greg Beaver

    Travis:

    The point of “no output unless there is an error” means that you can do things such as compare error codes by the PHP version being run (for instance required for any XML saxon parsing error, as they are different in 4.x, 5.0.x, and 5.1.x). In addition, instead of putting absolutely everything into a single file (i.e. using methods inside a PHPUnit/SImpleTest), you can put each test into its own file, and use good old:

    require_once dirname(__FILE__) . ‘/setup.php.inc’;

    to implement our familiar setup()

    and a –CLEAN– section with

    require_once dirname(__FILE__) . ‘/teardown.php.inc’;

    if necessary (–CLEAN– will be implemented in PEAR 1.4.7, btw).

    in terms of randomized testing, if you can choose random files (and you can, although it would have to be external to the “pear” command), you can do random testing.

    I’ve found the separation of each test condition into its own .phpt to be invaluable for another reason: a fatal error only kills 1 test rather than the whole suite… :)

    What Paul neglects to mention is that the output of a .phpt with the “no output unless there is an error” should in fact be a dump of expected and actual output, just as PHPUnit and company do in the test reports. This is all readily available in the .out file (testname.out if you have testname.phpt) after running the test. In addition, these files can be bundled together and posted online or mailed to the developer, simplifying remote testing by non-techie users.

    The only thing lacking is pretty-printed reports, so if you find that kind of thing essential, this isn’t for you. It does, however, spit out a log file containing a count of all tests run, and the paths to failed tests called “run-tests.log”.

    Oh, and one last VERY important thing. Tomas V.V. Cox did the initial port of php-src/run-tests.php into PEAR a long time ago, I have simply tweaked it (and only in minor ways) since its introduction. Credits where credits are due.

    Reply
  4. Mike Lively

    All the work trying to get TAP into general acceptance and you go and start using phpt :D. You are right though, one advantage it does have over PHPUnit and SimpleTest is it’s extreme simplicity. Then again one disadvantage it has is it’s extreme simplicity. I actually just recently used it myself to test a patch I am working on for PHP and you are right the innate run-tests support is pretty handy to have and is more simplistic and probably quicker to set up than the test harnesses for SimpleTest and PHPUnit.

    Travis you can use variable output. Check out http://qa.php.net/write-test.php for a good description of phpt.

    Reply
  5. Chris Shiflett

    Mike, you could always use the test-more.php library by itself:

    http://shiflett.org/archive/187

    This gives you TAP plus extreme simplicity (even simpler than phpt). I’ve used all four major testing libraries for PHP pretty extensively, and although I like a lot about phpt, it’s a very difficult approach to scale. In fact, using a separate file for each test is about the only way to provide precise feedback (which test failed?), and having a test suite that consists of several hundred tests is difficult to maintain across files like that.

    TAP is actually pretty descriptive by itself, so even without a testing framework like Apache-Test, you can still just test from the command line and get a decent report.

    Reply
  6. Mike Lively

    heh, you don’t have to tell me twice about the enormous number of test files you end up writing in phpt. The actual php test suite is up to 2000+ I believe :D. Running those tests with memleak checking takes me roughly an hour and a half.

    Also notice I didn’t mention test-more when it came to libraries ‘without’ extreme simplicity ;). Then again I guess I didn’t mention it at all…. :(

    Reply
  7. pmjones Post author

    Greg said:

    “Tomas V.V. Cox did the initial port of php-src/run-tests.php into PEAR a long time ago, I have simply tweaked it (and only in minor ways) since its introduction. Credits where credits are due.”

    Absolutely I want to give credit where it’s due; I’ve updated the first paragraph to reflect Tomas’ authorship. Thanks for the correction.

    Reply
  8. pmjones Post author

    Hi again Greg –

    You said: “What Paul neglects to mention is that the output of a .phpt with the “no output unless there is an error” should in fact be a dump of expected and actual output, just as PHPUnit and company do in the test reports.”

    Ah yes, I was not explicit enough about this. The Solar_Test_Assert methods use the expected value and the actual value as the fail() exception message (courtesy of var_export() to make them printable). So the unexpected output that causes the .phpt to fail does contain both values for logging purposes. Thanks for pointing it out.

    Reply
  9. Pingback: Paul M. Jones » Blog Archive » Solar 0.18.0 released, and more about testing

Leave a Reply

Your email address will not be published. Required fields are marked *