You said: “What Paul neglects to mention is that the output of a .phpt with the “no output unless there is an error” should in fact be a dump of expected and actual output, just as PHPUnit and company do in the test reports.”

Ah yes, I was not explicit enough about this. The Solar_Test_Assert methods use the expected value and the actual value as the fail() exception message (courtesy of var_export() to make them printable). So the unexpected output that causes the .phpt to fail does contain both values for logging purposes. Thanks for pointing it out.

“Tomas V.V. Cox did the initial port of php-src/run-tests.php into PEAR a long time ago, I have simply tweaked it (and only in minor ways) since its introduction. Credits where credits are due.”

Absolutely I want to give credit where it’s due; I’ve updated the first paragraph to reflect Tomas’ authorship. Thanks for the correction.

Also notice I didn’t mention test-more when it came to libraries ‘without’ extreme simplicity ;). Then again I guess I didn’t mention it at all…. :(

http://shiflett.org/archive/187

This gives you TAP plus extreme simplicity (even simpler than phpt). I’ve used all four major testing libraries for PHP pretty extensively, and although I like a lot about phpt, it’s a very difficult approach to scale. In fact, using a separate file for each test is about the only way to provide precise feedback (which test failed?), and having a test suite that consists of several hundred tests is difficult to maintain across files like that.

TAP is actually pretty descriptive by itself, so even without a testing framework like Apache-Test, you can still just test from the command line and get a decent report.

Travis you can use variable output. Check out http://qa.php.net/write-test.php for a good description of phpt.

The point of “no output unless there is an error” means that you can do things such as compare error codes by the PHP version being run (for instance required for any XML saxon parsing error, as they are different in 4.x, 5.0.x, and 5.1.x). In addition, instead of putting absolutely everything into a single file (i.e. using methods inside a PHPUnit/SImpleTest), you can put each test into its own file, and use good old:

require_once dirname(__FILE__) . ‘/setup.php.inc’;

to implement our familiar setup()

and a –CLEAN– section with

require_once dirname(__FILE__) . ‘/teardown.php.inc’;

if necessary (–CLEAN– will be implemented in PEAR 1.4.7, btw).

in terms of randomized testing, if you can choose random files (and you can, although it would have to be external to the “pear” command), you can do random testing.

I’ve found the separation of each test condition into its own .phpt to be invaluable for another reason: a fatal error only kills 1 test rather than the whole suite… :)

What Paul neglects to mention is that the output of a .phpt with the “no output unless there is an error” should in fact be a dump of expected and actual output, just as PHPUnit and company do in the test reports. This is all readily available in the .out file (testname.out if you have testname.phpt) after running the test. In addition, these files can be bundled together and posted online or mailed to the developer, simplifying remote testing by non-techie users.

The only thing lacking is pretty-printed reports, so if you find that kind of thing essential, this isn’t for you. It does, however, spit out a log file containing a count of all tests run, and the paths to failed tests called “run-tests.log”.

Oh, and one last VERY important thing. Tomas V.V. Cox did the initial port of php-src/run-tests.php into PEAR a long time ago, I have simply tweaked it (and only in minor ways) since its introduction. Credits where credits are due.

Actually, in some cases introducing a more robust testing framework can be a hindrence as you have to be familiar with at least the xUnit methodology in order to understand what’s happening. The one limitation that I see (and I might be wrong here) with the current testing is the lack of an ability to do any randomized testing if you’re checking output. Can you expect variable output from a phpt file?

That is definitely an interesting idea, though.

/**
* modify php include path to include parent directory
*
* this is required becuase the tests are run from the test subdirectory
* and the application is run from (and coded for) the parrent directory
* @ignore
*/
if (!defined(‘TEST_PATH_MODIFIED’)) {
ini_set(‘include_path’, ‘..:’.ini_get(‘include_path’));
define(‘TEST_PATH_MODIFIED’, true);
}