What Does This Say About Unit-Testing in PHP Land?

By | November 30, 2007

Without having done actual research, and depending on my personal experience alone, I would assert that in PHP userland …

  • There are maybe 20 or more general-purpose CMSes.
  • There are perhaps 50 or more public frameworks (of varying quality and approach).
  • There are easily 100 or more templating libraries (even if most of them can be grouped under 4 or 5 approaches).
  • There appear to be only 5 unit-test systems: PhpUnit, SimpleTest, Solar_Test, PHPT, and Lime.

Dispute the specific numbers all you like; they’re not the point. The point is that CMSes, frameworks, template systems, etc. seem to be a very common artifact among PHP developers, but writing (and then publishing) unit-test systems does not seem anywhere near as common an obsession.

What does this say, if anything, about the attention given to unit-testing among PHP developers?

Note that this is an *off-the-cuff opinion* and a *request for comment*, and nothing else. (Some folks get testy about testing; please remember to be nice while commenting. ;-)

Update: And there are only 3 or 4 documentation systems: PhpDocumentor, Doxygen, and Solar_Docs, along with the various combinations of DocBook tools that Zend, PEAR, and PHP are using. I wonder how that enters into it, if at all.

29 thoughts on “What Does This Say About Unit-Testing in PHP Land?

  1. Jan Schneider

    “What does this say, if anything, about the attention given to unit-testing among PHP developers?”

    It doesn’t say anything. Test systems are for low level php developers, i.e. people developing the 20 CMSs, 50 frameworks and 100 templating systems, rather not for the developers using those systems. And any sane php developer should use a unit tested framework/template/cms instead of creating yet another one. And it doesn’t make much sense, nor is it necessary to have more than 5 test systems for those.

    Reply
  2. pmjones Post author

    @Jan:

    To use your words in rebuttal …

    It doesn’t make much sense, nor is it necessary, to have more than 5 frameworks in PHP land … and yet we have *so many* frameworks and so few unit-testing systems.

    The disparity occurred to me late last night, because it seems that PHP developers love to write their own versions of relatively complex things — but not unit-test systems. I don’t know why this is, or what meaning it may have (if any).

    Reply
  3. Ivo Jansch

    I think there’s a relation between complexity/functionality and the amount of systems. There are so many frameworks out there because there are so many things that you can do differently in each of them. In Unit Testing, there are only a few approaches so any extra project is likely to be so very similar to the existing ones that it makes not much sense to create your own.

    Reply
  4. pmjones Post author

    @Ivo:

    I wondered about complexity/functionality too. The response I have for that is, there are only a few approaches to doing templating, and yet we have an overabundance of template systems — that’s one of the things PHP developers *really* love to build for themselves. And yet they don’t build unit-test systems for themselves. I don’t know why this is.

    Reply
  5. Sebastian Bergmann

    It does not say much. Why should there be a similar amount of unit testing frameworks for PHP as there are CMS systems written in PHP?

    Take a look at Java for which there are two predominant unit testing systems: JUnit and TestNG. Does this mean that unit testing is not a topic in the Java world? Not at all. Quite the opposite, actually.

    Reply
  6. Andrew Shell

    I think the numbers make sense if you look at the ideas behind each from a personal and business context.

    Templating systems are fundamentally simple, depending on the features. It could be as simple as a function wrapping a few lines of output buffers and an include, to something as complex as smarty. So if you aren’t happy with an existing templating system you just whip one up. That leads to a lot of templating systems. Frameworks are more complex but very personal. Everyone has their own idea how the apps should be architected and that is reflected in the number of frameworks.

    From the business side templating systems and frameworks are pretty devoid of propriety business logic so they can see throwing it out there for other people to use.

    Content management systems can take a much longer time to design and develop and there are a lot of businesses that sell them. So even though you say there are 20 content management systems there are really a whole lot more because I bet that there are a lot of companies that have proprietary product that they sell to their clients. My last two employers had them and you aren’t going to find them on a website. They were usually customized for each client.

    When it comes to unit tests, I suspect a great majority of PHP developers don’t use unit testing, so they don’t feel the need to write their own. No company that I’ve ever worked for did unit testing. Only one (my current employer) has even had version control. When I pushed for SVN at my last employer they pushed back and said that it was too complicated. They opted for a shared network folder. Needless to say I didn’t stick around too long. Last I heard the project is now three months late.

    Reply
  7. Regin

    If you look at all the crappy PHP source code out there – it should not come as a surprise that there the relations between the numbers of Frameworks, CMSes and Unit Testing systems are like this.

    Anyone remembering PHPNuke????

    Reply
  8. Dan Scott

    Much more interesting would be to find out how many of those frameworks, templating libraries, etc actually use unit tests (and to what degree of coverage). But that would take more work to come up with some valid results.

    On documentation frameworks – ReST is another one.

    Reply
  9. Pádraic Brady

    I think it does say something. To clear up one thing – I’m writing PHPSpec, PHPMock and PHPMutagen. I’ve been asked countless time by now why I’m writing a Beahviour-Driven Development framework, one for mocks, and one for Mutation Testing, and the answer is pretty simple. I use other languages enough that when something adopts a new testing strategy, or methodology or way of doing things I’m right there checking it out. Eventually it reached a critical mass – and I really needed to get some of them into PHP so I could use them. Nothing like a little frustration and need to get you to actually take action!

    If you counted up all the testing/designing related frameworks, PHP is far behind the curve. By a long long way. Adoption of “testing” is itself far behind other languages too – so there’s little doubt they’re related. It’s buried a lot since more prominent developers at least test (if not TDD/BDD). Since I started PHPSpec, I can probably count the number of people writing such systems in terms of fingers ;). Sebastian, Marcus, Travis, Perrick, Me, maybe a few others I’m less familiar with or are less visible.

    Should there be more testing/BDD/TDD systems? There should be – I keep wondering where TestNG for PHP is? Where’s Ruby Autotest? Where’s muJava or Jester? Where’s the folk picking and pointing fingers over each systems flaws and putting together some smaller efforts? Granted that’s my motivation for jumping in – but why isn’t it for far more people? I’m suspicious something so central to the development process lacks so much attention…

    Reply
  10. Travis Swicegood

    I didn’t even realize there were a full 5 testing frameworks… ;-)

    For what it’s worth, I wouldn’t consider Solar_Test a framework as it’s not independent of Solar. That falls into the class of internal testing. That being the case, every package that has “assert(is_string($string)’)” at the top of a method/function call could be qualified as some sort of a testing framework. I had never heard of Lime, but from what Google has told me about it, I wouldn’t classify it as an independent testing tool either. That means there’s four of us – SimpleTest, PHPT, PHPUnit, and PHPSpec, and three of the projects have me as a contributing developer… So what does that say about me? ;-)

    Reply
  11. Dougal

    It does to show. the majority of PHP coders are NOT software engineers. They are code hackers.

    Reply
  12. Luke Welling

    Here are a few possible theories:

    A: There are very few approaches to testing, so nobody is driven to create another system because one of the existing ones works just the way they want it to.

    B: Fewer people come to testing with rigid preconceived ideas about exactly how the system should work, so rather than rolling their own system to do it exactly how they want it to, they are happy to adjust their thinking and workflow to follow an existing system.

    C: Testing is boring (but important) so only crazy masochists want to spend their time writing an open source testing tool.

    D: Only the smartest 10% of programmers really want to systematically test their code. These same people are smart enough to check what is out there before wasting time reinventing the wheel.

    E: Testing is concentrated late in a project’s development when deadlines look more threatening. Downloading existing tools seems like a good idea when you are in that mindset. Writing yet another framework/CMS/template system might seem like a good idea early in the project life when you are spending time thinking about architecture and deadlines seem less threatening.

    Reply
  13. David Coallier

    It doesn’t say much I believe.

    One of the reasons why there are so many CMSes is simply because many many companies find it much easier to do their own CMS than using an open source one and (maybe) eventually having liability problems. Also, some of those CMS’s don’t have very good documentation so for a company to start searching for docu, etc etc, it’s wasted time. They figure, hey, we might as well do our own and later on we can sell it :D Until they realize they couldn’t sell it for crap and then they release it as open source ;-)

    Also, writings unit tests is something (to my great disagreement) that is costing and that you don’t see results right away. I agree that on the short term you will not see as much results, however, if a company wants to do serious business and have a great added value to their product, they HAVE to unit test their code.

    Douglas, to reply to your comment I have to say one thing, I have been consulting in MANY companies and the only reason why PHP “coders” are not engineers is not their fault but the fault of the people above. THEY are the one who think PHP is simple, quick and stupid and that they’ll make millions fast. As soon as you introduce the idea of investing money in the code by developing unit tests so that you know that later on, you will not be screwed when the junior developer changes a feature, they freak out. Why ? I have no clue? I guess I should read those IT magazines but that’s not my thing.. perhaps something like those very important business people read… business 2.0 ? :)

    And Travis what do you want us to say? Congrats and thanks ? Many of us to a LOT of things and we don’t get thanked ;-) You don’t have to ask for your thanks, it’ll come by itself.

    Than, my final answer to the original question is: Did people understand that it’s better to join efforts to make a good robust product? Or are they just too lazy to do their own unit testing frameworks ? Personally, I believe what’s out there is great and no-one should rewrite them just for the sake of it…

    Reply
  14. Elizabeth M Smith

    David, down boy, I think you missed the tag on Travis’s post. I think he was commenting on his masochistic tendencies, not doing a “look at me”

    I think you need to take a look at the number of PHP5 only (not that work in PHP5, that were written or rewritten FOR it) CMS’s, Frameworks et al compared to the number of testing systems written (or rewritten) for PHP5 – might have something to do with the correlation there. The simple fact is that people viewed PHP (and treated PHP) as a quick and dirty tool instead of a full fledged language until PHP5 came around (and it took until about 5.1 to even get passed beta quality). And the mindset is changing very slowly, so it’s taking longer for more advanced paradigms (such as $flavor_of_month testing, design patterns, MVC) to catch on with the masses.

    Reply
  15. Gergely Hodicska

    Dear Paul,

    May I ask you why did you decide to write an own unit testing/documentation framework? Did you find any limitation in the existing ones, or your concept was not using thirdparty tools?

    Best Regards,
    Felhő

    Reply
  16. Jacob Santos

    It means developing CMSes are sexy and developing Unit Test libraries are not sexy (unsexy). You can pretty much group everything into sexy and non-sexy groups.

    Writing Documentation: Not Sexy (and quite torture)

    Writing Unit Tests, Functional Tests, etc: Not Sexy (but awesomely fun, if you ask me, but still not sexy)

    Writing CMS: Sexy

    Writing Frameworks: Sexy

    I would wager a guess and say that businesses don’t hire you based upon how well you write documentation and unit tests, they want grease monkeys that fly in, code like a mad man, and leave to the next venture.

    Ah, the place I work for is one prime example:

    “I would like to write unit tests and use version control.” I say.

    “Why, what we have works fine.” says boss.

    “Look, we can either spend the extra five hours now and write tests that ensure that whatever change I make in the future doesn’t force me to spend an additional 2 to 3 hours fixing it and manually testing that the fix I implemented didn’t break anything else and actually fixed the problem (for every bug, not just for the first time).

    Not that I don’t like wasting time on mundane testing tasks that take hours on end, I do not true-ly. However, I believe firmly that the next programmer that comes in here, is going to scream and curse MY NAME and not yours.”

    I’m ashamed of the code I’ve written at work. I’ll admit that I don’t write unit tests, but lets just say, that when I go back to my code a few months later, I have no idea what I was writing (hint). I’m so going to programming hell.

    Reply
  17. till

    To follow up on Dan Scott – it would be great to get numbers on how many CMS, forums, whatever use tests.

    Reply
  18. Travis Swicegood

    @David – I think Elizabeth covered it for me. My question wasn’t a “look at me” comment; more of a “what crazy path to non-relevance have I chosen?” :-)

    Reply
  19. Pádraic Brady

    @Sebastian: It’s getting very close actually :)

    Another interesting metric – how many frameworks provide a test-execution API to simplify unit tests? In talking with a few people it seems the rule more than the exception not to apply unit testing/TDD to writing MVC controllers and models.

    Reply
  20. Pingback: PHPDeveloper.org

  21. Svetoslav

    –quote from Jacob Santo —
    I’m ashamed of the code I’ve written at work. I’ll admit that I don’t write unit tests, but lets just say, that when I go back to my code a few months later, I have no idea what I was writing (hint). I’m so going to programming hell.
    ——

    Do you write comments or inline phpdocumentation ?

    I’ve read somewhere that is better to write the documentation BEFORE the code, so your brain will think about it, and, it will be easier to program later.

    Svetoslav

    Reply
  22. Pingback: developercast.com » Paul Jones’ Blog: What Does This Say About Unit-Testing in PHP Land?

  23. Richard Orelup

    I think another issue is that it’s hard to see what adding this tests will actually do to help me notice an issue or troubleshoot my code.

    I sat through 2 PHP Unit talks at PHP|Tek last may and they did nothing to convince me of what this would do for me. I think mainly it was because the examples were so simple because they need to in that kind of introductory environment that it didn’t really show me what I was gaining with it.

    Anyone have any good resources that talk about this with better examples to show how this would actually help my applications?

    Reply
  24. Rich Vázquez

    I’m just moving to PHP full time and see the very few testing options, but haven’t been able to find a matrix comparing features and product maturity of the test suites..

    Reply
  25. Matt

    I think the main issue lies with the PHP developers. As one commenter mentioned, and I would guess, that the majority of PHP ‘developers’ are code hackers, and not software engineers. I would hazard a guess that a lot of PHP developers are uncertain of what unit testing is.

    I would love to see some figures on the number of PHP applications out there,
    and the percentage of which that are unit tested.

    Another topic along similar lines, is coding standards. I have seen some pretty ugly PHP code, including an earlier release of vBulletin. PHP is definitely behind other languages when it comes to best coding practices, no doubt about it. But I also put this down to how easy it is to adopt, and publish projects, which in itself is an advantage.

    As anybody ever been taught PHP in an academic environment or is it considered a hackers language?

    Reply
  26. PHP Encoder

    This situation exists because PHP developers abuse PHP to earn money ASAP, the code should just work, and only after when server hits million requests server admin wonders – what the hell is going on.

    Reply
  27. Pingback: ??????? » [Web] ????

  28. Steve Fenton

    There are 11 unit testing frameworks for PHP listed on Wikipedia:

    http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#PHP

    I wrote one of these and my motivation was that I came to PHP from languages where Unit Testing is more mature. In Java, C# or Python, you cannot be a professional developer if you don’t write unit tests – the whole community has rallied behind the importance of unit testing and test-driven development is commonplace.

    In PHP, unit testing is less common. Sebastian Bergmann has led the charge on this one and without his contribution to unit testing, PHP would be lacking something very fundamental.

    I have taken a good look at most of the common PHP unit testing frameworks (I would be an idiot to write one before doing this) – PHPUnit was definitely the best of the bunch. If you want to write unit tests in PHP, PHPUnit has to be on your list of candidates. I only created another unit testing framework (EnhancePHP) because I wanted less friction between my testing in other languages and my testing in PHP – so I have created a framework that is more fluid for my personal experience.

    This is where I attempt to answer the original question… does the number of unit testing frameworks define how much a community does unit tests? I don’t think there is a relationship between these numbers.

    I do believe that PHP across the board is less into unit testing, but this is because there are currently two classes of PHP developers. There are professional PHP developers, who make a living from writing PHP and there are hobby-developers who come to PHP because it is such a good language to get started with. This is one of the key strengths of PHP – it is really easy to write your first dynamic page in PHP, but it also has namespaces, classes, interfaces and all the other tools you need to write clean code.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *