Why I Prefer Test-Later

I remain unconvinced of the benefits of test-first and test-driven development (TDD) because I think the underlying principles of TDD are lacking, not because of the way TDD adherents talk about those principles. I believe I understand the test-first adherents very well, and I disagree with them.

I agree with Travis that many TDD and test-first devotees suffer from what Travis calls “expertise syndrome”. I think almost anyone who has a great deal of sophisticated and nuanced knowledge on a particular topic must actively look out for and modify the kind of behavior Travis describes.

But neither the “expertise syndrome” nor the sometimes religious zeal of many TDDers are the cause of my ambivalent feelings about test-first. Instead, I simply fail to see that TDD is really so useful as its adherents anecdotally proclaim. This is not because their communication skills are lacking in some way, but because I find the idea of TDD itself to be lacking.

I have various reasons for thinking test-first is not the fantastic solution its devotees claim. Chief among my reasons is this: a programmer not already familiar with the domain of a non-trivial problem generally doesn’t know enough to write meaningful tests in advance. The act of writing the code as a solution will train the programmer in the domain. He can write more useful tests afterwards to make sure future changes to the code do not break existing behaviors.

The Map Is Not The Territory

I think test-first is not likely to help most programmers understand the problem any better — test-first is only going to tell the programmer if his code works the way he thinks it should, not if it solves the problem he is addressing.

This is because the the map is not the territory. That is, a “paper abstract” map of the problem is not the same as the “physical concrete” territory of the problem. The map may provide useful hints, but until you have explored the territory, your understanding of reality is severely handicapped regardless of how smart, intelligent, or clever you think you are.

Test-first assumes the map of the problem is sufficient. In reality, unexplored assumptions and unexpected interactions abound. Test-later depends on you having explored the territory of the problem, becoming familiar with the reality of the situation by working through it yourself and gaining personal experience with it. Your solutions will be all the better for it, and your tests will then reflect the real territory, not the abstract map.

Thus, I believe test-later is more accurate and more useful, as well as a more effective use of your limited time when checking solutions in code, because it is based on practical experiment, not hypotheticals.

Testing Is Useful

Testing is useful and necessary. Unit, system, and integration testing are a great tool to help find, remove, and prevent new errors in working code. I do not think that test-first is any better than test-later, especially for programmers who have good self-management and self-discipline.

David Sklar has said in regards to testing that “tools are secondary, discipline is primary” (see slide 40). I heartily agree. In this sense, I think for programmers who have good discipline, test-first might well be a good tool. But test-later is just as good a tool, and perhaps a better one.

Similarly, with programmers who have mediocre or poor self-discipline, test-first and TDD may help them produce better code … but the quality of that code will still be lower than that generated by good programmers who take the time to explore the problem domain.

It appears that at least one bit of research supports my otherwise unproven assertions in this regard. Or, rather, the debunking of one bit of research.

  • The control group (non-TDD or “Test Last”) had higher quality in every dimension—they had higher floor, ceiling, mean, and median quality.

  • The control group produced higher quality with consistently fewer tests.

  • Quality was better correlated to number of tests for the TDD group (an interesting point of differentiation that I’m not sure the authors caught).

  • The control group’s productivity was highly predictable as a function of number of tests and had a stronger correlation than the TDD group.

So TDD’s relationship to quality is problematic at best.

Having a big clump in that upper left quadrant is troubling enough but then having the “Test Last” group almost double your “Test First” group in the over 90% quality range is something that should be noticed and highlighted.

While correlation doesn’t equal causation, the lack of correlation pretty much requires a lack of causation.

Read the whole article for yourself, as well as the original report, and draw your own conclusions.

Are you stuck with a legacy PHP application? You should buy my book because it gives you a step-by-step guide to improving your codebase, all while keeping it running the whole time.

15 thoughts on “Why I Prefer Test-Later

  1. I think I disagree with your first point, about not having enough knowledge on the subject. In essence you don’t have enough knowledge to write the test, but do have the knowledge to write the code? I think that leads to the common ‘hack something together’ approach. TDD here forces you to think about the problem before you start hacking away at it.

    Regarding the non-TDD control group case: in this ideal case, the testing at the end is done the right way. More often than not however, time constraints, project slack etc. and a looming deadline will shorten the test cycle at the end. Because of its ‘at the end’ nature, tests are the first thing that developers cut down on, when under pressure and when time is running out. TDD helps in this scenario because testing is mixed with the development and less likely to be the victim of deadlines.

    A third argument I would like to bring forward is that if you write a testcase after you write the code, you’re more likely to test only those parts of the code that you know work, because that’s what you have been thinking about while writing the code. Writing the test before hand let’s you think clearer about the various aspects your code is going to cover, instead of the aspects is actually covers.

  2. Ivo —

    “Hack something together” is not the same as “explore the problem domain.” Experiment. Explore. Build a couple small pieces to see if it works the way you think it should. You may not know enough to solve the problem yet, but you *definitely* don’t know enough to be able to write a test to check your solution, since you don’t even know the problems yet. Any solution-checking you build will have to be re-built as you learn more. Might was well build your solution-checks at the end when you know what you’re doing.

  3. Good thinking is always required before exploring any new problem domain, but I also tend to test later, exactly for the same reasons described hereabove.

    How would I know what to test if I do not know what solution will come out in the end of the proces?

    However, for refactoring, the TDD method is more than usefull,especially when working in teams.

    You could have the team leader or a a designated developer write all testing before the refactoring actually takes place.

    But what do we actually test on most of the times? Some input should produce some behaviour of the tested code and a certain output, and that’s it. Any erratic behaviour outside the scope of the test will not always rear it’s ugly head during testing.

    Any way, it could very well be that this is just me showing a lack in my test writing skills….

  4. From my point of view as a sometimes zealot, you’re missing the point that you use the tests to create the map of the terrain you’re covering as you cover it. No map has ever been permanent, nor should any tests be treated as such. My motto for tests is that they should be as read-only as pragmatically possible.

    As you gain familiarity with your surroundings those tests will help you address your changes by allowing your to quickly see where change 101 effects the original implementation and thought process in change 2. TDD is not the only path up that mountain, however, as a properly written test, regardless of when it is written, will give you the same benefit.

    The benefit I personally derive from TDD is that I know that no portion of my code is “detestable”[1] as none of my code was committed without a test that exercised it. Instead of containing conditional code on to address the “what if?”, I create the “what if” in an executable spec, then exercise the code via the test to make sure my assumptions are correct.

    I do whole-heartedly agree with one point that Ivo made. If tests are done last, they are the first to be cut. By utilizing a TDD approach where tests are created along side working code, you force the tests into the cycle of development in a place where they can’t be cut.

    Like any methodology or philosophy, TDD isn’t the one true path. If that were the case, there would only be one way to solve the problem and CPAN.org wouldn’t exist. 😉

    [1]: http://www.martinfowler.com/bliki/Detestable.html

  5. For me TDD is fine for libraries, or actually a must. For anything with a UI, I also remain unconvinced, mainly because I prefer a more design centric way of thinking there. That being said, as more and more things are done in a REST kind of way, more and more of what used to be UI logic is moved to nice API’s that fall more into the category of libraries. And all the other stuff is javascript, which I dont get 🙂

  6. Paul, one thing missing from your post and the comments thus far is that TDD is designed to be part of the larger picture of Agile development. Agile is all about working with your customer to determine the requirements in an iterative fashion, and adapting your code to those requirements based on feedback you receive as you build and release incrementally.

    Usually, the engineer does know as much about the problem domain as his/her customer. It is through working with the customer that you determine what tests to write, based on what they want their software to do. And these need not be just unit tests, but also functional (or acceptance) tests that are more clearly based on the UI or the expectations of the user. In fact, working top-down, you can start with acceptance tests which the client can articulate, and which verify the business rules of the application, and gradually work your way down into unit tests that verify that functionality on a class by class, or method by method, level.

    We have found test-first development to produce far better code–code that is designed from the perspective of the API, not the internals of that particular module–than when we write tests later for the same code. Test-first forces us to define the API first and clearly think about how our code will be used. As such, it encourages reuse and greater encapsulation. Combined with the practices of working in iterations, refactoring when needed, and running your tests in an automated, continuous integration process, you can derive the most benefit from TDD.

    Also, in my experience, tests are more likely to be written at all if you write them before you write the business code that will be tested. That said, there are many engineers who do write solid tests immediately after they finish writing some code, and having tests at all is far better than having none.

    I think the decision on whether to do test-first or test-later development is personal choice for the developer doing the work. It is a practice you can teach and encourage, but you cannot enforce it. On our team, not everyone codes test-first all the time. But we all recognize the benefits and strive to do it as much as we can.

  7. I can’t speak for all TDD adherents, but I’d agree that when you don’t know what you want, it’s hard to test for it.

    When I’m just noodling around trying to figure out an approach or a set of tools, I’ll write some scratch code to explore the problem space. Once I actually understand what’s going on, I generally throw that code out and start over, testing all the way.

    Why? Well, if I don’t know enough about what I’m doing to write good tests, I certainly don’t know enough to design things well, either. I also don’t know yet which bits of code will turn out to be the good ones, so there’s often a bunch of cruft lying around.

    Of course, I could try to clean up the experiments, but I find it faster just to start with a blank slate, glancing back at my experiments as a reference.

  8. Sam, I have to fully agree with you.

    I think Paul’s post is very nice and controversial. The thing I cannot agree with is that when I am not sure what the problem is, I start out with some code snippets and go from there. I mean how can you start writing code if you have no clue what you need to do?

    What I do in these cases is do normal paper sketches – define classes based on the nouns I write own when I describe the problem in three sentences. I then sit down, finding some mathematical dependencies and as soon as I have two or three of those I write the first tests and go from there TDD-wise. Actually that’s not true. For me a ongoing mix of test-first and test-after has been the best thing to do so far.

    Back to the I-dont-know-what-the-problem-is: Usually, the customer will tell you what they want.. Maybe I am getting something wrong? Paul, can you name a problem where you said down having no clue what to do and then starting out with some snippets?

  9. I agree with you Paul. TDD is great, but it has a flaw, the word “first” in its definition: “new functionality is written *first* as a test case”. I’m more into SCRUM than XP, and this doesn’t work for me. XP supports the TDD approach, they encourage you to Test first, then Design and Develop. While SCRUM, on the other hand, encourages you to Design first, then Test and Develop.

    Companies like Google and BBC are using SCRUM, and it works. While Testing is also a huge part of SCRUM, you are not forced to test a script before you design the component, can be a class skeleton, a use case or both.

  10. @Test-first forces us to define the API first and clearly think about how our code will be used.

    No it doesn’t. That’s just an excuse to justify the *first* mistake.

  11. Paul,

    I have to disagree with you here. Although running 500,000 tests does not guarantee quality (or any other metric), running the right tests as a matter of course is a boon to application security.

    I’ve been reviewing web apps since the late 1990’s before it was even known that this needed to be done. The first application I came across that made me sweat was from an agile team (again back in the days when if you did agile, you probably wore official Star Trek rank insignia to work on casual days).

    In the intervening years, there is a direct casual relationship between insecure software (which typically, but not always) has no tests and relatively secure software, which universally has tests.

    It doesn’t mean that the developers knew any more about web application security, for indeed no one (not even me!) has a perfectly secure application, as they tested their code extensively, and as a byproduct of that effort, made the application *accidentally* secure.

    They also had a handle on software engineering processes, which also helps application security. Stuff like repeatable automated installs. The costs to remediate cost which have testers in senior positions is nearly always cheaper to fix, because they don’t stress about changing a piece of code here or there as they know they can re-test it in a reasonable time frame and in a cost effective way. Folks who don’t do this, rarely fix my lower risk (notes, lows, and mediums, and even occasionally high) items as they cannot predict what impact the changes required would have on their system.

    So, although I am not impressed by test counts alone, and although initially skeptical about TDD’s / FDD / pretty much every agile methodology, I personally am now converted to the test first/write later paradigm through personal experience over nearly 6 years with many different dev teams in many different industries. They simply produce better, safer code, which is cheaper to fix and far, far faster from a “discovery of issue to fix (that works) running in production” point of view.

    These are THE metrics that are the true payoff for a business whose bread and butter may not necessarily be cutting code. They simply don’t care about the initial pass at cutting code nor how productive a developer is – coding fresh code – although the thing that makes us happy – is but a tiny, tiny fraction of the cost and time put into enterprise level projects over their entire lifespan.

    For example, I’ve reviewed production code that has portions older than me (and I’m 37). The devs probably wrote the initial pass in around a year or two, costing probably about a million bucks in today’s money. We’re talking a couple hundred thousand lines of COBOL here and several hundred core transactions. That company has invested hundreds of millions of dollars and many hundreds of developer years in looking after that code as it’s their competitive advantage and how they’ve done business for the better part of my lifetime.

    So I’m all for code we write today to be written as if it was written with a future of refactoring and maintenance in mind, even the small stuff like forums and our private collection of open source personal itches.


  12. I agree with you that during the development developer learns about the system they are going to develop. But those who proposes test-first will tell they can learn the system when they write test cases.
    I am opposed to this because this hampers my creativity. Test cases force me to focus on where system can fail, Instead of thinking how well i can accomplish my each task in system.

  13. I agree with the article, test first might force developers to write test, because some of them might get lazy if you ask them to write tests later, but as far as I know, if you are owning a business, you should minimize the reliance on people, E-myth explains this very clearly, you should not rely on people to be successful in business, you should rely on system, everything is system, and recently my whole purpose in our company is to redesign our system, I’m a developer myself, and a good one, I can compare most technologies, methods, and come up with really fascinating way to implement our next project, problem is, most of our developers lacks the knowledge, of course we can train them, but the more high tech ‘our way of doing it’ , the more we rely on personal knowledge, the more stress the new comers feel, I don’t want them to feel stressed, I want to build a system, which enables even the new graduates can contribute to our project, Just think about MacDonald, they don’t make ‘great’ hamburger, they don’t need first class expertise to operate their restaurant, but they have ‘great’ system, which enables the rookies to operate, so it can be copied rapidly.
    TDD is great for our system as it can improve our product quality to some extend, but if we enforce testing-first, as I observed, they really feel stressed, the idea of testing first even makes me stress my head, of course it’s much easier to test-later, I can do it gladly.
    So besides the reasons written above, I’d like to add that, from a system perspective , test-late makes it easier for developers to write test code, I also intend to separate testing/coding person ,which puts less burden on developer skills , no one has to know everything , or team members write tests for each other AFTER coding, which enables them to exchange ideas continuously.

Leave a Reply

Your email address will not be published. Required fields are marked *