I have to disagree with you here. Although running 500,000 tests does not guarantee quality (or any other metric), running the right tests as a matter of course is a boon to application security.

I’ve been reviewing web apps since the late 1990′s before it was even known that this needed to be done. The first application I came across that made me sweat was from an agile team (again back in the days when if you did agile, you probably wore official Star Trek rank insignia to work on casual days).

In the intervening years, there is a direct casual relationship between insecure software (which typically, but not always) has no tests and relatively secure software, which universally has tests.

It doesn’t mean that the developers knew any more about web application security, for indeed no one (not even me!) has a perfectly secure application, as they tested their code extensively, and as a byproduct of that effort, made the application *accidentally* secure.

They also had a handle on software engineering processes, which also helps application security. Stuff like repeatable automated installs. The costs to remediate cost which have testers in senior positions is nearly always cheaper to fix, because they don’t stress about changing a piece of code here or there as they know they can re-test it in a reasonable time frame and in a cost effective way. Folks who don’t do this, rarely fix my lower risk (notes, lows, and mediums, and even occasionally high) items as they cannot predict what impact the changes required would have on their system.

So, although I am not impressed by test counts alone, and although initially skeptical about TDD’s / FDD / pretty much every agile methodology, I personally am now converted to the test first/write later paradigm through personal experience over nearly 6 years with many different dev teams in many different industries. They simply produce better, safer code, which is cheaper to fix and far, far faster from a “discovery of issue to fix (that works) running in production” point of view.

These are THE metrics that are the true payoff for a business whose bread and butter may not necessarily be cutting code. They simply don’t care about the initial pass at cutting code nor how productive a developer is – coding fresh code – although the thing that makes us happy – is but a tiny, tiny fraction of the cost and time put into enterprise level projects over their entire lifespan.

For example, I’ve reviewed production code that has portions older than me (and I’m 37). The devs probably wrote the initial pass in around a year or two, costing probably about a million bucks in today’s money. We’re talking a couple hundred thousand lines of COBOL here and several hundred core transactions. That company has invested hundreds of millions of dollars and many hundreds of developer years in looking after that code as it’s their competitive advantage and how they’ve done business for the better part of my lifetime.

So I’m all for code we write today to be written as if it was written with a future of refactoring and maintenance in mind, even the small stuff like forums and our private collection of open source personal itches.

Andrew

No it doesn’t. That’s just an excuse to justify the *first* mistake.

Companies like Google and BBC are using SCRUM, and it works. While Testing is also a huge part of SCRUM, you are not forced to test a script before you design the component, can be a class skeleton, a use case or both.

I think Paul’s post is very nice and controversial. The thing I cannot agree with is that when I am not sure what the problem is, I start out with some code snippets and go from there. I mean how can you start writing code if you have no clue what you need to do?

What I do in these cases is do normal paper sketches – define classes based on the nouns I write own when I describe the problem in three sentences. I then sit down, finding some mathematical dependencies and as soon as I have two or three of those I write the first tests and go from there TDD-wise. Actually that’s not true. For me a ongoing mix of test-first and test-after has been the best thing to do so far.

Back to the I-dont-know-what-the-problem-is: Usually, the customer will tell you what they want.. Maybe I am getting something wrong? Paul, can you name a problem where you said down having no clue what to do and then starting out with some snippets?

When I’m just noodling around trying to figure out an approach or a set of tools, I’ll write some scratch code to explore the problem space. Once I actually understand what’s going on, I generally throw that code out and start over, testing all the way.

Why? Well, if I don’t know enough about what I’m doing to write good tests, I certainly don’t know enough to design things well, either. I also don’t know yet which bits of code will turn out to be the good ones, so there’s often a bunch of cruft lying around.

Of course, I could try to clean up the experiments, but I find it faster just to start with a blank slate, glancing back at my experiments as a reference.

Usually, the engineer does know as much about the problem domain as his/her customer. It is through working with the customer that you determine what tests to write, based on what they want their software to do. And these need not be just unit tests, but also functional (or acceptance) tests that are more clearly based on the UI or the expectations of the user. In fact, working top-down, you can start with acceptance tests which the client can articulate, and which verify the business rules of the application, and gradually work your way down into unit tests that verify that functionality on a class by class, or method by method, level.

We have found test-first development to produce far better code–code that is designed from the perspective of the API, not the internals of that particular module–than when we write tests later for the same code. Test-first forces us to define the API first and clearly think about how our code will be used. As such, it encourages reuse and greater encapsulation. Combined with the practices of working in iterations, refactoring when needed, and running your tests in an automated, continuous integration process, you can derive the most benefit from TDD.

Also, in my experience, tests are more likely to be written at all if you write them before you write the business code that will be tested. That said, there are many engineers who do write solid tests immediately after they finish writing some code, and having tests at all is far better than having none.

I think the decision on whether to do test-first or test-later development is personal choice for the developer doing the work. It is a practice you can teach and encourage, but you cannot enforce it. On our team, not everyone codes test-first all the time. But we all recognize the benefits and strive to do it as much as we can.