Today I had enough of assertions that Agile development is the One True Way, Waterfall management worked only when computers were big and slow, etc. I think this ignores one of the principles that I try to follow: Use the Right Tool for the Job. It's as silly to say that one development methodology is better than another, outside of any context, as it is to say that a spoon is a better tool than a fork. Try eating porridge with a fork and salad with a spoon, and you should see my point.
So. Harrumph. What sorts of jobs suggest each of these tools? I would propose that Waterfall works well when you know what to do but not how to do it, and Agile is a better choice when you don't even know what you are expected to do.
The US space program in the 1960s is a great example of the former. The task was difficult, but simple and completely specified: before this decade is out, to land a man on the Moon and return him safely to the Earth. There you go, rocket scientists -- the budget is not infinite, but we want to Get There and Back Again and we don't know, or much care, how you're going to do it. How is your job. The requirements are clear and will not change. Waterfall development is good for this. We can plan it all out in advance because we know precisely what Done looks like. This is good, because the consequences of failure are enormous -- we have to get it right in one. The program was to produce one unit of product, for a single use by one client, and so it had to work the first time.
So, for what sort of project do we not even know what we are supposed to do? One in which the client takes an abiding interest in details. Because often even the smartest, most thoughtful clients don't know what they want. Not in enough detail to make a complete plan that will satisfy. Agile methodologies formalize a process of generating a continuous stream of small prototypes with which to tease out the complete requirements when just asking will not work. The job to be done changes all the time, as we learn what it is. But because these steps are so small, we can afford to fail all the time, and learn from each failure like good engineers. The more users there are to be, the more likely you will find yourself in this position, because everybody is different.
I think that if, after completing an Agile project, you collect and merge all the user stories, you should find that you have -- gasp! -- a complete requirements document. You just wrote it piecemeal instead of all at once.
You may have noticed that I made some unusual claims about the space program: one customer, one unit, one use. Clearly that's at a very high level of abstraction. Which introduces another aspect of Right Tool for the Job: you may want to use different methodologies at different levels of abstraction, on the same project. As you drill down into that large but well-specified project, you are going to find that you don't actually know how the details should go. You are going to have to experiment. If it's big, like Apollo, you are going to need to break down the work into subtasks and subsubtasks and so on. In fact, the methodology tells you to do this until you can't go any smaller. Some of those subtasks are going to be inescapably ill-specified or simply experimental. Hello, Agile. The only rule that works all the way up and down the scale of abstraction is Use the Right Tool for the Job. This is not apostasy. A carpenter uses different tools to cut boards to size than to fasten them together.
So I would advise treating the selection of development methodology as craftsmanship rather than religion. First look at the job and let it tell you how you should start. As you gain understanding, look for patterns that tell you when you may want to use a different approach here or there.
