A Response to a Talented Author Related to Agile Pitfalls

Matthew,

This response is not as much for you as for anyone not deeply familiar with Agile. You show a great understanding in your article and I would love to talk to you about some ideas.
I am sure you are aware of corporate America’s statistics around software, with success rates at best in the 20% range, and no argument that almost certainly at least $100,000,000,000 a year is wasted.

As you clearly know Agile is a set of practices (mostly commonsense, but not all) that focus on what is typically the most important end state:

Success across time, budget, scope (feature set) and quality.

I would also include maintainability and an architecture that is well designed enough to separate intention (say in Interface) from Implementation, survived say 5 years of upgrades, and the ability to fundamentally change the implementation while maintaining the intention.
Agile is not perfect, that is for sure. Most agile projects fail to do proper QA (no system regression nightly inside iterations and no load/limit testing nightly) which technology today makes very achievable. Most agile iterations do not produce the Production quality that is expected (in spite of good compliance with Test First – really more a design benefit (especially with Framework/API/SOA Exposed Interfaces) with the added bonus of regression, continuous integration, continual refactoring and heavier business involvement – to name what I see work most often).

What I also see is a common lack of real Enterprise Architecture knowledge and no key Architectural choices made upfront (indeed Agile does not mean your key guiding ‘broad brush strokes’ should not be decided based on the nature of the problem domain.

For example a life critical system will demand far higher quality and therefore different metrics on what ‘good’ is). These are not something to expect to evolve like a ‘feature’, think 30,000 higher.

I am describing the fact that we know the patterns of how to create flexible Enterprise Architecture and these should not evolve from iterations. I think this is flawed thinking, especially as properly designed Intent can always be flexible in matching implementation. To be very specific, I would not enter iterations without deciding on say a ‘Client-Dispatcher-Server’ approach, which is then followed in implementing the iterations. Many will disagree with me on this I am sure, but I have found that jumping into iterations with no Enterprise Architecture framework at lease minimally defined, will typically require a lot more refactoring then necessary. To mean this is an ‘aspect’ not an evolutionary iterative discovery. Before true iterations start I have found (from a technical perspective) you should decide on an extensibility pattern that matches the levels of flexibility required. As I am building software for sale, my flexibility needs are far greater then a departmental app that might last a year. I must think about years of a product’s life, yet the real work (that is domain specific) is done in iterations (which is 90% of the work).

I have come to the conclusion that not defining some Enterprise Architecture and the inevitable set of implementation rules that follow (you need to create a Dispatcher that contains a collection of Resolvers to get implementation for your Interfaces for example, and follow the rules such that implementations can vary (often without a recompile, just a metadata change and the (.NET Specific) creation of new AppDomains to allow for this) of your Interfaces. To me and my experience, this should be decided on so that inside iterations these rules are followed. An Agile purist would say ‘where is the requirement for this’ and ‘we never predict the future’. True enough. But I do know change is a constant and the #1 role of an architect today in my opinion is (as much as possible) future proofing an application. If all other areas are met (quality, maintainability, scalability, etc) the #1 item to judge an architect on is how hard it is to make significant changes to an application and how much disruption it causes.
A common example: Many Agile teams start by cranking out concrete classes. Fine. But they often fail to refactor these into introducing abstract parent classes when they should and they often fail to start by defining Interfaces and seperating these interfaces (literally) from implementation. Where is the requirement for this? It can evolve, but the flexibility of Enterprise Architecture patterns such as Client-Dispatcher-Server will likely not show up in a requirement. You just know you will have to deal with change over time, and you want a system that can evolve with that change (and you know you cannot predict what it will be but you can put forces in place to make it far less painful). I no longer believe a system should iteratively ‘discover’ it needs these kind of fundamental structures and patterns in place. There are not many up front, I don’t like to do any more then absolutely necessary, but try refactoring a non-trivial system into one that is compliant with this kind of flexibility where the system is dominated by Abstract Classes and Concrete Classes, with few Interfaces.
Even Web Services should be Interface driven, both on the server (handler) side and the client proxy side. The implementations (in .NET code is generated and you have to manually add the Interface, or use a tool like WSCF – A much better option).

Also, flawed development to staging/user acceptance testing migration is all too common, teams developing in environments that do not try to be close to production (developing on XP with 1 CPU (and on many machines Hyperthreading is turned off by default so if you are thinking that might help you, it can but turn it on, deploying on 8 CPU Win2003 causing multithreading bugs to surface) as especially teams using session state ‘in proc’ and then at the deployment stage trying to do out of proc session state. All of these are easily fixed.
Going deeper, I see in most cases a flawed requirements process, where either too much or too little is done before iterations start. To make my position clear, a large, monolithic spec up front is almost always useless, while writing one sentence per major user activity (call it a use case, user story or whatever) without proper assignment of complexity, risk, or importance to the stakeholders who will judge your success or failure.
In fact, most of the ‘agile’ projects I have been hired to work with are actually NOT agile as the author points out. They just pay it lip service, with developers in closed door cubicles, a lack of pair programming (the most common) which means a lack likely of collective code ownership, and iterations that are driven by whim, not by a timeboxed and disciplined series of activities (and I mentioned the QA Failures).. Also I find it common for ‘agile’ shops to write their tests AFTER they write the code, missing the entire point (although it is arguably better then nothing I suppose.

Now let’s add the layer that is also lacking: A lack of knowledge of Design Patterns, Refactoring (Fowler type) and most commonly refactoring to patterns (Kerievsky). There is no culture of the daily stand-up or weekly brown-bad educational meeting. Agile has a reputation as being the domain for the lazy programmers who don’t want to do what needs to be done by the waterfall type advocates and in many cases I would agree.
But when an Agile team is hitting on all cylinders, every developer is amassing hundreds of test-first regressions (and the design has evolved a quality it would never had possessed if TDD were not practices), all source code modifications of the product are build tested with tools like CruiseControl, and QA is constantly building regression tests on the heels of the pairs who are completing work with nightly validation and separately automated load testing and code coverage analysis with tools like NClover (not to mention a culture of a ‘war room’ environment, white boards everywhere, and pattern speak (people say Chain of Responsibility, Factory, Bridge, etc. and can simply add the word to the diagram with team-wide understanding) and much more (some of it things I have developed and will be in my book), you cannot compare it to anything I have ever seen.
As for the article I will add my comments:

If your organization is trying to adopt (or considering) a more Agile approach to software development, you are probably hearing plenty of rah-rah stories of how great things will be.

“True but most do not do Agile, they do something they like to call Agile”
A rational response to those stories is to ask, “What can go wrong?” To discuss some of the traps to avoid, let’s consider what most companies are transitioning from: the waterfall model.
The waterfall model is the most common way for large organizations to write software today. It takes the complex, chaotic development process and turns it into something simple and clean:

Figure out what needs to be done (requirements).
Determine how to do it (architecture and design).
Write the software (coding).
Make sure that the software works (testing).
Deploy the system.

The only problem is, it doesn’t work.

“That is an understatement. It is a sure way to fail statistically”
I should probably clarify that statement a bit: The waterfall model often doesn’t work well. Companies try to turn the waterfall into an assembly line, with requirements analysts, architects, coders, testers, and project managers who oversee the assembly. Transferring information between these people is difficult, so they tend to rely on documentation, which is often vague and sometimes worthless. Worse, when the customer changes his mind after a decision is made, costs go up vastly—therefore, waterfall advocates suggest limiting or controlling change. So, when the software ships, it’s late, buggy, and doesn’t really meet the customer’s need; it meets what the customer thought his need was nine months ago…
“It is actually in line with American philosophy of doing business so it’s hard to fault the execs who are above it all. They had these ideas drilled into their head for decades and they do work in many domains, but not software”
Enter the Agile consultants. Instead of specialists, they suggest generalists, who do all the work, implementing features in slices, end-to-end, in very short timeframes.

“Not necessarily, but OK. I have never found Designers who could code at the level I needed or developers who could design at a level I needed. The designers typically are involved in the iterations but mostly to deliver UI supporting the iteration’s features that have been seen by stakeholders. It is kind of an ‘Agile Prototype’ and it has worked well for me. Before doing this I would have use cases/stories approved and implemented in an iteration, only to find that things are wrong (but they didn’t know it until they saw it). The ‘Agile Prototype’ means that the stakeholders can see the user experience and will usually enable much more accurate and revised direction from them on building the cases out. I would also (personally anyway) restate that a maximum impact Agile team is as small as it can be, but no smaller and is staffed with the absolute top quality possible, even if sacrifices elsewhere must be made to get these people. We know the best are 28x better then the worst [Facts and Fallacies of Software Engineering, Glass, R.]. It also has at least one dedicated QA person who is performing QA right along with the developers and at least one Business Domain Expert. So all in all the team is made up of some specialists”.
The customer can prioritize her features, getting the most important features first, meeting after every release to plan the next release.
“Technologists can move items up in priority if they are very complex and possibly not achievable, but still required for success to be awarded”
Productivity increases because problems aren’t falling through the cracks. There are no inter-team arguments, such as architects versus coders versus testers; everyone is on one team, focusing on delivering something of value. Testing moves up front, changing from a verification activity to a specification activity.
“I would add that QA is a peer of a developer as is a domain expert who can get answers if questions are in areas other then their area quickly”

Immediate Risks
Implemented well, Agile methods truly can deliver.
“Yes but they rarely are”
The customer sees working progress periodically, can change her mind after every release, and sees the most important features implemented first. But in my experience, Agile methods have three major potential risks:
* Agile methods are easy to misunderstand.
* It’s easy to think you’re doing Agile right, and be wrong.
* Agile methods make value visible.
“ABSOLUTELY!”
Agile Methods Are Easy To Misunderstand
Agile philosophy includes heavy customer involvement, responding to change versus following a plan, releasing software early and often, and focusing on individuals and interactions over processes and tools. Those Agile philosophies tie into Agile methods such as pair programming, the planning game, time-boxed iterations, test-driven development, and refactoring. (XProgramming.com is a wonderful resource on Agile methods.)
“his is describing XP more then Agile but it will do I suppose.”
The methods exist in order to enable the philosophy. For example, test-driven development combined with refactoring can make software malleable so it can change more easily, allowing the plan to change. Automated testing combined with continuous integration makes it possible to release often. Without automated testing and continuous integration, frequent releases create a huge manual testing burden that’s often unbearable.
“One thing that I mentioned that is not here: You must have an Enterprise Architectural foundation that can support Agile. This means ‘design by contract’ heavy work (look for my new article on thinkecture’s WSCF utility for Contract-First Development as an imperative and the declaration of intent and the flexible ability (via metadata typically) to find an implementer of the intent at run-time (for .NET people this means often a new AppDomain as you cannot unload the implementation (assembly) only the AppDomain. Interfaces (especially with the power of Generics) are critical, not abstract base classes in many cases due to the .NET architecture but I will try to stop the .NET focused stuff.”

Agile practices are like any other practices, though; they’re learned through example, application, and training. If the practices are disconnected from the philosophy, the result just won’t work.
“I would say that people almost universally separate architecture from process. In Agile I have come to believe this is a death-sentence. The Architecture must support the flexibility of Agile, or your going to have a very hard time. I cover this deeply in my upcoming book.”
For example, a few years back I worked with a company that wanted to be more “agile,” yet they still wanted to know exactly when projects would start and stop, before the requirements were defined. That’s a fine idea—in fact, Agile development can support this plan. Features are placed in a priority order; the team works on the highest priority first, iterating until time runs out. The problem was that the organization wanted an estimate for all the work to be done—before they knew all the requirements.

“As software is a ‘Wicked Problem” this fallacy of logic, that you can even do this, must end. The best you can do is provide risk adjusted ranges using the work of DeMarco and Lister in Waltzing with Bears as a fantastic example”.

The example above isn’t Agile—it’s psychic. Agile development specifically gives up on the psychic approach, instead choosing an adaptive approach.
It’s Easy To Think You’re Doing Agile Right, and Be Wrong
“YES!”
It’s really easy to throw out the big, thick methodology binder and the heavyweight requirements documents, but Agile development expects that something will replace them. Many organizations move to iterative development without automated testing—with the result that the testing burden grows exponentially—or move to iterative development but keep the hard deadlines and the expectation of full delivery on a ship date. Pair programming without communication and feedback is just someone breathing over your shoulder. Without testing skills, a developer won’t be able to do automated testing; worse, he won’t even realize this as he wastes hours writing so-called “tests.”
“I agree with that”
In other words, Agile techniques require depth: the ability to know the right techniques for the current project, and the ability to choose between them. Without direction, a team told to throw away its waterfall method will simply devolve into “code and fix.”
“Agreed again. It takes strong leadership and someone who can truly empower a team with Agile expertise – unfortunately these people are few and far between. I can think of around 10 out of 50 I know who can really deliver”

That isn’t Agile: it’s chaos.
Agile Methods Make Value Visible
Yes, believe it or not, this is a potential risk of agile adoption. It’s a political risk. In most large organizations, it’s possible to not work very hard, not contribute, and get by on political savvy and power. I refer to these folks as “roaches” because they’re essentially scavengers that hide from the light.
“Ha ha.. Yes”
Agile methods make progress visible. Unlike the complex, bureaucratic waterfall organization, which is opaque, if someone in an Agile shop isn’t contributing, the fact will become obvious—and fast. Think about the consequences of this fact for a minute. The roaches, often the most influential and politically savvy people in the group, are going to fight Agile methods tooth and nail. The misinformation campaign that a roach leads can be surprisingly effective, because Agile methods are easy to misunderstand and hard to get right.
“Absolutely. I like to say Agile creates a team with ‘nowhere to run, nowhere to hide’ because everyone knows if you are doing OK or not (if they don’t care you have much bigger problems).
Two Cautionary Tales
A few years ago, I presented a seminar on test-driven development at a large testing conference. Many of the participants were “doing Agile.” One in particular was having a very hard time. Her development group was rewriting a piece of core functionality from scratch in Java—a new language for them. They hired a group of Java coders and created a schedule with four-week iterations. Every four weeks, the developers were supposed to throw a chunk of code “over the wall” for the testers to test. The software was supposed to be developed in six iterations of four weeks each, with a final QA phase at the end.
“I am sure a disaster.. QA is part of an iteration, just as important as programming, not something for later”
The first iteration was two weeks late, and buggy. The second iteration was a week late (making the project now three weeks late), and completely buggy. The basic functionality didn’t even work, and it introduced a number of regression errors in the software, so the testers had to go rerun the original test plans for phase I. It didn’t take a crystal ball to foresee what the next five iterations would be like, and they weren’t good. Worse, the testing group was in trouble because they were the supposed bottleneck “holding up the Agile developers.”
In other words, the organization tried the following:
* “Big bang” style software scheduling with a new staff in a brand-new language. (But hey, they had iterations!)
* Iterative development without automated tests.
* “Iterations” that delivered incomplete work products (for instance, software that wasn’t ready to release to the customer).
* Agile development while keeping the developers and the testers in separate buildings, with testing as an afterthought, verification activity.

“As I said above, QA Testing IS PART OF AN ITERATION, not separate”

You can probably guess the results; they weren’t pretty. That sad part is that Agile was blamed for the failure of the project. Here’s the real truth: New people, new processes, and new technologies throw realistic scheduling out of whack. By focusing on delivering working software early and comparing expected features completed to actual features delivered, Agile development provides feedback, which can allow a management team to make good decisions.
A second organization was having integration problems; specifically, problems with different versions of the same libraries on different products. When management heard the Agile pitch of “continuous integration,” they were immediately sold, buying all of the developers copies of a famous book on Agile development. Management went to an iterative development standard, began daily standup meetings…and that was about it.
Without substance, the so-called Agile development conversion devolved into “work harder on tighter timeframes, with standup meetings.” The developers still lacked direct contact with the customers and had requirements that were even more light, vague, and unverifiable.
“Very common in my experience”

Twelve months later and the team is basically back to square one. That’s not Agile; that’s “code and fix.”
“Code and Fix is the most used method of software development.

Another name for Code and Fix is ‘no method’.”
Avoiding the Pitfalls
Taking a careful look at some of the failures I’ve described shows a few similarities:
* Massive, world-shaking change without experimentation.
* Expecting to have your cake and eat it too: All features, on time, on budget, on schedule, regardless of how realistic that schedule is.
* Implementation of Agile techniques without a belief in the core philosophies of responding to change and not planning everything up front.
“These should be obvious”
When an organization implements Agile in a “big bang,” all-or-nothing way…well, that is possible, but the results are unpredictable. They’re trying to develop in a new way on a project that’s never been done before, and the technical term for that is an experiment. Typically, you don’t want to use an experiment on a bet-the-company project, which is what many Agile projects end up becoming.
“Agreed in a way, but if you have the right amazing people you need in place, I would say go for it”
There is an alternative to declaring exactly how all development will be run, from this day forward, without ever trying it: Experiment on your existing projects. One place to start is with automated testing. If your team is constantly modifying the same code base, automated testing can pay for itself very quickly by lowering the overall testing burden. With automated testing in place, the team has a safety net, which can enable true iterative development and quick releases.
“I do think in many cases step-wise additions to process can make more sense then a full implementation all at once. It depends on the bandwidth of the team ad the presence of an Agile pro on the team”
Another way to be more Agile is to simply make decisions that conform more closely to the Agile manifesto—that is, decisions that value the following:
* Individuals and interactions over processes and tools
* Working software over comprehensive documentation
* Customer collaboration over contract negotiation
* Responding to change over following a plan
Add small, incremental changes to the way you develop software (your “methodology”) that support these values. Agility is not yes-or-no; it’s more-or-less—and don’t let anyone tell you otherwise.
A Final Reminder
Agile methods won’t result in an ability to ship infinite code in zero time, they won’t create discipline without effort, and they are possible to perform badly. Implemented poorly, Agile methods can leave an organization in worse shape than when it started, with all the same problems, less documentation, and worse architecture.
“Perhaps.. I am not sure where the studies that back this are. I suppose if people think Agile means you do no up-front Architectural stakes in the group (like a micro-kernel approach) then this could hold”
Implemented well, Agile methods provide working software quickly; they give the customer the ability to change his mind routinely; and, most importantly, by comparing actual to expected delivery, they provide insight into the actual progress of the project.
Knowing the progress of the project makes realistic schedules with realistic end dates actually possible. Adjusting the schedule to reality, on the other hand, takes guts, integrity, courage, and self-discipline.
Welcome to real software management. Buckle up, and get ready for the ride of your career.