Forrester notes the ability of Agile to deliver “improvements in time-to-benefits, overall quality and efficiency, team morale, the relationship between IT and business staff and responsiveness to change”.
Over the past few years many companies have started to turn to agile software development to replace more traditional waterfall software development methodologies that have often failed to deliver.
The “agile” vs. “waterfall” debate has caused much discussion in the software development arena. There are many evangelists for each approach that often have very polarized views.
Exponents of the waterfall or plan-driven approach describe how the step-by-step approach to gathering requirements, creating complete specifications, performing detailed technical designs, developing code and then testing in a sequential, waterfall manner will lead to a successful high quality solution. They will often describe agile software development as a hackers paradise, an approach where you do not know what functionality you will actually get until it is too late; in effect, they will often describe Agile as a high risk proposition without any realistic level of control where the deliverables are uncertain.
However, exponents of agile development will point to the relatively high level of failure in waterfall or plan-driven development projects. They will describe how it is very hard for users to know the detail of absolutely everything that they want at the start; how all projects will have an element of change in them; how people spend lots of time on waterfall projects creating documents that quickly become out of date and unused. In summary they will describe a waterfall project as a turgid affair, where it takes a long time and a large cost to implement a system that is probably not what the user originally envisaged and is incapable of accommodating change. They will then describe how taking an agile approach will ensure high-quality, flexible solutions that are delivered at great speed and at low cost where changes can be incorporated with no problems.
Is either of these two polarized positions totally accurate? In reality, of course they are not. We have probably all seen projects where the analysis phase goes on forever in an attempt to capture all the users requirement in one go. In turn, users often believe that this is their only opportunity to get requirements in to the scope of the project so they try and cover every possible eventuality that can conceivably think of for functionality that may never actually be needed. This subsequently leads to a very complex design and a very hefty development phase to deliver what is promised as being a highly flexible, future-proof solution. We will have all seen examples where these highly flexible, future proof solutions are very complex to use, hard to change and costly when compared to the original budget. Often these systems bear very little resemblance to the simple, powerful system that was originally envisioned.
So, does this mean that Agile is the one true solution to all software development issues? The answer to this can be a resounding no. In the above section we have described an example of a development project bogged down by complex analysis with noticeable knock-on effects. For those of us who have been in the software development industry for a number of years, we will have all often seen system built without the right level of analysis where the end result has simply not met the needs of the users. We have all seen developers who can cobble together a system very quickly where the bug list seems to go on forever.
The summary of all of this is that to simply say that an agile approach will work and a waterfall approach won’t work or vice-versa is just not true. We can all find examples of success and failures. The key itself is to recognize the strengths and weaknesses of an approach on an individual project and choose the right approach for the given project.
However, we should also recognize that there is a considerable amount of value in a lot of the principles that an agile approach brings (we cover this in more detail later). Indeed, whilst some Agile evangelists (or indeed some Waterfall evangelists) might not want to admit to this, it is clearly the case that a lot of the principles behind an agile approach have come from the experience of successful waterfall projects.
For example, one of the key principles of an agile approach is to create automated tests to check the functionality of the code. On many waterfall projects, we have all experienced problems when the lengthy, large-scale testing that is needed identifies a number of bugs. We then pass the results back to the development team who fix them and, given we are all human, introduce a few more. We then go through another lengthy testing phase and so on. Successful waterfall projects often create automated unit tests so that we can quickly and repeatedly check that code works and that we haven’t broken it when we change it.
Whilst there are a number of subtly different methodologies that fall under the Agile banner, the Agile Alliance identified four key principles that underpin all agile projects. It is perhaps easy to look at these and draw the conclusion that Agile is a high risk, uncontrolled, informal methodology but this is not the case. Under each of the key principles, we have set out how these principles are applied on successful agile projects in the real world. We go into this in more detail when we describe our agile approach.
Individuals and interactions over processes and tools
The essence of this principle is that we recognize that good people are needed to build good systems. This might seem obvious but it can be the case (perhaps more so in a waterfall project) that there is a belief that good process makes up for having average ppeople – experience shows that this isn’t the case.
This key agile principle also stresses the value of interaction over process and tools. Again, it is common sense that a successful project will need people who can interact with each other; people who can explain clearly and understand the needs of the user. Some waterfall exponents will describe how documents can be created that describe what is needed and it is common for a developer on a waterfall project to have to work only with written documentation and specifications, rather than interacting with users.
Our belief is having good people is critical to the success of a project. At esuasive we have over 15 years software development experience and have a very strong, proven, experienced and dedicated group of technical resources. Typically, the developers that we employ on an agile project will have in excess of 7 years development experience and are normally accredited Microsoft designers and developers. All of our consultants who work on an agile project have strong communication and analysis skills. Whilst it is sometimes possible for developers to work without having much interaction with the key users, it is common sense that a greater level of understanding will be obtained if the developer interacts directly with the user.
However (and this is often ignored by people who wish to disprove the viability of an agile approach), all Agile methodologies (whether they be SCRUM, Test Driven Development, XP etc) make use of processes and tools to help ensure the success of a project. The key difference between processes on a waterfall project and an agile project is that the processes on an agile project are totally geared towards helping the project members deliver the solution – i.e. to help good people build a good system. On a waterfall project there is a stronger emphasis on using processes to better manage and understand the status of a project – i.e. not all of the processes actual help the team create a good system.
Working software over comprehensive documentation
This key principle states a fact that should be obvious on any project, that the working software is the most important deliverable (from an IT development perspective).
In waterfall projects, there is an emphasis on creating many different types of documentation from requirements specifications to technical design documents. It is often the case that the project loses sight of the fact that these documents should really be only an interim deliverable on the road to building a working system.
It’s important to note that this principle includes the word “comprehensive”. Some critics of an agile approach will say that documents are not created; this is not the case. There is always the need to create the right level of documentation for the circumstances of the project.
Customer collaboration over contract negotiation
This is an important principle that is often misunderstood. Critics of an agile approach where the development team is made up of external developers from a third party will often mistakenly say that adopting this principle means that it is the customer who takes the risk on a project. This does not need to be the case.
The most important fact to take from this principle is that a strong level of collaboration with a customer is very important on a project. If everyone is working towards the same goal, the chances of success are far higher.
However, we recognize that there are contract considerations that must come into play. Some software development companies have tried to use Agile as an attempt to engage customers on a time & materials project with no defined deliverables – i.e. to put the risk on the customer. We simply do not do this as we recognize that this is commercially untenable for many customers.
Responding to change over following a plan
It is a fact of life that most software development projects will have an element of changing requirements. In Waterfall projects changing requirements are often seen as something to be avoided at all costs. This can lead to a circumstance where the initial analysis and design becomes bogged down as users and the project team attempt to cover all scenarios in huge detail.
The emphasis on an agile project is to identify and establish the key requirements first and then focus on delivering the known requirements quickly and robustly. Whilst there may not be a large scale Gantt chart containing hundreds of tasks in a plan, a good agile project will have clear visibility of what the requirements are, their relative priority, effort to implement and target release date.
Indeed we have found that, because an agile project works in short iterations (usually 1 month cycles) it is often the case that there is more certainty when a key piece of functionality will be delivered in an agile project than a waterfall project with much less frequent code releases.