Traditional Waterfall projects promise successful outcomes through meticulous planning and tight control of scope and risks. It is a promise not always fulfilled. The famous quote no battle plan survives first contact with the enemy neatly sums up the problem with waterfall in situations where the detail of requirements is not entirely clear, or where there are likely to be many externalities not easy to control along the way. Nevertheless, many software buyers still yearn for the certainty of a fixed price waterfall project.
Conversely, Agile acknowledges the fact that often we don’t have full understanding of requirements at the outset, or know the best way to achieve desired outcomes before we start, Agile believes it is better to figure things out as we go, but this too can lead to disappointing results, especially when agile teams lose sight of non-negotiable deadlines or ignore expectations that even though the project is agile, scope is fixed. Blessed are the flexible, for they can tie themselves in knots.
Both approaches work, in different circumstances: waterfall when requirements are very well understood and the deadline is set in stone; agile if service innovation is a desired outcome and requirements can evolve – as is frequently the case with a business transformation programme.
But what if that same business transformation programme encompasses both challenges?
- A clear need to move from one system to another at a given point in the future.
- An objective to create innovative processes spanning different business functions.
We faced exactly this challenge whilst working with a leading Welsh Housing Association, needing to replace their core finance system but recognising that this would be a ‘once-in-a-generation’ opportunity to innovate. To meet the challenge, we adopted a hybrid approach, implementing a new finance system (Microsoft Business Central) using waterfall, whilst simultaneously taking an agile approach to the innovation programme, enabling the client to control costs and plan with certainty in the critical accounting areas (waterfall) whilst shaping business change through continuous review and feedback (agile).
The Project Tradition
The Iron Triangle of Project Management, understood since the 1950s, lays out the inescapable fact that the outcome of any project will depend on:
- how much you want to achieve (scope)
- how quickly you want it delivered (time)
- how much you want to spend on resources (cost)
Setting objectives for any two of these inexorably determines what you will get for the third. This has been expressed succinctly as “Quick, Cheap, Good: Pick two!”. Waterfall methodologies seek to maximise the probability of successful outcomes by creating frameworks to make sure the three choices are in equilibrium before any significant effort, or money, is committed to a project.
Waterfall therefore seeks to manage risk and uncertainty from the outset, tracking progress through a series of stages (e.g. analysis, design, build, test, go live) to a predefined goal. This approach delivers projects as a sequential set of activities. However, despite the emergence of sophisticated project management methodologies such as PRINCE2, waterfall has a chequered history, and after each high-profile failure organisations have sought ever more control in an attempt to avoid similar embarrassment. Eventually, projects became scary things, shrouded in a mysterious set of rituals that could only be understood by ordained members of the PRINCE2 priesthood.
The Project Reformation
An alternative approach emerged following the publication of the Manifesto for Agile Software Development in 2001. In contrast to waterfall, agile prides itself on being a flexible approach that tackles planning, design, implementation and testing through a series of short iterative cycles. The manifesto opens with a declaration of its values:
- Individuals and interactions are more valuable than processes and tools
- Working software is more valuable than comprehensive documentation
- Customer collaboration is more valuable than contract negotiation
- Responding to change is more valuable than following a plan
The second half of each of these statements is anathema to the risk-averse methodologies associated with waterfall. Nevertheless, the prospect of speed, freedom and rapid progress has seen agile gain significant traction in the project community. Indeed, in the UK, Agile has gained sufficient momentum to displace PRINCE2 as the default project methodology in many organisations over the last 20 years, particularly when delivering bespoke ‘line of business’ systems.
The Counter-Reformation
Although Agile was principally formulated with software development in mind, its evangelists recognised the need for additional tools and techniques if it was going to become mainstream. These were required to maintain focus, sustain collaboration over an extended period and create the checks and balances to safeguard the project team and its stakeholders. The desire for credibility in these areas led to the emergence of agile methodologies such as Scrum, Kanban, Lean, etc.
Die-hards in the waterfall camp, though, still yearned for the certainty of fixed-scope, fixed-term, and fixed-price. Eventually, sequential and iterative methodologies were synthesised in the 2015 publication of PRINCE2 Agile, a methodology that seeks to combines PRINCE2 management controls with a broad toolset of agile delivery techniques and frameworks. For some, this is the best of both worlds. For others, an incoherent and illusory mishmash of fundamentally opposed attitudes.
The Dilemma
So, how to approach a project when some aspects lend themselves to waterfall, some to agile? Where some requirements are fixed, and clear, but there is also a strong desire to innovate. Choosing one methodology over the other cannot be the answer and despite its promise PRINCE2 Agile still requires proficiency in a baffling array of processes and frameworks that remains the preserve of an anointed few.
We recently encountered this situation with a leading Welsh Housing Association that was looking to replace the accounting system whilst simultaneously exploring opportunities to redesign cross-functional processes between Finance and Operations. The accounting system was in effect an important part of a wider finance transformation programme. This gave us the following dilemma:
- Waterfall was the natural choice for the accounting system due to the pressure to maintain control, support monthly accounting deadlines, handle a mid-year cutover, satisfy external auditors and meet non-negotiable deadlines
- But Agile was more appropriate where the objective was to revolutionise performance across functional boundaries through service innovation and end-to-end process transformation.
The Answer
Focusing only on replacement of finance processes on a like-for-like basis would squander the innovation opportunity. The transformation programme would then require “digging the road up twice” and would lose momentum. We needed a more fluid approach.
After ruling out PRINCE2 Agile, we knew we had to find a way to achieve the desired outcome without compromising either the rigour required for the accounting system replacement or the flexibility required for process innovation. The approach we chose was to incorporate both agile and waterfall work packages, clearly identifiable as such, nested inside fixed deliverables.
In practice, this involved the rapid implementation of a baseline version of Business Central pre-integrated with the client’s Housing Management System (Microsoft Dynamics CRM) which, in turn, enabled the finance team to go “hands-on” in the very early stages of the project. In a radical departure from the conventional approach to finance systems implementations, this allowed finance team members to take an active role in the configuration and design of core accounting processes through a series of “learning sprints”. The benefits quickly became apparent. Firstly, the team was able to understand the transformational potential of moving to a single, unified data architecture as something real and tangible, rather than as something abstract and conceptual. Secondly, the team was able to confidently decide which processes could be made cross-functional as part of the move to Business Central and which could be deferred. Thirdly, and perhaps most importantly of all, much of the thinking has already been done before the next phase of the transformation programme starts and, almost subconsciously, the finance team has embraced cross-functional working as “the new normal”.
This model has enabled us to implement Business Central as a well-defined and tightly controlled project whilst simultaneously allowing the finance team space and time to rethink Repairs, Asset Management and Rent Accounting as cross-functional processes within the client’s broader transformation programme.