Waterfall project management is a serial approach to the phases of a project. Proponents of waterfall methodologies hold it’s best to plan: gather requirements, design the solution, develop it, test it, deploy and maintain it in discrete steps. Critics maintain it is impossible to know all of the requirements prior to design or development.
by Peter Kemp / Paul Smith – Adapted from Paul Smith’s work at wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=10633070
Agile project management is an iterative process whereby developers focus on deliverables in short deliverable cycles. Proponents of Agile methodologies believe these practices more accurately reflect the complex nature of software development and place more decision-making with the developer. Critics (accurately) point out that Agile projects are fluid in schedule and impossible to predict – especially when it comes to time and money.
Which is Better?
The answer is: it depends. “What does it depend on, Andy? “ I’m glad you asked!
First, we need to accept that there are no pure Agile (iterative) or Waterfall projects. Every project is a blend. But every project also leans towards iterative or waterfall.
What do You Prefer for Managing Business Intelligence Projects?
I lean towards Scrum, an Agile project management methodology for managing business intelligence projects.
By Lakeworks – Own work, GFDL, https://commons.wikimedia.org/w/index.php?curid=3526338
The critics of Agile are correct: there is no way to predict the end date and, therefore, the cost of a project. From a business perspective, stakeholders often feel they are being asked to continue writing checks without knowing how many more checks they will have to write, or for how long, or for how much. The critics of Waterfall are also correct: some – most, in fact – software development projects are simply inestimable. I’ve written about software estimation in the past, sometimes in the context of project management.
Is there some way to limit the business risk? Yes there is…
Risk-Balanced Project Management
Remember, no project is purely Agile or Waterfall. I’ve been practicing a risk-balanced methodology for decades. How does it work? I combine the best of both worlds: Agile and Waterfall. Here’s an over-simplified explanation of how we deliver projects at Enterprise Data & Analytics: We treat each Scrum Sprint (iteration) as a small Waterfall project.
Sprints remain focused on deliverables. They must, or they’re not sprints. The developer makes the call about what’s in and what’s out in a given sprint. This works especially well with teams of developers who can practice Kanban or some other Theory-of-Constraints-based approach to problem-solving as a team.
Are There Daily Standups?
Yep. It’s not Scrum without daily standups. It’s important for the stakeholders to attend these meetings to maintain contact with the progress of the work. A Scrum Task Board – either virtual or physical – is a requirement. It can provide some feedback to stakeholders, but there is no substitute for stakeholders attending the daily standup meeting.
Why is it important for stakeholders to attend standups? Risks – time- and money-costing issues – usually surface in the standup meeting first. I measure the risk-awareness of a stakeholder by their standup attendance. You may have read that last sentence and thought, “That’s not fair, Andy!” Maybe not. Experience has taught me that it’s accurate, though.
Where’s the Waterfall?
We usually run 30-day sprints. We shorten the Waterfall cycle to 30-days and limit it to the deliverables identified for the sprint. We’ll do a couple days of discovery followed by a couple days of design. Development starts during design in business intelligence projects. Data integration is a large component in business intelligence – often the largest component. Data integration is also a bottleneck for most of the downstream parts of a business intelligence project. Testing (Validation) is tightly-coupled to development, and is vital. If you get nothing else out of reading this post, please remember this:
Deliver quality late, no one remembers.
Deliver junk on time, no one forgets.
We can run shorter sprints but my experience shows this actually delays completion of the project. Why? We need time to manage the (inevitable) issues that surface during a sprint.
How Does This Approach Mitigate Risk?
Believe it or not, business priorities shift. New information becomes available after the project starts. Maybe a competitor reveals they are more competitive than stakeholders believed. Maybe more marketing information shows a shift in customer demand. Maybe another internal enterprise project takes priority over the business intelligence project. Any number of market and business conditions can shift the necessity, priority, or direction of a business intelligence project.
Consider the impact of “re-Waterfall-ing” a business intelligence project during design, development, or testing. I’ve been there. It’s expensive for both the developers and business. Taking a phased approach allows an agile (double entendre intended) shift or graceful pause to the business intelligence project.
Are You Better Able to Estimate Project Completion?
Yes and no.
There are software and business physics in play. Laws that cannot be broken; principles that simply apply whether we like them or not. A phased approach allows us to place bounds around the unknown(s). This is, I believe, the most economical and the most reality-based methodology for delivering business intelligence solutions.
When I write economical, I mean experience informs me this approach costs the business less money than other approaches while delivering more with greater efficiency. How? Waterfall approaches often involve Change Orders with associated charges (do you remember that time a change order was free? Me neither). Some consultants win with the lowest bid and make up the difference with change orders. My credibility and delighting my customer is worth more to me.
When I write reality-based, I mean few software projects are completed when projected. On-time projects happen. But it’s rare. Why? Well, it’s either because a) all software developers and consultants are pathological liars; or b) software is inherently inestimable. I vote for b. When someone asks a software developer, “When will you be able to complete this task?” they are most often asking, “How long will it take you to figure out this completely new thing you’re tasked with figuring out?” I hear this question posed in many contexts. Sometimes I get asked the same question different ways. I’m not a fan of that kind of questioning, but the reason it works is bias.
Sometimes the most honest answer is, “I don’t know.” It’s ok to not know. It’s not ok to not know and not know how to find out; so when I don’t know, I honestly respond with, “I don’t know but I can figure it out.”
Conclusion
The goal of a phased approach is to balance risk for consultants, developers, project managers, stakeholders, and the business. A phased approach limits risk while preserving the options of all engaged. Should things shift, stakeholders can change the direction, priority, or (in extreme cases) the existence of the project in response; with minimal technical and financial impact to all involved.
A phased approach is a great way to mitigate risk for all parties. It works well for Enterprise Data & Analytics and our customers.
:{>
Need help implementing an SSIS solution?
Contact Enterprise Data & Analytics today!
Related Posts:
Value
Probably my biggest concern w/ the Waterfall approach is that I typically see requirements set in stone for lengthy projects. By the time the project is done, the requirements are just out of date and injecting change into that process is painful. Conversely, Agile has issues with scope creep and many projects that never feel “done” as a result. I like the Agile idea better, but need some safeguards to prevent projects from dragging on and on.
Great points, Peter!
Do you also take a blended approach?
:{>
if one is building an RDBMS based application, then waterfall is appropriate, since one must, at the least, decide on the principle tables and their natural unique (primary) key. if you say you can’t do that up front, then your better off just being a coding cowboy and build your Frankenstein application of random parts.
if one starts with an organic normal form schema, then adding/changing/deleting parts of the schema will only affect the parts of the code base that care about those data. in the end, if one starts with an organic normal form barebones schema, you start out with some waterfall (the up front tough part) and can then be agile in amending said schema.
Nice article – and I really like this – “Deliver quality late, no one remembers. Deliver junk on time, no one forgets.” 🙂
I’ve had mixed outcomes with agile in particular with “stateful systems” like complex relational databases, especially OLTP ones.
As long as you accept you may need to rework something you only just wrote in the prior cycle then its OK vs the trade off to major forward planning in waterfall.
In general yes agree small 30 day iterations of waterfall can work well – as long as you can pull apart a larger more complex project into strips of 30 day deliverables – in most cases this works pretty well.
Thanks Robert and Tom!
Tom, you wrote "…you may need to rework something you only just wrote in the prior cycle…" Doesn’t that happen anyway? I’ve been working with databases for a couple decades and I’ve yet to get the design right on the first try. In my opinion, Agile recognizes that no one gets it right on the first try whether they’re designing databases or applications. For that reason alone, I find Agile more closely resembles the real world.
:{>