Let's start with a bold proposition: Waterfall doesn't work for BI. Agile doesn't work for BI.
Why doesn't Waterfall work?
On the face of it, the reasons for Waterfall not working for BI are easy to tease out for those familiar with the field. Waterfall requires fixed inputs and outputs with a high degree of certainty over requirements. This works (to a limited degree) for some IT projects as it is possible to define this. Infrastructure is a good example. Even in application development there is often a reasonable degree of certainty in terms of requirements and desired functionality. In a BI project these are often easy to define to start with, but rapidly unwind as requirements change or are clarified (and often worked out as part of the projects progression) and the user realises what they asked for is either unattainable or was addressing the wrong question in the first place.
...but surely Agile fixes this?
Well, no. It should as in theory it adapts better to rapid change... but when Agile is attempted it rapidly descends into chaos due to a lack of deliverables coming out of the sprints, project owners and managers get nervous and Waterfall rears its ugly head again in an attempt to provide an illusion of certainty and control. Agile methods should help deal with the level of change that BI projects seem to inherently contain, but somehow struggle. Much of this is down to a common problem with Agile and Waterfall.
Is BI development the same as software development?
Here's the rub. In Larissa Moss's approach "Extreme Scoping: An Agile Approach to Enterprise Data Warehousing and Business Intelligence" she calls out that one of the biggest issues with trying to get Agile with BI projects is the data integration component. While in a BI project the output artifacts are not unlike traditional coding - reports, ETL jobs and the like - the input artifact - data - is nothing like an easily definable requirement. If your CRM data and your Web Analytics data need to be inter-meshed, there's a whole bunch of groundwork to do to uncover that. Common keys to be found, business rules to be teased out, access to be obtained. That may be as simple as looking up two common identifiers - in which case, good luck - but usually its far more complex.
This is what kills both Waterfall and Agile delivery. In estimation it's assumed that the coding is the work. When in reality the coding is really the expression of a data exercise which could be a matter of both interpretation and integration. In software development the coding is the expression of an idea - e.g. "I want to be able to use a shopping cart plug in to sell my product". In BI development this an expression of the data, which is bounded by the vagaries of the quality of the data, how it should be interpreted and how it interacts with all your other data. Resolving this extends far beyond calling an ETL job complex and assigning it 5 days in the project plan.
So what do we do next?
Whatever method we adopt, we have to listen to Larissa's message - that we cannot run a BI project the same way we run a coding project. Time needs to be allowed resolve the data issues - up front - before any estimate of difficulty is assigned. After all, once the problem is understood, defining a report as simple or complex is easy enough - but until the problem is understood you may as well be estimating with a dartboard.
Check out Larissa's book here: