Estimating Project Costs
Even though we've been estimating project costs since the beginning of software, it's remarkable to me how much confusion remains. I think the root cause of this confusion is that management needs cost information very early in the process, yet engineering doesn't have the information it needs to do a reasonably accurate estimate until much later in the process. The result is either premature estimates that prove wildly inaccurate, or surprises because people had different assumptions all along and when the accurate estimate comes in, management has a severe case of sticker shock.
Here's the process that I advocate. It is intertwined with the product development process I advocate, but it can be applied in most.
Recall that I strongly encourage all product efforts to start with a 10-question opportunity assessment (you can read more about these questions at www.svpg.com/blog/files/assessing_product_opportunities.html). This is what the management team uses to decide whether this is a problem worth trying to solve. There's no solution yet, just a problem, and an opportunity. But for most teams there's a need at this stage for a very preliminary estimate of scoping. Of course, since there's no solution spelled out at this stage, it's going to be very much a SWAG, which is why I recommend that the estimates at this stage are limited to "Small," "Medium," or "Large." It's usually fairly clear at this granularity what the cost will be, although there will still be occasional surprises.
If the opportunity looks good relative to the estimated cost, the management will likely allow the project to proceed to defining the solution (the spec). It is at this stage that the product solution is spelled out in detail, ideally with a high-fidelity prototype that you validate against real target users with prototype testing (see what I mean by a prototype-based spec at www.svpg.com/blog/files/revisiting_the_product_spec.html).
All through the process of coming up with the solution, the product manager and designer should be including a member of engineering to evaluate the various options and estimate costs for the different alternatives. That information is then considered by the product manager and designer and the product is adjusted as needed. But at the end of the spec process, there should be a very detailed and high-confidence estimate based on a detailed description of the product proposed to be built.
At this stage, the management team has a detailed product spec, and a corresponding high-confidence cost estimate, and they should be able to make a final decision on whether to proceed to build this product or not. It may be that the solution turned out to be more complex and expensive than they thought, or they might not like the actual solution, but if they do proceed the whole product team knows the cost and the product they'll get for that investment.
So to summarize, I'm suggesting a preliminary estimate at opportunity assessment time, followed by a detailed estimate that accompanies the final product spec.
As an aside, I also believe in tracking the cost of changes to that product spec and schedule ("churn") so that the organization understands the cost of change, and can improve by always trying to minimize this churn.