Estimates are “mission-critical” for Software Product companies because:

• In the short term, the quality of the development estimate can win or lose a deal
• In the medium term, on successful deals, the quality of the development estimate can drive profit or loss on a deal
• In the long-term, the quality of the estimates drives the evolution of the products – not just what we say we will do in the road maps but what we actually deliver in releases.

Keith Baxter of De-RISK, based in the UK, has developed a simple but effective way of capturing and visualizing the impact of risk on a project estimate.  The approach, called Quality Based Costing (QBC),  and some examples are provided in the following paper on Keith’s website:

Quality Based Costing Paper

QBC is based on the responses to structured questions about each project component which seek to identify four costs for that component:

• Absolute Minimum
• Best guess/realistic estimate
• Contingency added
• Disaster scenario

The approach is then to apply Monte Carlo simulation to derive a probability distribution of the likely cost.  I particularly like an estimation process that includes Monte Carlo simulation because it forces the project manager and organization to make an informed choice about the risk they wish to take on each project.

The May 19, 2010 edition of Computerworld magazine contained a couple of articles about the growing evidence that one of the reactions to tightening budgets in software development has been the increased use of open source software and the associated  “Hidden Snags”

It is clear that one of the attractions of open source is the availability of source code which is critical for integrating open source into the enterprise but this brings problems of its own (“Integration Jams”).  I have written before about the benefits of static code analysis and I really want to stress the value of getting a one-off analysis done of any significant open source code that you bring into your enterprise.  Such a one-off analysis can be delivered as a service and does not require purchasing or learning a new tool.

The SEER estimate by Comparison capability is something that I talk about a lot to company’s who are really struggling because their SWAG approach to estimating projects is neither repeatable nor coherent.  Often, the projects they are failing miserably to estimate professionally are mission-critical either for deliverables to customers or for internal investment (budget) planning.

At some point, these estimate amateurs may need to base their estimates on a repeatable sizing approach (of which Function Points is just one option) but they need to change their ways quickly so I often talk to them about doing estimates by comparison using SEER.

Even if you don’t see yourself buying a tool right now, I recommend investigating this approach so you can attend in real time or review later.  The technique is powerful even if the tool is not right for you.  DCG can also provide the tool as a Service so you can pilot the application or just use it a few times every year for this really important estimates.  For more information on this service, you can contact David Herron at d.herron@davidconsultinggroup.com.

The Estimate by Comparison application has traditionally been used to empower users to develop an understanding of software size; the single most significant driver of development cost, effort, and schedule for software projects.  However, SEER Estimate by Comparison has evolved and can be utilized with all of the SEER solutions to provide an insight into effective definition of scope through a series of project analogies and/or comparisons to a user’s repository of past projects, thus helping users to develop a reliable estimate on a project’s scope even when information is scarce.

SEER Estimate by Comparison further adds capability to the project team when used in a more contemporary manner. It can be manipulated in such a manner that a wide realm of subjective, qualitative alternatives can be evaluated in context of the project as a whole in a robust, repeatable and ultimately measureable manner.

I came across an interesting article in the May/June 2009 issue of IEEE Software by Magne Jurgensen of the Simula Research Laboratory in Norway which touches on the age-old challenge of deciding whether the lowest estimate is actually a realistic estimate.

From his research, Magne has derived 7 recommendations which I have paraphrased below (I suggest they have much validity for internal and external estimates):

1. If price is an important selection criteria, dont invite too many bidders if your requirements specification is incomplete.
2. Avoid using price as an important selection criteria if your ability to assess provider competence is low.
3. Ensure sufficient competence in assessing provider competence by hiring external experts if necessary.
4. When selecting bidders, compare the bids with your own independent cost estimate of the average of all bids.
5. Let providers collect sufficient information to help them analyze the projects complexity.
6. Don’t include potentially misleading information that could affect the estimation process (especially your budget expectations).
7. Be aware that a negotiation phase after receipt of bids may encourage even greater over-optimism.

Of course, “over-optimism” is Magne’s polite way of saying “stupidity.”  If these recommendations resonate with you, I suggest that you should talk to us here at DCG about getting a professional, objective view of your estimates and/or your estimation process.

I must admit that I rarely use the Standish Chaos figures on what percentage of software projects fail in respect of their original estimates of cost, time and functionality.  This may seem odd.  Surely, they support the business case for using consultants who specialize in improving software development?  Well yes but they never quite correlate with my experience and, perhaps as a result, the latest percentage never quite lodges in my memory.

My own  disquiet has always been based on the feeling that the statistics derived from any such survey may be more correlated to the type of questions asked than the reality on the ground.  For example, when a project finally reaches that point where the customers admit that they really didn’t know what they wanted when the project started, is that a project failure?  Probably it counts as a “yes” but I would argue that it was a controllable fault of the software developers or their process. Maybe that’s where the Standish publishers are going with it.

Anyway, I was pleased to see my concerns validated in an article by J. Laurenz Eveleens and Chris Verhoef in the January/February 2010 issue of IEEE Software.  With significant statistical analysis using reputable sources, the authors identify the following four problems with the Standish Group figures in their Chaos reports:

1. “They’re misleading because they’re solely based on estimation accuracy of cost time and functionality.”
2. The Standish estimation accuracy definitions are not sound.
3. If the Standish definitions are used to drive projects, they may cause large cost and time over-estimations.
4. The Standish figures cannot be extrapolated across organizations because large forecasting biases exist in any given organization which make extrapolation, in the authors words, “meaningless.”

Being able to estimate software development more efficiently and effectively is both possible and practical. The following key points will guide you to a successful outcome.

•    We need to recognize estimating as a problem and a potentially costly problem at that. Not until we fully understand that improper estimating is a potential barrier will we be able to consistently and successfully deliver software.

•    The way we think about estimating should be reframed in the context of managing expectations based upon the best information available at the time. Estimating is not a crystal ball used to predict the future.

•    We can no longer afford to compromise on what we need regarding the input components that make up a successful estimating model. It will require some investment of time and resources but the payback will be well worth the investment.

•    And above all, don’t overly complicate your estimating model. Adopt practices such as FP Lite to generate size information that is statistically accurate enough for the job of early estimating. Collect the baseline data you need and compute your own internal delivery rates of performance.

At DCG, when we work with clients we often conclude that spreadsheets may be the most appropriate way to manage a simple metrics program.  However, I was reminded this week (based on an incident at a client which caused me to look up the appropriate section of “Competing on Analytics” by Thomas H. Davenport and  Jeanne G. Harris) that there are a couple of common problems with basing your metrics program on spreadsheet repositories:

• Errors - Raymond Panko wrote a much referenced article back in 1998 based on his research which suggested that 20-40% of user-created spreadsheets contain errors.  The more spreadsheets, the more errors.
• Multiple versions of the truth - too often there is not a single, multi-user spreadsheet but multiple versions being spread like a virus across an organization by email, each with its own few unique data values.
• Unintended uses - the data in spreadsheets are very difficult to keep under control when key data elements are changed to meet new needs.  The knock-on effect on linked data can be catastrophic but not necessarily visible.

Worst of all, its almost impossible to debug or do a data integrity check on even a mildly complex spreadsheet.

The lesson - use spreadsheets as much and as often as you like but do not attempt to build a serious metrics or estimating application on them if you want it to have a useful life in your organization of more than one year!

One of the annual delights of being measurement and process improvement consultants for software development is the annual budget process.  We get involved in different ways every year.  Sometimes we are asked for independent validation of estimates on big or important projects, sometimes we are asked to help “size” (usually function point count) the requirements to validate them against capacity and sometimes we are asked to help develop, and/or facilitate (referee), a process for prioritizing the “too many” imperative projects against the “too few” resources available.

It is the “project prioritization” challenge that I want to pick up on today with a simple project classification idea that might help those of you who don’t have the final project portfolio for next year cast in stone yet.  I suggest that you could try to classify your candidate projects into one of the following three classes:

• Value Neutral Projects (VNPs) - these projects do not change the perceived value of the application or product to the end-user but may still be necessary.  Examples might be certain architectural changes, additional platforms supported or defect fixes.
• Value Add Projects (VAPs) - these projects add value for some users.  Examples might be new 3rd party interfaces, incremental functionality to some components, new foreign language capabilities or new reports.
• Value Multiplier Projects (VMPs) - these projects add value for all users.  Examples might be a new user interface, a new component or increased performance that leads to smaller hardware requirements.

Clearly, the project portfolio for any given year might include project from all three classes but the majority of the spend should be targeted at VMPs.  Of course, if you don’t have any VMPs in your planned portfolio, you might have to go back to the drawing board!

The same project may be classified in different ways in different circumstances so the input of the business continues to be crucial in this process.

Can anyone think of other examples of VNPs, VAPs or VMP’s?

To facilitate ongoing discussion and give everyone a forum to share their experiences, I have set up two groups on LinkedIn:

DCG Software Performance and Quality Benchmarking

DCG Software Measurement Experiences Exchange

I have launched these groups with an initial question on each but the real idea is for you to add your own experiences and, most importantly, your own questions.