The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

Interesting survey result in the June 2010 edition of CIO Magazine.  90% of respondents to a survey by Forrester Research reported that customer experience was “critical” or “very important” to the corporate strategy.  No surprise there.  The responses on the obstacles to improving customer experience were very revealing:

• 53% (over half) reported “No clear strategy for improving the customers experience
• 50% expressed a “Need for customer-experience management processes.”
• 49% are experiencing “insufficient cooperation across groups”
• 32% report a “lack of understanding about customers”

If any of this sounds familiar, you need help!  Give us a call or work it out yourselves.    All of these are solvable but the third bullet rings particular true across the business-IT interface.  We run across this all the time with our clients and it improvements in breaking down internal barriers DO benefit customers!

I see in a short report in the June 1, 2010 edition of CIO magazine that Forrester Research have coined the term “Agile BI” to describe an approach to business intelligence implementations that calls for incrementally delivering products versus a big-bang approach.  Having posted our Agile Approach to Consulting some months ago, it is nice to be able to say “Welcome aboard” to Forrester!

Hakan Erdogmus is the Editor-in-chief of the IEEE Software magazine.  In the May/June 2010 issue of the magazine, Hakan has written an impassioned plea for the importance of using evidence as the basis for software development decisions, especially when deciding on the adoption of software engineering ideas.

I totally support Hakan’s position.  Too many of us forget the “science” part of computer science and the “engineering” part of software engineering.  The testing of hypotheses in science and the use of prototypes in engineering are fundamental to those disciplines.  Yet in software development, we rely on methods more akin to building a bridge by adding one brick at a time on either side of the river and hoping that it will meet in the middle and be strong enough (while ignoring those budgets, deadlines, materials and colleagues that get swept away by the river during construction).

For deciding on the adoption of software engineering ideas, Hakan categorizes three types of evidence in increasing order of rigor:

• Feasibility Check - While it is the weakest of the three, a feasibility check should be able to:

- quickly evaluate major advantages, risks, limitations
- roughly estimate the size of the problem
- assess the novelty of the solution
- test the cost effectiveness
- help to decide if the idea is worth exploring further and under what controlled but real-world circumstances

• Anecdotal Evidence - This is effectively “crowd sourcing” your evidence gathering in support of a new idea with all of the advantages (the pioneers are the ones with all the arrows in their backs) and disadvantages (has the author really tested the idea under similar circumstances as you?  Do they have a vested interest in supporting the idea?) of that approach.  Nonetheless, external data points ARE valuable and a significant step up on a purely internal feasibility check.
• Systematic Evidence - Of course, researched and documented evidence of the merits of an idea is the best type of evidence but it is difficult to find or produce if the ideas are new.  Mature organizations can make this type of evidence much easier to discover and use by adopting a methodical approach to capturing lessons learned from projects.

Hakan makes the excellent point that the SIZE of the decision or idea should dictate the threshold of the quality of evidence required.

We often start our CMMI, measurement, estimating or other process improvement engagements with clients with a “gap analysis.”  Probably all of the readers of this blog have a pretty good idea what that means and so it is an easy entry point.

But I don’t like the term “gap analysis.” Why?  Because it carries with it the implication that it is an exercise in finding out everything that our client is doing wrong.  I prefer the term “opportunity analysis.”

In the April 2010 edition of SoftwareTech magazine, David Herron writes a very good perspective on this issue.  My own simple perspective is that every software development organization does some things well.  It must do otherwise it would not survive.  Hence, every software development organization must know how to do things well.  Finding the things the client does well is just as important as finding the things it does not do well because the pocket of excellence provide patterns for the improvement of the weak areas.

Further, identify the pockets of excellence is important because any changes must be carefully designed to ensure that the excellence is not compromised.  This is a classic mistake that we see too often - process zealots are allowed to bring everything the company does to the same “average” level.  This usually represents significant improvement is key problems areas but it can sometimes stifle the areas that were previously excellent and were making the company “special” to their customers.

Finally, for those of you who have time to go to read David’s article, I would also recommend the article by Capers Jones in the same edition, “Software Quality and Software Economics.”  Capers always delivers thought-provoking ideas and plenty of data points.

We have been using CMMI as a framework for our consulting initiatives for many years.  In recent years, it has become clearer that you can get a lot of value out of this approach without going all the way to a full accreditation.

In a recent (Jan/Feb 2010) article in CrossTalk, Jeffrey Dutton captured the essence of using CMMI in this way.  He puts forward three driving principles:

1. Focus on Business Issues and Performance Goals
2. Involved Leadership and Process Ownership by Process “Doers”
3. Improvements should be made at the Speed of Business

Jeffrey goes on to explain that the CMMI framework together withe three principles can readily accommodate different approaches like CMMI itself, Lean, Six Sigma and ITIL.

We are seeing more and more interest in this “multi-model” approach to get more value more quickly out of process improvement initiatives.

I must admit that I rarely use the Standish Chaos figures on what percentage of software projects fail in respect of their original estimates of cost, time and functionality.  This may seem odd.  Surely, they support the business case for using consultants who specialize in improving software development?  Well yes but they never quite correlate with my experience and, perhaps as a result, the latest percentage never quite lodges in my memory.

My own  disquiet has always been based on the feeling that the statistics derived from any such survey may be more correlated to the type of questions asked than the reality on the ground.  For example, when a project finally reaches that point where the customers admit that they really didn’t know what they wanted when the project started, is that a project failure?  Probably it counts as a “yes” but I would argue that it was a controllable fault of the software developers or their process. Maybe that’s where the Standish publishers are going with it.

Anyway, I was pleased to see my concerns validated in an article by J. Laurenz Eveleens and Chris Verhoef in the January/February 2010 issue of IEEE Software.  With significant statistical analysis using reputable sources, the authors identify the following four problems with the Standish Group figures in their Chaos reports:

1. “They’re misleading because they’re solely based on estimation accuracy of cost time and functionality.”
2. The Standish estimation accuracy definitions are not sound.
3. If the Standish definitions are used to drive projects, they may cause large cost and time over-estimations.
4. The Standish figures cannot be extrapolated across organizations because large forecasting biases exist in any given organization which make extrapolation, in the authors words, “meaningless.”

Why is it important to use fact-based decision making in software development?  Too many executives are prepared to treat software development as too complex to measure (or manage?). Instead, they operate in a state of continuous dissatisfaction based on occasional unsuccessful forays into the software development team which rarely yield significant change.  This is both uncomfortable and, potentially, illegal.  For example, as Davenport and Harris remind us in “Competing on Analytics,” the Sarbannes-Oxley Act of 2002 requires executives and other users of corporate data to demonstrate that their decisions are based on trustworthy, meaningful, authoritative and accurate data.  The act also requires executives to sign off that the data provides a clear picture of the business, major trends, risks and opportunities.

Can you or your colleagues really say this about your software development metrics?  Do you need help?

Consider the following 5 questions that you should ask yourself:

1. Is our software development data relevant?
2. Do we know where it came from (is it auditable)?
3. Do we have enough data (or too much)?
4. How can the data be made more accurate and valuable for analysis?
5. What rules and processes do we have in place to manage our software development metrics dat from its creation to its retirement?

At DCG, when we work with clients we often conclude that spreadsheets may be the most appropriate way to manage a simple metrics program.  However, I was reminded this week (based on an incident at a client which caused me to look up the appropriate section of “Competing on Analytics” by Thomas H. Davenport and  Jeanne G. Harris) that there are a couple of common problems with basing your metrics program on spreadsheet repositories:

• Errors - Raymond Panko wrote a much referenced article back in 1998 based on his research which suggested that 20-40% of user-created spreadsheets contain errors.  The more spreadsheets, the more errors.
• Multiple versions of the truth - too often there is not a single, multi-user spreadsheet but multiple versions being spread like a virus across an organization by email, each with its own few unique data values.
• Unintended uses - the data in spreadsheets are very difficult to keep under control when key data elements are changed to meet new needs.  The knock-on effect on linked data can be catastrophic but not necessarily visible.

Worst of all, its almost impossible to debug or do a data integrity check on even a mildly complex spreadsheet.

The lesson - use spreadsheets as much and as often as you like but do not attempt to build a serious metrics or estimating application on them if you want it to have a useful life in your organization of more than one year!

One of the annual delights of being measurement and process improvement consultants for software development is the annual budget process.  We get involved in different ways every year.  Sometimes we are asked for independent validation of estimates on big or important projects, sometimes we are asked to help “size” (usually function point count) the requirements to validate them against capacity and sometimes we are asked to help develop, and/or facilitate (referee), a process for prioritizing the “too many” imperative projects against the “too few” resources available.

It is the “project prioritization” challenge that I want to pick up on today with a simple project classification idea that might help those of you who don’t have the final project portfolio for next year cast in stone yet.  I suggest that you could try to classify your candidate projects into one of the following three classes:

• Value Neutral Projects (VNPs) - these projects do not change the perceived value of the application or product to the end-user but may still be necessary.  Examples might be certain architectural changes, additional platforms supported or defect fixes.
• Value Add Projects (VAPs) - these projects add value for some users.  Examples might be new 3rd party interfaces, incremental functionality to some components, new foreign language capabilities or new reports.
• Value Multiplier Projects (VMPs) - these projects add value for all users.  Examples might be a new user interface, a new component or increased performance that leads to smaller hardware requirements.

Clearly, the project portfolio for any given year might include project from all three classes but the majority of the spend should be targeted at VMPs.  Of course, if you don’t have any VMPs in your planned portfolio, you might have to go back to the drawing board!

The same project may be classified in different ways in different circumstances so the input of the business continues to be crucial in this process.

Can anyone think of other examples of VNPs, VAPs or VMP’s?