The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

In the June 28, 2010 print edition of Information Week, a long-standing friend of DCG’s, Capers Jones, identifies three keys to ensuring software quality:

1. Defect Potentials - Make sure they stay below three per function point
2. Defect Removal Efficiency - Keep it above 95%
3. Quality Metrics - Apply them from requirements through maintenance

For those of us who, like me, need an occasional reminder on definitions, Defect Potentials are the probable numbers of defects that will be found in the various stages of the software development life cycle.  Defect Removal Efficiency is the percentage of the defect potentials that will be removed (or actually are removed) before an application or project is delivered to users.

Which brings me to a conversation that I had today with a client team who are seeing the numbers of defects increasing on a monthly basis this year.  The explanation is that resources are being diverted from the maintenance teams so less bugs are getting fixed.  The client team has been told that all will be well when resources and budget ad diverted back to the bug fixers.

“Aaargh!” was my thoughtful and considered response.  The client team asked what they could do instead to improve the defect numbers.  I told them that the money =resources=time) needs to be invested further up the supply chain where it can make a much bigger proportional difference to quality.  Peer reviews of requirements, designs, code and test scripts are the most cost effective and quickest way for an organization to change quality outcomes.  To be fair, the client team then remembered that some development teams do conduct code reviews (”… and the quality is pretty good”) but the other teams “… don’t have time.”  This is a classic misunderstanding of the value of time spent in different phases of the software development life cycle.

So I get home this evening and start to catch up on my reading.  The first article I pick up is the Capers Jones article and my conversation earlier today jumps fresh into my mind when I see that Capers has included the following table:

Software Defect Removal Efficiency Ranges:

Activity Range

Formal Requirement Inspections                    50% - 90%

Formal Design Inspections                             45% - 85%

Formal Code inspections                                45% - 85%

Automated Static Analysis                              55% - 90%

Automated Unit Test                                       20% - 60%

Regression Test                                               15% - 30%

Integration Test                                               25% - 40%

Performance Test                                             20% - 40%

System Test                                                     25% - 55%

Estimates are “mission-critical” for Software Product companies because:

• In the short term, the quality of the development estimate can win or lose a deal
• In the medium term, on successful deals, the quality of the development estimate can drive profit or loss on a deal
• In the long-term, the quality of the estimates drives the evolution of the products – not just what we say we will do in the road maps but what we actually deliver in releases.

The SEER estimate by Comparison capability is something that I talk about a lot to company’s who are really struggling because their SWAG approach to estimating projects is neither repeatable nor coherent.  Often, the projects they are failing miserably to estimate professionally are mission-critical either for deliverables to customers or for internal investment (budget) planning.

At some point, these estimate amateurs may need to base their estimates on a repeatable sizing approach (of which Function Points is just one option) but they need to change their ways quickly so I often talk to them about doing estimates by comparison using SEER.

Even if you don’t see yourself buying a tool right now, I recommend investigating this approach so you can attend in real time or review later.  The technique is powerful even if the tool is not right for you.  DCG can also provide the tool as a Service so you can pilot the application or just use it a few times every year for this really important estimates.  For more information on this service, you can contact David Herron at d.herron@davidconsultinggroup.com.

The Estimate by Comparison application has traditionally been used to empower users to develop an understanding of software size; the single most significant driver of development cost, effort, and schedule for software projects.  However, SEER Estimate by Comparison has evolved and can be utilized with all of the SEER solutions to provide an insight into effective definition of scope through a series of project analogies and/or comparisons to a user’s repository of past projects, thus helping users to develop a reliable estimate on a project’s scope even when information is scarce.

SEER Estimate by Comparison further adds capability to the project team when used in a more contemporary manner. It can be manipulated in such a manner that a wide realm of subjective, qualitative alternatives can be evaluated in context of the project as a whole in a robust, repeatable and ultimately measureable manner.

In the April 26, 2010 print edition of Informationweek, the Dr Dobb’s Report contained a an article by Sid Sidner of ACI Worldwide describing his teams choice and implementation of a Static Code Analysis tool.  ACI looked at:

• AdaCore’s CodePeer
• Coverity’s Prevent
• GrammaTech’s CodeSonar
• Greenhill’s DoubleCheck
• Klocwork’s Insight
• Lattix’s LDM
• Microsoft’s StyleCop
• Ounce Labs’ Ounce Core

You can read Sid’s comments on each of these tools in his article.

The article is a very good description of how to integrate a software analysis tool into the development environment. Some engineering shops like ACI are a little bit ahead of IT organizations in building quality and security into their products.

Sid offers the following “Tips for Success”:

• Define an “initial issue” policy - what will you do with the code issues that identified by the first analysis run?
• Get the global mechanics working - many of the tools require license managers and centralized result servers
• Attack one product at a time
• Identify SMEs - those product experts who will also be experts in the tool
• Train the SMEs
• Work with the SMEs
• Train the developers
• Perform initial analysis on existing code and defer all issues - this is Sid’s suggested choice of “Initial Issue” policy.  Note that he suggested to defer not ignore.
• Deliver help from SME’s to developers as required
• Run the build anlaysis often
• Review deferred issues

I discussed this article with Lev Lesokin of CAST Software who have their own static analysis application, the Application Intelligence Platform (AIP).  Lev highlighted a couple of considerations which Sid’s paper did not cover:

1. ACI probably do a very good job building quality into their code at a module level, but they should also look at quality at a broad architecture level. Interactions between modules, between distributed components, and most importantly between the database and the code base. Some of the most risky quality issues actually reside in that application level and they are the hardest to identify by code walk-throughs or localized static analysis.
2. When analyzing your software, there’s an opportunity to also use that to introduce measurement. Sid could put some metrics in place to make sure his teams are getting better at building quality and security in. Also, risk metrics can be used to make go/no-go decisions at stage gates. This might be a point for his VP of Engineering, though. He/she can introduce productivity measures and drive the organization towards continuous improvement.

These two uses of software analysis and measurement products might not be as relevant for ACI, because much of the software they build is monolithic products that they ship and don’t see in a distributed environment until they implement at a customer. In an IT organization you will have many moving parts, connected through a services layer or various other interfaces. This is where app-level quality issues become much more present. Also, getting apples-to-apples measures across development teams in Java, .NET, DB stored procs, etc., is much more challenging. Once ACI’s customers integrate their newly acquired payments system into their bank’s accounts and cash management systems, they might consider an architecture-level analysis to determine how structurally strong their environment is in terms of stability, performance and security.

“AIE” is the last in the series of four metrics for measuring the business value of IT that I introduced in my posting of July 7, 2009.  The first three in the series were the “Business Value Index,” “Total Economic Impact” (TEI) and “Val IT.”

AIE is considered to be the most rigorous of these four approaches but it has not gained widespread use.  With its mathematical, statistical and economic underpinnings, AIE provides investment decision-makers with a high degree of confidence in its results but there is a steep learning curve and it requires significant expertise.  Key features of AIE include:

• Rigorous “Unit of Measure” definitions even for such normally subjective elements as customer satisfaction.
• Systematic uncertainty analysis - quantification of the risk of an IT investment in such a way that it can be directly compared with a non-IT investment.
• Calculation of the economic value of information - AIE works on the logic that information reduces uncertainty, less uncertainty improves decisions, better decisions result in more effective actions and effective actions improve profit or mission results.  AIE includes a mathematical framework for calculating an pricing all this in dollars!
• IT investments as an investment portfolio - this is perhaps obvious from the previous three features but the rigor of the previous three features allows the IT investments to be analyzed alongside, and compared using, the same tools as those used extensively for non-IT investments.

Here at DCG, we are unapologetic about our advocacy of function points as the best currently available, if not ideal, way to measure the (functional) size of software for the purposes of measurement, estimation and management of software projects and applications.  That said, we often implement non-function point solutions for our clients for various reasons.  One area where I believe that a non-function point solution can be valid is in the area of software rework or maintenance.  My theoretical basis for this lies in the six sigma philosophy of “possible errors.” Put simply, it seems to make sense that the number of bugs in a piece of software code could be broadly proportional to the number of opportunities there are to make mistakes which is the number of characters in the code or approximately the number of lines of code.

Edmund P. Morozoff of Medtronic has written a good article in the January/February 2010 issue of IEEE Software that makes this point in more detail (and even includes a nod in the direction of function points).  In particular, the article looks at the key issue of lines of coded added, deleted or modified.

I came across an interesting article in the May/June 2009 issue of IEEE Software by Magne Jurgensen of the Simula Research Laboratory in Norway which touches on the age-old challenge of deciding whether the lowest estimate is actually a realistic estimate.

From his research, Magne has derived 7 recommendations which I have paraphrased below (I suggest they have much validity for internal and external estimates):

1. If price is an important selection criteria, dont invite too many bidders if your requirements specification is incomplete.
2. Avoid using price as an important selection criteria if your ability to assess provider competence is low.
3. Ensure sufficient competence in assessing provider competence by hiring external experts if necessary.
4. When selecting bidders, compare the bids with your own independent cost estimate of the average of all bids.
5. Let providers collect sufficient information to help them analyze the projects complexity.
6. Don’t include potentially misleading information that could affect the estimation process (especially your budget expectations).
7. Be aware that a negotiation phase after receipt of bids may encourage even greater over-optimism.

Of course, “over-optimism” is Magne’s polite way of saying “stupidity.”  If these recommendations resonate with you, I suggest that you should talk to us here at DCG about getting a professional, objective view of your estimates and/or your estimation process.

There are a number of issues associated with allowing your staff to use Free and Open Source Software (FOSS).  There are more issues associated with allowing your software developers to incorporate FOSS in your projects and applications especially if you are in the business of reselling such software as part of a product or service.

High on the list of challenges in both cases are the license implications.  Equally high on the list must be concerns about the code quality of the FOSS.  When it comes to validating the quality of FOSS, there is a simple strategy that could allow you to have more confidence in FOSS than in the off-the shelf software packages that your organization probably uses.

How?  Well, the nice thing about FOSS is that you actually have access to the source code.  This means that you can run it through a source code analyzer such as those available from CAST Software and their competitors.  You can reasonably expect that a good source code analyzer will not only report on the efficiency and quality of the code but also on any potential security threats.  Using an analyzer in this way will provide a layer of certainty that you do not have over the packaged or COTS software you buy without access to the source code.

The business case?  The time saved in bringing in FOSS solutions over the time taken to write and evolve them yourself should cover the cost of the analysis tool.  If you use a lot of FOSS then it makes sense to invest in buying the analysis software yourself.  If you use FOSS only occasionally then DCG can help you by performing one-off analysis as a service using the CAST Software tool whenever you need it.

I must admit that I rarely use the Standish Chaos figures on what percentage of software projects fail in respect of their original estimates of cost, time and functionality.  This may seem odd.  Surely, they support the business case for using consultants who specialize in improving software development?  Well yes but they never quite correlate with my experience and, perhaps as a result, the latest percentage never quite lodges in my memory.

My own  disquiet has always been based on the feeling that the statistics derived from any such survey may be more correlated to the type of questions asked than the reality on the ground.  For example, when a project finally reaches that point where the customers admit that they really didn’t know what they wanted when the project started, is that a project failure?  Probably it counts as a “yes” but I would argue that it was a controllable fault of the software developers or their process. Maybe that’s where the Standish publishers are going with it.

Anyway, I was pleased to see my concerns validated in an article by J. Laurenz Eveleens and Chris Verhoef in the January/February 2010 issue of IEEE Software.  With significant statistical analysis using reputable sources, the authors identify the following four problems with the Standish Group figures in their Chaos reports:

1. “They’re misleading because they’re solely based on estimation accuracy of cost time and functionality.”
2. The Standish estimation accuracy definitions are not sound.
3. If the Standish definitions are used to drive projects, they may cause large cost and time over-estimations.
4. The Standish figures cannot be extrapolated across organizations because large forecasting biases exist in any given organization which make extrapolation, in the authors words, “meaningless.”