The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

In the June 28, 2010 print edition of Information Week, a long-standing friend of DCG’s, Capers Jones, identifies three keys to ensuring software quality:

1. Defect Potentials - Make sure they stay below three per function point
2. Defect Removal Efficiency - Keep it above 95%
3. Quality Metrics - Apply them from requirements through maintenance

For those of us who, like me, need an occasional reminder on definitions, Defect Potentials are the probable numbers of defects that will be found in the various stages of the software development life cycle.  Defect Removal Efficiency is the percentage of the defect potentials that will be removed (or actually are removed) before an application or project is delivered to users.

Which brings me to a conversation that I had today with a client team who are seeing the numbers of defects increasing on a monthly basis this year.  The explanation is that resources are being diverted from the maintenance teams so less bugs are getting fixed.  The client team has been told that all will be well when resources and budget ad diverted back to the bug fixers.

“Aaargh!” was my thoughtful and considered response.  The client team asked what they could do instead to improve the defect numbers.  I told them that the money =resources=time) needs to be invested further up the supply chain where it can make a much bigger proportional difference to quality.  Peer reviews of requirements, designs, code and test scripts are the most cost effective and quickest way for an organization to change quality outcomes.  To be fair, the client team then remembered that some development teams do conduct code reviews (”… and the quality is pretty good”) but the other teams “… don’t have time.”  This is a classic misunderstanding of the value of time spent in different phases of the software development life cycle.

So I get home this evening and start to catch up on my reading.  The first article I pick up is the Capers Jones article and my conversation earlier today jumps fresh into my mind when I see that Capers has included the following table:

Software Defect Removal Efficiency Ranges:

Activity Range

Formal Requirement Inspections                    50% - 90%

Formal Design Inspections                             45% - 85%

Formal Code inspections                                45% - 85%

Automated Static Analysis                              55% - 90%

Automated Unit Test                                       20% - 60%

Regression Test                                               15% - 30%

Integration Test                                               25% - 40%

Performance Test                                             20% - 40%

System Test                                                     25% - 55%

Estimates are “mission-critical” for Software Product companies because:

• In the short term, the quality of the development estimate can win or lose a deal
• In the medium term, on successful deals, the quality of the development estimate can drive profit or loss on a deal
• In the long-term, the quality of the estimates drives the evolution of the products – not just what we say we will do in the road maps but what we actually deliver in releases.

Phil Laplante of Penn State is a prolific and respected writer on many software and project topics as well as being part of my local IT community.  In the July/August issue of the IEEE IT professional magazine, Phil steps back and writes a thoughtful and though provoking article that is perfect for contemplation on the beach.

In “Nexialism and the Law of Unintended Consequences,” (there’s a search term you won’t use very often) Phil suggests that to understand complex interactions that could impact our project risk we need to spend at least some time thinking in simple terms about the big picture.  Nexialism is defined as “the science of joining in an orderly fashion the knowledge of one field of learning with that of other fields.”

OK so that sounds a little grandiose and you might be wondering how it impacts your project risks.  Think of it this way, how many times in a failing project did somebody know that it was going wrong early but was never asked?  Is the test manager communicating (talking and listening) with the coding team lead (and vice versa).  What is the opinion of the DBA’s?  The performance test team?  Like it or not, these are different disciplines (or fields if you like) with different perspectives and, too often, we do not take care to “join their knowledge in an orderly fashion.”

Keith Baxter of De-RISK, based in the UK, has developed a simple but effective way of capturing and visualizing the impact of risk on a project estimate.  The approach, called Quality Based Costing (QBC),  and some examples are provided in the following paper on Keith’s website:

Quality Based Costing Paper

QBC is based on the responses to structured questions about each project component which seek to identify four costs for that component:

• Absolute Minimum
• Best guess/realistic estimate
• Contingency added
• Disaster scenario

The approach is then to apply Monte Carlo simulation to derive a probability distribution of the likely cost.  I particularly like an estimation process that includes Monte Carlo simulation because it forces the project manager and organization to make an informed choice about the risk they wish to take on each project.

The May 19, 2010 edition of Computerworld magazine contained a couple of articles about the growing evidence that one of the reactions to tightening budgets in software development has been the increased use of open source software and the associated  “Hidden Snags”

It is clear that one of the attractions of open source is the availability of source code which is critical for integrating open source into the enterprise but this brings problems of its own (“Integration Jams”).  I have written before about the benefits of static code analysis and I really want to stress the value of getting a one-off analysis done of any significant open source code that you bring into your enterprise.  Such a one-off analysis can be delivered as a service and does not require purchasing or learning a new tool.

The SEER estimate by Comparison capability is something that I talk about a lot to company’s who are really struggling because their SWAG approach to estimating projects is neither repeatable nor coherent.  Often, the projects they are failing miserably to estimate professionally are mission-critical either for deliverables to customers or for internal investment (budget) planning.

At some point, these estimate amateurs may need to base their estimates on a repeatable sizing approach (of which Function Points is just one option) but they need to change their ways quickly so I often talk to them about doing estimates by comparison using SEER.

Even if you don’t see yourself buying a tool right now, I recommend investigating this approach so you can attend in real time or review later.  The technique is powerful even if the tool is not right for you.  DCG can also provide the tool as a Service so you can pilot the application or just use it a few times every year for this really important estimates.  For more information on this service, you can contact David Herron at d.herron@davidconsultinggroup.com.

The Estimate by Comparison application has traditionally been used to empower users to develop an understanding of software size; the single most significant driver of development cost, effort, and schedule for software projects.  However, SEER Estimate by Comparison has evolved and can be utilized with all of the SEER solutions to provide an insight into effective definition of scope through a series of project analogies and/or comparisons to a user’s repository of past projects, thus helping users to develop a reliable estimate on a project’s scope even when information is scarce.

SEER Estimate by Comparison further adds capability to the project team when used in a more contemporary manner. It can be manipulated in such a manner that a wide realm of subjective, qualitative alternatives can be evaluated in context of the project as a whole in a robust, repeatable and ultimately measureable manner.

The May/June 2010 edition of IEEE Software contains an excellent “Focus” on Software Product Line management.  The guest editors (McGregor, Muthig, Yoshimura, Jensen) introduction provides a great definition of an effective software product line strategy:

“The software product line strategy is a blend of business and technical actions that lets an organization satisfy a wide range of customers, gain leverage with suppliers, meet the threats of substitute products, and deter other companies seeking to enter the market.  The strategy is robust over a wide range of technologies and domains and organizations of different structures, cultures and goals.”

The guest editors assert that several elements are common to successful product lines:

1. The definition of which products belong to the product line provides the context within which my other decisions are made.  Note that this facilitates most branches from the core software line but some are considered members of the product line and some are not.
2. The organization uses a common software product line architecture as the basis for each product.  The architecture provides the basis for exploiting commonality and managing variation.
3. Commonality is sufficiently defined to realize economies of scale.  This is particularly true for reuse and maintenance.
4. Variation is sufficiently well managed to realize economies of scope.  The important message here is that the core has responsibility to support all variations within the product line.
5. The organization is structured and operated to facilitate building reusable assets and building products using those assets.  This sounds obvious but it is not an easy thing to implement.  Two keys to re-usability are documentation and trust.  The assets to be reused must be clearly documented so that new users understand what they do (and don’t do) bu then the new users must trust the documentation.  It is very tempting for the new user to trust their own new implementation more than the old existing one.

Hakan Erdogmus is the Editor-in-chief of the IEEE Software magazine.  In the May/June 2010 issue of the magazine, Hakan has written an impassioned plea for the importance of using evidence as the basis for software development decisions, especially when deciding on the adoption of software engineering ideas.

I totally support Hakan’s position.  Too many of us forget the “science” part of computer science and the “engineering” part of software engineering.  The testing of hypotheses in science and the use of prototypes in engineering are fundamental to those disciplines.  Yet in software development, we rely on methods more akin to building a bridge by adding one brick at a time on either side of the river and hoping that it will meet in the middle and be strong enough (while ignoring those budgets, deadlines, materials and colleagues that get swept away by the river during construction).

For deciding on the adoption of software engineering ideas, Hakan categorizes three types of evidence in increasing order of rigor:

• Feasibility Check - While it is the weakest of the three, a feasibility check should be able to:

- quickly evaluate major advantages, risks, limitations
- roughly estimate the size of the problem
- assess the novelty of the solution
- test the cost effectiveness
- help to decide if the idea is worth exploring further and under what controlled but real-world circumstances

• Anecdotal Evidence - This is effectively “crowd sourcing” your evidence gathering in support of a new idea with all of the advantages (the pioneers are the ones with all the arrows in their backs) and disadvantages (has the author really tested the idea under similar circumstances as you?  Do they have a vested interest in supporting the idea?) of that approach.  Nonetheless, external data points ARE valuable and a significant step up on a purely internal feasibility check.
• Systematic Evidence - Of course, researched and documented evidence of the merits of an idea is the best type of evidence but it is difficult to find or produce if the ideas are new.  Mature organizations can make this type of evidence much easier to discover and use by adopting a methodical approach to capturing lessons learned from projects.

Hakan makes the excellent point that the SIZE of the decision or idea should dictate the threshold of the quality of evidence required.

Software development for software products shares all of the opportunities and challenges of software development in corporations with a few majors quirks all of its own.  One of these is the issue of managing a code line for a single product while accommodating (or not) variations for specific customers.  If you are working or have worked for a software product developer, you will be familiar with this problem.

Essentially, the easiest, cheapest and most efficient way to maintain a software product is to have a single code line with all clients on the same, latest version.  In this ideal model, any defects can be fixed in the single code base and provided to clients in the next maintenance release (maybe with a short-term patch to get over any short-term delay).  This model is illustrate below:

At the other extreme, different clients for a software product each have different versions of the software in production with differing levels of maintenance releases and patched loaded.  Many of the clients have their own special customizations to the code some of which overlap with the core product code. A simple example of the challenges is illustrated below:

Many software products will acknowledge more complicated scenarios than this.  Over the years, managing these scenarios has been made slightly easier by more sophisticated configuration management software which manages code branches and code line re-mergers.  That said, such systems still require a level of discipline comparable to the complexity they offer to manage and it is crucial to remember that some code branches just cannot be “automatically” re-merged.

If it is so difficult to manage code branches, why do software product vendors allow or even facilitate the branching of the core product code line?  The following drivers are all relevant:

1. New software product releases of any kind require effort from the clients to acceptance test before loading into  production.
2. A big license deal requires some updates to the product that are needed before the next release will be ready so the software product vendor creates a custom version for the “special”client.  When the next release comes out, the client is in the middle of configuration and implementation and does not want to take the new release.
3. There is pressure on software product vendors to provide minimal patch fixes to a clients specific problems to avoid the need to load maintenance releases which fix every clients problems.
4. All clients believe that they are unique and need changes to software products to reflect their specific operating methods (instead of changing their operating methods to suit the software).
5. All clients understand the long-term costs of customizing software products and so strive to stay on the core product line without necessarily loading every release (Why load a new release if WE don’t need the new functionality or bug fixes?).