The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

In the June 28, 2010 print edition of Information Week, a long-standing friend of DCG’s, Capers Jones, identifies three keys to ensuring software quality:

1. Defect Potentials - Make sure they stay below three per function point
2. Defect Removal Efficiency - Keep it above 95%
3. Quality Metrics - Apply them from requirements through maintenance

For those of us who, like me, need an occasional reminder on definitions, Defect Potentials are the probable numbers of defects that will be found in the various stages of the software development life cycle.  Defect Removal Efficiency is the percentage of the defect potentials that will be removed (or actually are removed) before an application or project is delivered to users.

Which brings me to a conversation that I had today with a client team who are seeing the numbers of defects increasing on a monthly basis this year.  The explanation is that resources are being diverted from the maintenance teams so less bugs are getting fixed.  The client team has been told that all will be well when resources and budget ad diverted back to the bug fixers.

“Aaargh!” was my thoughtful and considered response.  The client team asked what they could do instead to improve the defect numbers.  I told them that the money =resources=time) needs to be invested further up the supply chain where it can make a much bigger proportional difference to quality.  Peer reviews of requirements, designs, code and test scripts are the most cost effective and quickest way for an organization to change quality outcomes.  To be fair, the client team then remembered that some development teams do conduct code reviews (”… and the quality is pretty good”) but the other teams “… don’t have time.”  This is a classic misunderstanding of the value of time spent in different phases of the software development life cycle.

So I get home this evening and start to catch up on my reading.  The first article I pick up is the Capers Jones article and my conversation earlier today jumps fresh into my mind when I see that Capers has included the following table:

Software Defect Removal Efficiency Ranges:

Activity Range

Formal Requirement Inspections                    50% - 90%

Formal Design Inspections                             45% - 85%

Formal Code inspections                                45% - 85%

Automated Static Analysis                              55% - 90%

Automated Unit Test                                       20% - 60%

Regression Test                                               15% - 30%

Integration Test                                               25% - 40%

Performance Test                                             20% - 40%

System Test                                                     25% - 55%

The May/June 2010 edition of IEEE Software contains an excellent “Focus” on Software Product Line management.  The guest editors (McGregor, Muthig, Yoshimura, Jensen) introduction provides a great definition of an effective software product line strategy:

“The software product line strategy is a blend of business and technical actions that lets an organization satisfy a wide range of customers, gain leverage with suppliers, meet the threats of substitute products, and deter other companies seeking to enter the market.  The strategy is robust over a wide range of technologies and domains and organizations of different structures, cultures and goals.”

The guest editors assert that several elements are common to successful product lines:

1. The definition of which products belong to the product line provides the context within which my other decisions are made.  Note that this facilitates most branches from the core software line but some are considered members of the product line and some are not.
2. The organization uses a common software product line architecture as the basis for each product.  The architecture provides the basis for exploiting commonality and managing variation.
3. Commonality is sufficiently defined to realize economies of scale.  This is particularly true for reuse and maintenance.
4. Variation is sufficiently well managed to realize economies of scope.  The important message here is that the core has responsibility to support all variations within the product line.
5. The organization is structured and operated to facilitate building reusable assets and building products using those assets.  This sounds obvious but it is not an easy thing to implement.  Two keys to re-usability are documentation and trust.  The assets to be reused must be clearly documented so that new users understand what they do (and don’t do) bu then the new users must trust the documentation.  It is very tempting for the new user to trust their own new implementation more than the old existing one.

Software development for software products shares all of the opportunities and challenges of software development in corporations with a few majors quirks all of its own.  One of these is the issue of managing a code line for a single product while accommodating (or not) variations for specific customers.  If you are working or have worked for a software product developer, you will be familiar with this problem.

Essentially, the easiest, cheapest and most efficient way to maintain a software product is to have a single code line with all clients on the same, latest version.  In this ideal model, any defects can be fixed in the single code base and provided to clients in the next maintenance release (maybe with a short-term patch to get over any short-term delay).  This model is illustrate below:

At the other extreme, different clients for a software product each have different versions of the software in production with differing levels of maintenance releases and patched loaded.  Many of the clients have their own special customizations to the code some of which overlap with the core product code. A simple example of the challenges is illustrated below:

Many software products will acknowledge more complicated scenarios than this.  Over the years, managing these scenarios has been made slightly easier by more sophisticated configuration management software which manages code branches and code line re-mergers.  That said, such systems still require a level of discipline comparable to the complexity they offer to manage and it is crucial to remember that some code branches just cannot be “automatically” re-merged.

If it is so difficult to manage code branches, why do software product vendors allow or even facilitate the branching of the core product code line?  The following drivers are all relevant:

1. New software product releases of any kind require effort from the clients to acceptance test before loading into  production.
2. A big license deal requires some updates to the product that are needed before the next release will be ready so the software product vendor creates a custom version for the “special”client.  When the next release comes out, the client is in the middle of configuration and implementation and does not want to take the new release.
3. There is pressure on software product vendors to provide minimal patch fixes to a clients specific problems to avoid the need to load maintenance releases which fix every clients problems.
4. All clients believe that they are unique and need changes to software products to reflect their specific operating methods (instead of changing their operating methods to suit the software).
5. All clients understand the long-term costs of customizing software products and so strive to stay on the core product line without necessarily loading every release (Why load a new release if WE don’t need the new functionality or bug fixes?).

In the April 26, 2010 print edition of Informationweek, the Dr Dobb’s Report contained a an article by Sid Sidner of ACI Worldwide describing his teams choice and implementation of a Static Code Analysis tool.  ACI looked at:

• AdaCore’s CodePeer
• Coverity’s Prevent
• GrammaTech’s CodeSonar
• Greenhill’s DoubleCheck
• Klocwork’s Insight
• Lattix’s LDM
• Microsoft’s StyleCop
• Ounce Labs’ Ounce Core

You can read Sid’s comments on each of these tools in his article.

The article is a very good description of how to integrate a software analysis tool into the development environment. Some engineering shops like ACI are a little bit ahead of IT organizations in building quality and security into their products.

Sid offers the following “Tips for Success”:

• Define an “initial issue” policy - what will you do with the code issues that identified by the first analysis run?
• Get the global mechanics working - many of the tools require license managers and centralized result servers
• Attack one product at a time
• Identify SMEs - those product experts who will also be experts in the tool
• Train the SMEs
• Work with the SMEs
• Train the developers
• Perform initial analysis on existing code and defer all issues - this is Sid’s suggested choice of “Initial Issue” policy.  Note that he suggested to defer not ignore.
• Deliver help from SME’s to developers as required
• Run the build anlaysis often
• Review deferred issues

I discussed this article with Lev Lesokin of CAST Software who have their own static analysis application, the Application Intelligence Platform (AIP).  Lev highlighted a couple of considerations which Sid’s paper did not cover:

1. ACI probably do a very good job building quality into their code at a module level, but they should also look at quality at a broad architecture level. Interactions between modules, between distributed components, and most importantly between the database and the code base. Some of the most risky quality issues actually reside in that application level and they are the hardest to identify by code walk-throughs or localized static analysis.
2. When analyzing your software, there’s an opportunity to also use that to introduce measurement. Sid could put some metrics in place to make sure his teams are getting better at building quality and security in. Also, risk metrics can be used to make go/no-go decisions at stage gates. This might be a point for his VP of Engineering, though. He/she can introduce productivity measures and drive the organization towards continuous improvement.

These two uses of software analysis and measurement products might not be as relevant for ACI, because much of the software they build is monolithic products that they ship and don’t see in a distributed environment until they implement at a customer. In an IT organization you will have many moving parts, connected through a services layer or various other interfaces. This is where app-level quality issues become much more present. Also, getting apples-to-apples measures across development teams in Java, .NET, DB stored procs, etc., is much more challenging. Once ACI’s customers integrate their newly acquired payments system into their bank’s accounts and cash management systems, they might consider an architecture-level analysis to determine how structurally strong their environment is in terms of stability, performance and security.

As often happens, i was looking for something else when I came across a 2008 article in IEEE Computer by Mary Lacity and Joseph Rottman (don’t worry - the subject matter is covered in their 2008 book, “Offshore Outsourcing of IT work).  Having just set one of our clients off on the implementation of some outsourcing options to supplement their in-house resources, the list of 20 major effects of offshore outsourcing reported by project managers caught my eye.  I strongly recommend that you think of this list as a set of risks that need to be mitigated in any outsourcing implementation.  With apologies to the authors, I have sorted their list according to my highest priorities.  The in-house Project Managers reported that they:

• needed a mentor the first time they managed a project with offshore resources
• had to motivate the supplier to share bad news
• had to make offshore suppliers feel welcome and comfortable
• needed to thoroughly verify the offshore supplier’s work estimates, which tended to be optimistic
• had to provide greater detail in requirements definitions
• had to do more knowledge transfer up front
• were forced to shortcut the knowledge transfer process because of deadlines set by senior IT leaders
• had to ensure that knowledge transfer was successful by testing the supplier employees’ knowledge
• had to set more frequent milestones
• needed more frequent and more detailed status reports
• required more frequent working meetings to prevent client-caused bottlenecks
• needed to accompany offshore suppliers to all client-facing meetings
• experienced higher transaction costs which threatened their ability to deliver projects on budget
• experienced project delays which threatened their ability to deliver projects on time
• had to guarantee that the supplier followed pre-agreed knowledge renewal practices
• had to ensure that the supplier transferred knowledge about new applications or technologies to the client
• had to learn about new applications or technologies independent of suppliers to ensure that the suppliers information and bids were valid
• had to integrate the suppliers CMM/CMMI processes into their own project management processes
• had to ensure that the supplier’s employees were fully trained as promised by the suppliers
• had to fill many of the roles the the PMO should have performed

An IDC report published in January 2010 (IDC Analyst Amy Konary) cited enterprise software customers (e.g. the CIO’s in corporations) as seeking the option from software product vendors to pay for their product based upon the in-house usage rather than the traditional model of up-front license fees followed by annual maintenance fees.

The corporate customers are seeking a pay-per-use model for their enterprise COTS applications similar to the sort of models that they are now getting from their adoption of cloud computing or software as a service (SaaS) models.  Software product vendors need to think carefully about this pricing option.  It offers a pricing model that sits halfway between the traditional licensing model and SaaS.  Interestingly, it may offer an unexpected side benefit to software product vendors by providing a means to get immediate value out of new enhancements to the core software.  These new enhancements are expected on a regular basis by corporate customers, partly in return for their maintenance fees but often the license plus maintenance model is stretched to incredulity over time.  This “you get what you pay for” model has the potential to yield a much more “honest” relationship between software product vendor and customer.  That said, the IDC report hints that all will not be plain sailing with the “pay as you go” model because CIOs will want assurance that costs wont suddenly spike as a result of unexpectedly heavy usage.  I’m sure they will be willing to trade this for similar assurances to the software vendor at the other end of the spectrum (just a small joke for my software vendor friends).

One of the major areas that will require serious thought for software vendors is the nature of the usage granularity they should offer.  An obvious cut would be user access over time similar to the monthly “per user” fees typically associated with SaaS.  Another idea would be to mix in user access to different subsets of functionality.  This will probably require some changes to the software either to proactively control access to subsets of the functionality or to retrospectively report access to subsets of the functionality.  An obvious approach to the functionality granularity would be to identify and group use cases or transactions (using function point analysis techniques).  This could be very attractive to clients - imagine only having to pay for the parts of Microsoft Word that you actually use!  Even if that meant paying a premium for those functions that you need but only use rarely.

Here at DCG, we are unapologetic about our advocacy of function points as the best currently available, if not ideal, way to measure the (functional) size of software for the purposes of measurement, estimation and management of software projects and applications.  That said, we often implement non-function point solutions for our clients for various reasons.  One area where I believe that a non-function point solution can be valid is in the area of software rework or maintenance.  My theoretical basis for this lies in the six sigma philosophy of “possible errors.” Put simply, it seems to make sense that the number of bugs in a piece of software code could be broadly proportional to the number of opportunities there are to make mistakes which is the number of characters in the code or approximately the number of lines of code.

Edmund P. Morozoff of Medtronic has written a good article in the January/February 2010 issue of IEEE Software that makes this point in more detail (and even includes a nod in the direction of function points).  In particular, the article looks at the key issue of lines of coded added, deleted or modified.

I came across an interesting article in the May/June 2009 issue of IEEE Software by Magne Jurgensen of the Simula Research Laboratory in Norway which touches on the age-old challenge of deciding whether the lowest estimate is actually a realistic estimate.

From his research, Magne has derived 7 recommendations which I have paraphrased below (I suggest they have much validity for internal and external estimates):

1. If price is an important selection criteria, dont invite too many bidders if your requirements specification is incomplete.
2. Avoid using price as an important selection criteria if your ability to assess provider competence is low.
3. Ensure sufficient competence in assessing provider competence by hiring external experts if necessary.
4. When selecting bidders, compare the bids with your own independent cost estimate of the average of all bids.
5. Let providers collect sufficient information to help them analyze the projects complexity.
6. Don’t include potentially misleading information that could affect the estimation process (especially your budget expectations).
7. Be aware that a negotiation phase after receipt of bids may encourage even greater over-optimism.

Of course, “over-optimism” is Magne’s polite way of saying “stupidity.”  If these recommendations resonate with you, I suggest that you should talk to us here at DCG about getting a professional, objective view of your estimates and/or your estimation process.

To some extent this is an age-old question that goes back to the first software and the “maintenance contract” trick that the industry managed to pull off in the early days (see my previous postings).  Of course, the answer is yes.  In the recent troubles that Toyota have had, there has been some discussion about whether there is or is not an underlying software problem.  I have seen no discussions of whether Toyota should be liable if these ir a software problem - the general assumption is that they would be.  So why not all vendors for all bugs?

I saw in a recent magazine article that procurement specialists from government (Homeland Security, NSA) and business are working together to agree on standard contract language to hold vendor liable for software with security flaws.   The work is taking place under the auspices of the nonprofit SANS Institute which publishes a Top 25 Most Dangerous Programming Errors list.  As the article commented, even this presents challenges.  Could a vendor be held liable for a flaw that did not emerge until after successful testing?  Most of us in the software industry would say “no” but think back to the Toyota example.