The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

In the June 28, 2010 print edition of Information Week, a long-standing friend of DCG’s, Capers Jones, identifies three keys to ensuring software quality:

1. Defect Potentials - Make sure they stay below three per function point
2. Defect Removal Efficiency - Keep it above 95%
3. Quality Metrics - Apply them from requirements through maintenance

For those of us who, like me, need an occasional reminder on definitions, Defect Potentials are the probable numbers of defects that will be found in the various stages of the software development life cycle.  Defect Removal Efficiency is the percentage of the defect potentials that will be removed (or actually are removed) before an application or project is delivered to users.

Which brings me to a conversation that I had today with a client team who are seeing the numbers of defects increasing on a monthly basis this year.  The explanation is that resources are being diverted from the maintenance teams so less bugs are getting fixed.  The client team has been told that all will be well when resources and budget ad diverted back to the bug fixers.

“Aaargh!” was my thoughtful and considered response.  The client team asked what they could do instead to improve the defect numbers.  I told them that the money =resources=time) needs to be invested further up the supply chain where it can make a much bigger proportional difference to quality.  Peer reviews of requirements, designs, code and test scripts are the most cost effective and quickest way for an organization to change quality outcomes.  To be fair, the client team then remembered that some development teams do conduct code reviews (”… and the quality is pretty good”) but the other teams “… don’t have time.”  This is a classic misunderstanding of the value of time spent in different phases of the software development life cycle.

So I get home this evening and start to catch up on my reading.  The first article I pick up is the Capers Jones article and my conversation earlier today jumps fresh into my mind when I see that Capers has included the following table:

Software Defect Removal Efficiency Ranges:

Activity Range

Formal Requirement Inspections                    50% - 90%

Formal Design Inspections                             45% - 85%

Formal Code inspections                                45% - 85%

Automated Static Analysis                              55% - 90%

Automated Unit Test                                       20% - 60%

Regression Test                                               15% - 30%

Integration Test                                               25% - 40%

Performance Test                                             20% - 40%

System Test                                                     25% - 55%

Interesting survey result in the June 2010 edition of CIO Magazine.  90% of respondents to a survey by Forrester Research reported that customer experience was “critical” or “very important” to the corporate strategy.  No surprise there.  The responses on the obstacles to improving customer experience were very revealing:

• 53% (over half) reported “No clear strategy for improving the customers experience
• 50% expressed a “Need for customer-experience management processes.”
• 49% are experiencing “insufficient cooperation across groups”
• 32% report a “lack of understanding about customers”

If any of this sounds familiar, you need help!  Give us a call or work it out yourselves.    All of these are solvable but the third bullet rings particular true across the business-IT interface.  We run across this all the time with our clients and it improvements in breaking down internal barriers DO benefit customers!

The May 19, 2010 edition of Computerworld magazine contained a couple of articles about the growing evidence that one of the reactions to tightening budgets in software development has been the increased use of open source software and the associated  “Hidden Snags”

It is clear that one of the attractions of open source is the availability of source code which is critical for integrating open source into the enterprise but this brings problems of its own (“Integration Jams”).  I have written before about the benefits of static code analysis and I really want to stress the value of getting a one-off analysis done of any significant open source code that you bring into your enterprise.  Such a one-off analysis can be delivered as a service and does not require purchasing or learning a new tool.

Software development for software products shares all of the opportunities and challenges of software development in corporations with a few majors quirks all of its own.  One of these is the issue of managing a code line for a single product while accommodating (or not) variations for specific customers.  If you are working or have worked for a software product developer, you will be familiar with this problem.

Essentially, the easiest, cheapest and most efficient way to maintain a software product is to have a single code line with all clients on the same, latest version.  In this ideal model, any defects can be fixed in the single code base and provided to clients in the next maintenance release (maybe with a short-term patch to get over any short-term delay).  This model is illustrate below:

At the other extreme, different clients for a software product each have different versions of the software in production with differing levels of maintenance releases and patched loaded.  Many of the clients have their own special customizations to the code some of which overlap with the core product code. A simple example of the challenges is illustrated below:

Many software products will acknowledge more complicated scenarios than this.  Over the years, managing these scenarios has been made slightly easier by more sophisticated configuration management software which manages code branches and code line re-mergers.  That said, such systems still require a level of discipline comparable to the complexity they offer to manage and it is crucial to remember that some code branches just cannot be “automatically” re-merged.

If it is so difficult to manage code branches, why do software product vendors allow or even facilitate the branching of the core product code line?  The following drivers are all relevant:

1. New software product releases of any kind require effort from the clients to acceptance test before loading into  production.
2. A big license deal requires some updates to the product that are needed before the next release will be ready so the software product vendor creates a custom version for the “special”client.  When the next release comes out, the client is in the middle of configuration and implementation and does not want to take the new release.
3. There is pressure on software product vendors to provide minimal patch fixes to a clients specific problems to avoid the need to load maintenance releases which fix every clients problems.
4. All clients believe that they are unique and need changes to software products to reflect their specific operating methods (instead of changing their operating methods to suit the software).
5. All clients understand the long-term costs of customizing software products and so strive to stay on the core product line without necessarily loading every release (Why load a new release if WE don’t need the new functionality or bug fixes?).

In the April 26, 2010 print edition of Informationweek, the Dr Dobb’s Report contained a an article by Sid Sidner of ACI Worldwide describing his teams choice and implementation of a Static Code Analysis tool.  ACI looked at:

• AdaCore’s CodePeer
• Coverity’s Prevent
• GrammaTech’s CodeSonar
• Greenhill’s DoubleCheck
• Klocwork’s Insight
• Lattix’s LDM
• Microsoft’s StyleCop
• Ounce Labs’ Ounce Core

You can read Sid’s comments on each of these tools in his article.

The article is a very good description of how to integrate a software analysis tool into the development environment. Some engineering shops like ACI are a little bit ahead of IT organizations in building quality and security into their products.

Sid offers the following “Tips for Success”:

• Define an “initial issue” policy - what will you do with the code issues that identified by the first analysis run?
• Get the global mechanics working - many of the tools require license managers and centralized result servers
• Attack one product at a time
• Identify SMEs - those product experts who will also be experts in the tool
• Train the SMEs
• Work with the SMEs
• Train the developers
• Perform initial analysis on existing code and defer all issues - this is Sid’s suggested choice of “Initial Issue” policy.  Note that he suggested to defer not ignore.
• Deliver help from SME’s to developers as required
• Run the build anlaysis often
• Review deferred issues

I discussed this article with Lev Lesokin of CAST Software who have their own static analysis application, the Application Intelligence Platform (AIP).  Lev highlighted a couple of considerations which Sid’s paper did not cover:

1. ACI probably do a very good job building quality into their code at a module level, but they should also look at quality at a broad architecture level. Interactions between modules, between distributed components, and most importantly between the database and the code base. Some of the most risky quality issues actually reside in that application level and they are the hardest to identify by code walk-throughs or localized static analysis.
2. When analyzing your software, there’s an opportunity to also use that to introduce measurement. Sid could put some metrics in place to make sure his teams are getting better at building quality and security in. Also, risk metrics can be used to make go/no-go decisions at stage gates. This might be a point for his VP of Engineering, though. He/she can introduce productivity measures and drive the organization towards continuous improvement.

These two uses of software analysis and measurement products might not be as relevant for ACI, because much of the software they build is monolithic products that they ship and don’t see in a distributed environment until they implement at a customer. In an IT organization you will have many moving parts, connected through a services layer or various other interfaces. This is where app-level quality issues become much more present. Also, getting apples-to-apples measures across development teams in Java, .NET, DB stored procs, etc., is much more challenging. Once ACI’s customers integrate their newly acquired payments system into their bank’s accounts and cash management systems, they might consider an architecture-level analysis to determine how structurally strong their environment is in terms of stability, performance and security.

We often start our CMMI, measurement, estimating or other process improvement engagements with clients with a “gap analysis.”  Probably all of the readers of this blog have a pretty good idea what that means and so it is an easy entry point.

But I don’t like the term “gap analysis.” Why?  Because it carries with it the implication that it is an exercise in finding out everything that our client is doing wrong.  I prefer the term “opportunity analysis.”

In the April 2010 edition of SoftwareTech magazine, David Herron writes a very good perspective on this issue.  My own simple perspective is that every software development organization does some things well.  It must do otherwise it would not survive.  Hence, every software development organization must know how to do things well.  Finding the things the client does well is just as important as finding the things it does not do well because the pocket of excellence provide patterns for the improvement of the weak areas.

Further, identify the pockets of excellence is important because any changes must be carefully designed to ensure that the excellence is not compromised.  This is a classic mistake that we see too often - process zealots are allowed to bring everything the company does to the same “average” level.  This usually represents significant improvement is key problems areas but it can sometimes stifle the areas that were previously excellent and were making the company “special” to their customers.

Finally, for those of you who have time to go to read David’s article, I would also recommend the article by Capers Jones in the same edition, “Software Quality and Software Economics.”  Capers always delivers thought-provoking ideas and plenty of data points.

How can too many customers be a problem for software development?  That’s the goal, right?  Well, yes, but here at DCG many of the suffering organizations we see are suffering because they are crossing a size threshold that requires them to scale up their processes and metrics to a new level of maturity.

I attended a meeting of the Eastern Technology Council recently (good networking organization for those of you in the Philadelphia region).  Scott Fraser, CTO of Portico,  presented his ideas on what you should include in a modern software development platform.  It was a good session so I will only pick out the (my) highlights here.

Scott’s basic premise was that, as a software product developer, your problems grow disproportionately to your customer base growth.  Making the software configurable rather than depending on customization is critical to persuading customers that your code is truly enterprise-ready.  Scott highlighted a number of ways to achieve this enterprise-ready recognition including:

• Architecting your software as a framework using scripting and meta data from which customers had instantiate implementations
• Building a graphical front-end for configuration to lower the entry barrier for staff who can do configuration (maybe, a little).
• Choose and use  “tools” that will lower you pain for achieving and sustaining enterprise readiness (see below)
• Implement strong QA on 3rd party vendors
• Need to be able to run all all platforms (see below)
• Integrate or DIE!

As examples of the tools that Scott and his team have found invaluable, he cited Crucible, a online tool for managing distributed code review (DCG actively promotes the value of formal and informal reviews throughout the SDLC), and Fortify360, a static code analysis tool (see other postings on CAST Software) focused on security.  Interestingly, i just read an article, “Integrating Application Security into Software Development,” by Jeffery Payne in IEEE IT Pro magazine, March/April 2010 edition, which supports Scott’s assertion of the importance of tools like Fortify360 and adds similar tools such as AppScan (IBM), HP WebInspect (HP) and Coverity5 (Coverity).

Scott confessed that some of the most important tools were the ones he didn’t need to buy.  His choice of Oracle and Java early on left much of the application interoperability heavy lifting on those (larger) shoulders.

Finally, Scott mused on the importance of good developers and what that means today.  He asserts that good developers are always “lazy” developers in that they design, build (and test) simple structures that they can reuse many times.  However, increasingly god developers are also polyglot programmers who have the knowledge and skills to think about the network environment and program for virtualization.

We have been using CMMI as a framework for our consulting initiatives for many years.  In recent years, it has become clearer that you can get a lot of value out of this approach without going all the way to a full accreditation.

In a recent (Jan/Feb 2010) article in CrossTalk, Jeffrey Dutton captured the essence of using CMMI in this way.  He puts forward three driving principles:

1. Focus on Business Issues and Performance Goals
2. Involved Leadership and Process Ownership by Process “Doers”
3. Improvements should be made at the Speed of Business

Jeffrey goes on to explain that the CMMI framework together withe three principles can readily accommodate different approaches like CMMI itself, Lean, Six Sigma and ITIL.

We are seeing more and more interest in this “multi-model” approach to get more value more quickly out of process improvement initiatives.

To some extent this is an age-old question that goes back to the first software and the “maintenance contract” trick that the industry managed to pull off in the early days (see my previous postings).  Of course, the answer is yes.  In the recent troubles that Toyota have had, there has been some discussion about whether there is or is not an underlying software problem.  I have seen no discussions of whether Toyota should be liable if these ir a software problem - the general assumption is that they would be.  So why not all vendors for all bugs?

I saw in a recent magazine article that procurement specialists from government (Homeland Security, NSA) and business are working together to agree on standard contract language to hold vendor liable for software with security flaws.   The work is taking place under the auspices of the nonprofit SANS Institute which publishes a Top 25 Most Dangerous Programming Errors list.  As the article commented, even this presents challenges.  Could a vendor be held liable for a flaw that did not emerge until after successful testing?  Most of us in the software industry would say “no” but think back to the Toyota example.