The Software Engineering Institute produces periodic reports called “Maturity Profiles” on the results of CMMI appraisal.  For those of you considering the value of CMMI, these Maturity Profiles make invaluable reading.

For example, in the latest maturity profile at time of posting (March 2008), we find the following food for thought:

• There have been just over 3,000 appraisals involving just under 15,000 projects.  This implies an average of slightly less that 5 projects per appraisal - a lower number than I would have expected.
• Just under 70% of the appraisals are non-US.
• About 75% of appraisals are at Level 2 or 3 for all appraisals.  That rises to 82% for more recent appraisals against v1.2.  Level 2 and 3 are still the “bread and butter” of CMMI despite all the relatively large number of Level 5 appraisals in recent years.
• By organization type, 72% are commercial/in-house, 23% are military or government contractors and 5% are military or government organizations.  Some people are not practicing what they preach!
• Numbers of CMMI appraisals reported to the SEI rose every year from 2002 to 2007.
• In 2007, there were 1006 reported appraisals versus 52 in 2002.
• The number of appraisals in 2007 was 21% higher than the number in 2006.

List of SEI CMMI Maturity Profiles

Latest SEI CMMI Maturity Profile - March 2008 - at date of posting

There are lots of “what went wrong” articles out there at the moment but when I came across the headlines for this one, I genuinely thought they were talking about IT projects not the financial system.  OK, so the fact that it was in the Harvard Business Review should have been a clue but I just assume that everyone has IT at the top of their minds!

The six sources of failure listed are:

1. Lack of appropriate data - insufficient metrics
2. Narrow measures of risk - poorly focused metrics
3. Overlooked risks - badly designed metrics
4. Hidden or unreported risks - “I knew that would happen”
5. Poor communication - over-complex or inadequate metrics can result in weak communication to executive of what is going on
6. Rate of change

The original article on financial system risk by Rene M. Stulz is in the March 2009 issue of the Harvard Business Review.

I received the following question in an email from a reader and I thought that it might be of more general interest:

“I am a 24 year old student from Austria (Innsbruck) and I study Management and IT. At the moment I am writing on my thesis that is dealing with capturing the business value of a test center. That’s why I am really interested in your book and work. Due to the fact that you are an expert in this area I wondered if you can suggest literature for my thesis. And do you have hints for me or suggestions concerning meaningful methodologies? The main challenge is that a test center has an indirect “business value” due to the fact that it is supporting business software in quality concerns.”

I replied:

“I confess that I have not had as much time as I might like to consider your thesis.  I have not done any online research but simply referenced books that I have on my bookshelves.  Nonetheless, here are some ideas that might be helpful:

1. The value of testing is essentially a risk management problem.  At one extreme, if we perform no testing, what are the potential financial or other value consequences to the business if/when serious and minor defects are discovered by the software customers?  At the other extreme, the cost of removing 100% of the defects in the software through testing would probably be too high so what level of defects is acceptable?

2. If we take the risk management approach to this, we would identify the risks, quantify their potential impact, quantify their probability and then use some sort of algorithm to aggregate the risk cost which we could equate to value of testing.

3. Steve Tockey includes lots of value treatments on software investments in his book, “Return on Software.”  In particular, he has a short section called “How much software testing is enough?” (page 474) in which he identifies this as an optimization question and cross references the other chapters in the book which will provide the tools to perform this optimization exercise so that, as he says, “… the question can be answered from a business perspective.”

4. You might also want to consider the risk management profile of alternatives to testing such as more rigorous design (lots of books on this topic), and peer reviews earlier in the process (see “Software Inspection” by Tom Gilb and Dorothy Graham).  How much these improve the risk management profile?  Generally, it is less expensive to fix a problem earlier in the process than it is at the test stage.

5. If you are interested in building a model of the software development process, take a look at “Metrics and Models in Software Quality Engineering” by Stephen H. Kan.”

Does anyone have any other suggestions?

I found an excellent summary article on COBIT by Adam Nelson in the December 2008 issue of Baseline magazine.  As we do here at DCG, Adam positions COBIT as the overarching control framework for other sets of best practices such as ITIL and CMMI.  I always talk of them as a set of “Russian Dolls” from the business perspective with CMMI inside ITIL inside COBIT.

You can access Adam’s article at:

Adam Nelson article - Building a Governance Foundation

One of the things that seemed to appear fairly regularly in those lists of the 10 most important things for CIO’s in 2009 that have appeared over the past three months is project management and project portfolio management.  One particular article I saw which expanded on this pick opened with the line, “In today’s time-sensitive and budget-conscious world, getting projects finished on time is paramount.”  I saved it because I want to check if they use this line every year.  Can anyone think of when this wasn’t true?  Well I guess it was #9 on that particular list.

More seriously, while project management must be on every CIO’s list every year, Project Portfolio Management is an area where real focus can yield real savings.  An initial review of the project portfolio by value will always yield a set of “least valuable” projects which can be culled.  This can be followed up by refinements in the IT governance process that would prevent such projects being started in the future. The use of more sophisticated software estimation techniques and some sort of sizing metric such as function points will add the dimension of effort to the value validation process (this is a too-clever way of saying that you should prioritize the low hanging fruit - smaller projects with higher value).