The SEI have just published a new report by Goldenson and McCurley. The report shows an interesting correlation between what high maturity organizations themselves report about uses of measurement and what appraisal results say about the organizations.

Eight out of ten business leaders say they “make major decisions with missing or untrusted information,” according to an IBM survey reported by Eileen Feretic in the print versions of Baseline (May 2009).  The best measurement systems and metrics are those that enable business leaders (including CIOs) to make important decisions. You would think that delivering this measurement information would be a key value contribution of the IT Department and sometimes that is true.  Unfortunately, all too often the IT Department is part of the problem.  The CIO cannot even provide metrics on the value of IT never mind the rest of the business.  Ms Feretic quotes from her interview with Dr David Friend, CEO of Palladium Group, “IT needs to be the keeper of the truth, the enabler of good decision making.  It can’t just spew out data.  It has to provide insight along with the information in order to help managers make good decisions.” If you are a CIO or a senior IT manager reading this, can you put your hand on your heart and say out loud that you are doing a good job of this?  What about for the IT department?

To some extent, this posting is basic knowledge in the metrics field.  However, all the best knowledge is simple AND valuable.  I based this analysis in an article by c. Warren Axelrod in “Information Control Systems Journal”, Volume 6, 2008.  Axelrod goes into more detail about his categories in his article but I have only included some of his and added some more of my own:

Existence:  This is usually the answer to a “yes/no” question e.g. Has the code review checklist been completed for this project?

Ordinal: This category describes judgemental metrics which can be estimated but not measured nor given a numerical value e.g. The project size is “small/medium/large.”

Score: Very similar to Ordinal but an estimate number within a fixed range e.g. experience of project team on a scale from 1-10 where 1 is bad and 10 is good.

Number (Cardinal): This is a count e.g. the number of defect reported today (which can be compared to the number yesterday and tomorrow).

Percentage: This is a well understood type of metric - a ratio.

Value: A measured value with a well understood and absolute unit of measure e.g. Function A executes in 0.05 seconds.

Calculation: A metric calculated from a series of other metrics of other categories from which the units may be implied.  The algorithm must be repeatable.  One example is productivity.  Another is Function Points.

Statistical: Really a special type of calculation, statistics can be very valuable but very easy to misinterpret.

Can you think of any I missed?

The following presentation offers another view of the deconstruction of the measurement process:

Paving the Road to a Software Measurement Program

Flicking through Information Week, I came across a couple of obscurely named but absolutely critical metrics to use when consider disaster recovery:

Recovery Point Objective (RPO) identifies how much data can be acceptably lost.

Recovery Time Objective (RTO) identifies the acceptable time for a system to be done (or the time it takes to bring it back up).

Clearly, such metrics require extensive discussion and explanations between the businesses and IT.  Both are functions of cost and it would be interesting a useful to ask for a plot of these against cost before making any decisions.

You can find the full, short article by Howard Marks at

Practical Disaster Recovery

The November/December Issue of IEEE Software celebrated the 25th anniversary of the publication.

software

As an IEEE Member of long standing and a long-time reader of IEEE Software, it is interesting to reflect that my reaction to most issues is based on how many things are changing and how many things are staying the same.   The commemorative issue reinforces this feeling.

In this issue, Shari Lawrence Pfleeger writes an interesting but brief article on, “Software Metrics: Progress after 25 years?”  To her credit, she focuses on how we can improve the path forward by offering up four challenges:

1. We must learn how to deal with considerable uncertainty in software development
2. We must recognize software development’s multidisciplinary nature and the concomitant need for measuring “softer” aspects of relevant variables, such as developer experience or expertize.
3. We must find a way to understand and anticipate some of the inevitable change we see during software development and maintenance.
4. We should make room for heuristics, based on careful study of their effectiveness.

These are interesting because they represent areas where DCG is focusing more and more of our energy.

However, if these are our future challenges, have we really made any progress in the last 25 years?

I would argue “yes” but what do you think?

Since our book was published, I am seeing the phrase “Business Value of Technology/IT” more and more.  I guess it’s a sort of affirmation or maybe even a compliment. I saw the above title on a short article by Faisal Hoque in the November issue of Baseline magazine.  Faisal asserts that “using the right metrics to assess the business value derived from technology is critical in demonstrating the effectiveness of business technology investments.”  I agree completely.  So what are they?  Unfortunately, while Faisal gives us some excellent pointer for a process to find the metrics, there are no examples of what the metrics might be.  Check it out for yourself at the link below:

Assessing-the-Business-Value-of-Technology

Doug Beizer wrote an interesting article in a recent FCW (link below).  I love the quote from Tony Mullen of PA Consulting Group, “The key, of course, to real success with dashboards for acquisition and other business intelligence is to choose good measures and to be rigorous in tracking them.”  In a side bar, Beizer highlights that the keys to doing software dashboards right are:

• The Right Metrics
• The Right Tool
• The Right Processes

I agree wholeheartedly.

154242-1.html

 With thanks again to the folks at InformationWeek for their excellent survey results (see reference at the bottom of this post), I was curious when I looked at their tables to see if the split between legacy and new project spending in IT budgets was consistent or different by industry sector.  As you can see from the chart, it seems to be that there is more consistency than variation particularly in the context of the second chart that shows much greater variation in percentage of annual revenue devoted to It in the different industry sectors.  A current rule of thumb for legacy spending seems to be about 65% or 60% which is much healthier than the figure of 80% than was doing the rounds a few years ago.  Of course, if we are facing a down turn then spending on new projects, which tends to be more discretionary, will drop while spending on legacy, which tends to be about preserving the status quo or keeping the lights, is less flexible and more likely to stay flat than drop. We may see some retrenchment in these percentages next year.

This chart shows that the Banking & Financial sector continues to spend twice as high a percentage of revenue on IT as its nearest rival sectors and four times as high a percentage as some.  It will be interesting to see if this changes next year as the Banking & Finance sector deals with the self-inflicted financial constraints of the credit squeeze and the demands for increased regulation (which will presumably require more reporting and more IT).

The above charts were created from data provided in the InformationWeek 500 published on September 15, 2008.  (reprints of that article which includes the data but not the derived charts are available at www.magprints.com/informationweek500

Earlier this week, I had the following exchange with a guy who had attended one of my seminar’s:

A question – if you were a Senior Manager, what would you be looking for from a Software Quality function? We changed our function’s focus from Process compliance to focusing on identifying and mitigating application risk from a perspective of Application Stability, Changeability and Reliability. What would some typical (quantitative) measures be?

Answer: I faced exactly this challenge in my days at Sanchez/Fidelity in two respects: maintenance and project completion.  In both cases, I found that the most useful metrics were related to change.  I had my software quality team gather metrics from the configuration management system about the total number of lines of code in the application or project.  We used this as a sort of baseline (not a real baseline because it was always changing) to track on a monthly, weekly or daily basis (depending on our sensitivity) the percentage of LOC’s being added, delete or changed.  Again, our configuration management system either generated this automatically or, for  some older code, we ran scripts to do it.  By running these metrics for a while we were able to get a feel for “normal” levels and “expected” fluctuations.  This in turn enabled exception reporting which kicked-off investigations.

You may think that this would help with stability and changeability but not reliability.  In fact, we found that if we extended the granularity of the metrics down to module levels, we were able to see outlying modules in particular applications that had either very few changes or very many changes.  Either very few or very many changes could often be attributed to problem modules.  For example, a module may have very many changes relative to its peers because it is a central module in the architecture that has to be changed by every project.  This module is a high risk module!  Equally, a module with very few changes is either very stable or not used (surprisingly many of them) or a module that is so critical and hard to maintain that nobody dares to touch it for fear of it failing.    This is a good example of why the metrics can only kick-off investigations rather than give you absolute answers.

IT spend as % of revenue is a frustrating metric because it places IT firmly in the sphere of costs that cannot be avoided and must be managed rather than an investment that should yield beneficial value.  Being above or below the average % revenue line is no great indicator of success other than the feeling that, as a CIO, you are doing are doing better or worse than your peers in the eyes of your boss (especially if your boss is the CFO).  Looking at the chart below (if you click on it you will get a version that’s easier to read), you can see that, for the mash of all industries, the percentage has stayed remarkably stable between 3% and 4% over the past eight years.  What does this tell us?  Two things:

1. I think its a number that business and IT need to be aware of (about 3.5% - more on the industry specific numbers soon)

2. Because this variable has been held steady, we can look at the other variables in a different light.  For example, have all the contributions to better performance over the past eight years (SOA, web services, ITIL, CMMI) been a waste of time or are the improvements perfectly cancelled out by the cost of the extra work we are able to do now?  What do you think?

The above chart was created from data provided in the InformationWeek 500 published on September 15, 2008.  (reprints of that article which includes the data but not the chart are available at www.magprints.com/informationweek500