In the June 28, 2010 print edition of Information Week, a long-standing friend of DCG’s, Capers Jones, identifies three keys to ensuring software quality:

1. Defect Potentials - Make sure they stay below three per function point
2. Defect Removal Efficiency - Keep it above 95%
3. Quality Metrics - Apply them from requirements through maintenance

For those of us who, like me, need an occasional reminder on definitions, Defect Potentials are the probable numbers of defects that will be found in the various stages of the software development life cycle.  Defect Removal Efficiency is the percentage of the defect potentials that will be removed (or actually are removed) before an application or project is delivered to users.

Which brings me to a conversation that I had today with a client team who are seeing the numbers of defects increasing on a monthly basis this year.  The explanation is that resources are being diverted from the maintenance teams so less bugs are getting fixed.  The client team has been told that all will be well when resources and budget ad diverted back to the bug fixers.

“Aaargh!” was my thoughtful and considered response.  The client team asked what they could do instead to improve the defect numbers.  I told them that the money =resources=time) needs to be invested further up the supply chain where it can make a much bigger proportional difference to quality.  Peer reviews of requirements, designs, code and test scripts are the most cost effective and quickest way for an organization to change quality outcomes.  To be fair, the client team then remembered that some development teams do conduct code reviews (”… and the quality is pretty good”) but the other teams “… don’t have time.”  This is a classic misunderstanding of the value of time spent in different phases of the software development life cycle.

So I get home this evening and start to catch up on my reading.  The first article I pick up is the Capers Jones article and my conversation earlier today jumps fresh into my mind when I see that Capers has included the following table:

Software Defect Removal Efficiency Ranges:

Activity Range

Formal Requirement Inspections                    50% - 90%

Formal Design Inspections                             45% - 85%

Formal Code inspections                                45% - 85%

Automated Static Analysis                              55% - 90%

Automated Unit Test                                       20% - 60%

Regression Test                                               15% - 30%

Integration Test                                               25% - 40%

Performance Test                                             20% - 40%

System Test                                                     25% - 55%

The SEER estimate by Comparison capability is something that I talk about a lot to company’s who are really struggling because their SWAG approach to estimating projects is neither repeatable nor coherent.  Often, the projects they are failing miserably to estimate professionally are mission-critical either for deliverables to customers or for internal investment (budget) planning.

At some point, these estimate amateurs may need to base their estimates on a repeatable sizing approach (of which Function Points is just one option) but they need to change their ways quickly so I often talk to them about doing estimates by comparison using SEER.

Even if you don’t see yourself buying a tool right now, I recommend investigating this approach so you can attend in real time or review later.  The technique is powerful even if the tool is not right for you.  DCG can also provide the tool as a Service so you can pilot the application or just use it a few times every year for this really important estimates.  For more information on this service, you can contact David Herron at d.herron@davidconsultinggroup.com.

The Estimate by Comparison application has traditionally been used to empower users to develop an understanding of software size; the single most significant driver of development cost, effort, and schedule for software projects.  However, SEER Estimate by Comparison has evolved and can be utilized with all of the SEER solutions to provide an insight into effective definition of scope through a series of project analogies and/or comparisons to a user’s repository of past projects, thus helping users to develop a reliable estimate on a project’s scope even when information is scarce.

SEER Estimate by Comparison further adds capability to the project team when used in a more contemporary manner. It can be manipulated in such a manner that a wide realm of subjective, qualitative alternatives can be evaluated in context of the project as a whole in a robust, repeatable and ultimately measureable manner.

An IDC report published in January 2010 (IDC Analyst Amy Konary) cited enterprise software customers (e.g. the CIO’s in corporations) as seeking the option from software product vendors to pay for their product based upon the in-house usage rather than the traditional model of up-front license fees followed by annual maintenance fees.

The corporate customers are seeking a pay-per-use model for their enterprise COTS applications similar to the sort of models that they are now getting from their adoption of cloud computing or software as a service (SaaS) models.  Software product vendors need to think carefully about this pricing option.  It offers a pricing model that sits halfway between the traditional licensing model and SaaS.  Interestingly, it may offer an unexpected side benefit to software product vendors by providing a means to get immediate value out of new enhancements to the core software.  These new enhancements are expected on a regular basis by corporate customers, partly in return for their maintenance fees but often the license plus maintenance model is stretched to incredulity over time.  This “you get what you pay for” model has the potential to yield a much more “honest” relationship between software product vendor and customer.  That said, the IDC report hints that all will not be plain sailing with the “pay as you go” model because CIOs will want assurance that costs wont suddenly spike as a result of unexpectedly heavy usage.  I’m sure they will be willing to trade this for similar assurances to the software vendor at the other end of the spectrum (just a small joke for my software vendor friends).

One of the major areas that will require serious thought for software vendors is the nature of the usage granularity they should offer.  An obvious cut would be user access over time similar to the monthly “per user” fees typically associated with SaaS.  Another idea would be to mix in user access to different subsets of functionality.  This will probably require some changes to the software either to proactively control access to subsets of the functionality or to retrospectively report access to subsets of the functionality.  An obvious approach to the functionality granularity would be to identify and group use cases or transactions (using function point analysis techniques).  This could be very attractive to clients - imagine only having to pay for the parts of Microsoft Word that you actually use!  Even if that meant paying a premium for those functions that you need but only use rarely.

Here at DCG, we are unapologetic about our advocacy of function points as the best currently available, if not ideal, way to measure the (functional) size of software for the purposes of measurement, estimation and management of software projects and applications.  That said, we often implement non-function point solutions for our clients for various reasons.  One area where I believe that a non-function point solution can be valid is in the area of software rework or maintenance.  My theoretical basis for this lies in the six sigma philosophy of “possible errors.” Put simply, it seems to make sense that the number of bugs in a piece of software code could be broadly proportional to the number of opportunities there are to make mistakes which is the number of characters in the code or approximately the number of lines of code.

Edmund P. Morozoff of Medtronic has written a good article in the January/February 2010 issue of IEEE Software that makes this point in more detail (and even includes a nod in the direction of function points).  In particular, the article looks at the key issue of lines of coded added, deleted or modified.

As you may have gathered from some of my postings, I am very interested in the potential of static code analyzers, such as those available from our partner CAST Software, to generate reliable metrics around software size in addition to providing reports on the quality of the code.  Dr Paul Black of NIST wrote a nice article on static analyzers in the March/April 2009 issue of “CrossTalk” - the Journal of Defense Software Engineering.  I refer you to that article if you need an introduction to static analyzers.

In support of better software and automated software sizing, DCG has recently joined the Consortium for IT Software Quality (CISQ).  We plan to take a leading role in helping to develop specifications for automated software sizing.   It was thinking about how we might develop this idea that I remembered a reference in Dr Black’s article to the Software Assurance Metrics and Tool Evaluation (SAMATE) Reference Dataset.  This dataset contains the code for thousands of programs that can be used to test and (I hope) calibrate static code analyzers.  Of course, you may also find other uses for this valuable resource!

Being able to estimate software development more efficiently and effectively is both possible and practical. The following key points will guide you to a successful outcome.

•    We need to recognize estimating as a problem and a potentially costly problem at that. Not until we fully understand that improper estimating is a potential barrier will we be able to consistently and successfully deliver software.

•    The way we think about estimating should be reframed in the context of managing expectations based upon the best information available at the time. Estimating is not a crystal ball used to predict the future.

•    We can no longer afford to compromise on what we need regarding the input components that make up a successful estimating model. It will require some investment of time and resources but the payback will be well worth the investment.

•    And above all, don’t overly complicate your estimating model. Adopt practices such as FP Lite to generate size information that is statistically accurate enough for the job of early estimating. Collect the baseline data you need and compute your own internal delivery rates of performance.

There are several approaches and sets of rules for counting Function Points (e.g. IFPUG, COSMIC, Mk II, etc.).  They are all designed to use that most powerful of processors, the human brain.  Designing a computer program to automate function point counting has proved impossible to date (and maybe forever) for two main reasons: the current rules assume the pattern recognition and variable input capabilities of the human brain.  Put more simply, the rules assume that the counter can receive information (in a wide variety of formats including subject matter interviews) and pick out relevant architectural information.

However,  software has evolved for other uses -mainly static code analysis - which uses algorithms to understand the structure of code to evaluate its quality from a code design and implementation perspective.  Clearly, such code analyzers can do a good job of measuring software size in the form of lines of code but as we all know that is not particularly useful.

Some vendors are now modifying these algorithms to develop software size metrics based on the underlying architecture and using the sort of rules that an IFPUG function point counter would use.  It is not realistic to expect these tools to exactly replicate IFPUG function point counts in all circumstances because the IFPUG rules (embodied in the IFPUG Counting Practices Manual) are simply not written in a way that allows a program to encapsulate them all (and the input format variability problem remains).  However, these tools are now exhibiting significant signs of being a disruptive technology in the Christensen sense in that while they may not be great at doing everything that a human function point counter can do, they may be better (cheaper) at doing some things e.g. a monthly or quarterly recount of an application portfolio.

Here at DCG, I am committing us to work with vendors of these new tools to see if we can develop them together to improve the overall software sizing services that we can deliver to users. For a longer explanation of where we are, please see the presentation that I made to the 2009 UKSMA Conference in London.

One of the ideas that I float in the presentation is the possibility of creating a different, new software size metric that can be automated - a sort of Automated Function Point.  A good client of ours, Thierry Fraudet, thought about this and wondered how it might work for COTS or SaaS software?  True to form, Thierry answered his own question by suggesting that users could require vendors of COTS or Saas to provide function point counts with their software.  What a great idea?  And why limit it to automated function points?  Why not start asking COTS and SaaS vendors for IFPUG function point counts now?  As Thierry says, “What an interesting way to compare license fees!”

If you are interested in automated function points, I would love to hear from you.

One of the annual delights of being measurement and process improvement consultants for software development is the annual budget process.  We get involved in different ways every year.  Sometimes we are asked for independent validation of estimates on big or important projects, sometimes we are asked to help “size” (usually function point count) the requirements to validate them against capacity and sometimes we are asked to help develop, and/or facilitate (referee), a process for prioritizing the “too many” imperative projects against the “too few” resources available.

It is the “project prioritization” challenge that I want to pick up on today with a simple project classification idea that might help those of you who don’t have the final project portfolio for next year cast in stone yet.  I suggest that you could try to classify your candidate projects into one of the following three classes:

• Value Neutral Projects (VNPs) - these projects do not change the perceived value of the application or product to the end-user but may still be necessary.  Examples might be certain architectural changes, additional platforms supported or defect fixes.
• Value Add Projects (VAPs) - these projects add value for some users.  Examples might be new 3rd party interfaces, incremental functionality to some components, new foreign language capabilities or new reports.
• Value Multiplier Projects (VMPs) - these projects add value for all users.  Examples might be a new user interface, a new component or increased performance that leads to smaller hardware requirements.

Clearly, the project portfolio for any given year might include project from all three classes but the majority of the spend should be targeted at VMPs.  Of course, if you don’t have any VMPs in your planned portfolio, you might have to go back to the drawing board!

The same project may be classified in different ways in different circumstances so the input of the business continues to be crucial in this process.

Can anyone think of other examples of VNPs, VAPs or VMP’s?

As some of you may know, here at the David Consulting Group we are acknowledged to be the worlds leading independent provider of function point analysis services.  I say that by way of an acknowledgement of potentially vested interests in what follows.

One of our clients is taking an honest and critical look at its software development metrics and processes.  It has bitten the bullet and is using DCG to help it review all its software development governance and measurement.  For them, as for many of you, its that budget time of year.  Accordingly, they have asked us to prepare “top-down” and “bottom-up” benchmarks (see explanations of these below) for their software development.

They are comfortable with the quality of their team but they want to be sure they are maximizing the amount of their budget that goes to new development  (as opposed to maintenance).  They are interested in validating such things as their location strategy and the span of control in the development management teams.  All good stuff - this is a client that takes measurement software development seriously

Enough background, my main point here is that this client once used function points but gave them up because they did not seem to be worth the effort.  Te business never looked at them and couldn’t see that the small incremental effort to generate them was justified.

They are now regretting this decision.  Our top-down benchmark shows that the cost per FTE of the development group is excellent but now we need to know how good is the output per FTE?  Are they getting what they pay for - less than their competitors - or are they truly getting comparable output for less cost?  They have asked me how they can measure output.  While we are function point analysis experts, we are not function point bigots so we have been working to try to find other units of output.  Its not easy.  Function points are not perfect but for measuring the size of software, they are the best that is available.  The best proxy for an non-FP output metric is often “number of projects in releases” but everyone knows how weak that is.  Of course, the client suggested effort hours but quickly agreed with me that effort hours are an input metric not an output metric.  We are now considering whether function points need to be a part of the clients future as well as its past!

*********************************************************************************************

For reference purposes, here at DCG, we describe “top-down” benchmarks as being benchmarks that look at high level aggregate data for a company e.g. revenue, number of development staff, number of IT staff, number of locations, etc.  to calculate metrics that can be used to compare companies.  This is the sort of information that is often available in company’s annual reports or other published sources.  It can also be found in the survey-based reports available from various companies.  It is an interesting but, in our experience, not very precise form of benchmark.  At this high level and when survey data is involved, it is almost impossible to be sure that you are comparing like for like.  That said, it has it’s place.  “Bottom-up” benchmarking is our preference.  For this we take or calculate data from a representative set of projects in the target organization and compare to data from other organizations projects in either our own extensive database or other publicly available project databases such as the ISBSG data.  While there is still the risk of some ambiguity, it is much easier to test for and control in “bottom-up” benchmarking.  If you are looking for a software development benchmark, please make sure that the consultant you choose does not do a top-down analysis and use that with some broad conversion factors (e.g. number of function points that one FTE can or should produce per day/month/year) to extrapolate “bottom-up” results.  That can produce wildly misleading results.

The May/June 2009 issue of IEEE Software contains a paper by Onur Demirors and Cigdem Gencel describing some excellent work they have done to build a universal model for functional sizing that incorporates models of the three main approaches: IFPUG, Mk II and Cosmic.  That this is possible should not be surprising because they are all derivatives of the original work by Albrecht et al.  What is refreshing is that someone has risen above the battles over the relative merits and provided a useful comparison of the techniques.  The conceptual map diagram in the article is particularly useful as a tool to explain functional sizing to new counters or managers.

The model doesn’t handle all the detailed rules of each approach but the authors have included a couple of case studies to illustrate their success with the model.  Obviously, two data points do not make a conclusive proof and I am not advocating that we should use the universal in place of the three more established approaches but this article is both interesting and useful.  I recommend it and would welcome feedback after you have read it.

Conceptual Association of Functional Size Measurement Methods