There are a number of issues associated with allowing your staff to use Free and Open Source Software (FOSS).  There are more issues associated with allowing your software developers to incorporate FOSS in your projects and applications especially if you are in the business of reselling such software as part of a product or service.

High on the list of challenges in both cases are the license implications.  Equally high on the list must be concerns about the code quality of the FOSS.  When it comes to validating the quality of FOSS, there is a simple strategy that could allow you to have more confidence in FOSS than in the off-the shelf software packages that your organization probably uses.

How?  Well, the nice thing about FOSS is that you actually have access to the source code.  This means that you can run it through a source code analyzer such as those available from CAST Software and their competitors.  You can reasonably expect that a good source code analyzer will not only report on the efficiency and quality of the code but also on any potential security threats.  Using an analyzer in this way will provide a layer of certainty that you do not have over the packaged or COTS software you buy without access to the source code.

The business case?  The time saved in bringing in FOSS solutions over the time taken to write and evolve them yourself should cover the cost of the analysis tool.  If you use a lot of FOSS then it makes sense to invest in buying the analysis software yourself.  If you use FOSS only occasionally then DCG can help you by performing one-off analysis as a service using the CAST Software tool whenever you need it.

I must admit that I rarely use the Standish Chaos figures on what percentage of software projects fail in respect of their original estimates of cost, time and functionality.  This may seem odd.  Surely, they support the business case for using consultants who specialize in improving software development?  Well yes but they never quite correlate with my experience and, perhaps as a result, the latest percentage never quite lodges in my memory.

My own  disquiet has always been based on the feeling that the statistics derived from any such survey may be more correlated to the type of questions asked than the reality on the ground.  For example, when a project finally reaches that point where the customers admit that they really didn’t know what they wanted when the project started, is that a project failure?  Probably it counts as a “yes” but I would argue that it was a controllable fault of the software developers or their process. Maybe that’s where the Standish publishers are going with it.

Anyway, I was pleased to see my concerns validated in an article by J. Laurenz Eveleens and Chris Verhoef in the January/February 2010 issue of IEEE Software.  With significant statistical analysis using reputable sources, the authors identify the following four problems with the Standish Group figures in their Chaos reports:

1. “They’re misleading because they’re solely based on estimation accuracy of cost time and functionality.”
2. The Standish estimation accuracy definitions are not sound.
3. If the Standish definitions are used to drive projects, they may cause large cost and time over-estimations.
4. The Standish figures cannot be extrapolated across organizations because large forecasting biases exist in any given organization which make extrapolation, in the authors words, “meaningless.”

As you may have gathered from some of my postings, I am very interested in the potential of static code analyzers, such as those available from our partner CAST Software, to generate reliable metrics around software size in addition to providing reports on the quality of the code.  Dr Paul Black of NIST wrote a nice article on static analyzers in the March/April 2009 issue of “CrossTalk” - the Journal of Defense Software Engineering.  I refer you to that article if you need an introduction to static analyzers.

In support of better software and automated software sizing, DCG has recently joined the Consortium for IT Software Quality (CISQ).  We plan to take a leading role in helping to develop specifications for automated software sizing.   It was thinking about how we might develop this idea that I remembered a reference in Dr Black’s article to the Software Assurance Metrics and Tool Evaluation (SAMATE) Reference Dataset.  This dataset contains the code for thousands of programs that can be used to test and (I hope) calibrate static code analyzers.  Of course, you may also find other uses for this valuable resource!

Mike Phillips and Sandy Shrum of the SEI have published a good article about the new CMMI v1.3 so rather than rehash it here, we have added the paper to our website.  It is recommended reading!

I’ve been talking to clients, colleagues and other industry experts about out proposed range of software development portfolio mix.  If you remember or choose to look it up (December 22, 2009), we have proposed a DCG Software Development Value Index which includes the concept of a mix of software development projects in the annual or quarterly portfolio that are either value neutral, value add or value multipliers.  We suggested that a good software development budget allocation for value neutral-value add-value multiplier would be $1-$2-$1 to $1-$2-$3.

That seemed reasonable to us from a customer (internal or external) priorities perspective. “Not so!” we heard. Readers and colleagues pointed out that our value neutral definition includes bug fixes and other software maintenance tasks.  What about the industry numbers that suggest that 60-80% of software development spend on average goes to maintenance?

Fair enough.  A $4-$1-$1 allocation may represent today’s reality for many organizations.  What is your split?  Do you think that is sustainable in the eyes of your customers?

Another thought on this is that you should remember to include customer (internal or external) funded development in your numbers.  Often, customers, which may be particular divisions in your company or actual clients if you a a software product organization) are funding work that is specific to their needs e.g. specific interfaces to other systems.  Under our assumptions, this type of customer funding goes into the “Value add” bucket.  Does that change your allocation much?

Interesting report by Mitch Betts in the the January 4, 2010 edition of Computerworld on how The Corporate Executive Board think the corporate IT funding model should evolve.

You know what this is all about - balancing the budget by transferring money from the profits centers to the costs centers (of which IT, and software development, are usually one of the largest).  Your organization probably does it in one of three ways:

• usage-based chargeback (which requires detailed cost accounting)
• lump sum payments (based on some arbitrary metric to divide up which profit center pays how much e.g. headcount)
• A mix of the two

The Corporate Executive Board believe that a much better way option for the future is to parcel up all of IT into 12-24 “business services” and assign costs for each one.  They recommend that the IT services should be described in business terms e.g. “video conferencing” not network “bandwidth.”  Although if video conferencing ios one service, how on earth do you describe It in only 12-24 services.

That brings me to my main point.  How would software development fit into these 12-24 services?

My guess is that it would be included in the services that it supports.  So if, say,  Service A needs to enhanced or modified, the software development cost would be included in the Service A cost to the business.   Hmmm.  That’s not so easy.  Presumably the typical cost of Service A is an operational cost.  How would you handle a one-off enhancement cost?  Would it be amortized over, say, two years and added to the operational cost?   What if the software development included some architectural changes that benefit more than one service?  Well you could make some arbitrary decisions about how to divide up the architectural costs among the services.  Oh!  Wait a minute! isn’t that where we came in?

Being able to estimate software development more efficiently and effectively is both possible and practical. The following key points will guide you to a successful outcome.

•    We need to recognize estimating as a problem and a potentially costly problem at that. Not until we fully understand that improper estimating is a potential barrier will we be able to consistently and successfully deliver software.

•    The way we think about estimating should be reframed in the context of managing expectations based upon the best information available at the time. Estimating is not a crystal ball used to predict the future.

•    We can no longer afford to compromise on what we need regarding the input components that make up a successful estimating model. It will require some investment of time and resources but the payback will be well worth the investment.

•    And above all, don’t overly complicate your estimating model. Adopt practices such as FP Lite to generate size information that is statistically accurate enough for the job of early estimating. Collect the baseline data you need and compute your own internal delivery rates of performance.

Why is it important to use fact-based decision making in software development?  Too many executives are prepared to treat software development as too complex to measure (or manage?). Instead, they operate in a state of continuous dissatisfaction based on occasional unsuccessful forays into the software development team which rarely yield significant change.  This is both uncomfortable and, potentially, illegal.  For example, as Davenport and Harris remind us in “Competing on Analytics,” the Sarbannes-Oxley Act of 2002 requires executives and other users of corporate data to demonstrate that their decisions are based on trustworthy, meaningful, authoritative and accurate data.  The act also requires executives to sign off that the data provides a clear picture of the business, major trends, risks and opportunities.

Can you or your colleagues really say this about your software development metrics?  Do you need help?

Consider the following 5 questions that you should ask yourself:

1. Is our software development data relevant?
2. Do we know where it came from (is it auditable)?
3. Do we have enough data (or too much)?
4. How can the data be made more accurate and valuable for analysis?
5. What rules and processes do we have in place to manage our software development metrics dat from its creation to its retirement?

“Val IT” is the 3rd in the series of four metrics for measuring the business value of IT that I introduced in my posting of July 7, 2009.  The first two in the series were the “Business Value Index” and “Total Economic Impact” (TEI).  Val IT is a complementary framework to the COBIT IT governance framework (described in our book, “The Business Value of IT” and elsewhere). While Val IT does not require a simultaneous COBIT implementation, it is clearly a good IT valuation option for organizations which already have COBIT in place.

Like TEI, Val IT started out focused on new IT investments.  Val IT is made up of three key processes which contain a total of 41 key management practices:

1. Value Governance (11 key management practices)

• Governance, Monitoring & Control
• Provision of strategic direction
• Define investment portfolio characteristics

2. Portfolio Management (15 key management practices)

• Identify and maintain resource profiles
• Define investment thresholds
• Evaluate, prioritize, select, defer or reject investments
• Manage the overall profile
• Monitor and report on portfolio performance

3. Investment Management (15 key management practices)

• Identify business requirements
• Develop clear understanding of candidate investment programs
• Analyze alternatives
• Definition and documentation of business case including benefits
• Assign accountability and ownership
• Manage through the economic life cycle
• Monitor and report on performance

“Total Economic Impact” is the 3rd in the series of postings on measuring the business value of IT that I introduced in my posting of July 7, 2009.  The 2nd in the series was the “Business Value Index” and TEI contains a number of characteristics that overlap with BVI:

• the use of a business case
• valuing intangibles
• calculating financial returns.

TEI adds:

• A methodology for quantifying risk
• The value of flexibility.

TEI has been used by Forrester for valuing IT investments.  It is of limited use in measuring the value delivered by IT in “normal operations” unless you take a view that the while IT budget is the investment. TEI is claimed to be more rigorous than BVI but the price of this is greater complexity.

TEI has four main components for assessing the total impact of a new investment:

1. Costs- the impact on IT

This looks at the costs incurred by IT compared to the costs of the status quo.  From the desired “Total Cost of Ownership” perspective, the “impact on IT” can be positive when money is saved or negative when money is spent.

2. Benefits - the impact on the business

Essentially, this is similar to the “Costs” analysis in that monetary flows, negative and positive, are aggregated, in this case, for all the non-IT departments of the organization.  Such flows would include improved productivity, training and so on.

3. Flexibility - Future Options

Derived from financial trading world, flexibility is measured as the value that this investment creates in terms of the ability to pursue other options in the future which are not currently available. The value of future options created is aggregated and added to the TEI,  For example, the investment being considered may be a change in software architecture which has a certain immediate value.  Additionally, at some point in the future, this change in software architecture could make it possible to switch to a cheaper hardware platform.  The switch to the less expensive hardware platform is not part of this project so the savings cannot be included in this TEI but there is some future value in creating the options and that value can be included in this TEI.

4. Risk

The first three types of analysis in TEI are additive.  The risk analysis reviews these the analysis of these three and generates an range of potential outcomes.  From this range, an expected or most likely value can be deduced and applied.  For example, if there is a 90% chance that the cost of training the business users will be $10k and a 10% chance that it will be $20k then the risk policy of the organization can be applied to make the risk adjusted cost $10k, or $20k or (10*0.9 + 20* 0.1) = $11k or some other value. Risk can be thought of as a multiplier of the sum of the first three to created “risk-adjusted costs” and “risk-adjusted benefits” to generate a “risk-adjusted ROI.”