Too often when I meet with executives I get confronted with, “Hey, you’re the CMM guy. How come my outsourcer is Level 5 and the software they sent me is full of bugs?”
CMM and its successor, CMMI, were never meant to be certifications for defect-free software products. Rather, they were roadmaps to help organizations adopt best software engineering practices that have proven over many decades to produce high quality software products.
There are many factors beyond process that affect the quality of a software product, such as:
- The skill of the developers and their knowledge of the domain of application,
- The novelty of the technology,
- The accuracy and volatility of the requirements, and
- The effectiveness with which a rapidly growing organization can sustain its high maturity development practices.
A high maturity process can mitigate the detrimental effects of these factors, but it cannot eliminate them. Ultimately, process only provides capability—the potential to produce high quality products. The achieving the full potential of an organization’s capability only occurs with sustained discipline and a dedication to managing factors that degrade it.
Most of us involved in the early development of CMM—Watts Humphrey, Mark Paulk, Charles Weber, and myself among others—believed that we did not even need to use the term ‘effective’ when describing the processes to be implemented. We believed that if developers would just adopt a practice or process, even if its results were less than expected, they would improve it over time to achieve optimal results. Unfortunately this did not happen as often as we had hoped.
Too many organizations were transfixed on getting their ‘Level 5’ for marketing reasons. They were more worried about compliance with ‘the CMM/CMMI standard’ than they were about implementing a process that produced excellent results. Robotic compliance does not ensure continual improvement.
Process appraisals compounded this problem. Appraisers were trained to evaluate compliance wih CMM/CMMI practices, but did not assess their result, that is, the quality of the product produced. For instance, in high maturity appraisals appraisers would inspect charts of defect data to see if the appraised organization had gotten defect rates under ‘statistical control’. Yet, there was no penalty for high defect rates based on the assumption that actions would be taken continually to reduce them. Although compliance to high maturity practices should help reduce defect rates, proof of significant reductions or even the achievement of a minimum threshold was not required to be appriased at Levels 4 or 5. The only guarantee of continually declining defect rates is a relentless devotion to product excellence.
This was the lesson of the Space Shuttle Primary Avionics Software System (SS-PASS), arguably the most process-disciplined software project in history. The SS-PASS team produced a product that was near-defect-free for most of its 30 year life. Before every shuttle launch the SS-PASS managers had to sign an affidavit affirming to NASA that to the best of their knowledge there were no safety-related defects in the SS-PASS. Consequently, the entire SS-PASS organization was laser-focused on the quality of their product.
The rigor of SS-PASS’s development process was only a means to gain the confidence needed to sign NASA’s affidavit in good faith. They were the living model we used in defining Levels 4 and 5 of the original CMM, but in our interactions they never claimed to be Level 4 or 5. However, they had no trepidation in signing NASA’s affidavit. Would your suppliers be willing to sign such an affidavit about the risks their product poses to your customers or your business?
The risk to the business is in the functionality, structure, and execution of the software product, period. A high maturity development process should reduce the risk, but many factors conspire to ensure that this is not always guaranteed. To paraphrase a famous quote about misplaced focus in past U.S. presidential elections (“It’s the economy, stupid”), good governance of IT investments must focus on the end result that is delivered to the business or customer—“IT’S THE PRODUCT, STUPID!”
Convenient governance is not necessarily good governance. Convenient governance accepts a process appraisal result delivered by someone else as ‘certifying’ that a supplier has a mature, perhaps even high maturity, development process. However, good governance requires that you perform the due diligence required to ensure that these processes are consistently applied by the department or location where your software will be developed.
Since a high maturity process enables but does not guarantee a high quality result, good governance requires more. You should evaluate a supplier’s historical data to verify they are achieving the product results that a high maturity process should deliver. If they do not have such data, run.
Most important, every organization that receives a software product from internal or external sources should proactively determine that its quality is sufficient to meet the needs of customers or the business without exposing them to unacceptable risks. There are at least four attributes that should be evaluated in exercising proper governance over the quality of a software product:
- The functionality is correct and sufficiently complete to meet user needs,
- The structure of the software does not exhibit weaknesses that expose the customer or business to unacceptable risks of outages, security breaches, data corruption, or other damaging operational problems,
- The structure of the software does not expose IT to excessive costs of ownership, and
- The software will behave as expected when loaded into the operational environment.
Good governance requires a Quality Gate where these attributes of the delivered software can be evaluated and approved before it is put into operation. Even if a Quality Gate is operated by contracted staff, the organization acquiring the software must oversee its operation and results. Among the attributes of a good Quality Gate are the following features.
- A senior member of the acquiring company oversees and reviews all results.
- This person has the authority to stop a software delivery from entering operations.
- The evaluation is conducted against quality thresholds that are written into the requirements, or for external suppliers, into a contract or documented agreement.
- Data is retained across deliveries for evaluating the capability of different sources.
The technologies for functional testing, static structural analysis, and dynamic behavioral analysis have matured sufficiently to provide strong, reliable feedback on the various quality characteristics of a software product. The 1990s was about trying to get software suppliers to implement a development process that was capable of delivering high quality products. The 2000s were about defining the architectural and structural rules for a reliable, secure, scalable software produc. The 2010s must be about creating the governance mechanisms that ensure the increasing automation of business decreases rather than increases risks to customers and operations.
In the face of serious operational problems, governance is only defensible if IT can affirm that it used state of the practice techniques to try and detect product flaws before going operational. However, in the absence of governing the quality of software products, IT stands fully exposed in front of customers and the CEO. CEO’s don’t care to hear about a developer’s processes or maturity. Once a software product goes operational, you own whatever happens next. Ultimately, it’s the product!