Dr. Bill Curtis
Director, Consortium for IT Software Quality
According to testimony before a US Congressional Subcommittee, government administrators knew about the performance problems of Healthcare.gov long before the American public were used as system testers. Of course they did. I have never seen a system disaster of this magnitude when the technical folks weren’t alerting management about operational risks long before the system went live, if it ever did.
I will leave it to journalists to report the decisions behind an immutable Oct. 1 go-live date regardless of the operational consequences. As a result of these and other decisions, the health and quality of the software in Healthcare.gov suffered from the engineering equivalent of medical malpractice. As with all uninsured patients, the costs will be borne by the public. This series of posts will focus on the constellation of snafus in requirements, code, acquisition, system integration strategy, etc. that collectively created the Healthcare.gov fiasco. So let’s start where the American public attempted to start, at the start page with the now banished smiling lady who was the friendly face of Healthcare.gov.
Worse than this were obvious and easily detected security problems. For instance, Michael Scherer writing for Time’s Swampland site reported that, “An error message from the site relayed personal information over the internet without encryption, while the email verification system could be bypassed without access to the email account. Both security vulnerabilities could be exploited to hijack an account.” 3 This latter security problem was independently verified, while at the same time discovering a new severe vulnerability that appeared to be created while fixing a website problem.1 The sloppiness of easily discovered security holes such as these reveal coding performed in such haste that novice-level mistakes were made without even minimally adequate attention being devoted to their detection.
The onslaught of Americans trying to explore health care options on Oct. 1 created a national denial of service attack on a website that reportedly failed with a load of only several hundred during pre-launch testing. Although the unanticipated size of the response was blamed for the website’s problems, the analyses cited here indicate that the website’s resources would have been overloaded by even the anticipated light customer load. The American public completed beta test before the system integrator, the Centers for Medicare and Medicaid Services (CMS) within HHS, had barely entered alpha test.
How could this happen at a taxpayer expense that will easily exceed the currently reported 9 digits. Here are a few hypotheses for journalists to explore. Some of these may be culprits and others may not be involved, but they are common problems that lead to these types of software quality fiascos.
The development team was too inexperienced at building high volume websites because either the contractor did not have adequate talent available or had to staff the project with less experienced developers to win cost-conscious competitive bid.
Development work started too late because of the government’s contracting bureaucracy or the inability to stabilize site policies and requirements until late in the schedule.
Changing requirements slowed development work and may have forced significant rework.
By the time development started, the schedule was too compressed to adequately test the system or optimize the front end code.
When signing up for an account became a requirement for using Healthcare.gov, the front end had to interact with so many different systems hosted by different agencies that interaction with the browser became hopelessly overloaded if the session was required to be conducted synchronously.
System integrators at CMS did not have sufficient experience with systems at this scale to anticipate the effort and time involved or the breadth of quality assurance practices required.
Warnings from knowledgeable IT people were ignored at many stages.
Political priorities were weighted far higher than software quality, user experience, protection of confidential information, effective operation, or eventual system costs in the go-live decision.
Usually in IT fiascos like this, the correction of the initial problems reveals new layers of problems. So it is with Healthcare.gov. As performance problems were addressed, new waves of problems involving incorrect data being transmitted to insurance companies have begun to emerge. As of this writing we are far from having an effective, efficient, and secure system. The answer to how long it will take to correct the problems is in the defect and incident logs, not in the promises of administrators. In upcoming blogs I will explore several dimensions of the software quality fiasco that is Healthcare.gov. However, the ultimate lesson is simple. Software quality controls its own pace, and not all the dictates of administrators, legislators, and even Presidents can shorten it.