Your $500MM Website Launching Next Week? Let’s Start Testing
Last month I was browsing the news when a headline caught my eye:
This was a few weeks after the troubled October 1st rollout of the site, when the details of what went wrong were starting to emerge. The article touches on the underlying issues causing the project’s derailment, which have since been widely reported and debated on Capitol Hill. In that time, the issue has morphed into a political football while those involved point fingers.
Reading these articles, I see many of the classic project management mistakes. These are mistakes that are made all the time, mistakes I’ve observed first-hand, even mistakes I’m guilty of making myself.
But seeing it all unfold in a public setting makes for an interesting case study. Like any train wreck, it’s hard to look away. So let’s take a closer look.
Lesson #1: Guess what? Your code doesn’t work.
I actually laughed out loud when I read the headline above. An anonymous source in the article states that “normally a system this size would need 4-6 months of testing and performance tuning, not 4-6 days.” And they’re right.
Clients sometimes balk at the added expense of testing, when in fact this is arguably the most important stage. Even the most talented programmers’ code is going to have bugs in it. It’s an expected part of the software development process, and the more complex, functional, and custom your website is, the more important this step becomes.
It varies from project to project, but for something highly functional and integrated with other systems, testing and fixes can represent up to 15 – 20% of the total budget. According to reports online, the cost of healthcare.gov is as high as $500 million, so lets do the math. At an hourly rate of $125, it would have taken 15,000 people testing for 40 hours over that last week to get it done correctly. Good luck with that!
Lesson #2: Deadlines are usually arbitrary.
Projects kick off with a launch date in mind. It is often an official-sounding date like “January 1st” or “the beginning of Q3.” While deadlines are ideally aligned with business goals, they are important regardless as projects can drag on indefinitely without them.
However, the question should be asked, “What happens on January 2nd if the project isn’t complete?” The answer is usually “nothing.” Business marches on, and the project continues until it is ready to be launched. Most projects don’t have a stakeholder as influential as the president or a goal so politically charged. In this case, the October 1st deadline was a firm and justifiable one.
But the adherence to the deadline caused them roll out a product that was woefully unready for mainstream use. The results speak for themselves.
When a deadline is approaching, weigh the risks of rushing to meet it against the cost of extending the timeline. Crashing a schedule often results in burnout and sloppy work, making your problems worse.
Lesson #3: Stick to the plan.
In what is reported as a “late decision1,” the healthcare.gov website was changed to require users to register before they could see prices of policies. The original plan was to allow users to “window shop” and only require an account at the point of purchase. This is consistent with the Medicare website2 and an e-commerce best practice. For reasons that are still being debated on Capitol Hill, the change came down reportedly a month before the go-live date.
While the implementation was completed before the deadline, this late change surely diverted resources away from the other activities that were scheduled during that time. It also introduced an untested change to the system, which can cause a cascading effect. On the launch date, the unplanned for load on the registration module was widely blamed for being unable to use the website2.
Requirements changes are part of any project. Technical limitations may not reveal themselves until later in the process. The landscape may change during the course of the project. But it should be expected that a significant change will create a corresponding delay and/or increased cost. As the project management triangle shows, there is fast, cheap, and good – and you can only pick two.
If something comes up late, decide if it’s something that must be part of the current iteration. If so, be prepared for the associated costs and time. If not, it can be added during subsequent enhancement phases. Fredric offers tips on how to launch a new website providing more detail.
Lesson #4: The left hand must know what the right hand is doing.
According to CNN, the healthcare.gov website was subcontracted out to 55 companies to build the individual pieces, but with no general contractor to oversee them3. Not until late October was Quality Software Services, Inc. (QSSI) tapped to oversee the fixing of the mess. Prior to that, the Center for Medicare and Medicaid Services (CMS) were acting as their own general contractor and systems integrator4, a role for which they were wholly unsuited.
Any project requires multiple skill sets and someone (or a group) to bring it all together. The individual modules built by the healthcare.gov subcontractors reportedly functioned independently, but the lack of “end-to-end” integration testing and subsequent failures reveal the lack of competent oversight.
Or you can think of it from the reverse angle; would you really want your project not to be managed?
How it applies to you
The Obamacare website is a massive project, far bigger than most of us will ever be involved in. The large scope of the project magnifies the effects of poor decisions and changing requirements. But the same principles apply to smaller projects. When planning, bidding, and building your project, be realistic about the effects of the decisions you’re making.
You don’t want the president on your back.