I have been doing more research on the programmatic and technical management of the Healthcare.gov program, and have identified the main reasons ObamaCare’s roll out was such a disaster. Bottom Line: Too much scope in too short a time. And the root cause of that was government taking on more than it knew how to do and not delegating enough technical management to contractors.

But before I get into those details, I must address this idea that it was the contractors’ fault and not the fault of HHS and CMS.

Health and Human Services Secretary Kathleen Sebelius will tell a House committee tomorrow the site’s botched rollout was the result of contractors failing to live up to expectations – not bad management at HHS, as the contractors suggested.

It most definitely WAS the fault of the government, who mismanaged this thing from the beginning. Not only did they not understand what they were doing, they took on the technical lead role to design and specify the solution, then integrate and test it. From what I have seen their intentions and some strategies were OK, but their execution was naive and clumsy.

Of course, the Sebelius CYA effort begs the question: if these contractors screwed up, why did HHS/CMS just appoint one to take over for them as technical lead? Got an answer to that?

And as for the idea HSS/CMS was not aware of the problems as their amateur effort began to fall apart this summer, the evidence is clear this was not the case. I noted this a while back in documents from state exchanges who were growing more and more concerned that CMS was losing control of the situation.  From this initial post on the subject [click to enlarge] is this fascinating report from the Colorado Exchange on Sept 13, 2013:

The 3rd issue notes The [Federal Data Services] Hub had just completed stand alone testing! End to End testing was yet to begin – and this is Sept 13? They ran out of time, and that is poor technical management. And then they deployed anyway – and that is political stupidity.

But an even earlier risk assessment from the Washington State Exchange effort from Jul 2013 pointed to the sane problems:

This “urgent” risk notes the interfaces between the State Exchange and the CMS Federal Data Services Hub [FDSH]. The conclusion (or warning to everyone state and federal) was:

These interfaces and associated systems are critical to the ability of the [Washington State] Exchange to be operational on the October 1 Go-Live date. … HBE [Washington’s Exchange] application processing has a high dependence on the [CMS FDSH] interface. As we noted in our June month-end assessment, CMS (verbally) re-confirmed a mid-August delivery of fully operational FDSH services.

OK, stop right there. In June 2013 Washington State (and probably all the 25 other states with exchanges being deployed) were pushing HHS/CMS to ensure HHS/CMS could meet schedule.  HHS/CMS promised an August delivery of the Data Hub (FDSH).  But we now know from the report from the Colorado Exchange from September that promise was never met.

Washington State goes on to conclude:

HBE will not be able to conduct full end-to-end testing of services until CMS makes this delivery. The probability of an on-schedule delivery of all FDSH services is, at best, questionable – and worrisome.

Look, Washington State is not creating and documenting these risks in isolation. They are being passed to HHS/CMS as well. And they were predicting failure in July.

Conclusion: HHS/CMS knew about these issues and their ramifications.

The one piece of this mess that HHS/CMS defends as ‘working’ each and very time is their precious Data Hub. Sebellius goes out of her way to make it clear the Data Hub is working. But the Data Hub is clearly the problem most noted here! There is something about the Data Hub, it seems to be the HHS/CMS Holy Grail.

As can be seen in this schedule that came out on Sept 13 in that Colorado State report, the Hub was being patched and tested all the way up to “Go-Live”

The Data Hub is on the fourth row under “CMS Required Testing”. Click to enlarge and you will see FDSH testing going on in parallel with end-to-end (E2E) testing. This is a sure sign of a program heading for a crack up.  If you are doing E2E in parallel with stand alone testing of your center element, you are in serious trouble. And it is apparently during this period that this happened:

Days before the launch of President Obama’s online health ­insurance marketplace, government officials and contractors tested a key part of the Web site to see whether it could handle tens of thousands of consumers at the same time. It crashed after a simulation in which just a few hundred people tried to log on simultaneously.

…

The Centers for Medicare and Medicaid Services (CMS), the federal agency in charge of running the health insurance exchange in 36 states, invited about 10 insurers to give advice and help test the Web site.

About a month before the exchange opened, this testing group urged agency officials not to launch it nationwide because it was still riddled with problems, according to an insurance IT executive who was close to the rollout.

How can HHS/CMS credibly claim they had not clue about the situation? CNN has blown the lid off that excuse.

So why do they continue to insist the Data Hub is the one piece working well?

This recent reporting would indicate just the opposite:

Connecticut residents trying to sign up for subsidized health care coverage through Access Health CT, the state’s health insurance exchange, won’t be able to complete the process because of an outage involving the federal system used to verify applicant information, Access Health announced Sunday afternoon.

The system, known as the federal data services hub, is used by the exchanges to verify information including citizenship and income of applicants. Access Health said in a statement that it received notice of the outage from the Centers for Medicare and Medicaid Services, which oversees the systems used for people to enroll in coverage made available as part of the health law commonly known as Obamacare.

While Connecticut does not use the problematic healthcare.gov website to enroll people in its exchange, as 34 other states do, the state’s exchange must still use the federal data hub as part of its enrollment process.

“Customers will not be able to access Remote Identity Proofing, Social Security Administration or Advanced Premium Tax Credits also referred to as subsidies. As a result, consumers will not be able to complete the sign up process for Medicaid or Advanced Premium Tax Credit determinationsm,” the Access Health statement said.

It is so bad Alaska has just put the stop to the whole mess until HHS/CMS can get the damn thing to work right:

Enrollment is suspended until a major issue is resolved, said Tyann Boling, the chief operating officer of Enroll Alaska. The issue is that the premium subsidy calculator for the Alaska exchange is underestimating the amount of the federal government’s portion of the total premium payment made to the insurance companies.

…

Friday Boling said she received an email from Susan Johnson, Region X director for the DHHS saying that the government was working to fix the Alaska subsidy calculator.

The Department of Health and Human Services expects to have the site fixed by the end of November.

There are 51 state Medicaid/CHIP systems (add DC) to interface with, half a dozen federal data systems such as the IRS and SSA (to name two), and 26 state Exchanges each implementing a subset of the Federal Exchange. That totals about 85 major and complex interfaces to work the ‘glitches’ out of.  Not to mention the hundreds of insurance companies trying to get valid data out the back end. Medicaid enrollment data to the states does even not “Go-Live” until this week!

I would wager a lot of the recent crashes have been attempts to fix the Medicaid Enrollment piece – which are adding more ‘glitches’ at each attempt.

So now we get down to the core problem. The wonders of this precious Data Hub were laid out by its author, Henry Chao, Deputy Chief Information Office and the CMS Technical Architect for what has happened. In the initial documents (see here for a primary one) you see two things: this wondrous ubiquitous system seamlessly (never a good word to see in a spec) tying to together around 85 state and federal data systems, and from there producing tailored products (insurance applications) for individuals by state – easy as pie.

The arrogance of such an endeavor is mind boggling. Tie together 85 data systems that are all independently  developed and implemented? Just mapping the data model across these would take 2 years! Each data system uses some ID to identify some attribute (e.g., name, given name, first name,  etc). All this information has to be collated in the exchange so that it can be used to calculate what options each individual has. But of course, it is not just individuals, it is families too, so don’t think one individual per application.

The first step HHS/CMS took in trying to tackle this entire effort all at once – was to tailor their process and technical documents to try and squeeze this 10 year effort into a 3 year program (section 3, page 11):

Given the need to respond quickly to the health reform requirements of the Affordable Care Act, CMS will use the Exchange Life Cycle as a streamlined model to guide and coordinate the development and implementation of the Exchanges. Various organizations will be involved in developing the Exchange IT solutions. Toward those ends, the Exchange Life Cycle accommodates all forms of systems development methodologies. CMS encourages the use of agile systems development methodology and will accommodate agile processes in the Exchange Life Cycle stage gate reviews as appropriate.

Full stop.  This is an enormous undertaking with lots and lots of moving parts. Each of the external systems is under independent development, many are legacy systems. You cannot use an Agile SW Process to tackle this job.  First and foremost you need to define all interfaces and get them on a coherent schedule.  Then you have to model all state and federal sources of data (i.e., define how each one labels common information and create a Rosetta stone to related each existing label in those data sources to a common definition, like “first name”) and then you need to model all data products (each state and each insurer in the state has unique definitions for data as well). Just understanding the format for “DOB” or “Date Of Birth” in terms of whether the day or month comes first is tedious, time consuming work.  But it must be CHECKED and verified if you want to get it right from the applicant’s input through onto the insurance enrollment form.

And of course, data modeling and translation is one of those things HHS/CMS tailored out to save time [click to enlarge]:

R = Recommended, which in tight programs means it won’t get done.

If you look at the above diagram and process, you also need to mentally apply the timeline across the top to see how ridiculous this plan was/is. The source document is dated March 2011.  All four of the purple-colored phases have to be done by October 1 2013 – around 30 months.  And most of the time (more than half) was dedicated to Concept definition through Requirements and Design. Development (3rd phase) did not begin until what looks like 20 months in.  Give it a year at best.

85 interfaces to build and test in 12 months (not including insurance providers)? With no overarching data model to provide a managed Rosetta stone between applicant data, federal databases, state Medicaid/CHIP databases and State Exchanges?

HHS/CMS developed a fictional program, that’s the problem here. They threw in everything they could possibly think of, used the sure-to-fail big bang approach (all at once), shunned critical aspects of the waterfall approach over agile wanderings, and left no time for testing and tuning. Worse yet, I found statements were this was patchwork funded across state and federal programs. And there was no way to hold states to any schedule (says so in the Statement of Work).  From top to bottom this was a fantasy.

If Congress does not hold someone accountable, then all our tax money can and will be thrown down the toilet because even obvious monstrosities of ineptitude like this will go unchallenged. Congress needs to independently investigate what happened and document ALL the lessons unlearned (and ignored). They need to make an example of programs like this which fly on good intentions yet fail on incompetence.

Tags: Alaska, CMS, Healthcare.gov, HHS, Obamacare