Major Update At The End

In my last post I assessed the technical challenges and issues with the launch of Obamacare. In that post I presented the clear signs of a government program technically in over their heads. I pointed to this chart of the core information traffic cop (the Federal Services Data Hub) as a clear sign CMS was blissfully naive of the sheer complexity of what they were attempting: [click to enlarge]

This simpleton concept diagram completely glosses over the challenges in design and testing needed to make Healthcare.gov work. Screw this up and nothing can fix the problems without a complete restart. What I exposed previously was the technical truth hidden behind the child-like simplicity of this chart:

When I see diagrams like this I see a technical project in serious denial and headed for a crack up. As I noted before, the right side is really something like 50 state Medicaid/CHIP systems and the left is the 24 State Based Exchanges. The right side also has all those Federal systems the HUB has to query for each individual [applicant].  The Hub uses a mush of messages, emails and files to interact with these entities, adding to the impossible complexity. And it translates a lot of unstructured data between all these outside systems.

…They show 6 entities, but the reality is 26 SBEs on the left, 60+state and federal systems on the right, and millions of transactions coming in to build a file for each of the 17 million applicants.

The group inside HHS responsible for this debacle is CMS. And CMS is clearly not staffed or experienced enough to manage something of this complexity – if this is their view of the problem in March 2013. Less than a year before getting ready to Go-Live!

Reader JWB alerted me to even more reporting that confirms my suspicions, and proves Healthcare.gov is going to go down in history as one of the biggest, most painful examples of why government cannot do certain things. Notice how insiders expose the very conditions I caught a glimpse of in the small snippets of documents I could scrounge up on the internet:

Federal officials did not permit testing of the Obamacare healthcare.gov website or issue final system requirements until four to six days before its Oct. 1 launch, according to an individual with direct knowledge of the project.

…

“Normally a system this size would need 4-6 months of testing and performance tuning, not 4-6 days,” the individual said.

The source said there were “ever-changing, conflicting and exceedingly late project directions. The actual system requirements for Oct. 1 were changing up until the week before,” the individual said.

This is supported by this schedule chart I pulled from a Colorado SBE site which shows Functional Testing (the red circle) bright red – meaning a very serious risk to the entire endeavor.

The report was from 9/11/13 (blue vertical line), the red dot is sitting 1 week out from there. Clearly, things slipped even in the last 3 weeks since the latest reporting is indicating that red dot slipped another week before “Go-Live” day – Oct 1, 2013

The insiders are confirming that this disastrous result was because CMS pretended it had the expertise to run such a program, which it clearly did not, and does not now, have.

The root cause of the problems was a pivotal decision by Centers for Medicare and Medicaid Services officials to act as systems integrator, the central coordinator for the entire program. Usually this role is reserved for the prime information technology contractor.

…

The individual described the project as suffering from a “lack of an end-to-end business and technology vision for the project,” adding that “the hardest part of any technology project is not the technology — it is the business process decisions, what is the system supposed to do and how it will it do it.”

In addition, “The challenge with this project was that the decisions were made very, very late in the project, and no one organization … seemed to know how this complex ecosystem of applications, interfaces, user processes and hardware should all work together.”

I also alluded to this being the probable situation in my previous post as well:

I tend to agree heavily with this commentary:

In a nutshell, many agile processes — and especially extreme programming — reject the big design phase as part and parcel of rejecting the waterfall methodology. Agile processes follow more of an “organic” software development, where developers start coding the smallest increment possible and “grow” the working software up, little by little, with constant customer feedback. These agile methodologies call for “user stories” to design each small increment of the system being developed. To be fair, agile can work for some software projects, but I assert that it is the kiss of death for projects with many moving parts, multiple organizations and complex interactions.

Personally, I am a fan of design and system architecture, and I have witnessed many successful projects that resulted from good design. Furthermore, I flatly reject the notion that user stories can suffice for large IT projects like HealthCare.gov that require scalability, data integration, numerous system interfaces and other complexities.

CMS relied on scrums and sprints when what they needed was a Systems Engineering/Architecture lead to scope the work, refine the interfaces between all the complex pieces, and ensure proper testing was done to burn out problems and enhance robustness. Things very few even in the DoD and NASA worlds can pull off.

“The system integration piece is an important one because that’s where it all comes together. And I don’t see any evidence in any of any of those contracts that says ‘system integration’ or overall management. So that leads me to believe that CMS — as others have said they did — chose to do the integration of the system of themselves.”

…

Other IT experts said it is highly unusual for a government agency to act as a systems integrator. “Literally, it’s crazy,” said IT consultant Charles Martin. “The reason government goes to external prime contractors to do integration is because they [government] know they’re no good at it.”

Martin has been an IT consultant to the U.S. Defense Department, the IRS and the Commerce Department.

“I don’t recall ever hearing that HHS was notably good at running big IT projects,” he said, referring to the Department of Health and Human Services, the parent agency of both the Centers for Medicare and Medicaid Services and much of the Obamacare infrastructure. “The government generally doesn’t do big IT projects. It hires somebody else.”

Since CMS  blew this in such naive fashion, it is obvious there is no way CMS is going to have the know-how to pull themselves out of their mess. They are still just becoming aware of how little they knew! Forget being able to do the surgery to fix it.

There are groups inside the government civil service corps that can Systems Engineer this kind of program. They exist in the DoD, NASA and a lot of dark programs. But even within NASA, DoD and other areas of the government, you can run into armchair Systems Engineers and Architects who think they know how to do this. But this is a highly specialized skill.

A skill I have been honing for a quarter of a century on this nation’s most challenging programs (think Space Station and Star Wars).

What we are seeing here is massive amateur-hour from HHS. I had no idea once the inside story got out it would be this bad. As bad as I painted it yesterday – this now looks to be much, much worse today. This is bordering on criminal ignorance. A prime example of waste, if not a fraud in the sense CMS and HHS pretended they knew what they were doing.

Heads should roll.

Major Update: This is an incredible peak behind the curtains posted late last night, which once more confirms what I suspected. It is nice to know my Systems Engineering and Analysis skills are still fairly sharp.

I will note this is from the perspective of a policy person, communicating with like kinds inside CMS and the insurance industry. And because of that, I will note where he may have drawn wrong conclusions, or where his sources have. But overall, the picture painted is bad – really bad.

So let’s get to it:

The assessment below summarizes conversations with five CMS officials and three insurance-industry insiders, …

The CMS officials are all career agency personnel, not political appointees; they are fairly senior people but not so senior as to be routinely privy to political discussions. All are involved in the exchange project in different ways. What they see is the nitty gritty operations of the program. But they are policy and management people, not information-technology experts.

…

The technical architecture of the federal exchanges and to a lesser extent the state ones has been very badly screwed up. The problem may be so bad as to render Obamacare’s rollout impossible in practice at this point. But it may not be. And right now no one knows if it will or will not.

First may I commend these people for letting the light onto this problem.  There is no way to get right in this kind of situation if the bunker mentality is in place. They are doing the right thing be exposing the situation, so saner heads can assess the best path forward (note I did not say the least painful, because the right path will be painful).

The reaction of these individuals to what has happened in the last two weeks is the reaction of people who are coming to realize that their expectations and understanding of web development were mistaken. … It is now increasingly obvious to them that this is simply not how things work, that building a website like this is a matter of exceedingly complex programming and not “design,” and that the problems that plague the federal exchanges (and some state exchanges) are much more severe and fundamental than anything they imagined possible.

…

The people I spoke with did all confirm the importance of one other detail in the Times story: that CMS did not hire a general contractor to manage the exchange project but handled that overall technical management task itself. None of the people I spoke with wanted to get into how this decision was made or at what level, but all of them agreed that it was a very bad idea and was at the core of the disaster they have so far experienced.

…

The tone of the CMS officials who spoke with me was a kind of restrained panic. Among the insurance company officials (who, I should stress again, work in the Washington offices of some large insurers, and so are basically policy people and lobbyists), there was much less restraint. The insurers are very, very worried about the viability of the exchange system—especially but not exclusively at the federal level.

…

Obviously what’s happening here is nowhere near that scale or significance [of Katrina], but for the people involved—for the officials in charge of running this system—this is a category 5 nightmare, and in a number of instances they traded in stories they’d heard from others which struck this outsider as basically impossible. They have been witness to problems in recent days that they would not have believed a month ago, and so they believe things about the extent and depth of their problems that may not be true.

As I have noted in this and the prior post – no surprise here at all. The documents I have read are heavy on policy, void of detailed technical architecture and planning. And the fact key late decisions undid years of prior work are a sign of this technical mismanagement. My comment above was these people are just now beginning to understand what the challenges were and how little they knew about how to deal with them. This reporting confirms this. Which means they are months away from being able to fix it. Doing this right takes a lot of skill and time. Undoing something this SNAFU’d takes even more, and that talent is even rarer.

So where are the ‘glitches’ – the Data Hub and the interfaces to all those systems of course. And here is where I take some issue with the insiders, only because their documentation says something very different:

But there are very serious problems beyond that, which are more like the sorts of problems people were predicting before the launch: database problems at the nexus of several federal and industry data sources. The federal data hub itself is so far doing reasonably well at its basic tasks, and that has come as a relief to CMS. But some of the site functions that rely on the hub, both in the federal exchanges and a number of the state exchanges, remain highly problematic. The calculation of subsidies continues to fail tests, and it’s pretty clear that some actual consumers have made actual purchases with bad information,

Full stop. The Hub is responsible for collecting the pertinent personal data from all those state and federal databases and then, from there, calculating premiums, co-pays and subsidies for the individual based on that information. The Hub is the universal translator of these sources and sinks. The Hub cannot be doing its basic functions of their is garbage going around.

Meanwhile, the back-end communication between the exchanges and the insurers has been terrible, as is increasingly being reported. The extent of these problems has also been a surprise to CMS, and here too an increase in volume if the user interface issues are solved could lead to huge problems that would be very difficult to correct. CMS officials and the large insurers thought at first that the garbled data being automatically sent to insurers must be a function of some very simple problems of format incompatibility between the government and insurer systems, but that now seems not to be the case, and the problem appears to be deeper and harder to resolve. It is a very high priority problem, because the system will not be able to function if the insurers cannot have some confidence about the data they receive.

Read the whole article, because its basic policy implications are clear. The Obamacare house of cards is about to come down. It is going to come crashing down because Obamacare was, and is, a vaguely thought out pipe dream. The author goes onto say critics never thought there would be a technical breakdown.

Well forgive me with disagreeing, but of course may of us did. We knew the challenges and barriers to trying to control 1/6th of our economy were impossible. We knew the Democrat faith in government genius and flexibility was pure fantasy. We who work on those modern marvels of national technology (think Space Station again) knew all along the 100 thousands lines of policy were going to require billions of lines of  code. We knew.

We just did not know HHS and the WH would expose the folly in such a spectacular and grand way.

Update: A consensus is building, and it won’t be good news for Obamacare, or Health Care in general in this country:

I noted yesterday how the most basic rules of software development were violated in this process. Failing, for example, to have a decent design before work started. Having specifications turning up in the middle, even towards the end, of the development process. But now we’re hearing even worse:

This is a Gov-Run Botch-Job of epoch proportions.

Tags: Healthcare.gov, Obamacare, Rollout