Major Update: Reporting by the Washington Post confirms just about everything I have been posting about Healthcare.gov. Their source – top administration officials:

President Obama’s top economic aides were getting worried. Larry Summers, director of the White House’s National Economic Council, and Peter Orszag, head of the Office of Management and Budget, had just received a pointed four-page memo from a trusted outside health adviser. It warned that no one in the administration was “up to the task” of overseeing the construction of an insurance exchange and other intricacies of translating the 2,000-page statute into reality. Summers, Orszag and their staffs agreed.

Not surprisingly, they were right.  But it get’s even better:

“They were running the biggest start-up in the world, and they didn’t have anyone who had run a start-up, or even run a business,” said David Cutler, a Harvard professor and health adviser to Obama’s 2008 campaign,

…

Soon, however, it became evident that the office — with more than 200 people — would not survive on its own. It lacked tools, such as the ability to award grants and outside contracts, that were vital to its mission,

…

The Medicaid center’s chief operating officer, a longtime career staffer named Michelle Snyder, nominally oversaw the various pieces, but, as one former administration official put it: “Implementing the exchange was one of 39 things she did.

…

That spring, CMS had begun writing specifications for the IT contracts to build the federal exchange, but the White House again insisted on caution. A larger number of states than expected were signaling that, under Republican pressure, they would refuse to build their own online insurance marketplaces and would rely on the federal one. The more states in the federal exchange, the more complex the task of building it. Yet, according to several former officials, White House staff would not let this fact be included in the specifications.

This last item is incredible in its incompetence. No surprise it came straight out of the White House – Incompetence Central.  How could you not specify the system you needed to win the political argument? Can we say “paranoid”!

Such delays were “a singularly bad decision,” said Foster, the former Medicare chief actuary. “It’s the president’s most significant domestic policy achievement,” he said, and the very aides who had pushed the law through Congress were risking bad implementation “for a short-term political gain.”

…

Inside CMS, meanwhile, some staffers were aware by late 2012 that the work of building the federal exchange was lagging, according to a former HHS official — a much earlier timeline than has been previously disclosed. Some employees in the main office involved with building the exchange repeatedly warned at meetings late last year and in January that so many things were behind schedule that there would be no time for adequate “end to end” testing of how the moving parts worked together, the former HHS official said.

…

A month earlier, on Sept. 5, White House officials visited CMS for a final demonstration of HealthCare.gov. Some staff members worried that it would fail right in front of the president’s aides. A few secretly rooted for it to fail so that perhaps the White House would wait to open the exchange until it was ready.

Yet on that day, using a simplified demonstration application, the Web site appeared to work just fine.

Much more here. And this is a must read.

As I wrote below before I read this – Healthcare.gov is an epic failure of activist government – end update

It is clear Healthcare.gov was not ready to “Go–Live” on October 1, 2013.  And those in government who either were ignorant of the problems or who willfully ignored them should be held accountable. This is one of the clearest cases of waste and fraud I have ever witnessed. While it was driven by lofty intentions, the end result is no different than any other case of wasting tax payer money and making false claims about the experience of the team and quality of its product.

If what we see today had been done by a private company under contract to the federal government, that company would have been fired and open to serious penalties. I have seen companies nearly go under for much less incompetence. There cannot be a dual standard here.  Whatever is proper punishment for a private contractor is proper for a government agency when they do much, much worse.

Here is what we have learned in the last few days:

• First, the system WAS NEVER properly tested.
• Second, the system WAS NEVER properly secured.
• Third, the system WAS NEVER properly designed.

All of these occurred because the project was epically mismanaged by HHS/CMS.  For these failures (and in some cases crimes if a government contractor was responsible), heads should role and careers ended. Plus some compensation must be made to all the victims of this (and the victims go beyond wast and fraud of tax payer funds, many are going to have their personal data exposed).

Let’s recap the evidence of lack of proper testing:

CBS News has learned the website failed with a small test pool of 200 to 300 people that included employees from the government and insurance companies. The government employees worked at their own computers and desks within the Centers for Medicare and Medicaid Services, which oversaw the health care implementation.

According to sources familiar with the process, CMS employees were provided fake personal information to enter into HealthCare.gov rather than their own data and were given a date that testing would begin. However, on that date, the employees were told it was being postponed.

When the testing finally took place in late September, the testers started trying to create an application. Just a couple of pages into the process, everything “ground to a stop,” said one source.

“It froze. It couldn’t go forward. It crashed,” the source said.

A couple of days later, testers tried again and had a similar outcome. They were never able to successfully browse plans for cost estimates.

“It was unequivocally clear from testing … this wasn’t ready,” said a source close to the testing.

Emphasis mine. CBS aired a damning news report on the matter:

Note this is HHS/CMS testing, and if anyone thinks the heads of HSS and CMS were not “in the loop” two weeks prior to ObamaCare’s roll they are just lying to themselves. I work these kinds of programs and launch date brings out ALL the big wigs. Errors, mistakes and failures percolate to the upper echelons with warp speed. Managers live and breathe every step in those last days. The fact is, the government employees would need to have charge codes and other authorizations (fake login IDs, etc) to do the testing. And who signed off on these fake IDs? Yep, top level people in CMS.

Contrary to Tavenner’s testimony, the pre-launch CMS test results were a clear sign of what would happen. Sure enough, it came to pass as we now see in meeting notes from the first few days of “Going Live”:

The website launched on a Tuesday. Publicly, the government said there were 4.7 million unique visits in the first 24 hours. But at a meeting Wednesday morning, the war room notes say “six enrollments have occurred so far.”

…

By Wednesday afternoon, enrollments were up to “approximately 100.” By the end of Wednesday, the notes reflect “248 enrollments” nationwide.

Same problem. barely anyone getting through the process. And I would wager those that did had errors. It get’s worse if we look at the details of the reporting from the very first morning, were we find major elements of Healthcare.gov just not functioning as they should:

• VA [Veterans Administration] system not connecting
• Experian creating confusion with credit check information
• residency issue has a script being developed for the 900 issues that occurred
• many agent-brokers have not signed up on the EIDM

These four items come from a single line in the status report. But each represents a major issue, something that should have been addressed way before “Go-Live”. For those not versed in these matters, let me expand on them.

The VA system is one of the many federal databases Healthcare.gov must integrate with. If there were problems connecting, it means the pre-launch testing was completely inadequate. And if VA was not connecting, it would suspend all applications as the system waited on the results that would never arrive.

Same thing with the Experian interface for credit checks. How could there be confusion if proper testing had been performed?

The ‘residency issue’ is huge. 900 problems identified! A properly designed and tested system would not be this bug-ridden at launch. The fact “a script” was all CMS thinks is required to fix this tells me the system testing was so lame that basic capabilities ended up being debugged post launch.

And why weren’t the agent-brokers signed into EIDM prior to “Go Live” (this is the new CMS identity management system, itself a huge endeavor and an indication of CMS taking on too much all at once)? This is a sign of incompetent management. We see more and more issues with the system not being fully configured or being mis-configured. One pre-launch check is to ensure all the necessary data is entered (i.e., agent-brokers have accounts and are able to access the system) and the data is entered correctly. Clearly, this was not done.

The status reports are riddled with examples like this, of ‘glitches’ that should have been dealt with at the design phase, the test phase, or the pre-launch check phase (known as the Operations Readiness Review). The issues popping up indicate to me HHS/CMS had no idea what it means to implement a program like this, and just failed over and over again.

Which brings me to the second clear issue with Healthcare.gov – securing personal information. It was discovered back in the summer, through a GAO report, that HHS/CMS had completely missed the mark on IT Security. Sadly, I have seen this scenario way too many times in Federal programs involving large computer systems and PII data.

Last summer (July) the GOA noted Healthcare.gov was way behind schedule, and trying to implement security way too late in the life cycle.

Federal watchdogs expressed concerns to lawmakers Wednesday about the security of information that will be traded through the Affordable Care Act’s (ACA) health insurance exchanges.

Federal tax information provided to the exchanges in order to determine benefit eligibility may not be adequately protected, an official from the Office of the Treasury Inspector General for Tax Administration said Wednesday.

…

“The lack of adequate testing could result in significant delays and errors in accepting and processing ACA applications for health insurance coverage,” Duncan said.

IT Security of PII data has to be designed in at the beginning. But this article (and others) clearly show HHS/CMS was trying to fold IT Sec onto Healthcare.gov after the design was done and at the last minute (August-September). Therefore it is not surprsing to find that CMS never made it through security qualification and the system went live on a ‘provisional’ basis

CBS News first found a Sept. 27 memo signed by Centers for Medicare and Medicaid Services Administrator Marilyn Tavenner stating that testing did not occur in a single environment and noting that applied patches meant the system wasn’t always stable over the course of testing.

That “exposed a level of uncertainty that can be deemed as a high risk,” the memo stated, although it noted that the FFM underwent three rounds of security control assessments.

But today’s headlines clearly show the warning flags of summer were spot on:

“I was in complete shock,” said Hadley, who contacted Heritage after becoming alarmed at the breach of privacy.

Hadley, a North Carolina father, buys his insurance on the individual market. His insurance company, Blue Cross Blue Shield of North Carolina, directed him to HealthCare.gov in a cancellation letter he received in September.

After multiple attempts to access the problem-plagued website, Hadley finally made it past the registration page Thursday. That’s when he was greeted with downloadable letters about eligibility — for two people in South Carolina.

…

After learning of the privacy breach, Dougall spent Friday evening trying to contact representatives from HealthCare.gov to no avail; he spent an hour waiting on the telephone and an online chat session was unhelpful. He also wrote to Senators Lindsey Graham (R-SC) and Tim Scott (R-SC), along with Representative Joe Wilson (R-SC).

“I want my personal information off of that website,” Dougall said.

This is a clear breach of PII data. And it is not the first one committed by HHS/CMS (see here and here). I am deeply concerned with the use of “Navigators” and other aspects of Obamacare. HHS/CMS is sending PII data to people, companies and organizations that probably have never been vetted for controlling PII. As a federal contractor myself, I have to commit to protecting PII and take measures to ensure we never release anything. Given how this disaster has been run, I would wager HHS/CMS never did the required due diligence on all the entities it is interfacing with on the behalf of people searching for insurance.

If CMS is telling Navigators to make sure they fax PII to the right phone number, they are not taking this responsibility seriously.

Which leads me to the final nail in the coffin. The system was never designed right. This article by someone in the business of developing large software systems (like me) makes a lot of salient observations. Specifically this one:

The bottom line is that those people who claimed that all would be well if HealthCare.gov had used an agile process are wrong.  The reality is that the developers did use agile, and the project failed miserably.

…

The process we are using begins with a detailed set of customer requirements that include both functional requirements (what the software should do) and performance requirements (metrics it should meet). This is painstaking work, as we analyze, debate, discuss and finally decompose hundreds of end-user requirements. … This is a robust process with dedicated and talented engineers that is working well. This will take a few months, but when it is done we will know exactly what we have to build.

…

A complex back-end data services hub — a piece of software with zero actual, living, breathing end-users — has to be described in terms of “user” stories. Does something sound off-key to you? It should, because user stories are great for user interfaces but poor, confusing and often misconstrued for non-observable behavior.

User Stories in the Agile Process are a vague, shallow descriptions of a complex system. It is akin to saying a car needs brakes to stop (duh) and does not address the engineering to include anti-lock braking, all-wheel disk brakes, etc. If your User Stories are vague and lack detail, the resulting system will be unable to handle all the technical details. They are important, because they describe the operations concept and flesh out how functions should interact. But they are never sufficient by themselves.

Worse than that, as I noted in my previous post, HHS/CMS decided that this huge, complex effort would best be served by skimping on the required technical design documents – a process called “tailoring”:

First and foremost you need to define all interfaces and get them on a coherent schedule.  Then you have to model all state and federal sources of data (i.e., define how each one labels common information and create a Rosetta stone to relate each existing label in those data sources to a common definition, like “first name”) and then you need to model all data products (each state and each insurer in the state has unique definitions for data as well).

…

And of course, data modeling and translation is one of those things HHS/CMS tailored out to save time. [In the above diagram] R = Recommended, which in tight programs means it won’t get done.

Leaving security to the last minute, skimping on design steps and artifacts, not vetting the consumers of the PII data Healthcare.gov collects from federal databases, sending PII to the wrong people, the lack of testing, the lack of operational readiness checks, the obvious lack of knowing what they were doing – all signs HHS/CMS has created one of the worst failures in government history.

The site should not have been authorized to Go Live, and those who did so should be held accountable.