First another clarification. I am not NASA or government employee, I work on NASA programs as a contractor. Just to make sure that phrases like “NASA Insider” are not taken to far out of context.

Due to some comments on my previous post regarding the Shuttle and its foam problems I decided to post an update this morning. Here is an article from just around the time of the Columbia that illustrate the foam has fallen off the orbiters for a long time, and a bit of history I had forgotten about (in a bit)

It was the second time in the past three launches of the shuttle that chunks of insulation had cascaded off the enormous fuel tank, which contains the liquid hydrogen and oxygen needed to fuel the shuttle’s trip into orbit.
…
The problems posed by falling insulation are well-known to NASA. Indeed, the 1997 incident damaged 308 ceramic tiles that protect the orbiter from re-entry temperatures of up to 3,000 degrees Fahrenheit.

Katnik’s report prompted NASA to take a hard look at the performance of the insulation. His report showed a “massive” amount of the fuel tank’s insulating foam crashed into Columbia when it blasted off from Kennedy Space Center on Nov. 19, 1997.

In a report posted on a NASA Web site, systems engineer Katnik found 132 tiles on Columbia with scars longer than an inch, with some as much as 15 inches long and an inch and a half deep. More than 100 tiles had to be replaced, he said, compared with only 40 after a normal mission.

A comment by Jeff at The Bernoulli Effect reminded me that the foam problem did escalate due to a change in formula hoisted on NASA by the EPA and environmental groups

Katnik speculated that the problem with the foam was caused by a new formula, which eliminated Freon from the mix in an effort to be “environmentally friendly.” He said an earlier 1997 shuttle flight — the first to use the new-formula foam — had experienced similar, though less-dramatic problems.

As everyone knows, Freon caused the damage to the Ozone layer years back causing a serious risk to many people. However, the amount of Freon released by the foam process and subsecquent burning up could be negligable compared to the amount of gases assaulting the ozone layer produced world wide. The Shuttles fly maybe 4 times a year, so you can see the formula change may have been an over reaction.

Here’s s a bit more on the role the foam plays

It was jacket weather in Florida, moist and mid-40s on the ground. Sixty feet up, on the launch platform, the temperature hovered below 50 degrees. The humidity was almost 100 percent when the launch team gave the go-ahead to start pumping super-cold liquid fuel into the massive tank.

Anywhere else, that’s just nippy weather. Up on the pad, it’s a different world. The presence of a half million gallons of liquid hydrogen, at minus 423 degrees, and liquid oxygen, at minus 298 degrees, changes everything.

That’s one reason why manufacturer Lockheed Martin sprays an inch-thick layer of polyurethane foam onto the tank. The insulation stems the growth of frost and ice that could come off the tank and pelt the shuttle during launch. But the deep freeze inside means temperatures on the insulation surface can be 10 to 30 degrees cooler than the air outside.

So with temperatures as high as 60 degrees, and high humidity, condensation can turn to frost and ice. That’s especially true where the foam is thinner, cracked or somehow altered, according NASA-sponsored research.

Some interesting points about how not all foam is the same (probably one of the first and easiest things NASA fixed after Columbia)

The fuel tank foam is closed-celled. That means individual cells are tightly packed together so other molecules, even water or gas, can’t get inside. The bulk of it is sprayed on the tank at a plant near New Orleans, mostly by robots. The outer layer hardens into a sort of rind, an orangish skin that further protects it from the moist air outside. This is the kind of foam shuttle program manager Ron Dittemore showed the news media in the days after the accident to bolster his point that the material is lightweight and impervious to moisture.

But the foam suspected of popping off Columbia’s tank is different. It doesn’t have that protective outer layer. It’s called “close-out” foam because it’s applied near the end of manufacturing, by workers using their hands, molds and tools. Some of the work is done in Louisiana; some at KSC where crews attach the tank to its orbiter and solid rocket boosters.

A perfect example of that kind of foam are the bipod ramps, the triangular blocks that fell off during Columbia’s launch and at least four previous launches. Workers pour the foam for the twin ramps into place near metal struts that attach the tank to the orbiter’s nose. They use tools to cut the foam to an aerodynamic shape.

In that general area, workers shave or sand other close-out foam. They also use what looks like a wire brush to poke tiny holes in large tracts of nearby foam. The process called venting was meant to let gas trapped inside escape instead of expanding and blasting the foam off the tank.

These treatments can provide a path for gas and moisture to get inside the foam. The workers are slicing open the walls of those closed cells and removing the polyurethane’s hard skin

Just to be clear, this was the pre-Columbia process for the attachment arms. The new design is for the connection arms to have heaters inmbedded in them and just not use the foam. I am not sure if that design was on Discovery, but it doesn’t matter – the foam came from a different area on the external tank in this flight.

Here is another good article that illustrates the balancing act every launch truly is.

But NASA still relies on the “normalization of deviance,” said Diane Vaughan, an organizational sociologist who has studied the agency extensively. That means components often fail, so the trick is realizing what will be catastrophic and what won’t.

“Having problems as part of the normal operation can really make it hard to realize when a problem is serious or not, and when you need to stand down and fix it,” said Vaughan, author of The Challenger Launch Decision: Risky Technology, Culture and Deviance at NASA. “If they don’t have enough information to fix it, they keep flying with errors.”

These systems and missions cannot be made perfect and error free. And when you constantly find and fix problems as you head to launch, you have the problem of detecting the problem that might get through your processes and solutions. The cascade of problems/fixes occur for months leading up to launch for shuttle, and is normal for every mission I have worked on in the satellite side. In those cases you do not have a repeatable process like shuttle, each one had major new features – though the systems are becoming re-usable ‘products’ so there is a lot more experience retained than in the days when shuttle was designed and everything was ‘handrafted’. But you live in a world where every day you have a new ‘issue’ (they are not problems until you cannot solve them within cost and budget) to address and adjust to.