- Politicians decide to take very big steps in space - rather than developing capabilities incrementally - because those seem more politically palatable.
- Big steps are expensive, so we only get to do it a few times - one urine recycler at one space station, rather than a dozen porta-potties at a dozen outposts.
- Since you have one of everything, it has to always work. More importantly, since there's only one of them, you don't have the option of six sigma or working out the kinks over time.
- Since you don't have that option, you need an army of bureaucrats (known by the euphemism "systems engineers") whose job it is to play the role of nature and nit-pick every inconsistency in results, test apparatus, interface (read: plug or bolt, literally), and so on.
- This is trading bureaucracy for causality. Instead of real-world shakeout, lots of people file enough paperwork to convince each other that the product will work.
- The result? Urine recyclers that don't work, falling foam that destroys multi-billion dollar space shuttles, a whole bunch of finger-pointing and denials, and more paperwork. NASA can't build a launch vehicle that blows up less than one time in 50... but I'd be willing to bet that every NASA employee can fill out at least 50 forms, perfectly, 999 times out of 1000, because they've screwed them all up once and got hounded about it.
So, kind aerospace industry non-expert, what is the alternative. I mean, this stuff is hard, right? Yeah, and so is keeping an airplane flying at 85% the speed of sound carrying a small town's worth of people 20 hours a day. And so is making a car that sells for $20,000 and but will run in all conditions for 200,000 miles before major overhaul. And so, for that matter, is making a million Dixie Cup that hold water, stay reasonably sterile, and won't spontaneously disintegrate or catch fire in a wide range of humidity, temperature, and better-idiot conditions. Engineering is hard. That's why we get paid the big bucks*.
There is another path. Some private industry is taking the path. Lots of people think space tourism is the killer app, but I tend to disagree. You want to fly enough to uncover gremlins, but humans are terribly fickle cargo. They have a habit of dying and leaving behind kin who in turn have lawyers. Their trips are often accompanied by TV cameras. Suborbital tourism makes a sense as a secondary market, but only because you've got room for bigger margins, and there are already a lot of safe small aircraft. People have already dies so that you may see black sky, in other words.
I think smallsats are the way to go for commercial business right now. They offer standardization in a repeatable package. So-called "cubesats," a standard form factor 10 cm on a side, are currently all the rave with universities, and companies are starting to develop standard power, data, and propulsion suites for them. You don't want to fly something for the first time, every time. Let Georgia Tech's solar array short out, so that yours doesn't have to. Orbital has built a successful business model of launching a bunch of smallsats with common form and systems on their own rockets. They are able to charge a premium of about 3X the market rate for the launch side of the service. Virgin Galactic is also very intent on the market, and for the life of me I cannot understand why SpaceX decided that Falcon I is a dead end for them... if anything I would have gone smaller, not bigger.
How about the human side? Everyone loves astronauts. A lot of us want to be one. The problems on that side are that, as we already established, humans make big, fragile cargoes. But the Russians have followed the incremental path. Their Soyuz rocket family has thousands of flights going back to the '50s, and they are remarkably reliable and inexpensive as a result. The point has been driven home by the fact that NASA will have to buy seats on them from 2011 until whenever NASA as we know it ceases to exists and embraces commercial American launch providers.
It all comes down to repetition. Six sigma will wipe the floor with systems engineering, every day and twice on Hump Day. A porta potty that works every time will beat a urine recycler that works when it wants to (and why would anything want to recycle urine?)
*I am currently unemployed. Hook a brotha up.