Commentary

Intellectual Inertia

Of the many unnatural demands that sciences makes on a scientists, the most burdensome is the demand to give up a cherished idea. Few scientists can give up an accepted idea without a struggle, and some scientists are incapable of ever give up an idea. This intellectual inertia is a constant of human existence.

Taking a page from Socrates, we can see this human trait by examining it within our institutions. Bureaucracies and large companies are notorious for clinging to an idea long after its usefulness has dissipated. How these organizations overcome their inertia depends on the external forces exerted on them.

NASA fell into the trap of clinging to a bad idea with the Space Shuttle program. The Space Shuttle was conceived as a reusable manned spacecraft that would provide safe, low cost, and frequent transportation into space. From the start, the spacecraft failed to provide either the low cost or the frequency of flights that had been promised. Later, evidence began to accumulate that the shuttle design is inherently unsafe because ice shed from the shuttle's external tank can strike and critically damage the shuttle. But NASA had staked its existence on manned space flight, which consumes half of its budget, and at the core of the manned space flight program is the Space Shuttle. Under this circumstance, failure of the Space Shuttle program was not an option for the institution, even when it had become apparent that failure was unavoidable. NASA clung to the failed Space Shuttle for a quarter of a century, when finally a second shuttle accident effectively brought the program to an end. If NASA were a public company, it would have gone bankrupt long ago, but, because the federal government provides it with money despite massive failure, NASA persists.

While the history of the shuttle program points to one reason we cling to wrong ideas, the history of many large private corporations points to a second reason: we cling to what has worked for us in the past.

In the 1980s, Digital Equipment Corporation (DEC) had success designing and selling minicomputers. Their model of the world was that users performed work by connecting through dumb terminals to a central minicomputer. I recall one day, when a graduate student at the Harvard-Smithsonian Center for Astrophysics, sitting at a DEC terminal when the chief executive officer of DEC, who was visiting as a member of a Harvard advisory panel, came by and peered over my shoulder. The terminal was a character terminal, but I had written a small program that placed various bits of useful information on various parts of the screen, plus some decorations, so the screen was more striking that the average dumb terminal on the floor. The CEO made some comments about what I had done, and then said “wait until you see the next terminal, it has more extensive graphics capability.” I said something polite in reply, but what I though was “what I really want is a personal computer that talks to the minicomputer so that the text editor would stop hanging when the user load got bad.” But this view of a dumb terminal talking to the minicomputer seemed to be DEC's focus, and because of it they missed out on the transition to workstations and personal computers communicating on a network. Instead they were caught in a market that was pressured from above by the IBM mainframe computers and from below by the workstation and personal computer makers. Eventually DEC was acquired by one of the personal computer companies.

The free market is very good at counteracting our natural intellectual inertia. If you have ever invested in the stock market, you know how effective the market is at destroying illusions, misunderstandings, and ignorance. Giving up a bad idea is much easier when that idea costs you a large sum of money. This discipline is strongest in the areas of the economy where the success of a business is most in doubt, and they are weakest in the areas where funding is most certain. Follow a mistaken idea, and a computer company such as DEC disappears, but a company like General Motors persists without obvious damage for decades. In a competitive industry, young businesses on the right side of a paradigm shift, as well as old businesses flexible enough to move to the right side rapidly, acquire business at the expense of static businesses in decline by being on the wrong side, and they can find funding from investors willing to take a chance on a new idea. This mechanism of funding young companies pursuing novel business ideas causes a smooth transition from one business paradigm to another.

This intellectual inertia that appears in large organizations are endemic among scientists. Scientists today are highly specialized, so they pursue the ideas they know and with which they are comfortable and successful. The acquisition of knowledge and of the tools for applying that knowledge (computer programs) carries a cost of time and money. More important, an astrophysicist very publicly places his prestige at risk whenever he develops a theory. The risk is not that the theory is poorly developed, but that the theory is not the correct theory for the observations. Beyond these forces are several much more personal forces that make us cling to an idea: our unwillingness to publicly admit mistake, our loyalty to our colleagues and the ideas they promote, and our desire for professional fame and success.

Long ago an astrophysicist proposed a variation of the big bang cosmology; in his theory, the early universe is cold and solid. This differs from the current theory, in which the early universe is extremely hot, so hot that at one point in the past thermonuclear fusion converted some of the hydrogen in our universe into helium. To its author, the cold big bang made logical sense, because such a universe would be evolving from a highly-ordered state in the past to the current highly disordered state. The problem, however, is that the cold big bang cannot explain the microwave background. The microwave background in the hot big bang is a natural consequence of thermal emission from a hot, expanding gas in the distant past. In the cold big bang, one needs a somewhat ad hoc process—the emission of radiation by dust in the not-so-distant past—to explain the phenomena. As a consequence, the cold big bang died as a plausible theory in everyone's mind but the originator's with the discovery of the microwave background. The originator continued to cling to the theory for decades, going so far as to highlight in his writings on the history of cosmology those scientists and philosophers who advanced correct theories that were not accepted during their lifetimes.

The scientific community is under more pressure to give up false ideas than government organizations, but it has much weaker mechanisms to encourage the development and adoption of better ideas. Observers are constantly providing more data that can support or falsify a theory, but this process is inherently slow. One observation does not falsify a theory; instead, a steady buildup of data forces the community to eventually drop a theory. But even when enough information is accumulated to disprove a theory, the community is often slow to discard the theory. Unlike businesses subject to the free market, the scientific community is subjected to forces that tend to reinforce our habit of clinging to outdated ideas. The public nature of scientific work, the pursuit of tenure and public fame, and the small number of money source, all controlled by committees of scientist, reinforce our desire to cling to theories. Finally, science lacks a mechanism, unlike the free market, that encourages and rewards independent thinking.

Eventually the data forces a change, but the change is catastrophic, like the breaking of a tree limb as more and more weight is hung onto it. No rapid evolution, like the rise of the personal computer or the Internet in the business world, but, as historians of science have noted, a revolution that in short order destroys the faith within the community in one theory, propelling the community into a new set of theories.