At times theory resembles stock-investing: when a new piece of information destroys a current notion, everyone rushes to profit from the change. The theorist tries to profit, or at least cut his losses, by rushing into print a paper that either touts a new theory or seeks to limit the damage to the old theory. Preprints flow into the physics preprint server at Los Alamos. Papers flow to The Astrophysical Journal Letters, to Nature, and other journals. If you're lucky, a sympathetic referee accepts the paper immediately, but if you're unlucky, your paper will be reject or delayed for months by a hostile referee. From this land-rush, claims on the next generation of theories are asserted.
I was involve in only one rush, the rush to locate gamma-ray bursts at the beginning of the 1990s. Gamma-ray bursts are common events that last from much less than a second to many hundreds of seconds. The longer gamma-ray bursts are now known to come from very distant supernovae, while the origin of the short bursts remains unresolved, but fifteen years ago most researchers believed that gamma-ray bursts were from neutron stars within our own Milky Way galaxy. The problem at that time was that no gamma-ray burst had a steady counterpart; without a counterpart, a burst cannot be studied with with optical telescopes, and a distance cannot be determined. At that time, we knew two things: gamma-ray bursts are uniformly distributed on the sky, and the bright bursts, the only bursts observed, are homogeneously distributed in space. These two pieces of information suggested to most researchers that the burst sources are relatively weak and nearby, so that the plane of the galaxy did not show up in their distribution on the sky.
This changed when an experiment on the Compton gamma-ray observatory found the edge of the distribution by observing fewer dim bursts than were expected for a uniform distribution extending to infinity. These dim bursts are still uniformly distributed on the sky, with no evidence of a galactic disk, so clearly we are at or near the center of a sphere of gamma-ray bursts. The only question that remained was what creates this sphere. The obvious answer is that the sphere is our observable universe, and the edge is produced by the rapid expansion of the universe, which redshifts the most distant gamma-rays into the x-ray regime. This was such an obvious solution (ignoring the difficulty of how the universe manages to create such an intense gamma-ray source) that most researchers immediately jumped onto this bandwagon. It was into this frenzy that I published a simple paper in Nature to remind my colleagues that there was still a second sphere that we were embedded in, the galactic halo, and that until we were able to collect enough gamma-ray bursts to show our displacement within that sphere (8 kpc from the galactic center), we would not be able to distinguish between a galactic halo theory and the cosmological theory. Eventually over time observations were able to distinguish the two theories, and the cosmology theory won.
In that paradigm shift from a galactic origin to a cosmological origin of gamma-ray bursts, there was rush to publish new theories, with every conceivable object, both real and imagined, proposed as a source of gamma-ray bursts. My own biases drove me to massive black holes at the centers of galaxies, but others favored moderately-sized black holes in the planes of galaxies. Some favored such fictitious peculiarities as cosmic strings and quark nugget, and one intrepid sole proposed the collision of antimatter asteroids with matter asteroids within our own solar system (would this produce an isotropic distribution of bursts?). Every one of these ideas has been disproven.
The papers in a theory rush are nothing more than back-of-the-envelope calculations. In the case of the gamma-ray burst theory rush, most of the calculations concern only the energetics required to produce the burst and the speed required to make the burst appear short—the apparent shortening of time from relativistic motion is present in these events. All other physics is submerged into a hope that nature can find a way of converting the energy produce in the theory into gamma-rays. In this simple-minded way, the groundwork is set for the next generation of theories.
But the real work comes in developing a full theory, and this is incredibly difficult work. The physics is generally exotic and wide-ranging. Even in studying the most run-of-the-mill star, the theorist must understand nuclear reactions, the propagation of radiation, the convection of hot plasmas, and the generation of magnetic fields. In more extreme objects, the theorist works with general relativity to understand gravity and relativistic quantum mechanics to understand matter under extreme densities. The theorist often must design his own computer codes to examine complex physical systems. This world of theoretical astrophysics is a world apart from the physics of a theory rush; it is among the hardest disciplines to practice well, but it is the only road to actually understanding anything in astrophysics.
More and more solid theories in astrophysics require use of sophisticated computer codes to calculate the flow of gases, the generation of magnetic fields, the thermonuclear burning of light elements, and the effects of general relativity. Where is most supernovae theory done? At labs such as Los Alamos, Lawrence Livermore, and Oak Ridge, laboratories with excellent computer codes to calculate the hydrodynamics of an explosion. Such codes are expensive to develop, which, in a world where dollars for theory are scarce, makes them scarce. This makes the practice of solid theory similar to making astronomical observations: only those with access to a scarce resource are able to practice the discipline. The spending on code development determines the direction of solid theory in the same way that spending on satellites and observatories set the direction of observation.
The incentives in theory are skewed. The theory-rush paper is easy to write, cheap to produce, and fast to publish. It produces instant recognition for the right ideas. Solid-theory papers are difficult to write, expensive in manpower to produce, slow to publish, and difficult to read. Often only the referee reads every word of one of these works. So theory coalesces into two camps, one that specializes in wide-ranging but simplistic theory, and one that specializes in narrow, detailed, and rigorous theory.
We are missing something in modern astrophysics: an incentive to develop unpopular theories. We have no mechanism for handicapping the multitude of proposed ideas, so that some amount of resources is placed into each. Returning to the analogy of the stock market, we do not have a mechanism that turns entrepreneurial ideas into successful enterprises; we lack the mechanism for turning any more than one or two ideas into solid theories.