When we think of a star's shape, we normally think of the Sun, which is a slightly flattened sphere. Most stars are an equilibrium of pressure, self-gravity, and rotation, and a flattened sphere is the the natural shape that these opposing forces create. If a star is in a close binary system, however, pressure is in equilibrium not only with the star's own rotation and gravity, but also with the rotation of the binary system and the gravity of the companion star. These additional bits of physics distend the star in the direction of the companion star. At its weakest, the effect is a slight tidal distortion, much like the distortion of Earth's oceans by the Sun and the Moon, that elongates the star, but at its most extreme, the effect molds the star into the shape of a raindrop, with its sharp point facing the companion star. This distortion of the photosphere in the direction of the companion is called the Roche lobe.
Tidal forces are a form of friction. In the Earth-Moon system, tidal friction caused the rotation of the Moon to synchronize with its orbit around Earth, so that we only see one side of the Moon. Tidal friction is causing Earth's rotation to slow. We see a similar effect in a close binary star; a star's Roche lobe dissipates rotational energy, driving the rotation period of the star to the orbital period. If the orbit is eccentric, the tidal distortion strengthens and then weakens over an orbit, causing a dissipation of orbital energy and making the orbit more circular.
There is a maximum size to a star's Roche lobe. For a star in a circular orbit, this maximum size is set by a point of unstable equilibrium between it and its companion star. This point is called the inner Lagrange point; it is often referred to as the L1 point. An object at the inner Lagrange point would stay on a line between the two stars in the system, maintaining a fixed separation with each star. If the object were displaces from this point towards one of the stars, that object would orbit that star. A star's Roche lobe reaches its maximum possible size when its touches the inner Lagrange point; any larger, and the star's atmosphere flows onto the companion star. A star that touches the inner Lagrange point is said to fill its Roche lobe. When a star becomes bigger, so that its atmosphere begins to flow onto the companion star, the star is said to undergo Roche lobe overflow. Roche lobe overflow alters both the orbit and the structure of the overflowing star.
As you would expect, a star's structure is altered when it undergoes Roche lobe overflow. The structure of the star changes to compensate for the reduction in gravitational force within the star and the removal of the star's coolest outer layers. The star reacts by mimicking the structure of a more-evolved star. The core of the star shrinks, but the outer layers expand, with the overall effect that the star puffs up.
How mass flow changes the orbits of a binary star system depends on whether the overflowing star is more or less massive than its companion. If the overflowing star is less massive, the distance between the stars and the orbital period of the binary increase over time, but if it is more massive, the distance and the orbital period decrease. This is a consequence of conservation of angular momentum. As mass flows from one star to the other, changing the mass of each and shifting the center of mass of the system, the system's size changes to preserve its angular momentum.
Why this occurs is easy to see if we start with two stars of equal mass. The center of mass for the system is halfway between the stars, and each star carries half of the angular momentum of the system. The angular momentum carried by each star is 2 π M R2/P, where M is the mass of the star, R is the distance to the center of mass, and P is the orbital period. If we then move almost all of the mass of one star to the other, but kept the angular momentum fixed, we would find that the system's center of mass is very close to the center of the large star. This short distance makes the angular momentum carried by the massive star much less than that carried by the two stars before we mass transfer. This means that the bulk of the angular momentum must be carried by the small star, much as in the Jupiter-Sun system, the bulk of the angular momentum of the orbit is carried by Jupiter, because the Sun's offset from the center of mass is so small. But for the small star to carry as much angular momentum as the previous system, the distance of this star from center of gravity must be much greater than in the previous system.
For two stars with much different masses, the orbital period is related to the separation between the stars as P2 is proportional to D3. Because the distance to the center of mass for the small star is effectively the separation between stars, the angular momentum carried by the small star is proportional to R1/2. Because the angular momentum is such a weak function of the separation between the stars, the separation between the stars must increase dramatically for the small star to carry the same angular momentum as the original system.
The position of the inner Lagrange point moves towards the star losing mass as mass exchange continues. When two stars have equal mass, the inner Lagrange point is precisely between the two stars, coincident with the center of mass of the system. If mass is transferred from one of these stars to the other, the inner Lagrange point moves in the direction of the smaller star. The question is whether this motion towards the small star is countered by the increase in the binary star separation. It turns out that while the distance from the inner Lagrange point to the small star decreases relative to the distance from the inner Lagrange point to the large star as the small star loses mass to the large star, the absolute distance from the small star to the inner Lagrange point increases. This means that as the smaller star loses mass to the larger star, its maximum Roche lobe increases in volume. Conversely, if mass is transferred from the large star to the small star, the maximum Roche lobe of the large star decreases in volume, because the inner Lagrange point moves toward the large star as the distance between the stars gets smaller.
We find that an overflowing star is unable to expand enough to keep its Roche lobe filled when that star is less massive than its companion. If only these mechanisms—the expansion of the maximum Roche lobe and the expansion of the star—were present, Roche lobe overflow could never occur, because the mass transfer would cause the star to cease overflowing its Roche lobe, bringing mass transfer to a halt. But there are other, slower processes that shrink the distance between the stars. Both a stellar wind and gravitational waves carry orbital energy and angular momentum out of the system, causing the binary orbit to shrink and the smaller star to fill its Roche lobe. Once mass transfer starts, the decay of the orbit comes into equilibrium with the expansion from mass transfer. In this way, the system transfers mass steadily from the smaller star to the larger star.
The converse occurs if the overflowing star is more massive than its companion. In this case, the separation between the stars shrinks, as does the distance from the inner Lagrange point to the massive star. Add to this the expansion of the massive star as mass is removed, and we have a system that is unstable to mass flow. Once the massive star begins overflowing its Roche lobe, the rate of flow increases rapidly, until so much mass has been pulled onto the companion star that both stars fill their Roche lobes. The continued expansion of the massive star then enshrouds both stars in a common envelope of gas; within this new, single star, two cores rapidly orbit each other, rapidly coming closer to each other as the orbit loses energy to the common envelope.