There were initially seven modes back in the era of modal music, but when tonality set in, the use of modes dropped to just two; Ionian and Aeolian, now the major and natural minor scales. Why were these two specifically chosen for tonal music? What stopped, say, the phrygian and lydian modes from becoming central to tonality?
There were initially seven modes back in the era of modal music is not really a correct statement. What you consider the era of modal music is an era of medieval music, which itself uses a not exactly correct reading of ancient greek music theory. There the number of tonoi depends a bit on which theorist you consider.
But in medieval modal music (most prominently chant) you get 8 different modes, which includes authentic modes and plagal modes (the difference mostly being that the authentic mode uses the final note or one step below the final as lowest note, while the plagal modes extend to a fourth below the finalis (but only to the fifth above it).
The modes we then get are (usually numbered like this):
- Hypodorian (the plagal mode to the Dorian mode)
So you see that modal music does not even have the modes Ionian and Aeolian. These alongside their plagal counterparts were only added in 1547. The final mode Locrian and Hypolocrian were only added for completeness in the late 18th century.
That being said, it would be wrong to say that these characters were not used before: Medieval music theory had a high and a low B, and occasionally the modes could flatten the high B to a low B. This then turns modern minor into a variant of the Dorian modes and modern major into a variant of the Lydian modes.
Now, what made the major and minor tonality to popular? It is hard to give a definite answer there, but some ideas about this: Keep in mind modal music did not have the same sense of harmony as we know it today. There are some concepts that became very popular in late medieval music, which is harmony based on thirds and sixths, the concept of a tonic, leading tones and thereby cadencial harmonics. And this means that the traditional modes are having deficits, and we even see a period of late medieval to renaissance music where music is notated in a traditional mode, but we see chromatic alterations (initially not notated by performed by convention).
Now if we look at leading tone harmonic we see that the only modes that include the lower leading tone to the final are the Lydian modes. But in a tonality which is built upon thirds and sixths there is an advantage to using Lydian with the low B: This gives us a pure fourth to the final, which is more usable in harmonies.
For minor we might propose that using the low B in the Dorian mode avoids the tritone between third and sixths to the final, which again would be more usable.
I want to clarify that this is just some ideas and not at all a researched opinion.
All the modes are equally different.
Your summary of the transition from the modal system to the major/minor system is mistaken, but those misconceptions are fairly common.
I found this article helpful to get a better understanding of the history of the modal and major/minor systems: Atcherson (1973) Journal of Music Theory - Key and Mode in Seventeenth Century Music. Scholarly reading about modal music is difficult. I think the Atcherson article is nice, because it is concise, and it focuses on comparing the two systems and the transition.
The well known Church Modes numbered eight and were divided into authentic and plagal pairs: dorian, phrygian, lydian, and mixolydian for authentic modes, and hypodorian, hypophrygian, hypolydian, and hypomixolydian for plagal modes. Notice that ionian and aeolian do not appear on this list!
Part of the transition from modes to keys, as described by Atcherson, is not the disuse of certain modes, but the recognition that the modes can also be group by whether the thirds above their finals (the modal equivalent of tonic) are major or minor thirds.
I have also read about an idea there was a reduction to three categories: major, minor, and phrygian. I don't have a reference handy for that.
The church modes system needs to be expanded with two details. Lydian mode very often would use a flat on
B which makes it sound very much like a major key. Final cadences needed to approach the mode's final by two steps in contrary motion, one step being a half step and the other step being a whole step. The motion is called the clausula vera. Lydian and phygian modes contain these two steps without alteration, but dorian and mixolydian do not, and so require their seventh degrees to be raised by a half step. Those raised tones act like leading tones in the major/minor system.
If we look for certain modes as ancestors for major/minor keys, rather than ionian and aeolian, dorian with a raised seventh degree is like minor, phrygian it just too different to become the ancestor to minor, and lydian with a
B♭ is like major. Mixolydian with a raised seventh degree is also like major, but as lydian has major triads on both the final and fifth degrees (the tonal tonic and dominant degree/chords), lydian seems the more like major.
If we try to summarize these ideas, it isn't really a development of preferring two particular modes out of many, but more about preferring certain intervallic relationship within the tonality: the desire for a whole step above and half step below a tonic, the desire for a half step above a major third mediant, the desire for a perfect fourth and perfect fifth above the tonic, etc.
Recognizing that just two modes, major and minor, can be transposed, was another importance factor in the decline of the modal system. The Atcherson articles discusses that too.
Based on how we subjectively experience the quality of intervals and harmonies, here is a guess. My main theory here is, that is has to do with the rise of music using functional harmony.
To have a stable Tonic, let's rule out all modes with a tritone instead of a fifth over the first step. Those modes can't form a tonic with a tonic without dissonance. Which is only locrian.
Ionic - Dorian - Phrygian - Lydian - Mixolydian - Aeloian -
Then, let's look at cadences. Only with the second step being a whole-tone away from the first step, we can have the 6-th chord over the fourth step, sixth ajoutee, or in so called jazz theory the IIm7. This rules out phrygian in favour of a stronger sub-dominant function. Note, that also Lydian technically can't form a real sub-dominant, but only a double-dominant. Also, without the second step being a whole-tone away from the first one, the dominant can't have a fifth, so there wouldn't be a stable dominant, which makes functional music hard to achieve.
Ionic - Dorian -
Phrygian - Lydian - Mixolydian - Aeolian - Locrian
Staying in functional harmony, let's look at the dominant seventh chord over the 5th step. It achieves it strongest cadence function by the tritone between it's third and seventh. In relation to our first step of the mode, this would be the perfect fourth, and the major seven. Which lydian as well as mixolydian don't have. So we rule out lydian and mixolydian in favor of a dominant seventh chord.
Ionic - Dorian -
Phrygian - Lydian - Mixolydian - Aeolian - Locrian
Lastly, to achieve a quality of minor, or the third of the finalis being a minor one, we have to decide between Aeolian, and Dorian.
Here's where it gets a little complicated. The small sixth is stronger associated with movement towards fifth, and the natural sixth is stronger associated with movement towards the septime. Here, using the argument of the first paragraph, I would favor a stronger fifth in tonal music, to indirectly establish the tonality even more. That's what (only kinda) rules out dorian over aeolian. Also, the subdominant being minor, it distinguishes itself from Ionian as our major scale a bit more.
Dorian - Phrygian - Lydian - Mixolydian - Aeolian - Locrian
And voila, we ended up with the two scales suited perfectly for music using functional harmony!
Historically, neither the Aeolian nor Ionian mode was used in the classification of Gregorian Chant (the primary reason for using modes.) Modes were based on D, E, F, and G. Those on D and E had a minor third above the finalis and the F and G modes had a major third above the finalis. These eventually sort of fell together and became minor and major keys. It's a complicated history.
Modern pop and jazz theories treat modes as permutations of an underlying scale. This use is fine for compositional purposes but it came long after the collapse of the Medieval mode classifications.
when tonality set in, the use of modes dropped to just two; Ionian and Aeolian, now the major and natural minor scales.
If by 'tonality' you mean functional harmony based on the dominant-tonic relationship, it wasn't major and natural minor, rather major and harmonic minor. Common Practice is 'all about the leading note'.