When tonal music replaced modal music, why were 6 of the 8 scales (modes) used in modal music abandoned in favor of only two, major and minor? Superficially this would look like a reduction in the options available to composers and therefore appears difficult to comprehend. Was there a reason why this needed to be done, or was it just an accident of history?
In his comment, Patrx2 listed the 8 traditional church modes: Dorian (and Hypodorian), Phrygian (and Hypophrygian), Lydian (and Hypolydian), and Mixolydian (and Hypomixolydian). The "Hypo-" forms are called plagal modes (as opposed to the four authentic modes). The plagal modes have the same "final" (tonic) and the same pitch classes as their corresponding authentic modes. The difference is primarily in the range a melody was permitted to use: e.g. the Dorian mode would go an octave from D to D, while the Hypodorian would go from A to A (but still have it's final on D).
This illustrates the weakness of the modal system. It was not strictly defined, as is done today, as a series of whole- and half-steps. Rather it was thought of as a combination of concepts: a specific final tone (no transposition!) and an associated reciting tone, as well as a specific range of pitches (plagal vs. authentic), a set of associated moods or emotions, a set of cadential formulae, and so on. In fact, the resulting pattern of whole and half steps that was formed was more of a coincidence that could be, and often was, modified through the use of musica ficta.
Musica ficta was the common practice of chromatically altering notes during a performance that weren't notated as such. Two examples include flatting the B (especially to avoid the tritone F-B) and raising the seventh to create a stronger cadence (creating a major third with the dominant, that resolved upward by a half-step, called mi-fa in the movable hexachord system). Thus, one could be playing in Dorian mode, and find the B flatted in some places, while finding the C sharped in other places. This is essentially where we get the various forms of the minor scale from.
Once these alterations began to be notated regularly (and once keyboardists started tinkering with improved temperaments), you could do something new and exciting: you could start transposing music to other sets of pitches, but keep the pitch relationships the same. For example, throw in a B flat, and now you could write in Dorian mode, but move its final to G. And following the same pitch conventions, you might occasionally need to flat the E, or sharpen the F. But now that you're no longer using the same pitches or final as the true Dorian, are you really still in Dorian mode? Or are you playing in an altered form of Mixolydian?
As theoreticians delved for a better way to describe modes, they realized that, for example, when playing in Lydian (or Dorian), the B would often be flatted so frequently that they were essentially playing the same series of pitches one would get if they started on C (or A). But those didn't correspond to traditional modes, so Heinrich Glarean invented the Ionian (and Aeolian) modes in 1547 (and claimed they were already the most frequently used modes) and Zarlino affirmed them a few decades later. Of course, there were traditionalists, who didn't like this new-fangled development, but they would ultimately lose the argument. You also start to see some growing confusion over just how a mode should be defined.
Another development was the recognition of inverted intervals and triadic harmony -- that all the various combinations of consonances could be consolidated into essentially two types of chords: major and minor. This paved the way for thinking about musical scales as being an elaboration of either a minor chord or a major chord (rather than ending on a final note). Coupled with the recognition that Ionian and Aeolian were really the only two modes being frequently used anyway, this eventually led to the formalization of major and minor keys. Werckmeister, a German organist and elder contemporary of Bach who had a firm grasp on the tonality and modality of the time wrote of this development:
If we take Lydian, on account of the tritone... there is such an unnatural progression in it that even the ancients themselves never or hardly ever used it. Who uses Phrygian in today's music? Nobody. Who Mixolydian? Hardly any. Therefore... according to today's style of composition, we want to maintain only two modes.
In fact, some musicians even referred (erroneously) to the new keys as being modes, claiming that there were now 24 modes: 12 major modes, and 12 minor modes. This demonstrates not only the confusion in the definition of modes at the time, but also shows how the newer key-based system was seen as superior to the older modal system that originally only had 4 finals.
I wrote a lot more about the transition from tonality to modality in my question and answer here: