I hope no one minds that I got curious, and did a bit of digging into this on my own. I discovered what appears to be an excellent resource answering this very question. The book is entitled Between Modes and Keys: German Theory, 1592-1802 by Joel Lester (1989). I do not have access to a copy of the book, but I've been able to see several relevant portions online (thank you, Google Books!). Even though it deals primarily with German music theory, it does occasionally touch on the rest of Europe. I'll try to summarize what I've gleaned so far, although I may not not be seeing the whole picture, since I'm not reading through the complete book.
The primary development that necessarily preceded the concept of major and minor keys, was triadic harmony (rather than intervallic harmony). Johannes Lippius (Synopsis musicae novae, 1612) coined the term "triad" to describe the three-part harmonies that had emerged in the late Renaissance, and, being part-theologian, compared the concept to the Trinity. He also recognized all the inversions of intervals (and triads), including the fact that major 3rds inverted to minor 6ths (and vice versa). Any set of consonances could now be reduced to a triad, and placed in root position. He classified the 6 pairs of modes as being constructed around either a major or a minor triad, with an additional interval of a fourth either above or below. This classification didn't take hold right away; indeed, Germany seems to have been especially resistant to abandoning the old concept of modes (this seems to be a major theme of the book). Apparently, as late as Haydn, traditional German musicians were still advocating the use of modes.
In fact, throughout the Baroque, there seems to have been widespread confusion, even among musicians, over the exact definition of a mode. Were modes defined by the octave range in which they sat? Were they defined by their final tone? Or by their "reciting tone"? Or by the mood they evoked? Or by the melodic and harmonic phrases and patterns that they accommodated (especially around cadences)? Or the patterns of tones and semitones they formed? What about transpositions? And what of the chromatic alterations that were becoming popular in theatric music? Are we better off sticking to the older traditional 4 pairs of "time-tested" church modes?
The practical solution, oftentimes, was to avoid the question altogether and just list out a handful of permissible "keys" (as in, on a keyboard) that could be used as the final, along with their corresponding signature (initially either blank, or containing one flat). The key of G, for example, could be used with either no signature, or a single flat. As additional keys were added to these key lists, they were classified, not by the quality of their third, but by whether they included flats, sharps, or neither in their root triads. Thus what we call "G-minor" would have been classed along with what we call "B-flat major" as a "flat key", while our "C major" and "D minor" would be examples of "natural keys", and so on.
Andreas Werckmeister, a church organist and theorist, whose works (and well temperament) were well-known to Bach, had a solid understanding of modes, and describes them as being important to understand (for which he would later be dismissed by proponents of keys). But he also acknowledged that, outside of chorales, only two were in popular use in his time. In Musicae mathematicae, (1687) he gives us this description of the system then in use, along with his own proposed nomenclature (which was not adopted):
Today's music is entirely different... and only some four modes are in
use: Ionian mixed with Mixolydian, and Dorian mixed with Aeolian...
[which each differ only in the upper fourth of their octaves].
Thus no more than two modes can now be established. And that is not so
unnatural... If we take Lydian, on account of the tritone... there is
such an unnatural progression in it that even the ancients themselves
never or hardly ever used it. Who uses Phrygian in today's music?
Nobody. Who Mixolydian? Hardly any. Therefore... according to today's
style of composition, we want to maintain only two modes. But because
these can take their names neither from the Dorians, the Ionians, nor
from any other nations (because they did not have our present style of
music), therefore we want to name them according to their nature and
character, so that they can be differentiated. The first can be named
the natural mode, because it always maintains the major third in
the beginning over the fundamental note... the second can be named the
less natural mode, because the root numbers in its natural progression
are further removed from perfection, and therefore do not
establish such a happy harmony as the preceding... We can also name
one mode perfect, and the other less perfect. Some performers
name them dur and moll; e.g., C E G is C dur, C E-flat G
is C moll... We are not happy with these names... nevertheless,
because these terms are now used so commonly, they will probably
persist.
Lester claims this may be the first published usage in German of the terms dur [hard] and moll [soft] to refer to major and minor keys, but the process was already said to be widely used by performers. Also mentioned is that Werckmeister originally envisioned the minor as being derived from Dorian rather than Aeolian.
Lester also shows that "By the late seventeenth century, French works routinely differentiated keys solely on the basis of major and minor." One reference refers to the distinction of "major" and "minor" keys as being according to the "French opinion". The first published recognition of all 24 major and minor keys is from a French mathematician named Jacques Ozanam, who in his Dictionaire mathematique (1691) explains:
There are twice as many modes as there are notes in an octave: each of
these notes gives its name to two modes, of which one proceeds by the
major third and the other by the minor. Since the octave contains
twelve notes, there are twenty-four modes.
Note here that each key is called its own mode, rather than transpositions of two basic modes. An earlier French quote, from a 1689 basso continuo manual, refers to two types of modes: a "sharp" type that "reduced to C," and a "flat" type that "reduced to D," (here again is the Ionian/Dorian distinction).
As noted by Robert Fink's answer, in Germany it is not until 1711 and 1713 that Johann Heinichen and Johann Mattheson enumerate a full list of all 24 possible major and minor keys along with their proper key signatures (using Aeolian as the relative minor). Heinichen connects them into a circle of major and minor keys, a formalization of a "well-known" device he learned from his organ teacher. There is also a series of letters between Mattheson and Fux, in which Fux complains that Mattheson's 24 keys are all just transpositions of two modes. Mattheson argues that each key is still distinct, due to differences in its temperament.
It's interesting to note that, as late as 1722, J. S. Bach's title for the Well-Tempered Clavier refers to the 24 keys rather obliquely: "preludes and fugues through all the tones and semitones both as regards the tertia major or Ut Re Mi, and as concerns the tertia minor or Re Mi Fa." (Re Mi Fa could even be seen as indicating that the minor third is still rooted in the Dorian mode).
In conclusion: If it could be said that there was one single thing that changed, which slowly but inevitably pushed music towards a natural distinction of major and minor keys, and made the modes obsolete, it was the recognition of triadic harmony, and the understanding that those triads could be inverted into various positions. Once that was recognized, early in the 17th century, it was only a matter of time before the concept of modal finals was replaced with that of a tonic triad, of which there could only be two types. Much of this process took place intuitively over the course of a century, with theoreticians describing the system after the fact, and traditionalists often regretting the loss of modes in the modern style of music.
