What is the trends that led to the fizzling of ars antique and to the rising of ars nova?
This has been discussed since the 1200s without coming to a unanimous conclusion. In practice, the minimum rather than the breve seems to be the main pulse note.
I have read various places that the big revolution (really a 200 year or so evolution) was the improvement in musical notation. As notation became able to show more complex rhythms and pitches, the limitations caused by only learning by ear could be relaxed. Thus composers could try more "interesting" compositional techniques and share these with everyone.
I recommend looking at music as an alternation between the influences of Apollo and Dionysus (look up this term, it's very important in the arts).
Apollo is the sun god, and the god of music. In art, Apollo represents formalism, intellect, and artifice. Apollo represents GREEK ideals.
Dionysus is the god of wine, and of nature. In art, Dionysus represents freedom, emotion, and chaos. Dionysus represents ROMAN ideals.
Thus the Classical period of Mozart, which is not too flashy and is highly formal, is mainly Apollonian. The Romantic period of late Beethoven and Liszt, is passionate (like Italians, aka Romans), so can be thought of as largely Dionysian.
So you can see Ars Antiqua as representing formal traditions, with highly ceremonial music, and Ars Nova as representing highly expressive, new rhythms and sounds.
Then comes Renaissance, which is the rebirth-- of Greek formalism, aka Apollonian Then the Baroque, which is quite flashy (highly technical keyboard solos for example)-- Roman, aka Dionysian Then Classical-- Apollonian Then Romantic-- Dionysian
After that, there has been so much going on that you can't divide it so easily into periods any more. However, you can see VERY strong influences-- techno and electronic music is highly structural therefore Apollonian, and some types of jazz are ultra-Dionysian.
Pretty long answer, but your answer can be inferred: Boredom with overly structural and intellectual music eventually led to a focus on secular music-- dance rhythms, fast and exciting melodies, and so on.
I'll leave you with an Ars Nova song. This is NOT performed by Gregorian monks!
You can check out the English lyrics here: saucy, irreverent stuff! :D