Classical music did use drums: timpani and bass drum for starters.
...I'm talking where the rhythm is one of the main driving forces
Rhythm IS a driving force in classical music.
I think what you mean is a combination of: why didn't classical music use constant, repeated, dance patterns? and why didn't classical music use percussion throughout the orchestration of music as a principle instrument?
To the extent that sonata form is the epitome of classical form, repetitious dance patterns were not the norm, because sonata form focuses on different treatment of rhythm. Basically sonata form uses repetition on a much larger scale. It's a bit like asking why a novelist doesn't write all their prose in rhyming couplets... they don't because it's a different form. Sonata form doesn't use dance rhythm, because it isn't dance music.
Why percussion isn't used constantly in classical music can also be explained historically. The history of composition which eventually leads to the classical style starts with purely vocal sacred music, as music evolved wind and string instruments supported the voices, keyboard instruments also developed during those times, secular forms (like the madrigal) used instrumentation more or less similar to sacred. Within that historic context percussion would be associated with folk dance or martial music!
Fast forward to the styles that evolve into jazz - featuring drums as a principle instrument - and we see that the march genre is the direct ancestor! March > ragtime > jazz > popular blues > rock > disco. In other words, martial music, the march, is the form that evolves into jazz and various pop forms familiar now.
So, classical music didn't focus on percussion as a principle instrument like today's pop styles, because the style evolved from sacred vocal music rather than martial music.
Of course the are exceptional cases like Ravel's Bolero featuring the snare drum. Is that classical? Regardless, such exceptions don't make the rule.
Just a comment about folk dance. Obviously folk dance historically uses percussion. I regard that music as not the historic source for the development of classical style. So, for this question I don't compare modern pop with folk dance. Classical music did borrow from folk dance for effect - ex. musette - but folk dance wasn't the evolutionary origin of the classical style.