If your audience is getting into the song and dancing (even 'dancing' in a low key way, like toe tapping / clapping), and you mess up the rhythm, you embarrass them - they suddenly look (and feel, mentally) out of time, relative to you. You've betrayed them! Shall we trust you again? Hmmm...
I imagine it might be similar if you got an audience to sing along, and then played a chord that made their singing seem very out of tune - they'd feel foolish and the emotional impact would act as a barrier to them getting into it again. However, if they were only dancing or toe tapping, the wrong note wouldn't mess with what they were doing, so it's much less of a big deal.
So although there may well be some senses in which rhythm is fundamentally more important than melodic or harmonic considerations in some contexts, I suspect that what you've observed may just be the fact that many live audiences tend to 'get into' the music rhythmically more than they do harmonically/melodically. This is even more the case when the song is unfamiliar to them, as the rhythm is likely to be much more predictable than the melody or chord progression.