When performing music solo on guitar (and singing) for an audience, I have noticed that the audience is very tolerant when I miss a chord or mess up a lick. Often they don't even notice.
But if I get my timing off on the rhythm and miss a beat, it totally kills the vibe. The audience may be tapping their toes, nodding their head, clapping their hands, smiling, and I can tell they are really getting in to the music. But the second I come in half a beat late on a chord change, I totally lose the audience. They stop nodding, tapping, clapping - they even stop smiling.
Even worse, it's very difficult to get them back into it on the next song. It's like they are saying "I'm not falling for that trick again!"
What is it about keeping the correct tempo (perfect time) that makes it seemingly more important to an audience than singing or playing the right notes?