Why is it important to stay within the key
It isn't, necessarily!
However, doing so is likely to help your music sound more stable. The fundamental frequencies of important notes within a key have a strong harmonic relationship with the tonic note of the key - for example, in a Major key, some of these ratios are 3:2 (perfect fifth), 4:3 (perfect fourth), 5:4 (major third); In a minor key, we have the minor third (6:5), which is also a simple relationship.
In nature/physics, a single resonating body often tends to have overtones that are integer multiples of a fundamental frequency, and therefore tend to have simple frequency relationships with each other. In music, using notes that have simple frequency relationships with the root makes the overall mixture of harmonics in the piece seem more like a single resonating body from a mathematical point of view; subjectively, it keeps the music sounding stable, and gives it an overall sense of being based around, or "wanting to return to", the root.
But is that a good thing? That sense of stability might not be what you want - and in any case, using some notes that clash with the root doesn't necessarily disrupt the sense of stability in an unpleasant way. Even in the major key, for example, the seventh clashes quite strongly with the root (although it is the major third of the fifth of the root - so it has a strong relationship 'by association').
My intuition is that if the song is supposed to be happy, ergo if it's to use major chords, then all chords would end up being major.
Some pieces, or at least parts of them, do exactly that - listen to the descending suspended and major chords in the main riff from Pinball Wizard from Tommy by The Who, or the chorus from the Timewarp. For a song that only uses minor chords, try Inner City - Good Life.
It can sound great. However, because it can mean using more notes that have a less strong relationship with the root than they have with other notes, it can cause the music to sound like it's 'jumping around' - which might be a bit too dramatic to do throughout a piece.
What's the deeper reality that has made diatonic chords be the thing to look for
If there is a deep reality, it's about maintaining the sense of stability that comes from using notes with simple frequency relationships with the root. Historically this was more important, as the 12-tone equal temperament system that we have today (which softens the clashes that we get from moving to different keys) wasn't in widespread use.
However, please don't be left with the impression that diatonic chords are "the thing to look for". They are probably the thing to look for if you want a sense of stability in your music, but if you want to add a bit of excitement, you may want to move outside of the notes from the key - whether that's "borrowing" a chord, using secondary dominants, working in one or more real modulations into your piece, or just using a non-diatonic passing note.