why is this pattern so crucial?
It isn't, really - it just happens to subjectively work pretty well, with most of its starting points (apart from, arguably, the Locrian) yielding modes that have interesting harmonic and melodic possibilities.
After all, a scale can be composed of any allegedly random set of intervals.
Well, it depends what you're trying to achieve. If you want to allow consonant harmony, it makes sense to have some notes that are consonant with your root note, and with other notes in the scale; If you look at the notes in the diatonic scale, it allows 6 fifths, 6 fourths and 3 major thirds, for a start, in a scale with only seven notes - which seems quite a good yield to me.
(The fact that there are 6 fifths relates to Michael Curtis' point that the diatonic tones can all be rearranged as ascending perfect fifths.)
And what explains the adherence to it?
I don't think there really is an adherence to it, if you look at a wide body of music. Minor key harmony? Usually goes outside of the diatonic scale. Modern major key harmony, for that matter, too. Whenever anyone 'modulates'? They probably go outside of the diatonic scale. 'Borrowed chords', likewise. Blues scale? Not diatonic.
Of course a lot of cases of these might be said to be almost diatonic, or represent a motion between one diatonic scale and another, or somesuch. Which is fine if it helps you as a way of communicating or doing your own analysis.
In case it sounds like I'm trying to denigrate or disenfranchise the diatonic scale, I am not. I like it!