Part of this can be answered by the idea of Function. Within the Western music tradition chords serve a function, that is they play a role. The most commonly known term from this thought process is the Dominant function, built on the V of the key. The Dominant chord is named such because it has the most tension and most desire to resolve back to the Tonic, or I chord. It is Dominant in that it demands to be resolved and therefore Dominates the harmonic progression. The IV is known as the Pre-Dominant and is frequently used to set up the Dominant chord. As an example, I | IV | V | I | is very common, or I | vi | IV | V | All of the chords have a functional name and different uses (like Pre-Dominant is not played exclusively before the Dominant) but that is, as others have suggested, quite a long discussion.
An interesting fact that seems less known or thought about could answer some of your question as well. It is common knowledge that within a given key there are 3 major chords but people don't always realize that all 7 notes of the scale can be harmonized with at least one of those 3 chords. This is why any diatonic melody (within the scale) can be harmonized with some combination of I, IV and V chords. If you add function to the mix, then the chord choices become more obvious. As an example, when notes within the melody are dissonant, you are likely to have the Dominant chord.
As others have suggested in their answers, tradition and cultural tendencies play a very large role. As the tradition evolved, people began to play with the 'rules' of Function. Composers would have non-diatonic melodies (consisting of notes outside the scale), which would require non-diatonic harmonies. Modal Mixture/Borrowing is a fun one that has continued to be used by seemingly all expansions/genres of western music. Modal Mixture/Borrowing consists of 'borrowing' a chord and/or melodic note from a parallel key (in A major you could borrow a chord from A minor). The Blues is a good example as well. The Blues is built on Dominant 7 chords. Using all Dominant 7 chords is playing with the idea of Function. Instead of the Dominant 7 being the Functional Dominant, it is the harmonic texture. The role of Dominant is still fulfilled by the V7 but the I7 and IV7 are not Functionally Dominant, as they do not have the same desire to resolve to the Tonic (the I chord). The Function of V is further played with by 'stressing' the Dominant. In the standard Blues progression the first V chord is followed by a IV chord, thereby denying its Function. The V only appears again in the turnaround, the last bar, and then fulfills its Function as Dominant to bring the progression back to the beginning. (Sorry for the extended examples but I think they help illustrate the point)
So these cultural tendencies, such as the Blues, lead us to feel certain chord progressions are obvious or the 'right' choice. If you gave a Blues melody to someone that had only heard Classical music up to 1750ish, they would not have the cultural experience to know which chords would accompany this. Jazz musicians use the ii chord as a Pre-Dominant. Aside from the differences in harmonic language, this would mean that the 'obvious' choice to the Classical composer is going to be different than that of the Jazz composer 99% of the time, just based on the cultural norm. Guys like Thelonious Monk (Jazz) and Claude Debussy (Classical in the Romantic era) are great examples of harmonizing melodies in unexpected ways.
There is sort of a scientific explanation but I don't believe there is an official theory per se. Mathematically, frequency determines pitch and when two pitches are played simultaneously, their consonance/dissonance (tension) can be measured in terms of a ratio. Unison is the most consonant 1:1, with the octave next 2:1, then the fifth 3:2, and so forth. The major chord is perceived as more consonant than minor, which often translates to happy/sad, because it is more consonant, which can be proven mathematically. Scales, and thereby keys, are mathematically determinable and naturally occurring. They are determined by the harmonic overtones produced by a single note, which I will not try to define as I am already making quite a lengthy answer. Nevertheless, scales do occur naturally and do pervade all forms of melodic music, including the Eastern traditions as well as indigenous tribes that had seemingly had no outside contact since the onset of music theory.
This next part is more conjecture, as I am not studied in Psychology or Neuroscience. I believe that once a key is established, our brains interpret that as home base and anything else is compared to that, which gives it a sense of consonance or dissonance. The further removed from the home base, the more dissonant it sounds, even if you play a consonant chord. The Dominant triad consists only of notes that are not in the Tonic, making it have the most desire to resolve back to Tonic. The Dominant also contains what has become known as the Leading Tone, the 7th degree of the scale. It is a half-step below the root of the Tonic and by this thought process, it is the most dissonant note in the scale. (In case there is any confusion: being 'further removed', as I described it, does not mean distance from a note. Combining this with the mathematical side, though a half-step is 'closer' to the root, it is a more dissonant interval than the whole step, making it 'further removed')
So we combine cultural tendencies and conditioning with sets of frequencies' ratios equaling consonance vs. dissonance and potentially my conjecture on our brains' interpretation of being in a key and to some extent you can see why there are 'right' or 'obvious' choices for a given melody. Cultural exposure will lead to melodic tendencies, which, through the same exposure, will have seemingly 'correct' harmonizations. So even if you are coming up with non-diatonic melodies, which means not naturally occurring, it is likely due to cultural conditioning, which has also provided you with a means of harmonizing it.
As a composer, I always try to break free of the obvious.