You’ve discovered a very deep fact about music that is referred to as “mode.” It isn’t ridiculous at all to say that the same notes can have a different emotional quality depending on how you use them, in fact that’s a huge part of what composition is about! Most western composers use the same twelve notes to express everything from rage to joy to anhedonia. The seven classic modes are all defined as a specific sequence of whole and half steps: some version of WWHWWWH. What I mean by “version” is that any starting point in that sequence can be used if we just rotate the sequence. For instance, I might start with the first half step: that gives me HWWWH, and then I finish up with the part of the sequence i skipped from the beginning, WW. The complete sequence is HWWWHWW. If I start my scale with an E, that gives me EFGABCDE, and this mode is called E Phrygian. You’ll notice this is the same notes as C Major—CDEFGABC—only rotated so that it starts on E.
But what a difference it makes! C is the home note of C Major, and it can be approached by a half step up from the B below it. E is the home note of E Phrygian, but it can’t be approached from below by a half step. However, it can be approached from above by a half step. The most common resolutions in C major involve an upward push, but in E Phrygian they involve a downward fall; that’s bound to have an emotional effect. There’s a reason that Phrygian scales are very common in heavy metal music, but C major is quite rare.
If I rotate the step sequence in a different way, I get WHWWHWW. If I start on A, that gives me ABCDEFGA. You referred to this as A minor, but actually it would be better to call it A natural minor, or, better yet, A Aeolian. That’s the name of this mode. It doesn’t have a half step on either side of the home note, and thus has yet another set of potential emotional qualities. However, the world of true A minor is more complicated: sometimes the sixth and seventh notes of the scale are raised by one half step from their “natural” state and at other times are left natural. You should look up harmonic minor and melodic minor scales for more details on this. The upshot is that A minor is more complicated than A Aeolian, since it sometimes use F# and G#.
One last example of how order can matter: the sentences “I’m a genius, am I not?” and “I’m not a genius, am I?” Same collection of words, different meaning.