The difference between the minor triad and the major triad is a good case study. People are often surprised when they hear that they have the same interval vector, since they "sound so different".
The commonplace that minor and major triads, which have the same prime form, sound very different, can be put into question. I'm pretty sure that it takes quite a while for music students to reliably tell apart minor and major triads by ear. However, if we compare, say, the minor triad and a three-note cluster (comprised of C, C#, D in some inversion or transposition, Forte number 3-1), I am pretty certain that people with no musical background at all can learn to tell them apart pretty much immediately. So yes, the major and minor triad are different, but by far not as different as many of the other possible three-note chords/chordioids.
The reason that minor and major triads do not sound all that different is that they have the same interval vector, etc. etc.. Now to get to your question: what feature makes that they nonetheless sound different? It is a good question, and actually a bit mysterious to me. The best answer I can offer is in the way the overtone series of the three tones interact. On most instruments, one cannot hear very much of the overtone series. So it's important not to overemphasize the contribution of overtones to the overall sound of chords.
In trying to work this out I added the first five harmonic overtones to each of the three chord notes of a major and a minor triad. The resulting pitch collections are not inversions of each other and have a different interval vector, so this could be a good answer to your question.
The first five overtones are three different octaves, a fifth, and a major third. So on the level of the pitch class, we only need to add the fifth and major third to all three of the chord notes.
For C major (C, E, G), we get (C, D, E, G, Ab, B).
For C minor (C, Eb, G), we get (C, D, Eb, E, G, Bb, B)
As mentioned, these are not inversions of each other. (They do not even have the same cardinality.)
In other words, although the fundamentals of the major and minor triad have the same interval vector, as soon as you start to consider overtones, some divergences start to occur, which could explain the difference in sound.
As a sidenote: This train of thought would predict that major and minor chords made with pure sine waves would be harder to tell apart than the same chords made with more harmonic content. (Perhaps interesting to test this empirically?)
Finally, perhaps a simpler answer is that minor and major chords are typically embedded differently in harmonic progressions. And they also have different melodic implications. Perhaps that is the simpler explanation.