This is a great question, and one that is logical. Yes, technically, major and minor triads are indeed built that way and can be thought-of as such. They can also be built and thought-of in the way slim described in his answer as well.
Interestingly enough, I would say that the majority of factors that influence how we perceive major and minor chords aren't actually theory related.
Much in the same way you use similar ingredients to make a large variety of food (take flour for example!) so too is intervallic function and pitch choice, and like many things in life, is largely dependent on context which I will expand upon in a moment.
In Set Theory you can actually prove that major and minor triads are exactly the same chord - which is always hilarious to watch a roomful of theory students as their minds explode. Mine certainly did when I first learned of it. But I digress.
When you hear a triad, I am willing to bet that you are probably not hearing it as a sum of two distinct intervals. Instead, you are probably perceiving the chord as a single, cohesive unit. Hearing distinct intervals would be pronounced in, let's say, an orchestral score where different instrument timbres can accentuate different pitches, but for sake of simplicity, let's discuss it as it occurs on a piano.
We're going to take a little shift here:
It has been proven in psychological studies that humans group together like stimuli - especially with respect to aural stimulus. Tangentially, but not completely unrelated, this is one of the reasons why you don't constantly feel your underwear throughout the day.
In another example, if you were listening to three different conversations simultaneously, it would be difficult to describe each conversation. However, if you heard a conversation, a jet going by, and a police siren, such disparate sounds would be easy to remember and describe.
Such is the case with major and minor chords. Because each chord contains a perfect-fifth where the fifth is harmonically supported by the root's overtone series, the fifth essentially becomes non-existent, as you would likely hear the overtone without the fifth's presence. This is why in four-part writing the fifth is usually the first note to be dropped from a chord voicing, and the same applies to Jazz as well.
The root establishes a "tonality" as it were, and the third merely denotes the "quality" or "color" of the chord, and the fifth is nothing more than overtone support. This is why chords that contain the same intervals sound differently from one another.
Earlier I referenced context, and context is important to note here as it can affect how a chord is perceived as well. For example, a major chord in the lowest register will sound very different from a major chord in the highest register (even using the same pitches!) Further, that same chord will sound differently played by a saxophone choir or tuned bass drums.
Apart from timbral context, major and minor chords can sound different depending on their harmonic function as well in a given key, tonal center, or pitch collection.
Hope that helps.