As someone with absolute pitch and trained in A440 12-tone equal temperament (i.e. the usual) with plenty of vocal music as a backup, I perceive notes that are several cents out of tune as "off" because I am not used to them. I have difficulty listening to music containing those out-of-tune notes unless the music is atonal. (Granted, I don't like listening to atonal music for the most part, probably because of my absolute pitch, but I did find out that I didn't think of quarter-tones as unpleasant in an atonal piece of music I listened to once.)
My experience with this:
as long everything is in the correct relation to each other why should it sound out of tune?
If I find that the out-of-tune notes are in tune with each other, I figure that the music or instrument is out-of-tune and may be able to accept the music more. (I'm interested in hearing what B quarter sharp major sounds like, for example, but not in the context of a C major piece.) If I find that only some notes are out of tune, I figure that the performance is off-key or the instrument was not tuned properly.
I'll have to note that I will not detect music as out-of-tune if it's not several cents off (unless the in-tune note is played at the same time--in that case, everyone can hear the beats that get produced). This probably does make listening to vocal music easier, as people probably go for Pythagorean tuning instead of 12-tone equal temperament within a specific key because Pythagorean tuning uses perfect ratios (e.g. 3:2 for perfect fifths) and 12-tone equal temperament doesn't unless they're octaves.
As for why blatantly non-A440-12-tone-equal-temperament pieces sound wrong to me, I believe this is because of my musical training. A432 sounds slightly off to me, but I suspect that if I were trained in A432, it would be A440 that sounds off instead.
To use your colour analogy, the monitor with the not-perfectly-RGB pixels is just like the out-of-tune musical instrument--the colours look different, the web-safe colours don't display properly, and this time, everyone notices. The extreme example is the dead pixel where one of them is black instead of RGB--people start throwing out their monitors or getting them repaired at this point. (You are right--"people define, very accurately on a physical level, what a standard red, green and blue are"!)
(Caveat: I developed absolute pitch starting in Grade 8 (age 13)--I started with middle C as a reference, needing to use classical music for key references, and writing sorely thin transcriptions by ear. I've read before that absolute pitch developed this way may not be true absolute pitch, and needs constant maintenance or else it will be forgotten. On the other hand, I can instantly recognize soloing on the dominant of minor keys (an unorthodox mode), can recognize modes increasingly well, and can figure out when polytonality (2 or more keys being used at the same time) occurs...)