I remember a masterclass where the piano teacher commented a piece from Chopin with something like "and after this modulation we enter the world of sharps".
This left me wondering if it was a poetic way of introducing a modulation or if it was really possible to recognize the difference of "color" between the sharp and flat tonalities without recognizing the tonality or the individual notes.
In context of a established tonality, modulating from the sharps to the flats brings at least two new notes (ex : modulating from G to F), so this could be a clue you're modulating a bit far, but it's not enough to know that you've entered in the flats (ex : from A to G).
In absence of established tonality, I think this would be analoguous to listening a perfect pitch (ex : E major is in sharp, F major is in flat, but there's only a semitone between the two scales) : you notice the scale, not the "color". The color always depends of the context : think enharmonics.
My conclusion is that distinguishing the sharps from the flats is rather a matter or familiarity with tonalities : you recognize the arpeggios from a piece you've already played, the scale fragments, maybe you can name the tonality, but you have no way to guess the color per se.
What are your thoughts about "color discrimination"? Do you think you can develop a feel for colors?
(English is not my native language, feel free to edit)