There are a number of reasons why this effect is sometimes observed.
Often, musicians and audiences will notice that modulations in the sharp direction (adding sharps = raising notes) often feel as though the key has been "lifted", as some notes are higher. In pop songs that modulate up a half-step (or even a whole step), often the same brightness effect is undeniable. If keys get too sharp, they may be perceived as flat, due to enharmonic properties.
Even when there is no context to compare the key to, often musicians will conceptualise sharp keys as brighter than flat ones, since they 1) know the key and 2) know that this key is often modulated to by adding sharps. Whether this is apparent in their interpretation of the music is unknown.
Listeners with perfect pitch will know what the key is in their heads, and even if they aren't musicians, they may be able to recognise that one song seems to be in a bright key compared to another. This may be similar to synesthesia, and synesthetes, of course, would perceive some keys as being brighter than others (but which ones can vary considerably).
Historically, composers would name their pieces after the key they were written in (Canon in D, etc.). This is a vestige of a time before equal temperament, when different keys really did sound different because the intervals were not equal. And of course, the whole "D minor is the saddest key" thing.
In music where equal temperament is not used, different keys could be brighter than others.
Finally, possibly, composers subconsciously compose music differently in sharp keys than bright ones. They might often be perceived as brighter if composers compose more brightly in sharp keys, whatever that could mean.