This isn't a music theory question, and there is no answer.
Why is a dog called "dog" in English, but "inu" in Japanese, etc. etc?
How do you describe the difference between crimson and vermillion without circular references to red?
Color are described as "warm" and "cold" even though color does not have temperature. People do that, because you can't describe color objectively other than a wavelength frequency. Even names like "green", "red", etc. are subjective. Where does green start and end on the spectrum?
Pitch is describe with various terms like "high" and "low", even though pitch doesn't have elevation. Objectively, pitch, like color, can only be described as a wavelength frequency. Music theory terms like "diatonic" and "chromatic" are fairly objective (although there is an argument about what diatonic means) but they refer to a system of pitches.
Why was the sense of sight, color, invoked to describe the concept of non-diatonic with the label "chromatic?"
Why is the sense of taste, spiciness, often invoked to describe the concept of dissonance?
These ideas are just metaphors. There is no logical reason to choose one particular metaphor over another or to explain the origin of the terminology. The terms could have just as well been "native", "foreign", and "exotic" instead of "diatonic", "chromatic", and "accidental", and the labels would work just fine even though pitches don't have Nationalities. Selecting a metaphor is arbitrary.
You also won't be able to pinpoint and origin for the terminology. Language emerges collectively out of a culture. The best you will be able to find for origin is earliest known (usually printed) usage. You might also find treatises, manifestos, etc. for the origin of terms. Examples from visual art like "Futurism" or "Dada" come to mind. But you still face the conundrum of such terms being in use before the publication.
There is no logical and historical answer.
That isn't satisfying.
If you want something that might satisfy the "why" of the linking, you might consider that the senses of sight and sound both deal with wavelength frequencies unlike taste, smell, and touch. Of course that isn't a historic understanding, but it does explain in part why the senses of sight and sound might be metaphorically linked.