It is impossible to say for sure, since the term chromatic has had a musical meaning since at least ancient Greek times (hence the use of a Greek word). More on that in a moment, but first, its worth noting that "color" has idiomatic meanings as well, such as these definitions for "colorful" in the Merriam-Webster Learner's Dictionary:
2. interesting or exciting
He gave a colorful account of his travels.
She has a colorful [=bright, cheerful] personality.
He was a colorful character. [=a very unusual person]
3. If you use colorful language you use words that are usually considered rude or offensive.
He's been known to use some colorful language when he starts talking about politics.
Either of these definitions could easily be applied to the case of chromatic notes in music. They add interest or excitement by using improper notes that are usually considered offensive (by virtue of being outside the scale).
As for the origins of the term... In Ancient Greek music theory, full scales were built by combining smaller tetrachords -- 4-note scales that spanned an interval of a perfect fourth. While the two outer notes had to be a fourth apart, the two inner notes could move around. Tetrachords were classified as belonging to a genus, depending on which of three distributions they had. The three genera were Diatonic, Chromatic, and Enharmonic. Note: the following explanations are only approximate, because the tuning system was considerably freer than what we have.
In the diatonic genus, the notes were distributed roughly as Half-step, Whole-step, Whole-step (HWW). Such as in E-F-G-A.
In the chromatic genus, the distribution was roughly Half-step, Half-step, Minor third (HHm3). Such as in E-F-G♭-A.
In the enharmonic genus, even smaller, microtonal intervals were used, such as E-E½♯-F-A.
As you can see, the diatonic genus primarily used whole steps, with half steps only to fill in the gap, which the chromatic genus used primarily half-steps (with a rather large gap). This is different than the modern usage of the term, to be sure, but the similarity should be evident.
The modern usage of the term can be traced back to this ancient usage through the Renaissance, and its interest in reviving ancient Greek culture. This can be seen especially clearly in the works of Nicola Vicentino. In Rome, 1551, there was a major music theory debate between him and Vicente Lusitano. The later argued that music could be understood through the application of only the diatonic genus, while the former was attempting to revive the importance of the chromatic and enharmonic genera. Although Lusitano was judged to be the winner of the debate, this didn't keep Vicentino from advocating for chromatic and microtonal music. In fact, in 1555, he designed and built a 31-note-per-octave keyboard, known as an Archicembalo, and published a treatise and music to go with it: L'antica musica ridotta alla moderna prattica (ancient music adapted to modern practice).
Similar instruments were built during this period, including one from 1606 which still survives and is actually called 'Clavemusicum Omnitonum Modulis Diatonicis Cromaticis et Enearmonicis' (roughly, something like "Keyboard to play all notes diatonic, chromatic, and enharmonic").
I found a translation of Vicentino's work on Google Books, and when introducing the three genera (which he calls "divisions" of the fourth) he has this to say about chromatic (emphasis mine):
The second division was called chromatic, which means transformed into steps unlike those of the first division [i.e. diatonic]. The third and last [division] was called enharmonic, which means composed of small parts.
Now "transformed" may seem an unusual translation for chromatic, but this is described in more detail later. Vicentino refers to the famous medieval theorist Boethius who wrote about Greek music theory in his 6th century work, De Institutione Musica (The Principles of Music).
Boethius says that chromatic means nothing more than the changing or the transforming of the diatonic sequence... transformed from one system to another. In calling this genus "colored" he did not mean to speak of color in similitudes rather than in literal terms, as some people believe. For just as mixed colors present diverse effects to the eyes, so transformed musical steps mixed in ascent and descent provide the ear with a variety of sounds.
Around the same time, Orlandus Lassus was experimenting with wildly chromatic harmonies in his set of motets, Prophetiae Sibyllarum, whose introductory movement, a sort of Prologue named Carmina chromatico, opens with the following exotic chord progression: C G B C♯m E.
Thus this use of the Greek term "chromatic" had become common place in the Early Baroque, where it was used to describe works like Girolamo Frescobaldi's Toccata Cromatica and Recercar cromatico post il Credo, or Tarquinio Merula's Capriccio Cromatico, his Sonata Chromatica, and his Toccata & Genus Cromaticum (the especially chromatic bit begins at 2:41). It would have been old-hat nearly a century later, when Bach wrote his famous Chromatische Fantasie und Fugue.
Side Note:
It's also worth noting that a decade before Vicentino re-appropriated the tetrachord-based use of the term "chromatic", it had acquired an alternative meaning. The Italian term note nere ("black note") was used to describe an ornamented style of madrigals that were written with shorter-than-usual durations (i.e. the "black" noteheads of quarter- and eighth-notes, as opposed to the usual unfilled noteheads of whole- and half-notes). Sometimes, such madrigals could also called "chromatic". It's not clear if this term was meant in a literal sense (e.g. colored notes), or a figurative one (the elaborate musical style being flamboyant and "colorful"), but the term had no connection to the Greek chromatic genus, as was noted by theorists at the time. However, this alternate association of embellished musical lines being "colorful" can still be seen in the term coloratura.
