Elfman is both right and wrong.
With respect to traditional music, he is very wrong.
Suppose an instrument is tuned to a specific key, not with equal temperaments. A tune played in the instrument's key will sound different than the same tune played in a different key.
For instance, using the diatonic scale, if an instrument is tuned to the key of "C", the interval between "do" and "re" will have a frequency ratio of 1.125 (9/8). If the same tune is played on that same instrument in the key of "D", that same "do-re" interval will have a frequency ratio of 01.111 (10/9), which will sound significantly different.
Classical composers were well aware of this difference and made use of it. That's why their works often include the key in their titles. Bach's Toccata and Fugue in D Minor would sound very wrong if played in any other key.
Bach knew very well what the various keys sound like in meantone temperaments, and in well temperaments. Organs he played were in various versions of meantone, as a standard. (And some still are.) Other wind instruments were also designed to use those specific twelve notes: A-B-C-D-E-F-G, F#-C#-G#, Bb-Eb. The other notes were by design not as well in tune in the temperaments in practical use. Composers understood this. This was practical musical knowledge. Temperament — Bach Cantatas Web Site
But since that time, instruments and music have changed, losing the tonality that defined classical music. In some ways this is a great loss (no more associated mood), but in other ways it is a win (any tune can be played on any instrument, or sung in any key).
Pianos, and many modern instruments, especially synthesizers, are tuned with equal temperament, in which the interval between any two adjacent notes, including semitones, has a ratio of the twelfth root of 2. Using this system the interval between "do" and "re" will be 1.0593781, a compromisean approximation of the 1.125 and 01.9111 of the diatonic scale. In fact, every whole tone difference will have that same ratio.
A piece written in equal temperament will sound exactly the same regardless of what key it is played in. And today, almost all music is written that way; it has no tonality.
So that is what Danny Elfman was referring to when he said "You can turn any key into anything you want.".