I've been facing this problem for quite some time now: some notes sounding bad with an overdriven amp. It's a 19 years old solid body Kaman GTX-33 (the maker of the Ovation acoustic guitar), made in Korea. Practically the same as this one, but with a better PING licensed Floyd Rose bridge.
I have found a good sound example in this youtube video. Approximately at 07:43, Jack (the guy on the right) stops playing for a brief moment, switches to the neck pickup and then you hear two weird, warbling/vibrating notes, they sound almost as if they were out of tune. My guitar has this exact same issue, it annoys me so much I lose concentration, sometimes. Except in my case, it happens on the 5th string, mostly, around the 6th - 9th frets, and the 6th, towards the end of the fretboard. Does anybody know what causes that? It could be similar to the problem mentioned in this question, possibly the same, but the question didn't make it clear and also no satisfactory answer was given, so I'm really not sure.
The string is absolutely not buzzing against the frets and I've double checked and ruled out everything I could about:
- loose parts
- old strings
- amp problem (happens on all of my 3 amps)
- excessive magnetic pull
- bad pickups (had them replaced just for testing, problem remained)
- floating bridge springs vibration
Important: it only happens with overdrive. With heavier distortion, it tends to disappear. With clean sound, I can only notice it if I listen very carefully and in a very silent environment. Seems worse when using the neck pickup.
The only other possibilities I ended up with are:
- a vibrating truss rod (unlikely),
- something about my floyd rose bridge, its saddles or how it resonates, or
- the neck/body junction, maybe the neck and body don't "like each other"?
My theory is that there is some bad resonance or frequency beating going on, either in the wood or the bridge. It's a solid body guitar, but it never had a fantastic sustain, so to speak.
I don't have many luthier options in my area, and the last time i tried one, the guy knew less than I do (I've been playing for 24 years). I will also post a more detailed sound sample ASAP.
EDIT: here are the sound files. Notes: the first one is the neck pickup, clean. I can't hear the problem in this recording.
The second one is the neck with overdrive. I let the background noise flow for a while to illustrate to @Yorik what I was saying about hum or RF. About the sound, If you listen loud with good speakers, the problem becomes quite evident: some warbling, some unwanted vibration, some additional noise under the note, making it sound weird. I feel like it could be an excess of bad harmonics as @Andy mentioned, but I'm just not sure.
The third recording is a trick I did just for this question: I've wired neck and middle pickups in series but inverted the wiring of one of them, so they go phase cancelling and kill the background noise (I know, it kind of kills the tone, too. It's not a real humbucker, after all). In this example we can hear the problem very clearly and this removes the hum/RF possibility. The hum is gone, yet the problem remains. The pickups are pretty far away from the strings to avoid magnetic pull and there is very, little fret noise in the recordings, it was my first guess, but I don't think it is the source. I'm running out of options! I still think it's a wood resonation issue or something with the floyd rose vibrating and interfering.