I am pretty new to improvisation. So far I have improvised over a 2 5 1 change using the diatonic 7th chords and the underlying scale tones that fit in with that chord. I am playing on a guitar.

As a precursor I will state what concepts I understand in music theory so far:

 - Scales (Major, Minor(Melodic/Harmonic/Natural))
 - Chords 
 - Diatonic Chords

I want to take it to a song, and I like the sound of "How Insensitive", here is the lead sheet I have:

[![How Insensitive][1]][1]


When I started, I thought that the key was C, because there was no key signature but someone told me it was actually in D natural minor. Is this common?

Next I wanted to see how these chords fit into that scale. For the D-9, I see that that's pretty much a diatonic seventh chord starting on the first note of the D natural minor scale, so I would improvise over it with that scale itself.

I checked it out the next chord, which is a C# diminished seventh, I know that C# isn't in D natural minor, but I do see that the melody is playing an A, so if I were to combine the notes of the C# diminished seventh and the A, we get A, C#, E, G, Bb ~ that is a A7 with a flat 9. 

At this point I was unsure about how to find a scale that would work under this chord, because C# is not in D natural minor.

I thought about two other minor scales:
 - D Harmonic Minor: D,**E**,F,G,**A**,**Bb**,**C#**,D
 - D Melodic Minor: D,E,F,G,A,B,C#,D

The D Harmonic minor looks quite similar to the chord, but am I allowed to play this scale in the background? Doesn't that contradict with the fact that the song is in D natural minor?

I moved to the C minor 6th which is C, Eb, G, A. I see that the Eb is in this chord, but not in D natural minor, so again, I need to find some other way to see what I can play in the background of it.

Based on the D Harmonic or Melodic Minor, it still wouldn't work as neither contain the Eb. I'm quite confused on how I could figure out what I can improvise with here. 

My two main question are:

 - Is there a process I can follow to determine what scale I can improvise with? Like with the C# diminished 7th It seems like the melody can help at times, but I'm not sure exactly
 - Can someone explain why lead sheets have non-diatonic chords, I'm confused because they might not have scale tones in them, especially like with the E7 near the end of the song.

Thanks for the help!

  [1]: https://i.sstatic.net/yf0Ku.png