Short answer: it's in D!
Long answer: Your approach of taking a chord/pitch inventory and comparing it to keys/scales is one possible way. The problem with this route, however, is that chromatic pitches will confuse the system and possibly results in an incorrect final answer.
For music like this, the best method is just to recognize what chord sounds like a stable point of repose. In this case, everytime the D chord appears (especially at the end) it sounds like it could be the end of the song; it doesn't need another subsequent chord to reach a completion.
However, the D# chords in the first two lines are not stable at all; they need to resolve to the subsequent E chord, so D# is a terrible candidate for the key of this piece.
Otherwise, you'll notice that the progression
A7--D happens three times throughout the piece. Look for this pattern of a "dominant seventh" chord resolving down a perfect fifth to a triad. In most cases, this will be a
V7--I motion and the latter chord will be tonic. (But note that this progression is becoming more and more rare in popular music, so this method doesn't always work either!)