Having a set of notes that fit in both the G major scale and D mixolydian scale how can I find out which key is it in? What if it's hard to even find the tonic, because maybe it is rarely used?
Look for cadence points. If the song has lyrics, these will be helpful. See what notes occur at cadences and what notes lead up to those notes. Esecially look for cadences for which the note value is "long" (like a half note within lots of quarters and eights). Often these cadences are at important chords like the tonic or dominant (or less often others.) The final phrase should lead to the tonic (normally). This is a good starting place. It's the same process as harmonizing a melody you wrote.
Practice doing this on many song.