I was learning modes. But I'm confused now. My problem is that I don't know how to apply the modes in other scale. For example I know in c major, mixolydian is form g to g and all keys are flat. But what is keys or notes should I play if I want mixolydian in d major scale?! Should I play c# and f# I this key or...?!
Modes are the result of playing a major scale but starting on a different degree. Here's how we derive all seven modes:
As you can see, there are seven different modes to be created from C major. But notice that these aren't only rearrangements of C major. We can think of them as scales in their own right!
C-D-E-F-G-A-B-C: C Ionian
D-E-F-G-A-B-C-D: D Dorian
E-F-G-A-B-C-D-E: E Phrygian
F-G-A-B-C-D-E-F: F Lydian
G-A-B-C-D-E-F-G: G mixolydian
A-B-C-D-E-F-G-A: A Aeolian
B-C-D-E-F-G-A-B: B Locrian
You also probably noticed that these aren't the same as the major scales on the root they have (except for Ionian, obviously). For example, D Dorian is not the same as D major. The word "Dorian" is therefore indicative of a different interval series! So how do we describe modes with relation to a major scale?
C-D-E-F-G-A-B-C: C major
D-E-F-G-A-B-C-D: D minor (♮6)
E-F-G-A-B-C-D-E: E minor (♭2)
F-G-A-B-C-D-E-F: F Major (♯4)
G-A-B-C-D-E-F-G: G major (♭7)
A-B-C-D-E-F-G-A: A minor
B-C-D-E-F-G-A-B: B minor (♭2,♭5)
Every mode can be thought of as an altered major or minor scale. To answer your question, "the mixolydian scale in D major" is a bit ambiguous. If you mean the mixolydian scale that is derived from D major, you mean A mixolydian (A-B-C♯-D-E-F♯-G-A). Notice it has the same notes as D major. If you meant a mixolydian scale built with D as the root, then that's just D mixolydian (D-E-F♯-G-A-B-C-D). One flattens the 7th scale degree (C♯) of D major to a ♭7 (C) to produce D mixolydian.
In order to do what you described as "applying the modes", one simply plays each mode over the correct chord in a key. For example, in B major, when an F♯7 chord is being played, one can play F♯ mixolydian scales over it (because F♯ is the 5 of the key of B major). Same notes as B major, but a different way to use them.
Two ways to look at modes. One - each mode has changed notes from its parallel major or minor scale set of notes. Thus D Dorian (minor) is D natural minor but its 6th note is a semitone higher. D Mixolydian (major) is D major but its 7th note is a semitone lower. Each mode has the same particular change to the note number/s.
The other way is to go back to the parent key. As in D Dorian is from parent key C. So it has no # or b (can be played on white keys on piano).D Mixolydian comes from parent key G. So there is only F# as an 'accidental'.
I suggest you consider both ways of perceiving modes - one will work better for you!
'Mixolydian in D' is ambiguous. Either it's Mixolydian of D or it's D Mixolydian. Think about it - The Mixolydian of D will use all of the notes from D major. D Mixolydian will use all the notes from G major. Close, but not the same!
@User45266 gives good advice with...
Every mode can be thought of as an altered major or minor scale.
...but if a picture is worth a thousand words, I offer this graphic showing the progressive alteration of the major scale/ionian mode by adding a sharp or adding more and more flats. I tried color coding major as blue and minor as red...
...so we can think of modes like this: lydian is a major scale with a #4, or mixolydian is a major scale with a b7.
With aeolian the same as the natural minor scale we can think of dorian as a minor scale without the b6 or phrygian as adding a b2.
Notice that all scale degrees can be altered except the tonic (degree 1.) If we skip the locrian mode (it was not one of the old church modes) then notice that the two most important tonal degrees - 1 and 5, the tonic and dominant - do not ever get altered.
Also notice that the "progressive" order of the modes in this graphic are...
...which is different that the order of the modes of the major scale...
You're on the right track. Basically, if you take a major scale as a reference point, you can get the modes by taking the same notes, but starting the scale at a different point. So as you say, if you take the notes from C major, but play the scale starting with G (the fifth degree of C major), you get G mixolydian. If you take the notes from D major (c# instead of c, f# instead of f) and start from the fifth degree of D major, which is the A, you get A mixolydian. For B mixolydian, you play the notes from E major and start on the B, and so on.
The modes are technically defined, as are scales, by the specific sequence of stepwise intervals that comprise them. In the case of mixolydian, that would be W-W-H-W-W-H-W, where W are whole steps and H are half steps.
To get modes in other "keys," simply apply this definition to any starting note you desire.
The modes that we most commonly talk about these days happen to all be "rotations" of the major scale, so the same W-W-H-W-W-W-H pattern that you get from the major scale is in there if you repeat the sequence of intervals and look for it. As you can see, the mixolydian mode also happens to be almost identical to the major scale rooted on the same note - only the last two intervals are swapped (H-W instead of W-H). This leads to the notion of the modes being "alterations" of some more familiar scale, which might help you figure out the notes more quickly given some arbitrary starting note.