I would like to learn about the art of establishing a mode. For example, how to make A minor sound like A minor instead of C major.
The basic idea on how to establish a mode is make the tonal center of the progression and melody center the tonic of the modal scale and use harmony that signifies the mode you are in.
For example this progression would signify C major.
C - F - G - C
The progression stats and ends with C and the melody to accompany this would use the notes of the C major scale that would start and end with a note in the C major chord.
This same progression in A natural minor would be:
Am - Dm - Em - Am
The progression stats and ends with Am and the melody to accompany this would use the notes of the A natural minor (C major) scale that would start and end with a note in the A minor chord.
However, most composers use the harmonic minor scale for minor keys by raising the 7th scale degree to a major 7th which would change the progression to.
Am - Dm - E - Am
It is still derived from C major/ A natural minor, but the scale makes A feel more like the center tone.
A very interesting sounding mode I use all the time is Phygian which is built from the 3rd scale degree of the major scale. In the Phygian mode the II chord is very strong and is a great for establishing the mode.
Em - G - F - Em
This progression diffres itself from a minor progression with the II and how II leads to i.
This is a very big topic, but this is the bare bones idea behind it. If you need more detail please ask.
@Dom answered this from the perspective of harmony, so I thought I'd complement that with an explanation about melody.
There is an old theory of melody (maybe it was in Fux' Gradus ad Parnassum? Don't have time to check) that a melody starts "at rest", then "moves", then "returns to rest". This pattern is generally found on at the level of the musical phrase, but can also be found at the level of multi-phrase systems, motifs, and the whole piece.
The "at rest" bits are typically at the tonic, and the "moving" part then passes through the other pitches, revealing what the scale is. This is the (a?) mechanism by which we hear mode established in melody. If the first note is a D, and in the subsequent melodic wanderings we hear F#s, Bs, C#s, As, Gs, and Es, and then the melody concludes on a D, our brains go, "Ah, D major!" If from that same D first note, in the subsequent melodic wanderings we hear Fs and Cs that are natural and Bs that are flat, then comes to rest on D, our brains recognize that as D minor. If from that D, we then pass through natural Fs, Cs and Bs, then back to D, our brains recognize that D dorian. And so on.
This is slightly oversimplified. The "at rest" initial note doesn't include pick-ups; it's really the first strong beat note. And you can manage to assert a mode without starting on the tonic if you start on the dominant or the third (i.e. in the tonic chord), but that can be tricky. You can also manage to assert a mode without ending on the tonic, in the same way, but I think you can't do both at the same time, because then the ear has no correct landmarks and will come to some other conclusion.
(I speak as someone who sings a lot of inner parts, where I often sing harmony lines that both do not start and end on tonic notes, and I promise you, when sung independently, those lines often sound like they're in a completely different mode. For instance, there's been many a piece in D minor or D dorian, an inner harmony part of which did a fine impression of F major. I discovered early that trying to learn my part independently of the ensemble would, if I didn't account for what everyone else would be doing musically when we put it together, be profoundly disorienting in context, because I'd be thinking, "The tonic is F and the mode is major! When we get to the end of this phrase it should be a major chord of which I will be singing the tonic!" and we'd get to the end of the phrase and "SOMETHING IS WRONG. WHY IS THIS CHORD MINOR? DID I SCREW UP? DID I MISS A REPEAT SOMEWHERE? AM I OFF PITCH?")
This is why it's common for arsis/thesis phrase pairs to consist of, first, a phrase that starts on the tonic and ends on something other than the tonic, and then second, a phrase that starts on something other than the tonic, and resolves to the tonic.
To expand on Dom's excellent answer, when you're in D dorian, it will sound quite minor, except that for the 4th chord, instead of the expected Gm, you'll use G maj.
On the mixolydian, G, it'll sound major, but a bit bluesy because of the flat 7.The 5th chord won't sound too pushy, though, as it's Dm rather than D maj. As above, you'll use all the same notes as C maj., but the home will need to be G rather than C.
When you are in F lydian, it'll sound like there's a #4 (which could be construed as a flat 5, as in blues) a sound that Vai uses.Obviously, centring will be around F.