I would like to learn about the art of establishing a mode. For example, how to make A minor sound like A minor instead of C major.
The basic idea on how to establish a mode is make the tonal center of the progression and melody center the tonic chord of the modal scale.
For example this progression would signify C major.
C - F - G - C
The progression stats and ends with C and the melody to accompany this would use the notes of the C major scale that would start and end with a note in the C major chord.
This same progression in A natural minor would be:
Am - Dm - Em - Am
The progression stats and ends with Am and the melody to accompany this would use the notes of the A natural minor (C major) scale that would start and end with a note in the A minor chord.
However, most composers use the harmonic minor scale for minor keys by raising the 7th scale degree to a major 7th which would change the progression to.
Am - Dm - E - Am
It is still derived from C major/ A natural minor, but the scale makes A feel more like the center tone.
This is a very big topic, but this is the bare bones idea behind it. If you need more detail please ask.
To expand on Dom's excellent answer, when you're in D dorian, it will sound quite minor, except that for the 4th chord, instead of the expected Gm, you'll use G maj.
On the mixolydian, G, it'll sound major, but a bit bluesy because of the flat 7.The 5th chord won't sound too pushy, though, as it's Dm rather than D maj. As above, you'll use all the same notes as C maj., but the home will need to be G rather than C.
When you are in F lydian, it'll sound like there's a #4 (which could be construed as a flat 5, as in blues) a sound that Vai uses.Obviously, centring will be around F.