Well yes, but it won't make any sense.
All of the modes you've mentioned are diatonic to C major. That means that regardless of the tonal center, you're only playing natural notes that also exist in the C major scale.
When you're playing music in a given key, you are tonicizing one note over another. Meaning, the "Do", or tonic note exists in your melodic line the same as it would in your harmony (as a I chord). When you play E phrygian, you are tonicizing E. The notes are the same as those in C major, but if the rest of the music is tonicizing C, then playing a line tonicizing E is not going to make a whole lot of sense!
You should think about modes not as different starting points on a scale, but as different alterations to a scale. If you're playing a piece of music in A minor, you might play an A phrygian solo over a pedal tone, but the notes are not going to be diatonic to C major or A minor! You will alter the 2nd of A minor, by flatting it. The notes will be diatonic to F major starting on A, but realize that F doesn't really have anything to do with the music you are playing--that's just one way to figure out what notes are in a mode.
The other way of figuring out which notes are in a mode is by learning the scale alterations. This will clarify other things for you as well. The simplest examples are lydian, where the 4th note of the major scale is raised; mixolydian, where the 7th note of the major scale is lowered, or dorian, where the 3rd and the 7th of the major scale are flatted. You already know Aeolian/natural minor, where the 3rd, 6th, and 7th notes of the major scale are flatted.
This question/answer will be helpful to you: https://music.stackexchange.com/a/6890/133
