On my acoustic guitar, whenever I strum my high E string my A string starts vibrating. This only happens with my high E string. When my A string vibrates it creates an echoing sound that won't stop unless I put my finger on the string. It is so annoying but I don't know what is wrong with it.
Sounds like a sympathetic vibration to me, also known as string resonance. From Wikipedia:
I get this on my acoustic guitars as well -- it's a natural part of acoustic instruments. I believe the specific strings/frequencies that vibrate depend on a variety of factors, such as how the top of your guitar is tuned (yes, the top of your guitar is likely "tuned," if it was built by a reasonably good luthier!), the bridge and which string/fret you play.
If you'd like to tone this vibration down a bit and you have room behind your bridge (e.g., for an archtop guitar or a Selmer-style gypsy-jazz guitar), you can wind some cloth in the strings behind the bridge. If you're playing a dreadnought, I'm not sure what you can do (sorry!).
Long story short, this vibration is a natural thing -- part of playing an acoustic instrument. Embrace it!