I consider myself an intermediate player and I have trouble with this, I think all guitarists do.
First -- you must practice regular. The ease of switching comes with a LOT of work. In time you develop an instinctive muscle memory and you don't have to think about it -- just like learning writing by hand.
Practice going from the E chord to the A chord.
Watch how your fingers move to do this. Practice each finger in the movement separately, then 2 together, then all three. (I assume you are playing the E chord with 3 fingers and the A chord with 3 fingers). Do this whenever you feel frustrated with the whole chords and keep at it. Try to keep the whole chord clean, all notes sounding --but don't fuss about that too much.
Otherwise just keep going back and forth from E to A.Just getting down these two chords is it for this exercise.
You don't want to be thinking : put your finger here, here and here. At first you do that but in time it becomes automatic, like writing. Until that point just work and stay at it.
BTW these two chords are very common in rock music. Try writing your own "songs" to these two chords "Play an E, Play an E again, then I go to A". That will help you get the rhythm side as well.
You should then start to add other chords as your skill improves.
It sounds like you have had some lessons. You are at the point where your muscle memory is not fully developed and your brain is thinking as opposed to your fingers.
I think the thing for you is to focus on steady practice, day by day. After ten days of ten minutes a day on this simple exercise I think you would see a big difference.