Assuming your neck and frets are ok and the E note on the second fret of the D string doesn’t buzz on its own there are a few options you can try. It sounds like you might have pretty wide fingertips that force your fingers into a diagonal angle when stacked.
You mentioned using your middle finger. The most typical way of playing an A chord is with the 1st finger on the D string, 1,2,3 on D,G,B strings respectively. If you are using 2,3,4 try 1,2,3, the conventional way.
The other option is to finger it low to high 2 1 3 as if you were playing an Amaj7 but with all fingers in the second fret. This places the 2nd finger right up against the D string fret.
Taller frets might allow you to put more down force on the string to make contact with the fret but of course they would all have to be taller, which means a re-fret job or a different guitar. Also you shouldn’t have to have to squeeze excessively to get the notes in a chord to speak properly. Lighter gauge strings might be an alternative solution.
I’m pretty sure at least one of these alternate fingerings will work for you.