From a technical point of view, inversions sound different from each other because of the relations of the upper notes in the chord to the overtones (harmonics) of the bass of the chord. The harmonics for C can be seen in the following picture (from Wikipedia):

Looking at the above picture, the first 6 harmonics of a note are:
I0, I1, V1, I2, III2, V2
where Roman numbers are the degrees of the scale and Latin are the octaves.
For A, C# and E the first 6 are:
- A0, A1, E1, A2, C#2, E2
- C#0, C#1, G#1, C#2, E#2, G#2
- E0, E1, B1, E2, G#2, B2
In an A chord, the notes from lowest to highest would be A, C#, E. We can see that the upper notes match the first 6 harmonics of the base. This gives a very "consonantic" sound. Indeed, the (uninverted) major chord is the most "consonantic" of all chords.
In an inverted chord with C# (first inversion - 6/3) as the bass, the G# is a consonance with C# and E, but E# is a consonance with C# and not with E. Since this clash happens only in the 5th harmonic, it is not as noticeable.
In an inverted chord with E (second inversion - 6/4) as the bass, the B is already a dissonance with C# though not with E and the G# we covered above. Here the clash happens already in the 3rd harmonic, it is a bit more noticeable. The second inversion can be considered a dissonance in some cases.
Although we disregarded harmonics of the higher notes and not really defined how much of a dissonance each note is with another, we do have a qualitative assessment and an understanding of the different sounds they give.