I find that whenever I improvise in E major, I naturally move back and forth between E major and Ab major key areas. It's like E major and Ab major feel especially connected in a way that say Bb major and D major do not feel as strongly connected at all but rather extremely contrasting. Both of those pairs are same # of accidental pairs, but of the 2, the E major and Ab major feel more closely knit than Bb major and D major. Here are those 2 pairs of keys on the circle of fifths:
And as you can see from the thickness of the lines, the E major and Ab major feel more connected to me than Bb major and D major do, despite the same relationship between both pairs of keys(Major chords a major third apart). This progression is an example of what I might use to seamlessly go from E major to Ab major:
E major -> B major -> Eb major -> Ab major E: I V Ab: V I
Of course, it probably wouldn't be quite that simple. Like B major might be followed by BM7(which would make there be 2 common tones between it and Eb major) or there might be some minor chords in between 1 or more of these chords, but these 4 major chords might very well be the basic progression that then gets harmonically embellished. I know that some chromatic mediant relations are stronger than others, and this 2 major chords is probably one of the strongest out there.
But still, why do E major and Ab major seem to be more connected than Bb major and D major? Is there just something about the range from 4 flats to 4 sharps that makes everything there feel connected to everything else, like perhaps the rarity of pieces in these keys, especially on the sharps side?