(I am referring mostly to the common practice period.) Before the advent of well temperament in the 1700s, moving very far from C was not done often because of the mean temperament causing out of tune accidentals. However, disregarding composition exercises such as Bach's Well-Tempered Clavier, why did it take so long for distant keys to become common? Was it because of the relative difficulty of playing in them? Dvorak and the middle romanticists seem to be the first ones to use them regularly, although I don't know every single romantic piece so I might be wrong. But is there a reason other than the relative difficulty that distant keys took so long to become common place after the advent of well temperament?
Apart from the practical issues of reading these key signatures (especially with transposing instruments), Matthew Bribitzer-Stull offers some possible explanations in his article "The A♭–C–E Complex: The Origin and Function of Chromatic Major Third Collections in Nineteenth-Century Music." (It's an award-winning article; if you're interested in the topic, I really recommend reading it!)
While close approximations of equal temperament in Western Europe were used for fretted instruments as early as the sixteenth century, true equal temperament on keyboard instruments was not universally accepted until 1917. Theorists and other musicians up through the nineteenth century espoused the virtues of equal temperament even though keyboard instruments of that century were almost universally tuned according to the principles of well temperament (170–2)
So one explanation is that, although equal temperament was well-known, it simply wasn't yet used enough to actually compose in keys with lots of accidentals.
While there were more tonally distant keys than A♭ and E (speaking in terms of C-centricity), these two keys often marked the outer limits of acceptable intonation on unequally tempered instruments . . . these keys were often invested with rich associations and served as tonal settings for composers' most profound musical utterances—a habit that persisted even during the theoretical hegemony of equal temperament.
In other words, keys with up to and including four accidentals had dramatic meanings associated with them. It's possible composers didn't fully utilize the remaining keys because they lacked these dramatic associations. Anyone interested in these associations may want to check out Rita Steblin's A History of Key Characteristics in the Eighteenth and Early Nineteenth Centuries.
A lot of the movement into keys with more sharps and flats had to do with the development of wind instruments. French Horns, for example, did not always have valves, and were limited to certain notes. In the mid-1700s, slides were developed, and in the mid-1800s, valves. This gave it much more versatility. More keys were given to the clarinet which also allowed it to play in more keys. Etc.
The changes in music were not just related to what keys composers wrote in, but also in what instruments were chosen for use in the orchestra. Keyboard and string instruments could play in more keys earlier on, but "full" orchestra music and music written for wind instruments was limited by what the instruments could play.