Before the time when sheet music was popular people were good at playing keyboard instruments. I am referring to the time when printing books was expensive. How did the students remember what to practise between lessons without any sheet music? What did they have back then that we have lost today? I just want to know what happened to our memorization. I am especially interested how one can learn from those people and apply it to our own lives.
Besides the copyist mentioned in the other answer, students could simply write on their own! No need to have someone copy it for them. Also it was not uncommon for teachers to write stuff down for their students. I have had teachers that give handwritten scores (or copies of them) and notes to their students.
This still is a way that is pretty cheap since it doesn't require hiring someone else, so if you had money for a teacher and paper you could easily study at home.
Students could memorize the stuff (good for memory practice) or copy out the examples. Copying out "old masters" is a very good way to learn. One picks up patterns and see stuff from a different point of view for that of just playing the piece (of course, it's not so easy to play a symphony as a single person.)
I have found this lots of help. Currently I'm copying (in Finale) the first movement of Haydn's "Emperor Quartet." I've already discovered that Haydn's rhythmic structures are much more complex that most popular music or lots of other classical stuff. It's a good way to learn something during "composers block" or when bored.
The question is fairly vague in asking for time-period, and music style. As such, I feel the best way to answer is to give you an impression of the various ways music was shared historically.
Regarding Written Forms
Firstly, written notation of music goes back multiple thousands of years - with examples of "something musically related" being found on a Sumerian tablet (~2000B.C.), and the oldest example of a full-melody being inscribed on an epitaph from ~100AD
Closer to what we'd consider "written music"; during the middle ages of western society (5th-15th century), and continuing somewhat after the proliferation of the printing press (15th century) - the music is written in manuscripts. Many of the examples of this are chants (unsurprising as the Monks/Nuns were one of the few literate groups). Due to the literacy rate, creating a copy of music is something that would be done by a "professional" copyist, hired by the Abbess (superior nun in charge of an Abbey).
After this, is a period where printing music was possible but expensive. As such, sheet music was used - but only affordable by the aristocracy.
Beyond this, after the proliferation of the western printing press, developments were made in printing music, and for the majority of the 16th and 17th centuries - allowed for a cheap printing process that opened up written music to a much wider audience.
While the above mentions the various ways written music was shared; it should not be taken as a given that the majority of people could read it, or copy it.
It is estimated that in the early 16th century; we're looking at approximately 10% of the male population of England being literate enough to "sign their name at marriage". While even as we reach Bach's death (1750) we're only just hitting a rate of 50%.
So how was it actually taught
We now have a clearer image of the situation:
For some small communities, of literate individuals, music was written - and shared by copying. The music could then be learned by this literate community and performed. In many of these communities (e.g. Monks), study was the main part of life - and so learning to perform music would be part of daily life.
Later, as writing developed, the aristocracy became more able to compose complex works and share them by having them manually copied. This opened up the ability for composition and performance to be studied by private individuals, and for apprentices to be tutored by their masters.
Finally, printing evolved to the point where complex pieces could be reproduced and shared easily (within the middle class) - allowing those who were educated, to learn in a style fairly similar to what you imagine in modern day (lessons and tutoring).
However, there is clearly an abundance of music that existed outside of these small enclaves and aristocracy. Looking at how this was shared, should give a good idea of the other side.
For this, it's worth considering two types of music that were shared. Firstly, folk music and secondly the music shared by Minstrels (the people you would imagine when you think of "bards").
As with many folk traditions, tales and practices. Folk music was an oral tradition - passed down through families and communities via practice.
In the same way as many scouting songs, simple hymns and nursery rhymes are learned in modern days - folk music would be learned by hearing it and taking part. This of course includes instruments, where individuals would learn by being shown how to play that community's songs.
Of course, it is also worth remembering that while using sheet music to learn may seem important in modern times - the complexity of most folk music isn't that high, and given that there were less things to do outside work; there was more time to spend passing on your skills each day. As such, it is far from a necessity to have written music, in order to pass it on.
This then leaves the slightly more organized side of amateur music; Minstrels.
While Minstrels originally played in courts for nobility, unlike the religious enclaves they often were not literate and relied on memorization and improvisation.
At this point, Minstrels were generally just "entertainers", and being able to construct songs and poetry was just another "talent" - similar to those who could juggle, or do acrobatics.
For Minstrels, it's not clear how they would originally learn their music - most likely from the same roots as those performing folk music. However, in the 14th century, guilds formed that allowed Minstrels to pass on their art from master to apprentice, as a profession (again, not necessarily via written medium).
During the 15th century, printing developments (noted above) then also helped to close the gap between the clerical (who were literate) and minstrels began to break down - as more Minstrels were able to pass on their music via written form.
The decline of Minstrels around this point, is probably a fair sign that this is where the majority of music began transitioning from oral tradition & improvisation, to a craft that could be taught in a structured way with sheet music.
Obviously, this is a deep topic with a lot of different areas involved. There are huge areas of music that have been missed here that don't fit any of the groups above (e.g. Troubadours, and many, many others).
However, I'm hoping this answer will have given some idea of where you might want to look further - and importantly, give an idea of how varied the situation was between different communities, social classes and times.
And to add salt; this answer has almost entirely focused on just a single culture
But, to answer the question most directly:
What did they have back then that we have lost today?
Time, and lots of it.
Music for the average person was far simpler than what the average player handles today. Importantly, the amount of time learning could be the majority of some people's lives - while today, it would be surprising to spend more than a couple of hours in the company of a "master" each week.
Beyond this, if you're looking for something to take away from this; it'd be, that for the majority of people, music was a way to bring them closer to the community, and part of daily life. Being similarly open to improvisation, and playing simple music with friends may be something to consider. But that feels overreaching, and the simple answer is "they were just humans, with more time, and less things to do".
I am referring to the time when printing books was expensive.
Before people had books, they had clay tablets, papyrus and stone carving.
(You can actually listen to someone from our time playing that score if you follow the link).
Writing into those things is way harder than writing on paper, and paper has been around for longer than books. Point being that no matter when you lived, if you could write you had no excuses.
You also have to remember that anyone rich enough to own a keyboard would be able to afford books. Tutorials and practice pieces made up a large proportion of early publishing (c. last 400 years).