It is possible to learn basic rhythms by ear but eventually you will have to learn standard notation to deal with the complex stuff. I have seen this first hand.
There is a Suzuki teacher in the family. Suzuki is a method that emphasizes the aural training from a young age. Trough this all though there comes a stage where the rhythms become too complicated to do by ear and you have to be able to read the scores to play the music correctly.
The family member does the basic aural training with the children and when it comes time for them to learn to read notes the family member does advise the parents to go for theory lessons.
With all this you do get those parents who of course refuse to accept this and don't send their children to theory lessons. I mean what can a Suzuki teacher with thirty years of teaching experience really tell people that have never played music about theory.
This has the effect of coming to a stage where all progress grinds to a spectacular halt. They want to progress trough the Suzuki books but they cannot play the music correctly because they do know what the notes mean. They play what they think they should play instead of what is written.
So yes you can but you are going to struggle eventually. I personally cannot understand some musicians insistence on denying themselves an education. Did these people also stop going to school in the 8th grade because some person made it big without a GED?
These concepts of music notations are not overly complex nor are they beyond the comprehension of the average intellect. You can learn basic notation along with every able bodied person in the world, for some reason some people don't. Not that this makes all that much sense to me. Music Tuition should also be seen as a broader attempt to educate the child