Staff music, and indeed pretty much any system of musical or rhythmic notation, is just a means to an end. You are free to invent any system you want, borrowing symbols from any discipline or making them up as you go. As long as they have meaning to you, that's the most important thing.
Since you don't want sheet music (understandable, given that sheet music, even a lead chart, can require several pages per song to convey all the information it provides), you will have to do at least some of this invention on your own; other than standard notation and drum tablature, there isn't any other standardized percussion notation, like there is for certain regions/genres for melodic lines and chord structures (like chord/lyric charts, Nashville Number System, etc etc).
Here's one system a drummer I worked with uses: For well-known rock/blues patterns he simply writes the name of the pattern shorthand, or builds it from the kick up in a way that makes sense. Example: "K1&3-S24-HC8" is "Kick on 1 and 3 with a third hit on the 'and' before 3, snare on 2 and 4, closed-hat 'clicks' on eighth notes"; pretty much your standard rock pattern. "FOF-S24-HC8O68" is "Four-on the floor kick, snare on 2 and 4, closed high-hat clicks on 8th notes, open the hat on the 6th and 8th clicks of the bar".
For fills, turns and complex lines:
Draw two lines across the page. Top line is your "hand" line. Bottom line is your "foot" line. You may choose to add an extra line or two to differentiate hand and foot; this can make it easier to read but requires more space and can be harder to make changes.
Divide this "staff" into as many bars as you need for a pattern.
Write the beats of each bar above the "staff", leaving space in between the beats and staff for additional marks. You may subdivide as necessary.
Now, pick or invent some symbols that are easy to draw, easy to tell apart, and have meaning to you, and assign each symbol to a particular instrument or manipulation possible with the kit. Things to think about:
- Most drums/cymbals have pretty much one way to hit them and so will only need one symbol. You can make special notations about exactly how to hit either with a variation of the symbol or by making notes above the line.
- For toms, consider using differentiable but related symbols common to that "family". Similar for cymbals.
- For the snare, you can either notate how to hit it (rim tap, off-center hit, center hit, rimshot) above the hits, or use different symbols.
Now, while listening, simply mark down the rhythms of each instrument by writing that symbol on the appropriate line, on the appropriate beat or subdivision. You can build it up in layers or, if it's easy to pick out, just write it all down left to right. If many things happen on one beat or subdivision, you can write them vertically above and below the line
If the rhythm isn't obvious, or a lot of hits come together, you can notate the rhythm in pseudo-standard notation above the line, or you can make additional notes of any type you choose.
In patterns where which hand (or foot) makes which hit are important, they can be notated easily with patterns of L and R written over each hit as necessary.
Rolls get a wavy line after the initial "hit"; you can increase or decrease the height of the line to indicate crescendo and decrescendo. If the roll's hits must be at specified intervals (16ths, triplet 16ths, 32nds) that can be written above.
Keep in mind that all of this notation is just a way to quickly take down a non-standard line or pattern, to remember said pattern for later practice. Most of these will be one or two bars; the basic pattern for a section should either be easy to remember or should be written in some acceptable shorthand, and variations of starting and stopping a particular layered rhythm can just be scribbled in next to the words where you should start it. You should never be trying to "sight-read" based on this, nor expect anyone else to do so.