There are a couple of things you should consider when sight reading:
As with any skill you need to start very slowly in order to build the physical and neurological mechanisms to operate faster and more competently. And when I say slow, I mean 30 or 20 BPM. If you have never payed with a click I suggest you begin to learn and as you get used to it, it will start act as a guiding hand, sharpening your timing and sense of rhythm. This leads me to my next point.
You know all the notes on the staff and that's great but how is your knowledge of rests? Knowing when not to play is as important as knowing when to play. Learning typical rhythmical groupings is also important because then you can view these groupings as single entities rather than as a collection of individual notes. I suggest you youtube drumming tutorials that have sheet music accompanying their instructions. You'll get a better sense of what I mean then. If you develop strong sense of timing and rhythm, irrespective of the key, melody etc.., and begin to see note placement patterns at first glance, you'll have moved a good way down the road of becoming a competent sight reader.
When you say you play it carefully once through and then memorise it, are you really playing it through or do you make a ton of mistakes on the first attempt and then begin studying the piece, foregoing any other sight reading? Improving at sight reading should be a long term goal with consistent observable progress. The little aphorism of 'little and often' should become your mantra - do a little bit of sight reading often. How often? Well, if you do twenty minutes of practise everyday devote 2-5mins to sight reading. Short focused blast of deliberate and intense concentration that will find satisfying and beneficial to your practise.
How little? Try one to four bars of music and then move on. Make mental notes of your weaknesses (you missed the rest, I can't read those ledgers lines, You've never seen so many accidentals!). Wash rinse repeat daily or whenever you practise ;). Consistency is King.
The torture of sight reading through 32 bars when trying to develop the skill should only be inflicted on you by your teacher.
- A little lesson plan for one bar of music to sight read in 4/4 at 120BPM:
You have 30 seconds to before you need to play:
(0-10 seconds): identify time signature, key and tempo. Mentally play the main scale or mock-play it (think of Adrien Brody in the Pianist). If there are awkward notes (accidentals and sharps for example) rest your fingers on them for a moment and internalize the fact that you will have to play them at some stage.
(10-20 seconds): hum the rhythm (or melody) of the piece while mock-playing what appears to be the supporting notes.
(20-25 seconds): Mock play the first phrase/chords/notes while looking ahead to the next.
(25-30 seconds): Steady and set yourself.
(31 seconds): Play...
On slow practise look at this: https://bulletproofmusician.com/is-slow-practice-really-necessary/
Free videos (my channel):
A free zip with 3 4-bar exercises with click and no click from 20BPM to 180BPM in 20BPM increments on mp3, pdf and with hands together and separated: https://www.patreon.com/SightReadingSounds