I've noticed that my mind always wants to make shortcuts when reading music. I rarely ever actually think about, or consciously process, what pitch the notes are in the sheet music.
This is completely normal. It's how minds work. When you see three apples sitting on a table you're more likely to process that as one triangle with an apple at each vertex. Similarly, when you see a D major triad on a treble staff you'll process it as a certain hand position with fingers 1, 3, and 5 playing simultaneously, whereas if you see an ascending scale comprising the notes D, E, F♯, G, and A, you'll process it as the same hand position with the fingers 1 through 5 playing in succession.
The main reason I'm adding this answer to the good answers that are already present is to note that it's entirely possible to be able to do this without knowing how the music sounds. What you describe is a mental link that is developing between your eyes and your hands. You will benefit in the long run if you also develop the link between your eyes and your ears.
Without thinking I can move from C to E by unconsciously playing with my thumb and then middle finger.
It sounds like you've developed an association between the first ledger line below the treble staff and the note immediately to the left of the group of two black notes on the keyboard, and between the bottom line of the treble staff and the note immediately to the right of that group. That's a fairly cumbersome way of identifying these notes! It is much quicker to call them C and E.
You don't necessarily need to be able to name them quickly, but you will benefit if you can. Don't stress out about it, but do practice. Play some slow scales or other simple tunes and sing the note names as you do. If the notes are out of your singing range, sing them in a different octave. Try speaking the note names instead of singing. Try singing without the names. Try singing solfege syllables. If the tune has words, try singing them. If it doesn't, make some up. All of that will help you become a better musician.