I would actually say that the opposite is true, namely, that study of music theory is what matters, and that even if you don't practice sight reading (though you probably should), it's the study of theory that will make the biggest improvement in your sight reading compared to anything else.
Sight reading is a tricky thing to do, there is quite a lot of information, and unlike the arabic letters of the alphabet, there is very little in the way of unique characteristic that helps to differentiate between notes of a different pitch.
Rhythm is actually far easier to read because there is far more that characteristically distinguishes each individual note, namely the stems, flags and fill of the circle.
The primary functions one must understand when reading music relate to theory, namely the key signature in which the piece is written and intended. When one observes which sharps and flats the key signature has, this illuminates almost immediately which notes are viable and which notes are not. In other words, the key signature eliminates a lot of possibilities and potential confusion because, to sound correct, there is only now a limited subset of notes to choose from, e.g. only the notes in an Eb scale perhaps, which means that you know several black notes and several white notes are out of the question.
This then means that one may focus less on the absolute note names and positions of each note on the staff, and may then refer to the relative distances between successive notes, combined with the scale which is being used, to determine which notes the music is probably referring to.
I for one, personally, have been almost always horrible at sight reading, but over the years of e.g. reading chord charts on both guitar and piano, and over the years of haphazardly attempting to solo, in an improvisational manner over chord changes, I have found that my sight reading ability has actually dramatically increased with pretty much no actual practice in actually spending time reading standard written musical staff based notation. This is because I have studied each of the 12 keys so in depth, and have learned so many songs, and have practiced my circle of 4ths, 5ths, 3rds, half steps, whole steps etc, that I just have a familiarity with each key and the territory and relationships across which music most often progresses.
This means that when I try to read music, so much of it is so much more immediately obvious to me that I don't need to actually "read" every single individual note, and this ability to read without reading is something most good teachers I've had have confirmed as being the general appropriate way to read music.
So, you really probably should practice reading music in its own right, but realize too, that study of theory and composition, of key and of modulation, are critically important to actually increasing your ability to read music as you practice that skill.