TL;DR
One should go for speed and difficulty simultaneously, where
- difficulty = only at the level where one can make music; and
- speed = only at the speed one can handle the difficulty.
Three axioms
(1) Speed comes from experience.
(2) Difficulty is the presence of lots of simple things in close proximity.
(3) Simple things are easiest to learn in a simple context.
Two corrolaries
(4) To read difficult things, one needs to read multiple simple things quickly.
(5) Experience with simple things in simple contexts allows one to read them quickly.
Thus, an effective formula:
Play music that can be easily and understood at sight. Choose practice material from the level at which you can easily read all of the music — notes, rhythm, dynamics, articulation, and any other instructions in the score (such as pedaling, for piano) — so that one is reading music, not notes. Sight-reading practice only works if you understand (i.e., can make music out of) what you're reading.
I highly recommend getting "book 1" of some teaching method series — or better yet, the "pre book 1" book — and read through it — making every exercise as musical as you can. A person who can make music out of a "song" with just quarter notes and one or two pitches, can make music with compositions where the musicality is "built in" by the composer.
Facilitated and reinforced...
"Sight-read" things multiple times — even to the point of memorization. This reinforces the connections between what one sees, how one executes, and what one hears. In this way, one is far better prepared when those things occur in a new, more difficult/complex context. They have become "easy" to read, so that one's attention can be focused on other aspects of the complexity.
Nothing makes sight-reading easier than pattern recognition. Study music theory, and practice scales, arpeggios, and chords. Pattern recognition is at the heart of these studies. It's far easier to read complex scale patterns if one recognizes the scales as opposed to reading each note separately. Similarly with arpeggios and broken chords, and so on.