First of all, as a general rule, it's a good idea to minimize the amount of looking at the keyboard while sight reading, but you have to realise that there is no taboo on looking at the keyboard. All pianists, even while playing from a score, have peeks at the keyboard, but they know how to do this without losing their place on the score.
This question elaborates on that:
Is there any evidence that suggest not looking at your hands increases ability to sight read on the piano?
Having said that, having a good idea of the topography of the keyboard and being able to blindly find your way is an essential skill for being able to sight read. The secret for this is that good sight readers feel their way on the keyboard.
There are two principles to this:
- Good sight readers use their proprioception, their sense of space. They have a generally good sense of how far they have to reach to get to the keys they need. It's essential for this to work that you sit always at the same place with your belly button in line with middle C.
- Good sight readers use the protruding black keys as signposts. They only need a (very) slight brushing with the black keys and, together with their general sense of space, they know where they are.
So you have to practice that.
You begin by finding the groups of two and three black keys with your eyes closed. Holding your hands in your lap and finding them in one smooth movement is what builds your proprioception of the keyboard. You have to be able to do this with your left and right hand.
When you can do this without hesitation you build on that. First, try finding the group of black keys in the neighbourhood of the note you need and then playing the note, all by feel. Repeat this until you can do it automatically. The longer you do this the less time it requires and the less you have to feel from the keyboard to know where you are.
And then you practice it hands together.
Howard Richman wrote a book 'Super Sight-Reading Secrets' that elaborates this method. It's very short and it has the title of a bad self-help book, but don't be mistaken--it's actually quite sensible. I think it misses a lot of aspects to be a real guide to learning sight reading but for what you are asking here it's a really good method.
One more thing: this takes a lot of time. So be patient!