Practice a variety of two part rhythms.
- use a metronome
- count out load while practicing (video on the psychology of counting out loud)
- use a good counting system (rhythm syllables like Kodaly) to develop awareness of the beat versus subdivisions
- pay attention to both composite rhythms and the individual rhtyhms, make sure the accents in each individual rhythm is correct
There are a lot of rhythm practice books available, or you could make your own exercises.
I didn't like the books I was finding - the rhythms seemed too irregular - so I made my own cards for simple & compound meters with also syncopation and polyrhythms. I made sure the counting syllables were clear...

...if you can't find a published resource for your needs, make your own.
A general though about sight reading:
I mostly play classical/romantic music like Chopin, Rachmaninoff, Debussy, Liszt, and Scriabin...
I've listened to it many times while reading the music so I know exactly what it should sound like
It sounds like you are reading that 19th century repertoire, but not sight-reading it. That's fine, but it's really two different kinds of reading.
I don't know that you have only difficult material from those composers, but I suspect that may be the case.
You want to make sure you are sight reading the right level of music. With that 19th century repertoire a lot was not meant for sight reading but virtuosic display.
- for 19th century style try finding simpler, shorter stuff like Schumann's Album for the Young, Schubert's keyboard dances, Burgmuller op.100, piano accompaniment parts from song albums might offer good material too. But dances sets and collections for young/beginners are a good place to start looking.
- 18th century classical style, Czerny is a great source. Lot of people dump on Czerny, but look at the Recreations, Progressive Studies, etc they offer lots of charming, modest pieces. Also typical keyboard instruction manuals from this time follow a format of music fundamentals and basic fingerings followed by a collection of fairly easy pieces. Clementi's Art of Piano Forte Playing is a good example. Also, get L. Mozart's Nannerl Notebook and W.A. Mozart's London Sketchbook. Those are filled with tons of short pieces.
- 18th century baroque style, get the Bach notebooks for Wilhelm Friedman and Anna Magdalena. Also, the Bach 371 Harmonized Chorales for keyboard are sight readable. There are endless dance suites and dance sets in this period. Telemann's 36 Keyboard Fantasias, Handel's sonatinas, and other small scale early sonata forms could be good sight reading material too.
A huge amount of the material above is available at http://imslp.org.
You probably can see there is a theme of dance forms. The important thing about that genre is much of it was meant for amateur players and home entertainment. Most of it wasn't virtuosic concert material. I think a lot of it was meant to be read and played without a huge amount of preparation. That kind of stuff should make good sight reading material.
Volume is important. Get enough material that you can't memorize it.
Simpler music isn't really much fun, that's why. And I have succeeded at learning many such pieces despite my limitations
This may be an impasse. If you enjoy the technical challenge of difficult music, then sight reading doesn't seem like a type of playing to satisfy you. "To each his own" I suppose. You don't have to sight read if you don't like to do it.
But I think you want to make a clear distinction what sight reading is about. I'm curious to see if anyone disagrees with me, but people don't sight read stuff like Debussy's Preludes or Liszt's Etudes. Sight reading is for smaller scale stuff of modest difficulty not virtuosic display.