I think they're usually referring to the style of drummingdrumming - listen to the first minute of Police - Every breath you takePolice - Every breath you take, for instance. The snare drum comes on beats 2 and 4, and you'll find this in lots of pop/rock etc. Reggae is also a clear example of putting rhythm on beats 2 and 4.
You're also right in that most simple chord changes occur at the start of a bar, or exactly half way through, hence the 1 and 3.
So in short, most harmonic changes occur on beats 1 or 3 (or various syncopations around that), but a lot of rock and jazz drumming uses beats 2 and 4. Why? maybe the snare gives a nice call-and-response bouncing off the rhythm of the harmonic instruments...