This depends on the context. In some styles (salsa dance music is one example), it's hard to catch the "2" beat in many pieces. Various styles distinguish breaking on 1 or on 2, but I have watched (supposed) experts miss as often as not. I do not do much better.
I have over the years noticed many dancers miss the down beat in ballroom (even waltz). Sometimes the arrangements lack a strong rhythmic feel.
However, as pointed out above, there is usually some other part of the music that carries the implication of the downbeat. Often this is the bassline, sometimes is the melody, and sometimes comes from other percussion instruments. Note that in the salsa I mentioned, the bass from beat 4 is tied to the bass on beat 1 of the next measure (with the second measure's harmony) and this is followed by the bass on the second part of beat 2, so there's no strong "2" beat for the dancers from the rhythm section.