My choir has a consistent tendency -- absent a metrenome or vigorous conducting -- to slow down for quieter dynamics (and then never regain the original pace). We have several individuals with good senses of pace; this is a collective problem. Why does this happen and how can we fix it? We don't want the director waving her arms that much on stage; we want to fix the root problem and perform with minimal conducting.
@Matthew Read gave some good suggestions, to which I'll add:
I can't speak to the psychological reasons or addressing them, but there are a couple things you can try that basically apply to all types of music.
1) Have the weaker members listen to and follow the stronger ones. Ensure they can hear them, of course; don't put them on opposite sides of the stage. The mediocre members will probably do well enough if the bad influence is removed.
2) The director doesn't need to flail in order to indicate the tempo. Small hand movements should be enough.
3) Not everyone can keep a consistent beat, but everyone can follow a beat. If choir members refuse to do the latter by following others or the director, they should be dismissed. (I'm assuming your choir is "serious". If not, it may be something you just have to deal with.)