I'll agree totally that the manual is not very informative, but my best guess (without the oppertunity to try it out on the device) is that it as to do with the messy world of trigger signals. The device simply needs to have some heuristics about what is a real stick hit, and what is just noise that comes from a particular stick's rebound, the vibrations and shock reflections from the mounting and stand and all kinds of unwanted signals.
This means that one setting of these parameters might work for you at home in a rehearsal situation, but other setting is needed on stage with a full band and PA. Also, one type of stick might need one setting, another stick need a different setting..
My best guess about the difference between 'mask' and 'reject' is that 'mask' is the initial time from the very first trigger signal (when the signal goes from 'low' to 'high'), and ignoring the very first repercussions of the initial hit, before considering another hit. This might be the most important setting by far.
I would then guess the 'reject' time is that when the signal went 'low' again (after minimum 'mask' time), it will reject further triggers ('high' signals) for the duration of 'reject' time. This could come in handy if your stick or mounting or stand or stage or stage dancers (!) are particularly bouncy.
If I am correct, you should experience that the two time settings sum up, fx if you set them both to the max value, I would expect a total of 3 seconds from the first hit and until the next hit will trigger a sound. This might behave differently depending whether you play a roll or just a repeated hit.
In practice, just set both settings as low as possible so you can play as fast as you like/need while avoiding unwanted double triggers, but beware that your surroundings might impact the settings.