Well, to give a boring answer to how it can be proved – since there are only finitely many subsets of the 12 tones (namely, 4096), you can just list them all and check exhaustively. But we can be a bit more efficient than that.
There's not really such a thing as limited transposition. What characterises these scales is that some of the transpositions will be enharmonically equivalent to the original. In other words, there's a transpositional symmetry in the scale. (Mathematically, you could say the group of transpositions contains nontrivial equivalences classes with respect to action on those scales.)
In effect, this boils down to these scales splitting up into several equal parts. Because 12 has the integer factors 2, 3, 4 and 6, that means you have the following options, omitting scales that are equivalent by global transposition and omitting the scales that have additional finer-grained symmetry:
- Size-2 blocks: s0=□□, s1=■□, s2=■■
- Size-3 blocks:
□□□≃s0, s3=■□□, s4=■■□, ■■■=s2
- Size-4 blocks:
□□□□≃s0, s5=■□□□, s6=■■□□, ■□■□=s1, s7=■■■□, ■■■■=s2
- Size-6 blocks:
□□□□□□≃s0, s8=■□□□□□, s9=■■□□□□, s10=■□■□□□, ■□□■□□=s3, s11=■■■□□□, s12=■■□■□□, s13=■■□□■□, ■□■□■□≃s1, s14=■■■■□□, s15=■■■□■□, ■■□■■□≃s4, s16=■■■■■□, ■■■■■■≃s1
Putting it back together again, that means we have these 17 note-sets which have a transpositional symmetry:
□□□□□□□□□□□□, ■□□□□□■□□□□□, ■■□□□□■■□□□□, ■□□□■□□□■□□□, ■□■□□□■□■□□□, ■■■□□□■■■□□□, ■□□■□□■□□■□□, ■■□■□□■■□■□□, ■■□□■■□□■■□□, ■□■■□□■□■■□□, ■■■■□□■■■■□□, ■□■□■□■□■□■□, ■■■□■□■■■□■□, ■■□■■□■■□■■□, ■■■□■■■□■■■□, ■■■■■□■■■■■□, ■■■■■■■■■■■■
From these we can manually filter out those that Messiaen considers “modes”, as those that have no steps larger than two semitones and aren't simply the full chromatic scale:
■□■□■□■□■□■□, ■■■□■□■■■□■□, ■■□■■□■■□■■□, ■■■□■■■□■■■□, ■■■■■□■■■■■□
For reasons that are a bit mysterious to me, he also includes ■■■■□□■■■■□□ and ■■■□□□■■■□□□, though these have bigger steps in them.