I suspect you are feeding the output of your looper into your effects pedal. With more layers, you get more volume, and that causes the distortion.
Make it guitar -> effects -> looper -> headphones.
Check with a clean setting on the effects unit, and set the levels so you get a clean sound when you strum hard. It's most important that the effects unit's output is at the same level expected by the looper's input.
With this setup, you have the advantage that you can apply different effects to each layer: e.g. loop a clean chord sequence, play a distorted solo over it.
To play this at volume, instead of through headphones, use a clean channel on your amp, or even better use a PA amp rather than a guitar amp: your amp modelling effects unit is giving the sound character.
Edit: All of this assumes your looper works at line level or headphone volume. If your looper is designed to work at instrument level, which is much quieter than line level, that has two repercussions:
- You'll need to turn the output level of the multi-fx unit down to instrument level
- The output of the looper will be quiet. You'll need to amplify it. If you don't have an amp with a headphone output, you could spend $20 on a headphone amp. This will also deal with splitting the mono signal into both stereo headphone channels.