EDIT: Learn what kind of FX loop your amp has. If it is parallel consider the advice below.
If your amp has a parallel FX loop, I would recommend using the delay pedal in there. An amp basically works like this, please excuse the crudeness:
GUITAR -----> PREAMP SPLIT 1 ----------------- (wah/distortion/etc)-----------------> POWERAMP JOINS
PREAMP SPLIT 2 --> FXSEND --> (Delay/Chorus/Phaser/Flanger etc) --> FXRETURN --> POWERAMP
Lots of guitar players prefer to have their delay/chorus/etc unaffected by the pedals they run in the front of their amp (volume/compression/wah/distortion/etc). Routing a guitar rig in this manner allows the pedals in the FX loop to sound more clear and not be obfuscated by the pedals in front of the amp (a separation of concerns, if you will).
A simple example: you wouldn't want to distort delayed signal (guitar--delay--distortion); you want the distorted signal to be delayed (guitar--distortion--delay). A parallel FX loop does just that!
You should see two input jacks on the back of your amp labeled (FX SEND and FX RETURN). If you wanted to use a Delay in this, you would need an extra pair of cables, but you would route it like so:
FXSEND --(instrument cable)--> Delay/Chorus/Etc --(instrument cable)--> FXRETURN
Since a delay pedal doesn't have a litany of knobs, I'd recommend tinkering around until you find that "neat" sound. Knowing how to use a pedal on the fly can help a ton during a gig! (Plus, like you have stated..it is fun!)
This is a great article to pad your understanding: http://www.jamplay.com/articles/5-guides/182-signal-chain-a-look-at-effects-routing-part-1