I think the most important point when holding auditions is knowing what your own goals are for the band. It sounds as though you are looking for more than one player, which makes me think you are starting a new band from scratch, or maybe picking up the pieces from the last one.
If this is the case, what is the band going to be doing? Do you have gigs lined up, or do you plan on learning some tunes and then getting gigs? Or is it more a just for fun thing?
Whatever your plans, you probably should make sure that anyone you invite to audition has the same goals as you, and this, of course, means you must know what those goals are and be able to clearly state them. A lot of bands fizzle because no one really knows what the goals are, or even if there are any. Oh, sure, they jabber about maybe someday getting this gig or that one, but no one ever really does anything about it. ;)
If someone you audition wants to play in a just-for-fun party band, and you want to go gig ASAP, chances are you wouldn't get along for very long, because you would not be working toward the same goals. Or if you get an experienced player who just wants to get out and play gigs, they probably aren't going to be interested in coming to rehearsals for several weeks while everyone else learns their parts.
As to the other things, I think asking someone to learn a piece and then play it for you by themselves is definitely not the way to relax them and get the best out of them. :) In most cases, their ensemble playing is going to be far more important than their solo playing, anyway.
For the auditions themselves, I would sort of go with a structured but unstructured jam. Structured in that you keep goofing off and noodling around to no purpose to a minimum, but unstructured in that you don't need a setlist or anything like that, just pick some standard tunes everyone knows and play them; see how you sound. If it becomes obvious after a couple of tunes that it is not going to work out, don't be afraid to tell them so, politely. Thank them for coming, tell them goodbye. Don't feel obligated to play with them for hours; they are there to audition, and not as your guest.
The other answers here bring up a lot of good points that definitely should be considered, but in my experience having the same goals is most important and most often overlooked.