As you say, in session playing, sets are often not prescribed. Thus when they are notated in books, tunes are listed alphabetically, or by key or by type (jig, reel etc.). Popular sets will vary from region to region (or even from session to session or pub to pub!) so there has been no 'need' to standardise and name sets.
Two exceptions:
first if a tune set is so popular that it has become 'part of the
folklore' then it may well be listed with its 'partners' in that set,
and it could be named after the band/musician that popularised it.
Second, a Ceilidh band or similar might want to list tunes in sets
which accompany specific dances. In both of these cases, the name of
the set is that of the first tune (more often than not), which like
you say might not be the most descriptive way.
When listing tunes, you might take inspiration from the excellent site The Session, where tunes are linked according to whether they appear together on recordings (this is a fairly good approximation for determining how often they are played together in local sessions).
For example: the tune The Blackthorn Stick has 96 listed recordings, and is listed as having been recorded with The Kesh Kig 6 times, amongst other tunes. The implication is that these tunes often go together, but there are many instances when tunes are partnered with others.
Obviously this is a good format for a website, which allows you flexibility in the way you display tunes, but for a book it might be best to use another format. In addition, naming the set isn't so important when you can just access a list of associated tunes and of course it's flexible and allows for 'overlapping tunes' i.e. tunes that appear in multiple sets.
You don't say for what purpose you're transcribing sets, however here are a couple of ideas:
- If it's for dance you could easily put together sets and call them by
the name of the dance(s); this is what bands I have played with have
tended to do.
If it's to record tunes typically played in a pub session (n.b. some
purists might frown upon this sort of thing, but obviously it would
be a good idea in some cases), then you might just want to list all
the possible combinations of tunes in a glossary at the front of your
book and then write all the tunes according to their type so that
readers can look up the right tunes.
The names of the sets would be up to you - you could name different
sets after different pubs/sessions or according to the keys and
styles of the pieces ( e.g. The Dog and Duck Slip Jigs or similar).
There is no convention other than using the name of the first tune so
you would be in uncharted territory using a new naming system!
One more thing - as tunes often have different names in different regions, especially when someone has put words to them, you could look at all the different names a tune/song has and use those to differentiate sets, although this could be confusing if the same tune appears many times throughout sets you are recording.