Is there a precise rule (or set of rules) by which I can determine if a note should called sharp or flat?
Consider the two scales below. In the first C#/Db is C#, and in the second it's Db. By which rules would I be able to make this decision, given only a pitch class set like 0, 1, 2, 4, 5, 6, 7, 9, 10?
Scale 1715: "Harmonic Minor Inverse"
For the second, each note has its own letter name, there's no duplication, be it ♯, ♭ or ♮. that makes it easier to write - each note has its own line/space on the stave - and also easier to read, there being no need for accidentals.
That's pretty straightforward for heptatonic scales. However, when it comes to scales with more than seven notes, it can't really matter. There are two Cs and two Fs in that scale. Or, maybe, two Ds and two Gs! As a 'rule of thumb', the chromatic naming works well. Use C and C♯ when they follow each other, or D and D♭ when they follow. That saves using natural signs to cancel a previous note.
But as far as where they occur in the dots, when writing music which uses these scale notes, keeping to the same sign as much as possible seems a good idea.
For the first one, it's a forced approximation, translating a more-than-seven-notes-per-octave language to a seven-notes-per-octave language. Something gets lost in translation, and in this case something that wasn't there gets added, namely the sharp/flat aspect. I don't know what this Youlan scale is, but most likely there were no sharps and flats where it came from. And there probably was no A, B, C, D, E, F, G either.
The sharp/flat note naming system has evolved for use in a culture where scales have seven notes per octave. That's why there are seven note names: A, B, C, D, E, F, G. The Western staff notation system corresponds to this. There are seven staff positions per octave. It's a language - it naturally lends itself to expressing certain things well, but some other things, not so well.
When you try to describe scales that have more than seven notes per octave, the seven-notes-per-octave note naming and staff notation systems are not ideal. But since they are so commonly used, that's what often gets used.
There are other kinds of notation systems and "languages", for example guitar tablature. There you could write 0,1,2,4,5,6,7,9,10 without having to ask the sharp or flat question. Do you speak tab?
For the second scale, it has seven notes per octave, and you can derive it from the C major scale by making two scale degrees flat. The logic is, each letter-name should occur exactly once. You cannot have C and C# in the same scale, that would be against the whole letter naming idea. Each of the seven scale degrees can be basically in a sharp, natural or flat position. By saying "C#" you declare that "my C slot is tuned sharp". The tuning switch cannot be both natural and sharp at the same time, in that sort of way of thinking.
BUT like already said, that logic only applies to systems that can be seen as somewhat compatible with the Western seven-notes-per-scale way of thinking.
The web site cited for the scales says...
...more than you ever wanted to know about...
...apparently that doesn't include how to fit more than seven tones in the western seven tone gamut.
I wouldn't use the work "precise." A better word might be "conventional." If the tuning is equal temperament then enharmonic equivalent tones like C#/Db are precisely the same pitch. But if an instrument isn't tuned by equal temperament, then enharmonic equivalents aren't necessarily the exact same pitch. A violinist may play half steps smaller than those of equal temperament. So, C C# D and D Dd C might not be exactly the same intervals on violin.
There is a convention to write chromatic half steps using sharps for ascending steps and flats for descending steps (or the corresponding naturals/doubles.) The cited web page shows the ascending usage...
...but it doesn't show the scale descending. Following the convention of flats for a descent it would be...
If the scale has only seven tones, you should spell it with only one letter for each tone.
I would say that there can't be an unambiguous set of rules. Its more like natural language - part always depends on culture, context and what you're trying to express. As a practical suggestion, I would say pick something and stick to it, and when you have to break your own rules try and do so in a way that's a 'gentle' deviation from the style you've established. An aesthetic choice of course but probably simpler to do than to talk about! That way the reader will come to understand your way of communicating, which is the end goal of course. (For context , I work microtonaly so my bias is I'm not a big fan of 'rules' by instinct :) )
