I am trying to think about what defines a musical scale. To do this, I am trying to consider all the general categories of scales that exist. How can I know how many possible types of scales exist? Is there some way to prove this?
There are a few things that can drastically change how you look at this so let's first look at the definition of a scale is defined as:
A scale is any set of musical notes ordered by fundamental frequency or pitch.
So at heart, a scale is just a set of notes so to carry out the basic calculations, we'll talk about this in more set theory terms.
Consider our typical system of 12 named notes
(C, C#/Db, D, D#/Eb, E, F, F#/Gb, G, G#/Ab, A, A#/Bb, B) and let's label them 0 to 11. We can make a scale out of any set of them including the set that contains all of them or a set that contains none of them which is also known as the null set. For the specific calculation of how many possibilities there are you can use combinatorics where you are choosing to use a certain amount of a set.
So considering this system you have the formula C(12,r) where r is the number of notes per scale and the sum is the total number of scales you can consider in the system.:
# of notes per scale # of possible sets 0 1 1 12 2 66 3 220 4 495 5 792 6 924 7 792 8 495 9 220 10 66 11 12 12 1
So in this system, the number of scales you can have is 3964.
If you also want to consider modes, the formula changes because of the sets only account for 1 of each set. The modified formula would be C(12,r) * r Although you would have to add 1 to include the null set.
Now let's consider just start thinking in terms of quarter tones which utilize 24 distinct notes. We can do the same calculation again, this time with the formula adjusted to the sum of C(24,r) which for now I will not fully calculate, but the process is the same.
Let's also consider the 12 note system, but one where we use different reference pitches. The standard is A440, but if we were to use let's just say A441, note wise our scales would be the same, but the scales would sound slightly different. Considering these two different systems the previously calculated result would double and introducing more has the same effect.
In practice there's a lot to think about in terms of these calculations. When writing, are you really going to consider the set of no notes as a specific scale? Probably not. These calculations give you a baseline of distinct scales, but one thing to point out is that these are just bare calculations. Some of them do reduce to the same scale as the set of notes that make A major is different than the set of notes that make C major.
I'm going to put another finger on the scale (though there may be something fishy about these ideas):
1) Nothing requires a scale to repeat within only one octave.
2) While there are theoretically notes too close to each other to be distinguished by (normal) listening, once you add the possibility of multiple notes and beat frequencies you can achieve much greater precision... and the differences do matter; that's one reason all the debates about temperament and microtones and "stretch tuning" arise.
3) It is possible to construct instruments (usually electronically; it's much easier that way) in which as you go up the scale higher harmonics die out and subtones strengthen, so that no matter how high or low you climb you keep coming back to exactly the same sounds. Is that really a scale? (I would say so, depending on what musical note each of these was perceived as, in what tuning...)
4) If you play a scale on an instrument which produces a completely different sound for each note -- the General MIDI percussion set, for example -- is that a scale? What if we modify that so each sound is tuned so it's recognizably notes in sequence?
For what it's worth, I'm a definite fan of Harry Partch's compositions and instruments. As one set of liner notes described it, "he wasn't satisfied with our music so he created his own." I believe his instruments are tuned to a 31-note scale.
I'm going to take a slightly different approach from Dom, defining a "different" scale differently. I'm positing that the pitch of the tonic is irrelevant to the scale's designation. For example, a "Major" scale is a single class, so Cmajor, Dmajor, etc are the same so far as the interval sequence goes. Following that rule (which you may or may not choose to do), there is only one (1) 1-note scale. Similarly, there are only six two-note scales: pick a pitch, then pick any of the other eleven pitches and you have one scale. But, a scale with a fifth between them is sonically the same as a scale with a fourth between them (it's an inversion). Since the tritone is it's own inverse, there are six sonically distinct 2-note scales. And so on. If you prefer to treat inversions as different scales, which is equally reasonable, there are 11 2-note scales.