The Locrian Mode is considered unstable - my question is what in it makes it unstable? What makes a scale stable (have better relation between the keys)?
To establish a modal final (which is what we call a mode's melodic keynote) as a tonic, you've really got three things that you will use:
- a hierarchy of root relationships within the mode that confirms the tonic;
- conjunct melodic motion, notably by semitone, that pushes into the tonic; and
- the stability of the most important harmonies within the mode.
The root motion that confirms a tonic most strongly is 5-1, because the chord on 1 tends to subsume the chord on 5, the overtones of 5 being also overtones of 1. Thus movement from 1 to 5 tends to create tension; movement from 5 to 1 resolves it.
Root movement from 1 to 4 tends somewhat to undercut 1 for that reason: 4 subsumes 1 and acts like a resolution, so there is a balancing act involved. This is why the subdominant tends to be used as a pre-dominant, e.g., in I-IV-V-I, and why plagal (IV-I) cadences sound less conclusive than authentic (V-I) cadences. In I-IV-V-I, the relaxation of the move to the subdominant is balanced by a tension-creating move to dominant, and the final drop of a fifth confirms the tonic. The subdominant, however, is still closely related harmonically to the tonic, and, when used with materials that tend to move very strongly to the dominant, serves to counterbalance the dominant and confirm the tonic. (This is why you will often see references to the subdominant key in the recapitulations of sonatas whose expositions featured a strong move to the dominant key.)
Movement to other roots is less powerful than these root movements to a varying degree, so there is a hierarchy of strength of root movements involved.
Points 2 & 3 are fairly obvious, I think. 7-1 melodic movement (e.g., B-C in the key of C) is very strong indeed, and you already know how unstable the tonic triad is in Locrian mode.
But it is the degree that these elements line up and work together that determines how easy or rough a time you'll have tonicising the modal final, and what steps you'll need to take.
For instance, the Ionian (major) mode does make things fairly easy: the I, IV and V chords are all very stable (all major); the root relationships between I on the one hand and IV & V on the other are all perfect intervals; the third of V is 7 (the leading tone); and the mode's other melodic semitone interval, 3-4, forms on the one hand a rising leading tone relationship between the third of I and the root of IV, and a falling semitone relationship between the dominant's seventh and the third of the tonic. Even the location of the mode's diminished triad is fortuitous: vii° forms the top notes of the dominant seventh chord, and makes that chord unstable enough to demand a resolution to the tonic.
Now let's consider the other end of the spectrum of commonly used modes, the Phrygian mode.
In Phrygian, the melodic semitones are between 1 and ♭2, and 5 and ♭6. This means the key degrees are i and iv (both minor), and v° (diminished).
Because of the tritone, v° is deeply ambiguous: the root of the chord isn't well-established. In the Ionian (major) mode, because it contains the leading tone and the falling seventh, the diminished chord (vii°) can stand in for a dominant 7th that's missing its root; in the Phrygian mode, it is going to suggest that same Ionian dominant seventh (which, of course, wants to resolve to the Ionian tonic a minor third below the Phrygian final), and thus undercut the tonic. Also, it only contains the falling leading tone, ♭2.
So, to establish the Phrygian final as tonic, you have to use what the mode gives you: a reasonably stable subdominant and a falling leading tone. Note that the two don't go together, so you have two kinds of cadences to lead to the tonic: a fairly standard iv-i plagal cadence, and a cadence that features ♭2-1 in the bass. In pop music, the latter cadence is often ♭II-i, but that doesn't make for great voice leading using common practice harmony, so in Classical music, you use the so-called Phrygian cadence, ♭vii6-i.
You do have an option of musica ficta, which is the introduction of chromatic notes into a mode to create leading tones, but you have to be careful about how you go about it, or you will either undercut the tonality or undercut the mode. In the Phrygian mode, you can sharp the sixth of ♭vii6 to produce an augmented (Italian) sixth chord that has both the rising and falling leading tones. You don't sharp ♭2 in the bass, or you lose the sense of being in the Phrygian mode.
Other modes have their own "gotchas" - in Lydian mode, for instance, you have to watch carefully that ♯4 doesn't tonicise the dominant - but, with care and an analysis of the mode's characteristics, you can stabilise a modal final as a tonic, so the stability of the various modes is relative, and, as often as not, it is as much relative to your ability to work out the requirements of a given mode as it is relative to the other modes.
That's even true of Locrian mode (see Locrian Harmony), but make no mistake: Locrian is very, very difficult to stabilise. (I actually wrote a piece to study the ramifications of the mode when I answered that question, here. I think I succeeded in making the final B major chord a logical ending for the piece, but that was one tough exercise.)
The stability of a scale or a mode is directly related to the stability of its tonic chord. Locrian is considered unstable because it contains an unstable tonic chord with a flat 5 (m7b5) instead of a perfect fifth. This chord wants to move one, it does not sound resolved. A m7b5 chord is usually perceived as a II chord in minor, not as a I (tonic) chord. Having said all that, it is important to realize that the concept of stability is nothing absolute or objective. This question and its answers give you some more information on the stability of chords.
The key is not only the the harmony as Matt suggests, but also the melody and what notes sound at rest and what notes don't . To demonstrate this, play any Locrian scale. How it will sound is unfinished as in the ascending scale you will want to go up another half step and decending you will want to go back up a half step. You melody will never sound at rest on your tonic even though that is supposed to what you are trying to base your melody and harmony.
Tonal harmony is based around the concept of the leading tone which would be the root note in the Locrian scale so when thinking about how it's used it makes sense that we would not feel at rest on it.
So pretty much if the root note of the scale causes issues when you try to build the melody and harmony of a scale then it is unstable. If it doesn't then it is stable. Note how you approach the melody and harmony in different scales/modes and being unstable may be a destination in certain contexts.
For people used to more melodically based music, the notion of scales like the Phrygian scale and the Locrian scale being unstable seems strange. Surely, consonant melodies can be played within them, so why are they unstable? What does this mean?
However, in a different paradigm, the notion makes sense. I am talking about the paradigm of functional harmony, which is a chord based paradigm.
In this type of music, triads are used extensively, and these chords aren't just the side effect of multiple simultaneous melodies, or an embellishment to modify timbre. Here, chords are fundamental building blocks, like tones in a scale. They are used to set up tension and release. There are three main kinds of chords:
A tonic type chord is to functional harmony what the pitch centre is to melodic / modal music; it feels like a centre, like a home. Tension is resolved by returning to this chord.
The dominant type of chord does not feel like home, and has tension because it points toward the tonic type. There's a pull from the dominant type, toward the tonic type. A purely melodic equivalent would be sitting on a scale degree close to the tonal centre, especially if the melody up to that point had been angular; you can just feel that the tonal centre is coming in a situation like this.
The subdominant type of chord does not feel like home, but it also doesn't point to home. It doesn't introduce tension, and going from subdominant to tonic is not a strong resolution.
Locrian can't perform triad based music that has tension and resolution, because the tonic triad is very dissonant, being a diminished chord with a tritone in it. Generally the tritone wants to resolve inward or outward, thus it is unstable. A major triad on the other hand doesn't inherently want to go anywhere, so it is stable. You can't have tonic function without stability. This is why Locrian is unstable: If you're going back to the home chord of Locrian, that very chord wants to change to a different chord that's not in Locrian, which would make you leave the scale. Of course, if you're just playing melodies, then Locrian is a stable scale as there's nothing pulling it away. You can stay in Locrian all day playing modally.
Phrygian is also unstable because the triad built on the fifth scale degree, which tends to have dominant function, is diminished, meaning that there's a pull toward leaving Phrygian, by going to a chord not in the sacle.