when you're figuring out a melody by ear, are you listening to the
sequence of notes and figuring out where each note sits relative to
the root of the key? Or are you figuring out what the next note is by
the interval from the last note?
All of those, and none of them.
The thing is, I don't "figure it out", and it's not a matter of reasoning or logical thinking. It happens, I see the incoming melody notes, chords and bass notes in my mind on a piano keyboard or guitar fretboard. If the melody line is too fast and unfamiliar, I don't see it, or if the harmony is completely strange i.e. something I've never played, I might only see rough guesses and movements. And even if I see the notes completely clearly, it might be wrong! I have to play it to verify my guesses.
The learning process is an action-reaction feedback loop and it goes like this:
- Hear a note or chord (or whatever aspect you're trying to reproduce)
- Guess something as an immediate reaction, not a logical "calculation". Even a wild random guess is ok.
- Test the guess by playing it.
- If it was wrong, make a correction.
Monkey see, monkey do. Monkey don't figure out.
The important thing to realize about the process is, the guesses might be right or wrong, and the accuracy of the guesses improves over time, through practice. But it's not about logical and intellectual reasoning, it's about training and doing, practical exercises. The ability to see the notes without actually playing has come through lots of playing by ear in practice, and when I don't actually play, it's some kind of a mental simulation of playing.
If I interpret your question about tonic (which you meant by "root"?) vs. relative steps to fit this process: I "see" all aspects that I've trained myself to produce.
- A tonic/key/reference point is always there and all notes are located somewhere relative to a tonic. Actually it's completely relative - I can re-calibrate the "seeing" at will to any key, because I've trained myself to play any song in any key. That's what I've had to do. For example, if I want, I can see any incoming song in F minor or Ab major, without changing the audible pitch I'm imagining or humming.
- Chordal harmony inside the key. What chords would I play to reproduce this progression.
- Melody notes. How would I play this melody.
- Bass inversions. What would I play with bass notes to reproduce what I'm hearing.
The "next note" / "previous note" thing is an aspect as well - even if my guess is wrong about the melody's relation to the chords and tonic, the melody notes move in steps up/down the scale. Sometimes it happens, particularly if I'm "looking" at a song on the guitar (instead of piano keyboard which is the stronger instrument for me), the melody I see (=guess) on the (imaginary) fretboard is a fifth or fourth off, particularly with some fast jazzy/bluesy lines, I might confuse notes that form extended chords, for example if the melody goes G-E over an Am9 chord, I might see it as C-A. Because the Am9 is kind of Am and Em played at the same time, I imagine the wrong minor third. But if I switch my imagined instrument to piano, I might see the notes differently.
I suppose people who are good at sight reading and composing by writing music, the music staff is like an instrument they play, so they can "see" things they hear on music staves. But that's not the case for me, I need to imagine playing piano or guitar.