When writing a melody on top of a riff, is there a rule for what types of notes you use?
For example, if your riff has goes A, B, C continuously, do the notes in the melody have to be certain notes with each note of the riff, or can they be anything?
Rather than directly choosing melody notes based on the notes in your riff, it would be more common to consider what tonality (or tonalities) your riff could suggest, and then write a melody line that has the same tonality.
For example, a riff that goes A, B, C continuously could (depending on its phrasing) suggest the key of A minor, so you could take notes from the A minor scale as your starting point (though as Neil Meyer points out in his comment, that doesn't mean you're restricted to that set of notes!). Alternatively, you could work in a blues tonality rooted on A. Another possibility might be C major.
However, there's no one rule, and there are all sorts of possibilities. If the riff was slower, you could choose notes from G major over the A and B, and notes from C minor over the C. Or you could play a melody that sometimes followed the phrasing of the riff but a perfect fifth higher, creating a kind of 'oriental' sound... or just muck about and try things and see what sounds good. usually if you're composing, you have the time to come up with something interesting by trial and error!
The number of rules is so huge, that you can actually justify absolutely any sound on top of any other sound. The only rule concerning a single sound that still stands is "avoid notes which are one half tone above the base note" (for A avoid A#, with B avoid C and with C avoid C#). But even this last rule can have exceptions - note that dissonant sound is not an "error".
Most of rules for "good sounding" harmonies depend on context which is only present in imagination of your listeners. Exactly the same chord can sound strange and dissonant or quite tame, depending on the current melody.
Your riff (A B C B etc...) will probably evoke the feeling of A minor. Unless bass plays F or D (then people will imagine F lydian or D dorian). The most consonant notes for this setup would be A C D E G (note that C will clash with B, so you might want to be slightly careful here).
