Is there any way of knowing what note in a scale would fit with a chord, or is it all trial and error, or do you just have to have that not you play in the melody feature in the chord you are playing?
You can always try and see what fits, but there are some notes that might fit better than others.
Every chord consists of some notes. If you play those notes as a melody on top of the chord, it will sound nice.
Chords belong in scales. So, you can play notes from the scale you are in, on top of a chord. Not all the notes of the scale will sound consonant, but you can try and see which ones you like.
There are many nonchord tones you can use. Wikipedia has a nice list. I will just include the list, but you can look at the page for more info.
- Neighbor tone
- Incomplete neighbor tone
- Escape tone
- Passing tone
The methods used in this list with the nonchord tones, will create a slight dissonance, and you might not like them at first, but they are widely used.
Ultimately it is 'what sounds good' or fits the emotion/aesthetic/story/etc. you are trying to convey.
Personally I have not beeen helped by theory/rules when it comes to playing a melody over a chord progression.
You can learn (induce) the rules/theory that make up a particular style/aesthetic by learning to play, playing along or experimenting over a piece of melody in that aesthetic.
To get an idea of what has been done in the past (and thus use as a model), the book by Goetschius: https://books.google.com/books?id=HV4QAAAAYAAJ&printsec=frontcover&dq=free+pdf+goetschius+melody&hl=en&sa=X&ved=0ahUKEwjK0ZHT7IfMAhXK7CYKHcjaCXkQ6AEIHDAA#v=onepage&q&f=false is pretty good. It's a bit old-fashioned but discusses lots of stuff that is missing in most other discussions of melody.
It's easy to harmonize a melody (or write a melody to chords) using only harmonic notes (like harmonizing a C chord with C,E,G or maybe A or harmonizing an E7 chord with E,G#,B or D). The trick (which allows for more interesting melodies) is to see how to use non-harmonic notes (like D or G# or F#) over a C major harmony. The book also goes into the difference between accented non-harmonic tones and un-accented non-harmonic tones. Also things like, handling chromatic tones (which not only do not belong to the current chord, but not even to the current scale): for example over a C chord, G# works well if the melody goes G,G#,A or the like; however, if the melody goes from D down to G#, then G works less well but A works well. (Chromatically raised tones tend to move upward; chromatically lowered tones downward.)
The above mostly applies to "Common Practice Period" "classical" music. Jazz and rock have different esthetics.