Do I sing the pitches I hear within the chord progression, or there is another way to do it? What if I want to sing notes that are not in the chord progression how do I sing them?
Learning to improvise, like other kinds of learning, is best done with the educational idea of "scaffolding," that new skills build on prior skills. So start simple and build up.
Yes, singing chordal tones is a great way to start. When a violin or flute player is dropped into a rock band and handed a sheet of chord symbols, a good starting strategy is to simply play the letters they see, the roots of the chords. Say, if it's a simple I - IV - I - V pattern, C - F - C - G, then they could simply play those notes and be fine.
The next step might be to play other chord members to create a more linear melody. For instance, from those chords, you could choose E - F - E - D and have stepwise motion. (Hey, we invented Tom Petty's "Free Fallin'"!)
The next step might be ornamentation with non-chordal tones. For instance, you might fill in gaps with scalar motion. Going back to the root pattern of C - F - C - G, you could simply fill in with C (D E) F (E D) C (D E F) G. (Hey, we invented "Lean on Me"!)
There are some music theory labels to non-chordal tones; you can explore those. They don't all have to fill in gaps; some can simply start with notes that "don't fit" and then change them to ones that do.
Finally, as you feel confident with all these, you can get more daring with non-chordal that don't move to chordal tones. Like, say, I could imagine simply holding a D through that whole progression. It's not a member of the C chord, but it is of all the others, and the temporary tension during the C resolves as you get to the G. Or, even more daringly, I could imagine holding a B flat. This isn't a member of any of them, and it creates a very strong dissonance against the A that's in the F chord and the B natural that's in the G chord, but the "seventh chord" feeling it gives to the Cs might excuse it. Maybe. I'm not sure I'd have the chutzpah to avoid moving by the time we got to the final G chord.
Finally, I'd advise studying simple music theory. For one thing, the progressions of chords are often very predictable, and understanding their function will help you predict them. The question of "what makes a good melody" or "what sounds good" often means simply "what rules, habits, and conventions govern the musical practices that we're deeply familiar with." As you find these codified and described, you'll be able to replicate them, and eventually break them at will for artistic impact.
For me, this goes entirely on the intuitive level, no conscious thinking about it. The old "play it by the ear", or in this case, "sing it by the ear". Often I don't even give any thought to the chords, just play what intuitively feels like a good idea to bring on next. Most of my songwriting is done this way, playing chords, and the melody just condenses around those, I'm tempted to say on its own. Notes outside of the chord appear in it, too, generally on the less prominent positions in the rhythm, just as you'd naturally expect.
I'm not saying a more analytical approach won't work for you, but for me, "don't overthink it and leave it to the subconscious" is the productive way. (Well, tune-wise. I definitely involve my conscious in putting together the lyrics.)