A big majority of pop-type songs use diatonic chords mainly. Those are chords made up from the notes in their scale. Knowing the 'chord families' will be of great help.
There are 3 major and 3 minor chords in each family. I, IV and V being major, and ii, iii and vi being minor. So, in, say, key C, the majors are C, F and G, the minors Dm, Em and Am. Were it a minor key - use vii as the tonic - the other 5 will remain the same as quoted. Each key will have its own set of 6, and without knowledge of those, you could be searching randomly amongst any of the 24 majors and minors. There is viio, but that occurs so infrequently, I'm omitting it for these purposes.
You say you can fing a key. That's a good start - it gives you the other 5 chord choices quickly. Most pop-type songs will start on the tonic, so if you can tell if that's major or minor it's a great help.
As Albrecht says, the bass note is very helpful. Bass players often play one on one. That's the root note on the first beat of a bar. A really good guide as to what the chord can be called, letter wise. Don't worry about inversions - they won't change what the chord is - Am is still Am in root, 1st ot 2nd inversion.
A good move is to write out the bars on paper, 4 er line, and while listening, put in the tonics as you go. Keep that 'feel' in you, and each time you hear a tonic bar, write it in. Listen to any bar after a tonic bar, and decide (if tonic's major) whether the next bar is major or minor. If it's major, then there's a 50:50 as to IV or V. Not bad odds! Even if it's minor, there's a 33% chance of guessing correctly.