For instance, in the key of Cm we could borrow Am7-5 from C Dorian mode. But also C Jazz minor has Am7-5. Also F7 from C Dorian mode and F7 from C Jazz minor. There will be more in different modes if I keep looking. But what are the differences in these situations? Is it ultimately the scale??
'Borrowing' is just a device for making you happy that using a certain chord is 'allowed'. If you're preparing an excursion to another key, 'borrowing' some chords from that key can be effective - for instance if you're in C major and contemplating a move to Eb major (a very common modulation, beloved of the 'Golden Age' songwriters) it can be a good idea to throw in some Fm chords. Fm is a very common chromatic colour in C major, and is also a gateway to a ii, V, I modulation into Eb, that would be Fm7, Bb7, Eb.
Apart from that, I really wouldn't worry too much about 'borrowing'. If you're noodling around in C major, like I said Fm is a common chord to use. If it ISN'T the preparation for a modulation, 'borrowing' isn't a particularly useful concept. Just let it be a modified diatonic chord. It fits in well because two of its notes are diatonic. That's sufficient 'permission' to use it.