You've fallen into the common trap of seeing a scale, and the triads that can be formed from it, as a restriction.
Yes, there is a set of chords that can be constructed from the notes of a scale. But that's no more than a mildly interesting fact. A song in a given key will use some of the diatonic chords a lot, others not so often. And other, non-diatonic chords will also very likely appear. The diatonic, scale-derived chords are not a restriction, not the chords that you MAY use. They're just the diatonic chords.
For instance, consider the C major key. C, Dm, F, G, Am will probably be used a lot. But D7, C#dim, Bb, Fm are more likely to be used than the (diatonic) B dim. And, if you're playing blues-based styles, C7, F7 and G7 are the basic chords, two of which are non-diatonic!
THE SCALE IS A FRAMEWORK, NOT A RESTRICTION.
Once you absorb this, a lot of the 'borrowing' thing becomes unnecessary. Where a chromatic chord IS used as a gateway to a modulation, there's sense in calling it borrowed. An Fm chord in C major MIGHT be a pre-echo of a Fm7, Bb7, Eb modulation to Eb. So calling it 'borrowed from Eb major' is useful. But C C7, F, Fm, G7, C at the end of 'When the Saints' doesn't go anywhere. Just let C7 and Fm be chromatic chords. 'Borrowing' them doesn't help.