Is it not considered "chord borrowing" in the sense that you're not using notes in your chord that derive strictly from the parallel key/mode of the song and you are simply changing the key of the song?
Generally speaking, there are two types of chromaticism in tonal music:
You can have what is called mode mixture, where you borrow pitches/chords from the parallel key. (Hence this matches your term of "borrowing chords.") If you're in C major, the IV chord, F-A-C, is major. But you can "borrow" the Af from the parallel minor, meaning you "borrow" the iv chord as well.
The other type of chromaticism could be called "secondary" or "applied" chromaticism; this is when chromaticism tonicizes another key. If you're in C major, the ii chord, D-F-A, is minor. But if you want D-Fs-A, this suddenly becomes a temporary dominant to G, and we label it a V/V ("V of V").
If you're asking about chromaticism that stays in the current key and is not borrowing, you're probably looking at examples of "secondary" chromaticism. (So named because of its relation to the "secondary" dominant.)
But if you're changing key, this is something altogether different: modulation!
Often a 'borrowed chord' doesn't constitute a change of key. Possibly a slight modulation, but that's all.If there was, for instance, in C, a C+ with C-E-G#, usually leading to an F, the G# isn't from the parallel key, (as you are saying), but that's not changing the key of the piece. I suppose it could be argued that the G# is found in the relative, though. There are only 12 notes to play with, 7 get used in a major, another 3 in parallel minor. How many 'unrelated notes' are there? And I'm sure someone can justify their existence !
No most of the time you are not changing keys. Being in a key gives you two pieces of information one being the note that you will sound at rest on and two the overall harmony of harmony used in a piece. For example in a song in the key C major will make C the note that you always want to come back to and you'll find most of the harmony comes naturally from the major scale. You can easily deviate as seen in the progression C-D-G-C. You wouldn't consider the D chord as being borrowed from a parallel mode, but as a dominant from the key of G major. In Roman Numerals the progression is I-V/V-V-I.
The only time you actually change keys in a song is when your tonality and/or you tonic change are established and persist. For example if instead of the pression above we did this one instead C-D-G-Am-D-G the tonality has now shifted to G and if it says there we have now modulated.