Body building, weight lifting etc affect the voice indirectly. A pretty important factor for voice quality is controlling the breath pressure and airway resonance actively and passively. "Passively" means "riding" on the breath and its support in the lower body. When you are training your muscles, you are losing elasticity in your breathing apparatus and consequently your breath supply.
There is also the problem that the body building posturing requires a totally different way of breathing than singing: good singing relies on a "big bag of air" around your waist. Which is exactly what you are trying to avoid when posturing.
If you want to see an example of "too much of a goodness", take a look at old "Conan the Barbarian" movies from the time where Arnold Schwarzenegger was "Mr. Universe". And particularly scenes where he is "running". It's obvious that his running muscles are quite detrimental to actually moving forward speedily.
At any rate, once bodybuilding gets to the stage where it is changing your constitution and metabolism, it will not just cause differences to your breathing but actually affect the voice itself as you have different waste products running through your body (and that's assuming that you stay off steroids).
So there are a lot of things where bodybuilding and voice building are not independent. And if you look at the faces weightlifters make: that requires serious strength in muscle groups (neck, chest, others) which you don't want to tense while singing. Making those muscles stronger and habitually tensing them is going to make control of them more tricky.
So it all boils down to an emphatic "depends".