If I were to change the size or shape of the resonance hole of a guitar, how would it affect the sound of it? Would it be totally ruined or is it only a matter of retuning it?
By most accounts, not much. There is presently a huge variety of soundhole designs, here's a little Wiki stub article with some references to others:
Currently we see round, oval, "f-hole", multiple/little, soundholes on the sides of the guitar, none whatsoever, etc, etc. Much of the sound emanates from the entire vibrating surfaces of the guitar, including the back... There are several designs without any holes at all.
I think much of the variation we see is more in the name of style rather than function.
The hole itself doesn't affect the sound of the guitar. It is more the actual top (soundboard) of the guitar that generates the sound by vibrating like a drum skin. Usually, up to 80% of the sound is created from two vibration points to either side, and just below the bridge.
If you want to see which part of your soundboard is generating most of the sound - sprinkle some flour or fine sand all over the soundboard and pluck some open notes. You can then see the flour/sand settle into areas where there is no movement, and be bounced away from areas with lots of movement.
The trick is then not to damage or change the areas of maximum movement on the soundboard too much, as this will drastically affect the sound of the guitar.
Then again, if your guitar is TOO bright and you want to dampen the sound down a bit, you can stick bits of blu-tack or plasticine under the soundboard at these vibration points to dampen them a bit. Classical guitarsts do this quite often.