If I were to change the size or shape of the resonance hole of a guitar, how would it affect the sound of it? Would it be totally ruined or is it only a matter of retuning it?
By most accounts, not much. There is presently a huge variety of soundhole designs, here's a little Wiki stub article with some references to others:
Currently we see round, oval, "f-hole", multiple/little, soundholes on the sides of the guitar, none whatsoever, etc, etc. Much of the sound emanates from the entire vibrating surfaces of the guitar, including the back... There are several designs without any holes at all.
I think much of the variation we see is more in the name of style rather than function.
The hole itself doesn't affect the sound of the guitar. It is more the actual top (soundboard) of the guitar that generates the sound by vibrating like a drum skin. Usually, up to 80% of the sound is created from two vibration points to either side, and just below the bridge.
If you want to see which part of your soundboard is generating most of the sound - sprinkle some flour or fine sand all over the soundboard and pluck some open notes. You can then see the flour/sand settle into areas where there is no movement, and be bounced away from areas with lots of movement.
The trick is then not to damage or change the areas of maximum movement on the soundboard too much, as this will drastically affect the sound of the guitar.
Then again, if your guitar is TOO bright and you want to dampen the sound down a bit, you can stick bits of blu-tack or plasticine under the soundboard at these vibration points to dampen them a bit. Classical guitarsts do this quite often.
As a luthier, when I want more bass resonance I will design the soundhole smaller, and to decrease bass resonance I will increase soundhole size, regardless of soundhole shape. Keep in mind wood type, grain, cellulose density, and thickness also plays a role in tone, but to recap, smaller soundhole=more bass, bigger soundhole=decrease in bass.
The size/shape of the sound hole does affect the sound: you could model guitar body as a Helmholtz resonator (though this is an extremely simplified approach, see for example Helmholtz resonances and guitar section). The effect of it is that it makes some frequencies more pronounced, while reducing the other.
Changing the hole size moves the primary resonance frequency a bit, which roughly may be described as "the smaller the hole, the lower the frequency". The shape also has some effect, but it should be a much smaller change in the response than the one achieved by changing the area of the hole.
As an experiment, try closing part of the hole (e.g. by taping a piece of plastic with a smaller hole in it) and seeing what effect it produces. Another interesting effect can be achieved by attaching a tornavoz — a tube attached to the edges of the soundhole that projects into the guitar body. This effectively increases the length of the neck of the Helmholtz resonator.
Adding multiple holes may cause cancellation of some frequencies due to interference of sound waves emanating from different sound holes, that could definitely ruin the sound if done without proper research.
If you enlarge your hole in the guitar, it provides a higher, louder sound. If you want a lower, richer sound, get a guitar that has a smaller soundhole.
I hope this helps ;)
A bluegrass modification on D-28's is to expand the sound hole to the first inlay, just shy of it, to increase punch. It actually has a slight high ender shimmer dulling effect, so I guess it boosts the mids and sacrifices the highs.