If I have a guitar plugged into an amp and I remove the cable from the guitar I get a buzz or hum. Why is this? Shouldn't the amp be quiet since the circuit is open and supposedly "off"?
The circuit isn't completely open. The cable acts as a capacitor (around 500 pF, depends on cable length and insulator properties), parallel to the input. That by itself wouldn't cause hum (it just “couples” some ground into the input, but ground is quiet). However, for an unplugged cable the tip is exposed, thus you basically get another capacitor with a terminal to everything else in the room. In other words, the tip acts as an antenna that picks up electromagnetic waves. And if there are electrical appliances in the room, these will cause such waves in an audible frequency range. Fluorescent tubes, dimmed light bulbs and CRT monitors used to be particularly bad offenders creating buzz; nowadays a bigger problem tends to be the burst noise from switching power supplies.
The solution is to avoid exposing the tip of the cable. E.g. plug it into a switched-off effect pedal; its metal housing will shield the tip from such interference. Make sure it's a pedal in true-bypass mode, else it will actually connect the tip to some circuitry inside, which causes yet different outcomes. Alternatively, you can wrap first the tip in adhesive tape (as an insulator) and then the entire plug in aluminium foil. To ensure the foil has a good contact to the shaft, fixate it tightly with another layer of tape.
If the noise sources are particularly strong, you might still hear some of it even with shielded tip, in particular if the cable is not so great. Instrument cables are coaxial, i.e. the ground “wire” is really a wrapping around the signal wire that should shield it from all interference, but in practice this shielding isn't implemented perfectly so even then some noise might leach in.
Actually, the problem with your understanding is your belief that disconnecting the instrument will turn the amplifier circuitry off, it doesn't, it only disconnects the instrument and causes the input circuit of the amplifier to be classified as an open circuit, similar to a radio antenna, which will pick-up and amplify any kind of electrically broadcast hum from fluorescent or neon lights, power supply transformers, electric motors, lighting dimmers, and all sorts of other electrical devices. In order to stop the buzz, the input needs to be grounded to the amplifier ground. This is almost universally done by using an input jack on the amp that connects to amplifier ground automatically when the guitar cable is unplugged at the amplifier end of the cable. The amp circuitry will not be "off" until it is disconnected from it's power source either by unplugging the amp from the wall or using the power switch.
Guitar amplifiers (and most other audio amps) use a Class A amplifier circuit (or more probably class AB which is a hybrid of classes A and B).
The Class A amplifier is biased (set up) to idle at the middle of its amplification band. So, it is ready to amplify any signal it receives. Even noise or feedback. See the reference to Class AB below.
When you remove the jack and let it idle there is some inherent noise in the circuit that gets amplified since there is nothing connected to it to pull it down to zero. If you plug in a guitar or other device, it can “zero the signal out” so to speak, keeping it from idling and therefore keep it quieter. If you take your hands off the strings it can still hum from the noise of the pickups.
This is an oversimplification of what the amp is doing but the logic holds that the amp is waiting for a signal to amplify and when there is none, the "noise" on the guitar cord is amplified as a humming noise since the signal is floating. If you hold the open jack in your hand it will amplify your body's noise also.
Hopefully, this makes a little sense.
Amplifier Classes "The AB classification of amplifier is currently one of the most common used types of audio power amplifier design. The class AB amplifier is a variation of a class B amplifier as described above, except that both devices are allowed to conduct at the same time around the waveforms crossover point eliminating the crossover distortion problems of the previous class B amplifier."