This is pretty self explanatory. The moment I stop holding the E string peg in place, it immediately goes back down to an Eb. Why is this happening? Is there a good fix to this?
Why is this happening?
The most likely cause is that the peg is loose in the hole.
If the string changes tune slowly over the course of a few minutes and by a small amount (E to Eb) then this is normal for a new string. New strings stretch slightly over time (not instantly the moment you release the peg) and you just have to retune every 10 or 15 minutes as it gets slightly flat for the first few days. Different makes of string will take different amounts of time to settle down.
Is there a good fix to this?
There are several fixes.
First, make sure that you are winding the string in such a way that it doesn't block the peg from going all the way into the hole. I normally aim to start with the string close to the middle of the peg and wind it towards the side which is on the same side as the handle of the peg, pushing the peg in as I go. If you are new to tuning your violin this is a likely first cause.
Use pressure when you turn the peg to tune so that you are pressing the peg into the hole. Usually this is enough.
Put a drop of peg drops onto the peg at the two places where it contacts the wood of the peg box.
Rub a tiny amount of talcum powder into the side of the peg, again where it contacts the peg box.
Finally, if there is no way you can stop the peg from slipping then it is likely that the peg doesn't fit the hole. The only answer is to take it to a luthier who will likely have to fit a new peg.
Loosen the tuner fully before you begin tuning (turning it counter-clockwise to loosen). (As Brian said, make sure the string has had time to stretch out, if it's a new string that's never been used before. This might take several days of multiple tunings.) Use the peg to get the string to approximately the correct pitch, and then use the tuner (the metal screw below the bridge) to fine-tune. The tuner should be able to correct for a half-step, if it was fully loosened at the beginning. In the picture, the tuner is a gold color. However, often they are a silver or black color.
If the peg doesn't want to stay put, either follow Brian's procedure with "peg dope" that you will have bought at a music shop, or take it to a luthier, your teacher, or a more experienced string player, to have them apply the peg dope.
Note that string instruments are sensitive to changes in humidity, and can suddenly start behaving differently. It can be helpful to buy a hygrometer and keep an eye on the humidity in the room where you keep your violin and practice. String instruments do especially badly in very dry air. If the humidity is low (I'd guess below 30%), then you risk one of the seams opening up due to dry glue and shrinking wood, and you should humidify your home. You can use a humidifier, or boil a big pot of water every day, or run a hot shower with the tub stopper in place and the door open, or use a damp-it, moistened and inserted in an f-hole of your violin.
One other thing to check is whether the bridge is at an unhealthy angle or skewed. If this happens, tuning one string can cause other strings to go out of tune. Until someone has taught you how to correct a bridge angle, please don't try to do it yourself, though.
One more thing that can go wrong is that the bottom of the string might not be properly seated in the tuner, and in this case, some creeping can occur, with the string mysteriously lengthening and going down in pitch at odd moments. To remedy this, loosen the string until it's a bit slack, with the peg, and re-seat the bottom of the string in the tuner arm, and then tighten the peg, with inward pressure on the peg as Brian described.