When I connect my Yamaha Pacifica guitar to Vox DA 5 amp and connect the amp to the M audio 2496 sound card input by taking the output from the Vox phones output, I get a hum while playing the bass strings and recording it in the software. How to rectify this problem?
Hum is typically caused by ground loops, i.e. multiple points where the signal chain is grounded. By far the best way of dealing with this is to use balanced inputs and outputs everywhere but the manuals of your gear didn't say and I doubt it's balanced. The second best way to get rid of hum is to use an isolation transformer or direct box with a ground lift between the amp and the sound card. Something like this http://audiopile.net/products/DI_Boxes/FDB-101/FDB-101_cutsheet.shtml (disclaimer: I'm not affiliated with the company in any way, but consider them an excellent source) with the right adapters to make it work. It's also possible that the power supply of the host computer isn't that great. That's easy to to test by recording from a different source and see if that's clean. A good choice for a recording test is an battery powered player such an IPod, since there is no possibility of a ground loop.
It sounds like you are saturating ("peaking") the signal.
Turn down the volume on the amp and adjust the input volume on the 2496 so that the meters in your recording software never show the input signal in the red.
Play hard on the guitar, to ensure you are testing actual playing levels rather than just plucking notes while testing the input volume. Some notes will peak the input more than others (some chords as well).
There are very basic stages to recording: 1) getting good samples ("clean" signals); 2) mixing those clean signals to the volume you want for the end result.