Title essentially sums up the problem. I have a VHT AV-RL-20R combo amp that I am trying to hook up to my computer for recording, but every time I plug in my amp via the output jack on the rear panel, both the internal speaker and my PC's audio monitors start giving some awful feedback and make it impossible to record anything. I thought it may have been the cable I was using at first, as it's one of the cheap-ish Real Tone cables used for the game Rocksmith, but after plugging it in directly to my electric guitar, it produces a very clean sound free of noise, which hopefully means I've managed to isolate the issue. Any advice is greatly appreciated, it's one of the only issues I haven't been able to solve yet.
The best way to record guitar using the equipment you have would be connecting the guitar directly to computer using the RealTone cable and an amplifier modeling plugin in the DAW. The RealTone cable is a small audio interface in itself.
If you want to record the tone of your amplifier, you need a microphone.
Recording the speaker output does not work, both because the electrical properties of the speaker output are very different from guitar or line output and because the guitar speaker itself affects the tone by filtering out high frequencies and adding some distortion. There are dummy load boxes that take the speaker output and emulate the speaker, but these are made for recording powerful tube amps at low volumes and you probably don't want to pay for one. If you are happy with the tone from the amp's headphone output, plugging that to a line input may work, but the RealTone cable probably won't work with it.
The socket on the rear is a speaker output. I'm surprised you haven't blown a circuit yet.
You cannot even plug the headphone output directly into a computer's headphone input, the pins won't match. It's not really designed to do what you're trying to do with it. You'd have to get a cable made up to take one side of the stereo headphone output to the mic input of your computer's headset socket… & even that would really be too powerful for it.
Better to either mic it, or get a proper USB interface that can take line level.