Of course the only way to actually make a professional sounding recording is to have it made by a team of professionals using high end equipment. But you can get fairly close without all that stuff and you can definitely make recordings that sound great.
A great master relies on a great mix, and a great mix relies on great recording. So you really want to think about all parts of the chain, but mostly about how to get better recordings. That's because you can actually "mix" during the recording process and "master" during the mixing process if you do it right. By the time you get to mastering, you'll be starting from such a good place that your main goal will be to not screw it up.
A complete answer to "how to do all this well?" wouldn't fit in this space. But I will break down the three elements that changed over the years for me that took my recordings from "clearly bedroom" to "where did you do this?"
Those three ingredients are:
- At least one good room (you'll wish you had three or more)
- At least one good microphone (you'll want like fifty if you stick
with it long enough)
- At least five to ten years of study and practice
What's a "good" room? Well first of all, the more you know about how to use a room, the better you can do with less good ones. There's a lot to know about rooms, but at least knowing how important they are to the final sound should get you started.
Usually for the "home" recordist, all rooms are inherently bad and the goal is to reduce how much badness gets into the mic. In a bad room, walls are like sound sources that we don't want to capture. We can "reduce the volume" of walls by treating them with diffusing and/or absorbing materials. Aside from commercial products, there are DIY options for room treatment, and items like bookshelves and curtains make a big difference when placed effectively.
The second way to reduce the amount of sound from a wall that gets into the mic is by putting the mic farther away from the wall. So bigger rooms are better. Actually bigger rooms are better in all ways when it comes to that quality of sound that they have, for most instruments. After a certain point, it's possible for a room to be too big, but rooms that size are rare for home recordists who can't also just buy themselves their own studio. Note that vocals are often recorded in fairly small rooms that are fairly "dead" sounding - meaning as much sound as possible is absorbed instead of reflected back into the mic. You can replicate this at home with a walk-in closet and some treatment, or you can just record vocals in any decent room of any size.
Finally, we can reduce the amount of room in a mic by careful selection and placement of the mic. Miking falls mainly under the "practice" category of our three ingredients so I will touch on it later.
As a recordist, the microphone is like your musical instrument. It doesn't have to be a Stradivarius, but if it's My First Microphone by Kenner it will frustrate you and hold you back. Like a musical instrument, there is personal preference involved in mic selection.
That said, if I had to record everything with only one mic, I would start with a large diagram multi-pattern condenser with an integral pad and maybe bass tilt switch if I can get it. Dropping the bass tilt and multi-pattern requirements would come before dropping the pad, in my opinion. In the US market, there are great options here in the $500 - $1000 range, and plenty of decent choices under $500.
My second mic or second choice for only mic would be a high quality cardioid dynamic mic. You can get a lot of mileage from just a humble SM-57, but jumping up to an SM-7, RE-20, or MD-421 would really set you up.
If you are going to be recording acoustic drum kits then that's a whole nother topic that won't fit here, but it can be done with just a couple mics. You'll quickly wish for all kinds of great sounding rooms and an economy car's worth of microphones.
It's helpful (and natural) to approach good recordings backwards, in a way. By that I mean you might be listening to one of your recordongs and think "I really need to master this better". Then you get back into the mastering and you start pulling your hair out thinking, "I wish the instruments were better balanced in this mix". So you go back to the mix and soon you're frustrated and wondering "why are all of these tracks fighting each other so hard?" The answer is because of the recording.
So on your next project, you vow to record tracks that practically mix themselves, and you learn that mic placement can make a big difference and you start placing mics in better and better places until one day you just can't find a good spot on the snare drum for the mic. At that point, it could be a bad room, or maybe a bad snare, or it could be badly tuned, or most likely it's a bad drummer.
This is where producers and A&R folks would normally be trying to help you out by only bringing you good artists or by being in charge of the project enough that they can specify that a session artist be brought in to play a rented snare tuned by a professional tech in a brilliant room with a $5000 microphone on it.
But as an amateur, you don't have those things, you just have the problems that those things are meant to solve. So you'll want to learn as much as you can about all the instruments you want to record, how they can be made to sound their best, what mics in what places give you what sounds, and which of those sounds will sit better in the mix without even any compression or EQ.
The biggest thing the recordist can do to improve the sound of recordings, mixes, and masters, is to place microphones perfectly, or as close to perfectly as possible. Instruments are not point sources. Different parts of the sound come from different parts of the instrument. You can back the mic off to capture the whole sound, but then you get more room noise, which is why a good room helps so much - it gives you more micing options. Mics also do not pick up the same sound at all distances and angles. Turning the mic and moving it closer and farther away dramatically change the frequency response.
With mic placement, you can pretty much control EQ and reverb (depending on the room) for an instrument right when you record it. When it comes time to mix, part of your job is already done. Mic placement is usually a process of trial and error: Move the mic, play the instrument, listen, repeat until you've found the best spot. Worst case scenario, you make several test recordings with the mic in different places and choose the best. Best case, you have a separate room to listen to the mic in real time while an assistant moves the mic and a musican plays the instrument. At home, you have to compromise. A good pair of isolation headphones (not noise cancelling!) or in-ear monitors with a feed coming from the mixer/computer interface to listen while you move the mic and someone else plays can make a big difference in your recordings.
That's just a window into part of the huge topic of what affects the sound of a mastered recording. Pros are pros because they do it for more than 40 hours a week and they are learning something new every time. Time spent learning and practicing is the most important thing for making better recordings, and you should expect it to take many years, unless you can quit your day job and just do recording all day.