Why doesn't my synthesized note sound natural?

Question

I'm currently developping a game, and I'm trying to synthesize a music note from various instruments in C++. For now, I'm only trying to replicate an A4 440Hz with the help of an existing audio file as a reference. I've tried my best to make my signal look the same as the original in terms of frequency and harmonics, but the note still doesn't sound natural. I'm kind of new to music synthesis, so I hope you will be able to solve my problem. Links to the audio files:

• Original Piano A4

• Synthesized Piano A4

My program only adds sine waves with different frequencies together. Here is the comparison of the two audio files:

Spectrogram (left=real, right:synthesized)

FFT (peak) (left=real, right=synthesized)

(the peaks are the same with a ~=3hz difference)

Signal (left=real, right=synthesized)

Phase shift on peak (left=real, right=synthesized)

Period envelope

Real piano

Synthesized piano

What have I missed? I've tried my best to match the harmonic frequencies, but they still sound different. Same for their envelope. Should I have more harmonics? I've tried adding more, but the sound doesn't really seem to change much. Are the harmonics between 15kHz and 20kHz important?

PS: I don't want to use existing note nor music for my game, since I'm doing this to understand how music synthesis works.

Answer 1

I haven't been able to listen to the files, but I think Pat's comment is very likely to be closest to the mark. From the spectrographs, it looks like you may be missing two things:

• Inharmonic partials - Because of the stiffness of the piano strings, the various sine waves generated are actually not exact even multiples of the fundamental. Instead, they are sharper than expected. That leads to the tuning of the strings to be slightly sharper for the highest notes and slightly flatter for the lowest notes. Here is a paper with some measured and theoretical numbers on this effect.
• Transient noises - There are many important parts of the sound of a real piano that are not captured in the steady-state string vibrations. There are some transient frequencies and partials, and there are also several noises. The most important of these is probably lower key noise, which is the sound of the key striking the keybed. It tends to resonate briefly through the piano cabinet and it creates a low frequency, wooden "thump" sound. Listen closely to piano recordings and you should hear it. Play an actual piano and you can also feel it. As Pat commented, there are also transient high frequency string noises created by the striking of the hammer. The noise components and their amplitudes depend on the hardness of the hammer felt, and are heard as the brightness of the piano.

Some suggestions:

1. Read the article that Pat linked, it has a good overview of the challenges of piano synthesis and strategies for making it work: https://www.soundonsound.com/techniques/synthesizing-pianos
2. Experiment with subtractive and FM synthesis techniques. The technique you are using currently is additive synthesis, which has its advantages and disadvantages. I think you could learn a lot about inharmonicity by working with a digital FM synth engine to re-create an electric piano sound. Subtractive synthesis is a type for re-creating lower key noise.
3. Consider learning some percussion synthesis techniques using subtractive synthesis. Even if you use additive synthesis in the end, practicing making thumps and noise bursts with a subtractive synth engine will help a lot in understanding how to get the transient sounds you'll want.

---

Edit after hearing the file:

The sound of a piano note changes over time. Your synthesized sample is only 1 second long. You should be comparing the sound for the entire life of the note. Once you have a decent sound, one thing you might do is double it, since most of the range of the piano has two or three strings per note instead of one. Once it's doubled, slightly detune one (or two) of them. And I mean slightly. Good piano tuners get them very close but they still might differ by .5 Hz or so. Note that a beat frequency of .5 Hz takes 2 seconds to complete a cycle, so that's another reason why a 1 second sample is not enough listening time. Overall I think your synthesized sample is pretty good. It is missing some of those components mentioned above, and you should also start comparing lower and higher notes to see how your algorithm holds up.

Answer 2

The single biggest thing you're missing is an attack. Pat Muchmore's comment is exactly right:

One thing that’s definitely missing: a real piano sounds has tons of non-harmonic components during the attack of the sound. The overtone series will help with the sustain and decay, but the attack needs lots of noise and inharmonic components.

Nearly all instruments have a brief, non-pitched, percussive sound that plays for a tiny fraction of a second when a note first starts playing. If you listen to your sample piano note again, you'll notice that the first split-second of the note is much louder than the rest of the note. You might also notice that it sounds a little bit like a snare drum is being struck when the note starts playing. That's the attack. (On an actual piano, this sound is caused by the movement of the hammer sticking the string.)

In contrast, your synthesized note is at its quietest when it starts playing. It has no attack. This missing attack is incredibly important! I believe that if you add an attack to your note, it will sound convincingly like a real piano.

You can do this by either playing a shaped burst of white noise when your note starts playing or, somewhat more convincingly, playing a sample of a percussion instrument, mixed in subtly, each time the piano plays a note. (Even though a white noise burst sounds pretty artificial, I've had very good success adding white noise attacks to synthesizers in my own music. It really is convincing enough to work in a lot of situations! Todd Wilcox's comment gives good advice on making white noise work for an attack.) The most convincing approach would be to find a recording of someone isolating the sound of a piano hammer striking without the sound of the piano string ringing and use that for your attack sample.

---

I did notice that your synthesized tone has a percussive element in it already. However, it sounds partway through the note, not at the beginning. The attack has to reach its peak at the precise moment the note starts playing. You added this graph to your post:

Notice that for the real piano tone on the left, the loudest point of the sound is all the way to the left edge of the sound wave. In contrast, your tone gets louder for half a second before reaching its loudest. Unfortunately, this is too late. The percussive sound needs to start at the beginning of the tone to be a convincing attack.

In addition, your percussive sound is very low-pitched. A piano typically has an attack that also has some mid-range and high-pitched elements. I suggest using a snare drum hit, soft wood block hit, or, as I mentioned above, a recording of a piano hammer strike to stimulate your attack instead of the deep handdrum sound you're using now.

---

Other than the attack, to my ear, your synthesized tone is almost there. I personally think that once you add an attack, you'll have a synthesized tone that's as high-quality as what you'll find in most $100 MIDI keyboards. It will still sound artificial, but it will sound pleasant and it won't seem strange.

Answer 3

Just to add to the previous answers regarding the initial transient of the sound. There are two things missing from your synthesis, both of which affect the initial sound.

1. There is the mechanical noise of the key mechanism, plus the "thump" of the hammer hitting the strings which comes from the static parts of the piano, not from the strings themselves. Those noises are similar for every note on the instrument (and it's a fairly good approximation to assume they are exactly the same for every note) and they typically decay to nothing in around 0.1 seconds.

2. The second effect is something that isn't usually mentioned at all in "high school physics" level descriptions of how a stretched string vibrates, but …

The sound of the piano comes from transferring the energy of the vibrating strings into the soundboard, which also vibrates with a small amplitude but has a large surface area which moves the air to create the sound that you hear. Therefore, the ends of the strings are not fixed absolutely rigidly, because there has to be some movement of the soundboard at that point. The amount of flexibility is different for motion in two different directions, perpendicular to the plane of the soundboard and from side to side parallel to the soundboard.

The different flexibilities mean that a single string vibrates simultaneously with two slightly different frequencies, perpendicular and parallel to the soundboards. Also, the energy in transferred to the soundboard faster from the "perpendicular" mode because that is the more flexible direction.

The consequence of all this is that what you measure as "a single harmonic" is really two harmonics, with slightly different frequencies, one of which decays faster than the other. For a note with a frequency around 500 Hz - 1 KHz, the lower frequency vibration starts with a higher amplitude but decays quite fast (say in 0.5 seconds) while the slightly higher frequency decays slowly and extends for the full length of the note (several seconds).

As a further complication, the two vibrations start in phase with each other but the phase changes with time because the frequencies are slightly different. If you look at a piano sample recorded close to the instrument to eliminate the acoustic effects of the room the piano is in, you see that the combined amplitude starts high and quickly decays almost to zero as the two vibrations go out of phase and cancel each other out, and then rises again. By this point one of the vibrations has almost died out, and the rest of the note is the slow decay from the other one.

Actually, this is easier to measure and understand on an acoustic guitar instead of a piano, because there is only one string per note and the difference between the two vibration frequencies is bigger. On a guitar you can also change the initial amplitudes of the two frequencies, by "plucking" perpendicular to the soundboard (as in classical guitar technique plucking with a finger nail) or "strumming" the strings sideways parallel to the soundboard.

On the piano, there is the further complication that most notes have two or three strings, not just one. The strings will not be tuned perfectly in unison and (as another answer said) a skilled tuner will adjust the shape of the combined sound envelope of all 6 vibration modes of three strings to create whatever "piano sound" is desired - a Bosendorfer Imperial and a Yamaha acoustic grand piano intended for rock and pop music both "sound like pianos" but sound very different from each other!

Answer 4

I'd say you're learning just how subtlely the human ear can detect small differences in phases and frequencies. Your synth appears to have some slightly off- peak frequencies at higher amplitudes than the real one.

All I can say is that there's a reason all high-end digital pianos use samples rather than raw frequency generators.

Answer 5

Your amplitude graphs and your envelope graphs make it clear that you have not matched the phases of the components. This causes the attack to be smeared out in time. It also causes the lack of "lean to the left" of the components in the envelope -- the envelope of the piano is more sawtooth than your band-limited triangle synthetics. Shifting some of your components to earlier phases would help.

One way to address this is to track the argument (complex phase). I see that you have matched amplitudes in your periodograms (absolute FFT amplitudes), but there is evidence that you have not matched the arguments.

As an aside... You have picked one of the hardest instruments to additively synthesize. The spectral content shifts nonlinearly with amplitude. There are several harmonic and enharmonic hums by inductive transfer to the undamped strings. The direct oscillators (strings) exhibit three degrees of freedom, so there are three natural spectra for each string and most notes are produced by two or three strings. If you must start with a keyboard instrument, something like a celesta may be substantially less frustrating for you.

Answer 6

As many others have pointed out, you are only emulating the spectrum of the harmonic phase, but not attack and decay of the piano.

Here are some options:

Use Samples.

Yes, more than one sample, ideally one per note, but probably one per octave should be enough for a start. Pitch the sounds +- 6 halftones. Separate the attack, the harmonic and the decay phase, and loop only the harmonic phase. Use different samples for damped / undamped strings. Get different samples for different dynamics, because the harmonics, attack, and decay characteristics change. You could probably get away with 20 samples per octave to make a somewhat convincing piano. And you will start recreating something very complicated and expensive.

Model the Physics.

Instead of thinking about harmonics using additive / subtractive synthesis, think about the way the sound is created. String instruments use some excitation on a string, which brings the string to resonation and only harmonics of the string's physical characteristics are amplified, other frequencies get damped. An easy algorithm for that is Karplus-Strong.

You could also use a physical modeling library, e.g. STK, which has some e-piano like demos, but this is probably nearly as hard to get a natural sounding piano as with the sample based approach.

But what to use?

If you only want to play some notes with a fixed range of dynamics, use one sample. When notes sound unnatural, add a second sample and then try to find a way to load the best of several available sample based on pitch, dynamics, duration.

Theory

Depending on how deep you want to dive into the subject, there are several books dedicated to the subject, some free ones:

• The Theory and Technique of Electronic Music
• Numerical Sound Synthesis
• Programming Electronic Music in Pd

Using Puredata for synthesis may be a good choice, because you can build some higher level sound abstractions while using Pd as a library in your later program via libPd.