41

I'm currently developping a game, and I'm trying to synthesize a music note from various instruments in C++. For now, I'm only trying to replicate an A4 440Hz with the help of an existing audio file as a reference. I've tried my best to make my signal look the same as the original in terms of frequency and harmonics, but the note still doesn't sound natural. I'm kind of new to music synthesis, so I hope you will be able to solve my problem. Links to the audio files:

My program only adds sine waves with different frequencies together. Here is the comparison of the two audio files:

Spectrogram (left=real, right:synthesized)

enter image description here

FFT (peak) (left=real, right=synthesized)

enter image description here

(the peaks are the same with a ~=3hz difference)

Signal (left=real, right=synthesized)

enter image description here

Phase shift on peak (left=real, right=synthesized)

<Picture>

Period envelope

Real piano

enter image description here

Synthesized piano

enter image description here

What have I missed? I've tried my best to match the harmonic frequencies, but they still sound different. Same for their envelope. Should I have more harmonics? I've tried adding more, but the sound doesn't really seem to change much. Are the harmonics between 15kHz and 20kHz important?

PS: I don't want to use existing note nor music for my game, since I'm doing this to understand how music synthesis works.

  • 1
    Answers that are just links to other sites aren’t appreciated here, but you might find this article helpful. soundonsound.com/techniques/synthesizing-pianos – Pat Muchmore Jun 1 '18 at 14:20
  • 13
    One thing that’s definitely missing: a real piano sounds has tons of non-harmonic components during the attack of the sound. The overtone series will help with the sustain and decay, but the attack needs lots of noise and inharmonic components. – Pat Muchmore Jun 1 '18 at 14:22
  • 5
    @PatMuchmore exactly! As a historical side note, this is something the designers of the Roland D50 knew and used to great effect by mixing a sampled attack with a purely synthesized sustain. – Tobia Tesan Jun 1 '18 at 19:24
  • 1
    Focus on the attack. It’s the initial ~100ms of sound that needs the most work in my listening. Zoom so that you are looking at the very beginning of the waveforms for comparison. Your synthesized waveform has a bit of a ‘chop’ at the beginning. – Matthew James Briggs Jun 2 '18 at 7:11
  • 1
    Potentially helpfully worth noting is that in the world of synthesis, this technique (of adding sine waves) is classified as "additive synthesis". You may wish to take a look at "subtractive synthesis" methods, in which some filter function is applied to the resulting wave data. The parameters of this filter can be modulated over time, often synchronized to note attacks, so the initial part of a note could have high-frequency inharmonic components and then as it plays these can be gradually faded out. Techniques like this are often used to emulate "natural" instruments. – Darren Ringer Jun 2 '18 at 16:06
33

I haven't been able to listen to the files, but I think Pat's comment is very likely to be closest to the mark. From the spectrographs, it looks like you may be missing two things:

  • Inharmonic partials - Because of the stiffness of the piano strings, the various sine waves generated are actually not exact even multiples of the fundamental. Instead, they are sharper than expected. That leads to the tuning of the strings to be slightly sharper for the highest notes and slightly flatter for the lowest notes. Here is a paper with some measured and theoretical numbers on this effect.
  • Transient noises - There are many important parts of the sound of a real piano that are not captured in the steady-state string vibrations. There are some transient frequencies and partials, and there are also several noises. The most important of these is probably lower key noise, which is the sound of the key striking the keybed. It tends to resonate briefly through the piano cabinet and it creates a low frequency, wooden "thump" sound. Listen closely to piano recordings and you should hear it. Play an actual piano and you can also feel it. As Pat commented, there are also transient high frequency string noises created by the striking of the hammer. The noise components and their amplitudes depend on the hardness of the hammer felt, and are heard as the brightness of the piano.

Some suggestions:

  1. Read the article that Pat linked, it has a good overview of the challenges of piano synthesis and strategies for making it work: https://www.soundonsound.com/techniques/synthesizing-pianos
  2. Experiment with subtractive and FM synthesis techniques. The technique you are using currently is additive synthesis, which has its advantages and disadvantages. I think you could learn a lot about inharmonicity by working with a digital FM synth engine to re-create an electric piano sound. Subtractive synthesis is a type for re-creating lower key noise.
  3. Consider learning some percussion synthesis techniques using subtractive synthesis. Even if you use additive synthesis in the end, practicing making thumps and noise bursts with a subtractive synth engine will help a lot in understanding how to get the transient sounds you'll want.

Edit after hearing the file:

The sound of a piano note changes over time. Your synthesized sample is only 1 second long. You should be comparing the sound for the entire life of the note. Once you have a decent sound, one thing you might do is double it, since most of the range of the piano has two or three strings per note instead of one. Once it's doubled, slightly detune one (or two) of them. And I mean slightly. Good piano tuners get them very close but they still might differ by .5 Hz or so. Note that a beat frequency of .5 Hz takes 2 seconds to complete a cycle, so that's another reason why a 1 second sample is not enough listening time. Overall I think your synthesized sample is pretty good. It is missing some of those components mentioned above, and you should also start comparing lower and higher notes to see how your algorithm holds up.

  • -1 The biggest thing that the sample is missing is an attack. Everything else you added will help the synthesized sound become even better once an attack is added, but without an attack, the sound won't ever sound convincingly like a piano. I don't believe your answer emphasizes the need for an attack enough to be helpful. – Kevin - Reinstate Monica Jun 1 '18 at 20:07
  • 6
    @Kevin Fair point. I wonder if by "attack" you are talking about what I've called, "transient noises". They are heard at the very beginning of a piano note and then die off very quickly. The parts of a piano sound that are not the attack are sometimes called the "steady state vibrations", even though they are not truly steady state. – Todd Wilcox Jun 1 '18 at 20:22
  • 6
    Shoot, you're right. You and I are talking about the same things. Thanks for responding to my comment! I'll upvote your answer if you make a small edit. – Kevin - Reinstate Monica Jun 1 '18 at 21:29
  • 1
    most of the range of the piano has two strings per note instead of one Three, not two. – Ben Crowell Jun 2 '18 at 12:45
  • 1
    @BenCromwell It varies from piano to piano, but I can’t even find one count of any particular piano for how many notes have one, two, or three strings. So I’ll change it to “two or three” to be safe. Thanks. – Todd Wilcox Jun 2 '18 at 12:55
23

The single biggest thing you're missing is an attack. Pat Muchmore's comment is exactly right:

One thing that’s definitely missing: a real piano sounds has tons of non-harmonic components during the attack of the sound. The overtone series will help with the sustain and decay, but the attack needs lots of noise and inharmonic components.

Nearly all instruments have a brief, non-pitched, percussive sound that plays for a tiny fraction of a second when a note first starts playing. If you listen to your sample piano note again, you'll notice that the first split-second of the note is much louder than the rest of the note. You might also notice that it sounds a little bit like a snare drum is being struck when the note starts playing. That's the attack. (On an actual piano, this sound is caused by the movement of the hammer sticking the string.)

In contrast, your synthesized note is at its quietest when it starts playing. It has no attack. This missing attack is incredibly important! I believe that if you add an attack to your note, it will sound convincingly like a real piano.

You can do this by either playing a shaped burst of white noise when your note starts playing or, somewhat more convincingly, playing a sample of a percussion instrument, mixed in subtly, each time the piano plays a note. (Even though a white noise burst sounds pretty artificial, I've had very good success adding white noise attacks to synthesizers in my own music. It really is convincing enough to work in a lot of situations! Todd Wilcox's comment gives good advice on making white noise work for an attack.) The most convincing approach would be to find a recording of someone isolating the sound of a piano hammer striking without the sound of the piano string ringing and use that for your attack sample.


I did notice that your synthesized tone has a percussive element in it already. However, it sounds partway through the note, not at the beginning. The attack has to reach its peak at the precise moment the note starts playing. You added this graph to your post:

enter image description here

Notice that for the real piano tone on the left, the loudest point of the sound is all the way to the left edge of the sound wave. In contrast, your tone gets louder for half a second before reaching its loudest. Unfortunately, this is too late. The percussive sound needs to start at the beginning of the tone to be a convincing attack.

In addition, your percussive sound is very low-pitched. A piano typically has an attack that also has some mid-range and high-pitched elements. I suggest using a snare drum hit, soft wood block hit, or, as I mentioned above, a recording of a piano hammer strike to stimulate your attack instead of the deep handdrum sound you're using now.


Other than the attack, to my ear, your synthesized tone is almost there. I personally think that once you add an attack, you'll have a synthesized tone that's as high-quality as what you'll find in most $100 MIDI keyboards. It will still sound artificial, but it will sound pleasant and it won't seem strange.

  • "Nearly all instruments have a brief, non-pitched, percussive sound that plays for a tiny fraction of a second when a note first starts playing" Yes, and the piano has several non-pitched, percussive sounds that play for a tiny fraction of a second when a note is struck. One of the most important ones that the piano has is lower key noise. There's also the upper key noise and hammer noise and the upper harmonic transients. I think we are using different words to talk about the same things. – Todd Wilcox Jun 1 '18 at 20:24
  • 2
    I don't agree that white noise alone will do. Lower key noise is not even close to white. it's most like tapping a large wooden box with your finger. It's very low frequency and doesn't have the high frequency components that white noise has. It's also very percussive. A burst of filtered noise would work for the inharmonic transient partials, and then maybe a square wave swept downwards very fast, like an 808 kick sound, for the lower key noise. – Todd Wilcox Jun 1 '18 at 20:28
  • @ToddWilcox Shoot, you're right. I didn't read your answer very carefully. I'm sorry for being so antagonistic towards you and your answer, especially when you're right that we're saying the same thing. – Kevin - Reinstate Monica Jun 1 '18 at 21:27
  • I think you're right about the white noise, too. I have noticed that whenever I use white noise for attacks on my synths, I usually make it low frequency and sweep it down several octaves within a few milliseconds, like you suggested. A plain white noise burst would be very obvious. And even the artificial attacks I use are on clearly synthesized sounds, not on sounds that are supposed to sound like acoustic instruments. – Kevin - Reinstate Monica Jun 1 '18 at 21:28
  • You can tell just from looking at the waves. "This is too late." +1. With just five minuets of doing several fade ins, adding a 5 voice chorus, and a straight up 3db increase - I've got it sounding at least 10 more percent like a real piano ;) Prior to that I would of sworn it was being played backwards. – Mazura Jun 1 '18 at 21:51
8

Just to add to the previous answers regarding the initial transient of the sound. There are two things missing from your synthesis, both of which affect the initial sound.

  1. There is the mechanical noise of the key mechanism, plus the "thump" of the hammer hitting the strings which comes from the static parts of the piano, not from the strings themselves. Those noises are similar for every note on the instrument (and it's a fairly good approximation to assume they are exactly the same for every note) and they typically decay to nothing in around 0.1 seconds.

  2. The second effect is something that isn't usually mentioned at all in "high school physics" level descriptions of how a stretched string vibrates, but …

The sound of the piano comes from transferring the energy of the vibrating strings into the soundboard, which also vibrates with a small amplitude but has a large surface area which moves the air to create the sound that you hear. Therefore, the ends of the strings are not fixed absolutely rigidly, because there has to be some movement of the soundboard at that point. The amount of flexibility is different for motion in two different directions, perpendicular to the plane of the soundboard and from side to side parallel to the soundboard.

The different flexibilities mean that a single string vibrates simultaneously with two slightly different frequencies, perpendicular and parallel to the soundboards. Also, the energy in transferred to the soundboard faster from the "perpendicular" mode because that is the more flexible direction.

The consequence of all this is that what you measure as "a single harmonic" is really two harmonics, with slightly different frequencies, one of which decays faster than the other. For a note with a frequency around 500 Hz - 1 KHz, the lower frequency vibration starts with a higher amplitude but decays quite fast (say in 0.5 seconds) while the slightly higher frequency decays slowly and extends for the full length of the note (several seconds).

As a further complication, the two vibrations start in phase with each other but the phase changes with time because the frequencies are slightly different. If you look at a piano sample recorded close to the instrument to eliminate the acoustic effects of the room the piano is in, you see that the combined amplitude starts high and quickly decays almost to zero as the two vibrations go out of phase and cancel each other out, and then rises again. By this point one of the vibrations has almost died out, and the rest of the note is the slow decay from the other one.

Actually, this is easier to measure and understand on an acoustic guitar instead of a piano, because there is only one string per note and the difference between the two vibration frequencies is bigger. On a guitar you can also change the initial amplitudes of the two frequencies, by "plucking" perpendicular to the soundboard (as in classical guitar technique plucking with a finger nail) or "strumming" the strings sideways parallel to the soundboard.

On the piano, there is the further complication that most notes have two or three strings, not just one. The strings will not be tuned perfectly in unison and (as another answer said) a skilled tuner will adjust the shape of the combined sound envelope of all 6 vibration modes of three strings to create whatever "piano sound" is desired - a Bosendorfer Imperial and a Yamaha acoustic grand piano intended for rock and pop music both "sound like pianos" but sound very different from each other!

4

I'd say you're learning just how subtlely the human ear can detect small differences in phases and frequencies. Your synth appears to have some slightly off- peak frequencies at higher amplitudes than the real one.

All I can say is that there's a reason all high-end digital pianos use samples rather than raw frequency generators.

  • 1
    Can humans hear phases in a mono signal? – CodesInChaos Jun 1 '18 at 19:24
  • 2
    all high-end digital pianos use samplesactually some Very High End ones such as the Roland V-Piano, the RD-2000, Pianoteq (or the hardware instrument based on Pianoteq released a few years ago by, I think... Viscount?) do not :) – Tobia Tesan Jun 1 '18 at 19:29
  • "Can humans hear phases in a mono signal" Sure. Play a recording of a late-1970's arena rock song with heavily phased lead guitar in mono. You'll hear the effect. The result of phase effects being applied to a signal is... A different signal. – Beanluc Jun 1 '18 at 21:50
  • 3
    @Beanluc "Sure" is the correct answer (at least in some circumstances) but your example is a poor one, since in reality "phasing" as a guitar effect does a lot more than just changing the phase of some harmonics of the signal. – user19146 Jun 2 '18 at 9:06
  • 5
    @CodesInChaos Humans can’t hear phase in isolation, so if I played a cosine wave and played a sine wave you couldn’t tell the difference. But in any sound that isn’t purely sinusoidal, we can hear variance in phase between different partials. There can be a significant sound difference between two sawtooth waves with different phase variances in the upper partials. – Pat Muchmore Jun 2 '18 at 13:21
4

Your amplitude graphs and your envelope graphs make it clear that you have not matched the phases of the components. This causes the attack to be smeared out in time. It also causes the lack of "lean to the left" of the components in the envelope -- the envelope of the piano is more sawtooth than your band-limited triangle synthetics. Shifting some of your components to earlier phases would help.

One way to address this is to track the argument (complex phase). I see that you have matched amplitudes in your periodograms (absolute FFT amplitudes), but there is evidence that you have not matched the arguments.

As an aside... You have picked one of the hardest instruments to additively synthesize. The spectral content shifts nonlinearly with amplitude. There are several harmonic and enharmonic hums by inductive transfer to the undamped strings. The direct oscillators (strings) exhibit three degrees of freedom, so there are three natural spectra for each string and most notes are produced by two or three strings. If you must start with a keyboard instrument, something like a celesta may be substantially less frustrating for you.

1

As many others have pointed out, you are only emulating the spectrum of the harmonic phase, but not attack and decay of the piano.

Here are some options:

Use Samples.

Yes, more than one sample, ideally one per note, but probably one per octave should be enough for a start. Pitch the sounds +- 6 halftones. Separate the attack, the harmonic and the decay phase, and loop only the harmonic phase. Use different samples for damped / undamped strings. Get different samples for different dynamics, because the harmonics, attack, and decay characteristics change. You could probably get away with 20 samples per octave to make a somewhat convincing piano. And you will start recreating something very complicated and expensive.

Model the Physics.

Instead of thinking about harmonics using additive / subtractive synthesis, think about the way the sound is created. String instruments use some excitation on a string, which brings the string to resonation and only harmonics of the string's physical characteristics are amplified, other frequencies get damped. An easy algorithm for that is Karplus-Strong.

You could also use a physical modeling library, e.g. STK, which has some e-piano like demos, but this is probably nearly as hard to get a natural sounding piano as with the sample based approach.

But what to use?

If you only want to play some notes with a fixed range of dynamics, use one sample. When notes sound unnatural, add a second sample and then try to find a way to load the best of several available sample based on pitch, dynamics, duration.

Theory

Depending on how deep you want to dive into the subject, there are several books dedicated to the subject, some free ones:

Using Puredata for synthesis may be a good choice, because you can build some higher level sound abstractions while using Pd as a library in your later program via libPd.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.