2

I am working on a project using the Unity game engine, The idea of the project is that the user plays a note on a guitar (and other instruments) and the app should display the note frequency that the user played, I have found this block of code that uses Unity API to get the spectrum data based on FFT algorithm:

 using UnityEngine;

 public class AudioMeasureCS : MonoBehaviour
 {
     public float RmsValue;
     public float DbValue;
     public float PitchValue;

     private const int QSamples = 1024;
     private const float RefValue = 0.1f;
     private const float Threshold = 0.02f;

     float[] _samples;
     private float[] _spectrum;
     private float _fSample;

     void Start()
     {
         _samples = new float[QSamples];
         _spectrum = new float[QSamples];
         _fSample = AudioSettings.outputSampleRate;
     }

     void Update()
     {
         AnalyzeSound();
     }

     void AnalyzeSound()
     {
     
         GetComponent<AudioSource>().GetSpectrumData(_spectrum, 0, FFTWindow.BlackmanHarris);
         float maxV = 0;
         var maxN = 0;
         for (i = 0; i < QSamples; i++)
         { // find max 
             if (!(_spectrum[i] > maxV) || !(_spectrum[i] > Threshold))
                 continue;

             maxV = _spectrum[i];
             maxN = i; // maxN is the index of max
         }
         float freqN = maxN; // pass the index to a float variable
         if (maxN > 0 && maxN < QSamples - 1)
         { // interpolate index using neighbours
             var dL = _spectrum[maxN - 1] / _spectrum[maxN]; //This line 1
             var dR = _spectrum[maxN + 1] / _spectrum[maxN]; //This line 2
             freqN += 0.5f * (dR * dR - dL * dL); //This line 3
         }
         PitchValue = freqN * (_fSample / 2) / QSamples; // convert index to frequency //This line 4
      }
  }

I have tested this code and it seems it works well only if I play a pure sine wave or pure note (no harmonics).

I have a few questions:

  • I don't understand the logic in the last 4 lines (theoretically) in the code which is critical, I understand that we searched the max amplitude index (maxN) but why do we need the next element and the previous element? and why it is divided by the value at index maxN, I will appreciate it if some one could give an explanation of the logic in the code.
  • The code above has some issue, most of the time it shows the frequency of the harmonics and not the fundamental, for instance, if I play a string with frequency 65HZ the code returns 130 or 195, What I can do to get the fundamental frequency or all the harmonics frequencies including the fundamental frequency
4
  • 1
    There is a DSP stack where you might get better answers. Sep 20 at 22:08
  • I fear the missing fundamental will foil your plans of obtaining the fundamental frequency.
    – Dekkadeci
    Sep 20 at 23:12
  • I spent a couple of weeks trying to find heuristics and failed hard. I decided to just show the whole spectrogram and draw lines where the notes should be instead, much easier. I noticed that the fundamental often did not have the largest amplitude and that sometimes I got a large response for frequencies lower than the expected fundamental (perhaps some weird resonance). So I think if you want to go this route you need machine learning or a very very good heuristic (I even tried optimizing for harmonics and which strings where physically possible to play simultaneously and it still was crap).
    – Emil
    Sep 21 at 5:52
  • Four methods with optional improvements are demonstrated here, with "pros" and "cons" for each method: gist.github.com/endolith/255291 (1) count distance between zero-crossings, (2) find peaks in FFT, optionally interpolating between bins for more accuracy, (3) find peak in autocorrelation, (4) find peaks in harmonic product spectrum Sep 21 at 9:03

2 Answers 2

3

The FFT gives the amplitudes for a finite set of frequency, it is digitised. The first three lines you wonder about are a simple interpolation to improve the resolution on the frequency: it tries to see if the peak you detect is more toward the low frequencies or the high frequencies.

The last line you wonder is the standard way to compute the frequencies from a FFT: a set from 0Hz to Nyquist (fsample/2) with a number of different frequencies equals to the number of input samples.

In order to detect the fundamental and not the harmonics, you can try to weight a bit your FFT, for instance by dividing each element of your FFT by its index (equivalent to 1/f normalisation).

(Sorry, on a phone hence a bit limited, will expand if needed, hope it helps already !)

6
  • Hi Tom, thank you for the answer it is already useful, Yes please I would be very happy if you could expand more about this subject with more details in your free time, I was searching for this a lot. including materials/resources is also welcome, thanks in advance
    – WhiteNight
    Sep 20 at 20:39
  • @WhiteNight the subject is very broad, is there any specific part that need to be expanded or the whole? Note that this forum is focused on music practice and theory. Imo this falls into practice because that close to the way I'm doing music but I'm sure some will it as outside of the scope ;)
    – Tom
    Sep 20 at 20:54
  • You are right :), I am having difficulty understanding the samples array that the FFT collects, I don't know what kind of manipulation/interpolation to do to get the results I desire, but I know the array contains amplitudes values for frequency ranges depends on my resolution (bin) size, for instance, an array of 1024 samples at index 0 will have an amplitude of frequency range 0-43.06 and next index will have the amplitude of the frequency 43.06 - 86.12, and so on, but I don't understand the interpolation part, is it a mathematical term? Is it integral?
    – WhiteNight
    Sep 20 at 21:16
  • No, it simply computes how big are the neighbouring bins compared to the max and shift the max toward the highest neighbor. Imagine you have a max, and it's t left neighbor is very close to that max (but not the right). Then the true maximum is between the max you found and the next bin to the left. This is just a simple way to estimate the true position of the maximum of a digitised signal.
    – Tom
    Sep 20 at 21:35
  • 1
    To understand this interpolation, i suggest to draw a curve with a maximum somewhere, on paper. Then, mark points at fixed x-interval (if you have a good on paper just use that) on that curve, that's the FFT estimation. Your points won't necessarily fall on the true maximum, but a bit next to it probably. The question is where is the true max compared to the point one. For that, the code you give just compare the neighbors and estimate the true max from that. (Would be easier with a blackboard ;))
    – Tom
    Sep 20 at 21:40
3

Four common frequency estimation i.e. pitch detection methods with optional improvements are demonstrated here, with "pros" and "cons" for each method, along with links for more reading: https://gist.github.com/endolith/255291

  • (1) find distance between zero-crossings
  • (2) find peaks in FFT, optionally interpolating between bins for more accuracy
  • (3) find peak(s) in autocorrelation
  • (4) find peaks in harmonic product spectrum ("HPS")

The examples posted in the gist above are in Python+Numpy+Scipy, and you'll learn more about it by tweaking the Python code. I won't try to translate that for Unity.

Real learning happens by doing, not by reading. It's the same in music, but also in mathematics and programming, and learning this by using Python is a good choice. Python has all the needed signal processing and audio i/o handling tools like FFT, (auto)correlation, windowing, WAV reading/writing, live audio streaming, etc. readily available in library packages you can install with the pip package manager. When you've learned how it works, you have a much better chance to make it work in Unity.

I've tried using the Python methods in my own music-making, and they do work. The FFT peaks + parabolic interpolation method is very accurate, down to at least a 10th of a cent, i.e. 1/1000th of a semitone, and it has given the same numeric results as the Tuner plugin in Ableton Live.

In any case, you'll have to adjust, mix and match several different methods for your specific application needs. For example, how to deal with noise and how to set a threshold between note and non-note. "Note onset detection" could be a possible additional search term.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.