I'm looking to design a mobile app which displays piano notes from a MIDI file and follows the user playing, warning them when they make a mistake and giving them a score at the end.

I'm trying to do something similar to this question but I already know what the music should be, hopefully making things easier.

A simple approach would be to use some fuzzy sound matching algorithm against a synthesised version, but this has a number of drawbacks including difficulties with different types of pianos or electric pianos with different sound fonts.

While I could just use MIDI directly for this, programs for this already exist, such as synthesia. I'm trying to create something that users could just pickup and use, without MIDI equipment, even on real pianos!

What approach can I use to find differences between a MIDI file and a recital? Notes are a start, which then could be expanded to include commentary on timing and volume.

  • 3
    I believe you’d be interested in the literature surrounding score following. It’s in no way a trivial task.
    – Édouard
    Commented Nov 14, 2014 at 4:17
  • 1
    @BardiHarborow I don't mean to discourage you, because I think this would be a great application, especially for piano students and self-learners... However you have to take into consideration that this is a multi-facet project where each individual piece is in and of itself complex in nature. Signal processing (to guess the notes a user is playing) is complicated and not perfectly accurate, as I mentioned in my answer. By all means though, if this is what really interests you, study computer science and musicology! It's what I'm doing for computational linguistics ;) Commented Nov 14, 2014 at 4:50
  • 1
    Cool. . it sounds like tecnologically speaking what you're after is a mixture of (the astonishing) shazam.com and something to show the notes on the screen as people play. Both existing technologies - maybe it's worth researching how shazam works ? though I daresay the detail is an industrial secret Commented Nov 14, 2014 at 10:02

2 Answers 2


I was going to leave a comment, but it turned out to be too long; I'll answer, but this is mostly advice based on what I think you're trying to do (as I think it's a great idea!):

Signal processing

Not trivial... especially if you want an application to be 'real time'. This answer on Signal Processing.SE gives some interesting feedback as to how frequency detection could work, but it's not an easy task. Algorithms are complex, and false/negatives grow in frequency as the complexity of the signal increases (the more notes you play, the more likely you are to get wrong answers).

You may have better luck, as you suggested, by trying to approximate the mixture versus the correct mixture that you have stored in the application. I imagine that you would use an approximation algorithm to get the pitch, then use the Expected Maximization algorithm to compare the two distributions, and decide whether or not you think they were close enough to be correct.

This is especially true of sustained notes, chords, etc. Imagine Moonlight Sonata's 3rd movement, and trying to process the notes in real time - I'll just say "good luck!" :)

Limited computing power

Despite the time complexity of pitch analysis, you also run into data complexities. Frankly, I doubt an iPhone 5 could analyze the complete Greensleeves piece in less than 5 minutes. The fact that tables/phones are almost always RISC architecture, things get complicated - you'll be running so many floating point operations in this that the likelihood of it being realtime is incredibly low. There just isn't enough power in mobile to support this.

More realistic approaches

Keyboard to USB [MIDI-direct]

Though I only have an iPhone 4, I've heard of phones (and obviously tablets) that have the ability to attach USB. Take, for example, some Android models.

What that means is that, if a user were to have a MIDI keyboard with USB, they could theoretically plug their keyboard directly into their phone/tablet. Now, not being a device driver expert, I imagine that you would have to write a 'middleman' in order for the keyboard to connect directly. Since Google/Apple/Microsoft all have their own operating systems, this might not be a trivial task.

REST API (on the server)

Another possible approach would be to create an API where users can submit MIDI files from their computer (after recording or during), and have your application (on the server) analyze the differences between submitted and expected. You could then send the results back to the user via the mobile application, or display them on the website directly (or both, your call).

The advantages in this approach are that you don't need to write software to have the keyboard communicate with the device directly. Keyboard-->computer is easy, and most people who have keyboards and would use this application know about MIDI. They can send the MIDI files directly to your website, and you can send the results back however you want. This simplifies the transcoding of sounds to MIDI, because it's already in MIDI at that point!

Now, take what I've said with a grain of salt as I'm obviously not an expert on any of the above subjects, just throwing out ideas for you. There may be technologies available that accomplish this, or make it easy. I have not researched those; as you're building the application, it's up to you!

I would say frequency/pitch analysis should be a last resort, because those algorithms are not trivial to implement, and they are usually exponential or worse in time complexity.

As Édouard mentioned, you have multiple facets to a complex project. I really only got into signal processing and touched on some technical ways you could implement this. This may be a project suitable as an undergraduate capstone, or more likely a master's project (in musicology, for example). You'd likely need to take some signal processing and artificial intelligence courses on the side in order to obtain the skill set needed to do this effectively.



is your solution but interfaced (chained) with live MIDI stream input via virtual keyboard taps (I am just working on) Synthesia supports virtual keyboard taps while MIDI is played back but generally fails to process polyphony multitaps correctly (have a try yourself)

If bugs with synthesia are solved correctly I go to Piano2MIDI or Sound2MIDI applications to implement chaining, pipelining via display taps.

The problem is tap's duration is not supported via adb shell scripting, wait in seconds only, so I test if looped taps can simulate note key pressed

What comes next is my search for Audio2MIDI or Piano2MIDI live streamer

Hardware solution is already supported with usb output but I am looking for software based solution to be run on 1st smartphone, generating adb shell script taps send via usb cable or wifi to 2nd smartphone for animation in synthesia

  • @Brian Towers (not converted in user's live link) thank you for your edits How to enter web link as a direct link since via standard editor at SE, what is generated is [1] link
    – darius
    Commented Aug 11, 2020 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.