3

I'm having a hard time trying to do something presumably very simple, and it's driving me mad.

I'd like to use javascript to extract from a midi file: lyrics + related time stamps. Of course, I'd like to have the time stamps as absolute time in seconds from the beginning of the song. As far as I understand, the native midi format provides times in the form of relative times (delta vs previous event in the list), measured in ticks (rather than seconds). So what I've found & achieved so far is the following:

I can use the midi-parser-js library to parse a midi file and retrieve the lyrics & delta times. https://www.npmjs.com/package/midi-parser-js I can somehow compute the absolute times in seconds when the tempo doesn't evolve, or when tempo changes are conveniently located. As soon as tempo changes occur between notes, I cannot seem to find the proper formula, which looks awfully complex, and the more I search, the further I get from the solution.

I think that the difficulty partly lies in that the tempo is located in a separate track, and as a result, delta times retrieved from the lyrics track do not take the tempo changes into account. So I thought that parsing a midi file in format 0 (rather that format 1) would help, since all events would be merged into the same track, but the tempo still seems to be on a separate "track 0", rather than in "track 1" with the lyrics events.

So I found another javascript library, Tone.js, that provides a midi parser: https://tonejs.github.io/Midi/ Which seems almost perfect, because it provides absolute times in seconds (and also note durations, which are not available as such in the native midi format). However, this library doesn't seem to expose the lyrics data! (or if it does, I couldn't find how). The user discussion / support forum for this library seems to be down.

Any idea or suggestion someone, please?

6
  • Are you interested in JS only or other programming languages are acceptable too? For .NET/C# I can recommend my library – DryWetMIDI – which can solve all your tasks (automatically taking care about tempo changes). The code will be extremely simple with the library.
    – Maxim
    Jan 13 at 10:06
  • I already stumbled upon your library :) Actually, I might use another language indeed, but sofar, I have never used C#, so javascript was the easiest for me to set up.
    – Denis
    Jan 13 at 11:02
  • 1
    Well, I've created a gist – Get lyrics with timestamps in seconds – that shows how to solve your task with DryWetMIDI. I'm not posting it as an answer here since it can be downvoted as I'm showing C# instead of JS. If you want, I can post an answer. In the gist lyricsWithTimestamps will be an array where each object contains two properties: Text with lyrics, and TotalSecondsFromStart with timestamp in seconds from the start of the file.
    – Maxim
    Jan 13 at 13:01
  • I will definitely try that, thank you very much. :) I'm not sure about the "answer / not answer" part...? I for one will not downvote anyway! And so if I may ask, while writing your library, did you encounter the same issues as did? Am I taking them from the wrong end?
    – Denis
    Jan 13 at 16:54
  • 1
    I've tried to set up a .NET project, but I still lack the most basic knowledge in that matter. I'll try harder, and I will let you know how it goes...
    – Denis
    Jan 17 at 17:06

2 Answers 2

2

In order to translate relative tick-based timestamps to absolute seconds-based times, you need to use a tempo map. Actually, the MIDI file format specification explains that this is assumed.

... To make it easy for the synchronizer to extract this data from a MIDI File, tempo information should always be stored in the first MTrk chunk. For a format 0 file, the tempo will be scattered through the track and the tempo map reader should ignore the intervening events; for a format 1 file, the tempo map must be stored as the first track.

It is assumed that there is a tempo map reader, which extracts a tempo map from the MIDI file. A tempo map is needed to synchronize the musical tick/beat based sequencer with other time bases, like the wall-clock seconds you're after. The MIDI file specification has a scenario with a SMPTE synchronizer machine, but we can treat the lyrics display as having a similar role, if it has to operate in a non-musical time base.

I couldn't find a tempo map reader feature in the midi-parser-js library, so I wrote a rudimentary tempo map add-on that can be kludged over the test-script-element.html example in the midi-parser-js examples folder. This tempo map works for type 1 MIDI files, at least the one I tried it with. It utilizes the objects parsed by midi-parser-js's MidiParser.parse() method. There's no error checking and we hope that everything is as assumed. To work with type 0 files, you may have to tweak the code.

class TempoMap {
    constructor(sourceTrack, timeDivision) {
        // default initial tempo is 500000 microseconds per quarter-note = 120 beats per minute
        let mspqn = 500000;
        this.timeDivision = timeDivision;
        this.tempoMap = [{timeTicks:0, timeMicroSeconds:0, mspqn:mspqn}];
        let timeTicks = 0, timeMicroSeconds = 0;
        sourceTrack.event.forEach(event => {
            timeTicks += event.deltaTime;
            timeMicroSeconds += (mspqn * event.deltaTime) / timeDivision;
            if ((event.type == 0xFF) && (event.metaType == 0x51)) {
                mspqn = event.data;
                this.tempoMap.push({timeTicks:timeTicks, timeMicroSeconds:timeMicroSeconds, mspqn:mspqn});
            }
        });

        this.tempoMap.push({timeTicks:0xFFFFFFF, timeSeconds:undefined, mspqn:undefined}); // terminator item
        // the tempo map now contains at least two entries
    }

    getEventTimeInSeconds(eventTimeTicks) {
        // find what tempo is active at the requested tick-based time
        const tempoEntryIndex = this.tempoMap.findIndex((tempoEntry) => tempoEntry.timeTicks > eventTimeTicks) - 1;
        const tempoEntry = this.tempoMap[tempoEntryIndex];
        const eventTimeMicroSeconds = tempoEntry.timeMicroSeconds + ((eventTimeTicks - tempoEntry.timeTicks) * tempoEntry.mspqn) / this.timeDivision;
        return eventTimeMicroSeconds / 1000000.0;
    }
}

How to call it, added to the test-script-element.html example:

window.onload = function(){
    // configure MIDIReader
    var source = document.getElementById('filereader');
    MidiParser.parse( source, function(obj){
        console.log(obj);

        // Build tempo map from track 0
        const tempoMap = new TempoMap(obj.track[0], obj.timeDivision);

        // Add a timeSeconds attribute for each event on track 1
        let timeTicks = 0;
        obj.track[1].event.forEach(event => {
            timeTicks += event.deltaTime;
            event.timeSeconds = tempoMap.getEventTimeInSeconds(timeTicks);
        });
        document.getElementById("output").innerHTML = JSON.stringify(obj, undefined, 2);
    });
};

The TempoMap constructor builds a list of tempo changes with both tick-based and microseconds-based timing values for each tempo change.

A tempo map entry (record) has three attributes:

  • timeTicks: time of the tempo change in musical ticks, with the tick rate defined by the timeDivision attribute of the MIDI file, for example 480 ticks per quarter note (beat)
  • timeMicroSeconds: wall-clock time position of the tempo change, 0 being the beginning of the song
  • mspqn: tempo (musical speed) value in microseconds per quarter-note (beat). 500000 mspqn means 120 beats per minute.

The getEventTimeInSeconds() method first locates the tempo map entry that's "active" at the requested tick-based instant. It does this with a linear-search array find(), which could be slow with a file with a very large amount of tempo changes. This could be trivially sped up significantly by caching the last-used tempo map index and starting the search from that, instead of from the beginning every time, because events are usually parsed and tempo values queried in linear order. But the idea is there, and it works for tempo changes between events, including lyric meta events.

3
  • 1
    It DOES work indeed :) As far as I can tell, it's an elegant solution, and it was beyond my reach obviously. Thank you very much, this is brilliant. I'm going to try and figure out how & why it works so well.
    – Denis
    Jan 17 at 17:03
  • @Denis Great to hear that you got it working. What kind of an application is this, do you get a lot these lyric MIDI files from somewhere? I had to do some digging to even find a scoring application from which I can export such files. I ended up using GuitarPro 7.5 for making a short test file with notes and lyrics, and with tempo changes in funny places. MuseScore 3's MIDI export didn't seem to save lyric meta events and it was more difficult to place a tempo change in the middle of a note. Jan 18 at 6:38
  • I use Cakewalk by Bandlab. I have a number of original songs I've worked on with a friend, and I had turned all the lyrics to music sheets (melody + text, mostly so that musicians can have a copy of their intrument's music sheet along with the lyrics, to use as a guide). At some point, we thought we'd like to use the lyrics as karaoke or subtitles. So I thought I could extract that data from my files, rather than processing it all manually. Cakewalk happens to have its own programming language, but it seemed to lack the ability to output text for external use. So I switched to javascript.
    – Denis
    Jan 24 at 15:18
3

If you're OK with C#, I can suggest to use my library DryWetMIDI. Your task can be solved easily then:

var midiFile = MidiFile.Read("FileWithLyrics.mid");
var tempoMap = midiFile.GetTempoMap();
var lyricsWithTimestamps = midiFile
    .GetTimedEvents()
    .Where(e => e.Event.EventType == MidiEventType.Lyric)
    .Select(e => new
    {
        Text = ((LyricEvent)e.Event).Text,
        TotalSecondsFromStart = e.TimeAs<MetricTimeSpan>(tempoMap).TotalSeconds
    })
    .ToArray();

lyricsWithTimestamps will be an array where each object contains two properties: Text with lyrics, and TotalSecondsFromStart with timestamp in seconds from the start of the file.

You can learn more about time/length conversion from the library docs - Time and length. Also Getting objects article can be useful too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.