12

During the last years I tought myself to write scores using LilyPond and furthermore to make file handling easier by using Bash scripts. Recently I started a project a little bigger than only one or two pages and once again I came across the article about Makefiles in the LilyPond Documentary.

Even though reading this article it was kinda hard for me to grasp the actual technique behind the template Makefile also because the article does not supply the file repository it is working with.

So I though to take the time to make up an example project to ask, how a Makefile in this scenario could look like. How I would build it step by step and how I actually run the Makefile. (Edit: referring to a make man page I got my understanding as far as being able to say that one uses make as interpreter like bash on a shell-script.sh. The command then simply looks like make -f Makefile run in the root directory of the project.)


The Project has a file-structure like this:

├── Book.ly
├── Book.pdf
├── global-files
│   ├── copyright.ily
│   ├── Frontpage.ily
│   ├── header.ily
│   └── paper.ily
├── input-files-voiceI
│   ├── Nr_01-voiceI.ily
│   ├── Nr_02-voiceI.ily
│   └── Nr_03-voiceI.ily
├── input-files-voiceII
│   ├── Nr_01-voiceII.ily
│   ├── Nr_02-voiceII.ily
│   └── Nr_03-voiceII.ily
├── README.md
├── single-pages-voiceI
│   ├── MIDI
│   │   ├── Score-Nr_01-voiceI.midi
│   │   ├── Score-Nr_02-voiceI.midi
│   │   └── Score-Nr_03-voiceI.midi
│   ├── PDF
│   │   ├── Score-Nr_01-voiceI.pdf
│   │   ├── Score-Nr_02-voiceI.pdf
│   │   └── Score-Nr_03-voiceI.pdf
│   ├── Score-Nr_01-voiceI.ly
│   ├── Score-Nr_02-voiceI.ly
│   └── Score-Nr_03-voiceI.ly
├── single-pages-voiceI_a_II
│   ├── MIDI
│   │   ├── Score-I_u_II_Nr_01.midi
│   │   ├── Score-I_u_II_Nr_02.midi
│   │   └── Score-I_u_II_Nr_03.midi
│   ├── PDF
│   │   ├── Score-I_u_II_Nr_01.pdf
│   │   ├── Score-I_u_II_Nr_02.pdf
│   │   └── Score-I_u_II_Nr_03.pdf
│   ├── Score-I_u_II_Nr_01.ly
│   ├── Score-I_u_II_Nr_02.ly
│   └── Score-I_u_II_Nr_03.ly
└── single-pages-voiceII
    ├── MIDI
    │   ├── Score-Nr_01-voiceII.midi
    │   ├── Score-Nr_02-voiceII.midi
    │   └── Score-Nr_03-voiceII.midi
    ├── PDF
    │   ├── Score-Nr_01-voiceII.pdf
    │   ├── Score-Nr_02-voiceII.pdf
    │   └── Score-Nr_03-voiceII.pdf
    ├── Score-Nr_01-voiceII.ly
    ├── Score-Nr_02-voiceII.ly
    └── Score-Nr_03-voiceII.ly

The input-files of both voices have a format like this:

\relative c {
  \clef bass
  \time 3/4
  \key c major

  c4( d e f      | %01
  g1) \bar "|."  | %02
}

The Score files have the purpose to be compiled for output of PDF and MIDI. and simply look like this (despite the fact that the Scores for two systems contain another Staff):

\version "2.18.2"

#(set-default-paper-size "a4")
#(set-global-staff-size 22)

\include "../global-files/header.ily"

\score {
  \new StaffGroup = "" \with {
    instrumentName = \markup { \bold \huge { \larger "1." }}
  }
  <<
    \new Staff = "celloI" \with { midiInstrument = #"cello" }

    \include "../input-files-voiceI/Nr_01-voiceI.ily"
  >>
  \layout {}
  \midi {}
}

The Book part is the part I'm still pretty unhappy with. I would prefer this pretty simple with just using the Score*.ly files as \includes, but I get problems with the \includes that are already in the Score.ly files, since they do not only contain the \score block to be compileable by them selfs.

Well I could use a \book with setting a book output name like \bookOutputSuffix "OutputName", but then my Book.ly would become a massive file, taking quite a long time to be compiled, even for a small change on a single piece.

So right now my Book.ly file has the following format and the only purpose to compile the whole book with two voices in two staffs, but with all pieces, here 01-03:

\version "2.18.2"

#(set-default-paper-size "a4")
#(set-global-staff-size 22)

\include "./global-files/paper.ily"

\book {

  \include "./global-files/Frontpage.ily"

  %%%% Score Number: 1 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "1." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_01-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_01-voiceII.ily"
    >>
    \layout {
      \printTupletBow
    }
  }

  %%%% Score Number: 2 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "2." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_02-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_02-voiceII.ily"
    >>
    \layout {}
  }

  %%%% Score Number: 3 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "3." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_03-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_03-voiceII.ily"
    >>
    \layout {}
  }
}

My workflow is the following:

  1. I write the input files: input-file.ily
  2. I run a bash-script.sh that creates the compileable Score.ly files from the input-files/*.ily
  3. I run a bash-script.sh that creates the compileable Book.ly file from the input-files/*.ily
  4. I compile the Score.ly files one by one or run a simple for file in *.ly; do lilypond "$file"; done loop, but in each of the three Score directories. I use a script to move the PDF and MIDI files into their corresponding folders.
  5. I simply run lilypond to compile the Book.ly file.

DONE


The actual project this question is asked for can be found here on GitHub


Update 1:

My system:

    Operating System: Debian GNU/Linux bullseye/sid
              Kernel: Linux 5.3.0-2-686-pae
        Architecture: x86
        GNU LilyPond: 2.18.2
My Editor - GNU Nano: 4.5
      Guake Terminal: 3.6.3
            GNU Make: 4.2.1

I added my shell-scripts to a separate Git-Repository


Update 2:

This is a very simplified chart of the dependencies. Assuming there was only one voice:

./infiles/                        
  infile{01..03}.ily -------------> ./Book.ly ===> Book.pdf
     |                                 ^ ^ ^
     *---------> Scores{01..03}.ly === | | |=====> Score{01..03}.pdf
                      ^  ^         === | | |=====> Score{01..03}.midi
                      |  |             | | |
./global-files/       |  |             | | |
  header.ily    ------*  |             | | |
  copyright.ily ---------+-------------* | |
  Frontpage.ily -------------------------* |
  paper.ily     ---------------------------*
  • 1
    Do you have the bash-script.sh files available anywhere? I'm having some difficulty grokking the whole procedure. – luser droog 2 days ago
  • 2
    Also do inform abt OS and your editor of choice – Rusi 2 days ago
  • @Rusi I updated my question at the very bottom, with all relevant system INFOs – nath 2 days ago
  • 1
    FYI: I updated my answer to provide some more context and a brief technical description of the mechanics. – wchargin yesterday
13

Invocation

make will search the current directory for a file named Makefile or makefile, so it's often simplest to name it one of these two choices and then invoke with the simple command:

$ make

If you use the uppercase 'M' then the file will usually be listed at the top according to alphabetical or collation order.

Rules

make operates by using rules for how to create an output from an input. If you don't specify a target on the command line, it will invoke the first rule it encounters. So a common technique is to make the first rule a "dummy" rule which doesn't produce a file but simply collects all the steps or outputs together. Eg.

all: output1

Invoking make with this makefile will try to create output1 if it doesn't exist. If there's a rule for creating output1 later in the makefile it will use that.

For your case, I suggest making a top level rule to create Score.ly and Book.ly

all: Score.ly Book.ly

Pattern rules

To replace your shell loop, you can use a pattern rule.

%.pdf: %.ly
    lilypond $^

This rule says: To make a .pdf file, run lilypond on the corresponding .ly file.

Note that the command to execute must start with a literal TAB character. The %^ variable refers to the input file mentioned above. Other useful variables are $@ for the target, and $< for the first input if there are more than one.

This handles just part of your shell loop, defining the transformation from an input file to an output file. For the other task of generating a list of files, there are some special variables available in GNU make for this.

inputs= $(notdir $(wildcard ./*.ly))
bases= $(basename $(inputs))
outputs= $(patsubst %,%.pdf,$(bases))

After these definitions, you can use $(outputs) as a dependency in a rule, like:

pdfs: $(outputs)
    mkdir -p PDF
    mv *.pdf PDF
    mkdir -p MIDI
    mv *.midi MIDI

I actually don't know how to inform make that lilypond will create both a PDF and a MIDI file, so this rule just handles the MIDI files with extra commands. Again, each of these commands must be indented with a bona fide ASCII TAB character.

(I have more to add, but I wanted to get a head start.)

  • That didn't work. Trying <tty><pre>. – luser droog 2 days ago
  • 2
    Works now. However a semantically loaded initial tab is so unnatural, not just to humans but even well-meaning editors that a stronger caveat may be in order?? – Rusi 2 days ago
  • 1
    True. That's how make has always worked. But I'll try to flag it more loudly and more often as I fill out the answer. – luser droog 2 days ago
  • 1
    I found this in the LilyPond Docs looks like the way to handle both formats PDF and MIDI, as far as I can tell from my bash knowledge it also looks like they use an if statement to find out weather the files exist or not and then run the mv command on it. – nath 2 days ago
  • Instead of lilypond $^, I would use lilypond $<. This lets you use \include and specify the dependencies in the makefile. – Dietrich Epp yesterday
3

luser droog’s answer gives a good overview of make itself. Here’s an example of how to apply that to a real-world Lilypond project with the following characteristics:

  • Multiple parts: piano, bassoon, etc.
  • Multiple movements.
  • Multiple desired outputs: a single “master” (conductor) score, a master score for each movement, and a part score for each instrument.
  • Both PDF and MIDI outputs.
  • Common definitions (macros) for both music and presentation.

For instance, if your Book.ly file includes your Nr_01-voiceI.ily file, which in turn includes your macros.ily shared definitions file, then if macros.ily is changed, your Makefile needs to know that it needs to recompile Book.ly to update Book.pdf.

On the othre hand, if you change just one part, it is very convenient to be able to write make parts and have Make only recompile the part that was changed, rather than wasting time recompiling all the others. And, if you just want to listen to a segment, you should be able to make midi to skip the slow typesetting and only re-do the necessary sequencing.

I’ve written a shell script to generate a Makefile that has all these properties. In particular, it automatically traverses the \include graph of your project to figure out which Lilypond files depend on which others, and will incur the minimal set of recompilations for any particular change. You just tell it what “main files” you have, and which ones you want in PDF and/or MIDI form, and it will do the rest.

Here’s the script: https://github.com/MutopiaProject/MutopiaProject/blob/918971593735f2dbf4864f289767b8d59a7d950e/ftp/MozartWA/KV488/Mozart-KV488/Mozart-KV488-lys/create_makefile.sh

Here’s an example of the output Makefile: https://github.com/MutopiaProject/MutopiaProject/blob/918971593735f2dbf4864f289767b8d59a7d950e/ftp/MozartWA/KV488/Mozart-KV488/Mozart-KV488-lys/Makefile

The core mechanism is to create a secondary target for each Lilypond file, which I’ve called its “lydep” (“Lilypond dependencies”) file, such that a file’s lydep is dirty if and only if any of the file’s transitive sources is dirty. In practical terms, this means that each lydep target needs to depend on its source file plus all the lydep targets on which it has direct \include dependencies. Then, Make’s automatic resolution takes care of the rest.

The script is tailored to a particular project, but you should be able to keep the core infrastructure in place and customize the makefile function at the bottom to substitute in your project structure.

I’ve written this script for GNU/Linux, and it might need some minor adjustments for macOS/BSD (e.g., swapping out readlink -f), but it’s pretty straightforward overall. It does require bash and GNU make (for secondary targets). I see that you’re on Debian, so it should run just fine as is.

This is all released under the MIT License. Please feel free to take whatever you find helpful.

  • yuhu man, this is quite a couple of lines to read through. Thanks for the answer, and for sharing your work! <3 – nath yesterday
  • 1
    Sure thing. I’ll see if I can explain/summarize the gist of it when I get a chance later today. – wchargin yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.