How do you process instrument samples to make convincing sustain phase loops for sample-based synthesis?

Question

I'm interested in coding up something like a crude sampler (instrument). From what I've read, you can put samples of real audio into a sampler and pitch-shift them for different notes.

I want samples that represent a single note of an instrument I'm trying to mimic. I'm envisioning a note sample as something with an attack phase, a sustain phase, and a decay phase. The sustain phase will be recorded as a cycle that I repeat for an arbitrary length of time; simply repeating a cycle will almost always sound like a fixed pitch and volume.

I'm having trouble getting samples for instruments whose notes often have long sustain phases. Notes can vary in pitch and volume during this phase, so it's hard to get a cycle. I figure one would alter the note's audio to fix its pitch and volume so there is approximately a repeated cycle, but I don't know how I would go about it. Has anyone else who worked with samplers have a method for getting a sustained note?

EDIT 1: After learning about the existence of formants, the pitch-fixing portion of this post seems very iffy right now. If I want to mimic instruments, then I need to model their pitch-invariant formants. Altering the pitch in the audio likely will not preserve those, like if I stretched audio in time to pitch-shift.

Answer 1

What you were initially trying to do might not really be possible with currently known technology. You can probably achieve something useful for some instrument types, but I can't promise what. :) After the discussion in the question's comments, it looks like you were trying to do the following:

• Take an existing full mix audio recording, or at least a recording that's just a performance of a song or a phrase that wasn't played specifically to be used for a sample-based instrument.
• Extract a single note of a single instrument from the recording.
• From that single-note audio, produce a playable high-quality instrument that retains the timbre and expressive characteristics of the original instrument and player (singer perhaps) who produced the original note.

There are many challenges with this. Depending on what sort of rocket science you're aiming for, you'd have to do things like:

• Isolate the frequencies originating from the one instrument/player (section of players maybe, for a "string section" instrument?) in question.
• Isolate the components of the sound, like how much of the sound is breathing noise, how much is vibrations of the instrument. When you pitch-shift, the noise part doesn't necessarily shift, even though it's an elemental part of the instrument's sound.
• Isolate/detect the pitch of the sound. Which note was played?
• Isolate/remove pitch changes resulting from vibrato and other pitch expression
• Isolate the formants of the sound, i.e. the overall big-picture shape of the spectrum of the instrument's sound. Formants are produced by the instrument's (or singer's) body shape, which acts as a resonator, amplifying and attenuating certain characteristic frequencies, which make the sound recognizeable. Every vowel and other sound in spoken/sung languages has a number of formant frequencies which are required for identifying the sound.
• Maybe you need to isolate formant changes, for example in sung vocals?
• Perhaps you need to detect or automatically reverse-engineer types of instruments, such as blown/sustained wind pipes, bowed strings, plucked strings, membranophones, ...

Developing a Model

Whatever you do with the samples, you're going to have two things:

• (1) some kind of a model of the world, and
• (2) a model/architecture for the synth that's supposed to reproduce the phenomena in the world model.

What components does the target sound consist of, and what components does your synth/sampler consists of, and how are they connected?

Your initial idea for the synth model was very simple:

• synth = sample player with a loop

That's such an incredibly simple synth model, it's surprising that any sounds at all could be modeled with it. But it's useful for modeling many real-world instruments such as pianos and drums.

But then let's say you discover formants. A new feature was added to your world model! This new feature will need some changes to the synth model. What's a formant? It is the overall invariant shape of the instrument's sound spectrum, independent of the pitch of the note that's played. The original sample player model cannot reproduce this! In a plain sample player, the overall spectrum moves up and down together with the note pitch. How can you reproduce a pitch-independent formant?

In multi-sampled instruments, reproducing formant behavior relies on having a separate sample for every target note range. In all of the samples, the formant frequencies are "baked in" the PCM samples at the same frequencies, and changing target note switches between samples. As long as the target note ranges are narrow enough, the formants aren't pitched too far away from the original, creating an illusion of there being a formant component in the synth model.

But you want to do this starting with just one single sample, so multi-sampling and sample-switching is not an option. Let's say, you try to reproduce the spectral shape using an array of 5 parametric filters that stay on the same frequencies regardless of what note is played. Low-pass/band-pass/high-pass, etc. Now the revised synth model/architecture becomes

• synth = sample player with a loop + a filter bank

Then you discover the breathing noise issue (so it was probably not a guitarist - you don't care about guitarists breathing), and you want to model noise as an independently controlled separate layer

• synth = sample player 1 with a loop + filter bank
• ... + sample player 2, for breathing noise, with a loop point + another filter bank?

Then let's say you discover vocals and want to have words

• synth = sample player 1 with a loop + filter bank
• ... + some sort of programmable envelope curves changing the filter parameters, so you can do lyrics
• ... + sample player 2, for breathing noise, with a loop point + another filter bank?

(Then you realize that getting the envelope curves just right is extremely difficult, so you ditch that idea.)

Etc. Using sample players as components in a synth's architecture for modeling all sounds is one idea. But there are many, many other models. Basically, each and every sound-producing software has its own model and architecture, and they all have different expressive characteristics. There are many categories and perspectives for classifying synths. Analog modeling synths, physical modeling, PCM sample players, subtractive synthesis, additive synthesis, FM synthesis, ... The idea of using several component samples, and controlling an array of such components with instrument behavior models reminded me of Wallander Instruments. Some sample players have support for formant filters and/or impulse responses, such as NI Kontakt. Convolving a sound with an impulse response recorded from an instrument (or room) can be used for modeling an instrument body, and it should give you at least some of the formants you're after. But you cannot extract an impulse response from a performance! You'll have to specifically record one (somehow). And AFAIK you can't record an impulse response from a person's body. (Interesting idea though)

If you want to experiment with different synth architectures, including sample-based ones, you can use prototyping environments such as Pure Data, Max/MSP or NI Reaktor. Or even samplers, but dedicated samplers won't give you the sort of freedom for architectural experimentation you probably need. Anyway, I encourage you to go ahead and try your wings with it. You'll learn a lot more things very quickly, and you'll make new discoveries. Many of the discoveries will already be known by other people, but the longer you continue, the more likely you will discover something completely new. The most important thing is to keep trying. :)