The MIDI Protocol

The MIDI protocol was released in 1982 as a means for connecting electronic musical instruments. First synths to feature the new technology were the Prophet-600 and the Jupiter-6. Although limited in resolution from a recent point of view, it is still a standard for conventional applications - yet to be replaced by the newly released MIDI 2.0. Besides rare mismatches and some limitations, MIDI devices can be connected without complications. Physically, MIDI has been introduced with the still widespread 5-pin connector, shown below. In recent devices, MIDI is usually transmitted via USB.

MIDI jack  (5-pin DIN).

MIDI jack (5-pin DIN).



Standard MIDI Messages

MIDI transmits binary coded messages with a speed of $31250\ \mathrm{kbit/s}$. Timing and latency are thus not a problem when working with MIDI. However, the resolution of control values can be a limiting factor. Standard MIDI messages consist of three Bytes, namely one status Byte (first bit green) and two data bytes (first bit red). The first bit declares the Byte either a status Byte (1) or a data Byte (0).

/images/basics/midi-message.png

Standard MIDI message with three Bytes.


Some of the most common messages are listed in the table below. Since one bit is used as the status/data identifier, 7 bits are left for encoding. This results in the typical MIDI resolution of \(2^7 = 128\) values for pitch, velocity or control changes.

Voice Message           Status Byte      Data Byte1          Data Byte2
-------------           -----------   -----------------   -----------------
Note off                      8x      Key number          Note Off velocity
Note on                       9x      Key number          Note on velocity
Polyphonic Key Pressure       Ax      Key number          Amount of pressure
Control Change                Bx      Controller number   Controller value
Program Change                Cx      Program number      None
Channel Pressure              Dx      Pressure value      None
Pitch Bend                    Ex      MSB                 LSB

Pitch Bend

If you are stuck with MIDI for some reason but need a higher resolution, the Pitch Bend parameter can help. Each MIDI channel has one Pitch Bend, each with two combined data Bytes, resulting in a resolution of \(128^2 = 16384\) steps.


System Exclusive

SysEx messages can be freely defined by manufacturers. They are often used for dumping or loading settings and presets, but can also be used for arbitrary control purposes. SysEx messages can have any length and are not standardized.

Getting Started with SuperCollider

Supercollider (SC) is a server-client-based tool for sound synthesis and composition. SC was started by James McCartney in 1996 and is free software since 2002. It can be used on Mac, Linux and Windows systems and comes with a large collection of community-developed extensions. The client-server principle aims at live coding and makes it a powerful tool for distributed and embedded systems, allowing the full remote control of synthesis processes.

There are many ways of approaching SuperCollider, depending on the intended use case. Some tutorials focus on sequencing, others on live coding or sound design. This introduction aims at programming remotely controlled synthesis and processing servers, which involves signal routing and OSC capabilities.


Getting SC

Binaries, source code and build or installation instructions can be found at the SC GitHub site. If possible, it is recommended to build the latest version from the repository:

https://supercollider.github.io/download

SuperCollider comes with a large bundle of help files and code examples but first steps are usually not easy. There are a lot of very helpful additional resources, providing step by step introductions.

Code snippets in this example are taken from the accompanying repository: SC Example. You can simple copy and paste them into your editor.


SC Trinity

SuperCollider is based on a client-server paradigm. The server is running the actual audio processing, whereas clients are used to control the server processes via OSC messages. Multiple clients can connect to a running server. The dedicated ScIDE allows convenient features for live coding and project management:

/images/basics/supercollider-components.png

Server, client and ScIDE.


sclang

sclang is the SuperCollider language. It represents the client side when working with SC. It can for example be started in a terminal by running:

$ sclang

Just as with other interpreted languages, such as Python, the terminal will then change into sclang mode. At this point, the class library is complied, making all SC classes executable. Afterwards, SC commands can be entered:

sc3>  postln("Hello World!")

ScIDE

Working with SC in the terminal is rather inconvenient. The SuperCollider IDE (ScIDE) is the environment for live coding in sclang, allowing the control of the SuperCollider language:

/images/basics/scide.thumbnail.png

ScIDE


When booting the ScIDE, it automatically launches sclang and is then ready to interpret. Files opened in the IDE can be executed as a whole. Moreover, single blocks, respectively single lines can be evaluated, which is especially handy in live coding, when exploring possibilities or prototyping. In addition, the IDE features tools for monitoring various server properties.


Some Language Details

Variable Names

Global variables are either single letters - s is preserved for the default server - or start with a tilde: ~varname). Local variables, used in functions, need to be defined explicitly:

var foo;

Evaluating Selections

Some of the examples in the SC section of this class are in the repository, whereas other only exist as snippets on these pages. In general, all these examples can be explored by copy-pasting the code blocks from the pages into the ScIDE. They can then be evaluated in blocks or line-wise but can not be executed as complete files. These features help to run code in the ScIDE subsequently:

  • Individual sections of code can be evaluated by selecting them and pressing Control + Enter.
  • Single lines of code can be evaluated by placing the cursor and pressing Shift + Enter

Parentheses

Parentheses can help structuring SC code for live programming. Placing the cursor inside a region between parentheses and pressing Control + Enter evaluates the code inside the parentheses.

(
      post('Hello ');
      postln('World!');
)

Wavetable Oscillator with Phase Reset

The Faust oscillators.lib comes with many different implementations of oscillators for various waveforms. At some point one might still need a behavior not included and lower level approaches are necessary.

This example shows how to use a phasor to read a wavetable with a sine waveform. This implementation has an additional trigger input for resetting the phase of the oscillator on each positive zero crossing. This can come handy in various applications, especially for phase-sensitive transients, as for example in kick drums.

The example is derived from Barkati et. al (2013) and part of the repository:

import("stdfaust.lib");

// some basic stuff
sr = SR;
twopi = 2.0*ma.PI;

// define the waveform in table
ts =  1<<16; // size = 65536 samples (max of unsigned short)
time = (+(1) ~ _ ) , 1 : - ;
sinewave =  ((float(time) / float(ts)) * twopi) : sin;

phase = os.hs_phasor(ts,freq,trig);

// read from table
sin_osc( freq) = rdtable(ts ,sinewave , int(phase)) ;

// generate a one sample impulse from the gate
trig =  pm.impulseExcitation(reset);

reset = button ("reset");
freq = hslider("freq", 100, 0, 16000, 0.00001);

// offset = hslider("offset", 0, 0, 1, 0.00001);

process = sin_osc(freq);
  • Karim Barkati and Pierre Jouvelot. Synchronous programming in audio processing: a lookup table oscillator case study. ACM Computing Surveys (CSUR), 46(2):1–35, 2013.
    [BibTeX▼]
  • NIME 2020: Spatialization

    Virtual Source Model

    Spectral spatialization in this system is based on a virtual sound source with a position an space and spatial extent, as shown in [Fig.1]. The source center is defined by two angles (Azimuth, Elevation) and the Distance. The Spread defines the diameter of the virtual source. This model is compliant with many theoretical frameworks from the fields of electroacoustic music and virtual acoustics.

    /images/NIME_2020/source_in_space.png
    [Fig.1] (1, 2) Virtual sound source with position an space and spatial extent.

    Point Cloud Realization

    The virtual source from [Fig.1] is realized as a cloud of point sources in an Ambisonics system using the IRCAM software Panoramix. 24 point sources can be controlled jointly. The following figures show the viewer of Panoramix, the left half representing the top view, the right half the rear view.


    [Fig.2] shows a dense point cloud of a confined virtual sound source without elevation:

    /images/NIME_2020/panoramix_confined.png
    [Fig.2] Confined virtual sound source.

    The virtual sound source in [Fig.3] has a wider spread and is elevated:

    /images/NIME_2020/panoramix_spread.png
    [Fig.3] Spread virtual sound source with elevation.

    For small distances and large spreads, the source is enveloping the listener, as shown in [Fig.4]:

    /images/NIME_2020/panoramix_enveloping.png
    [Fig.4] Enveloping virtual sound source.

    Dispersion

    In a nutshell, the synthesizer outputs the spectral components of a violin sound to 24 individual outputs. Different ways of assigning spectral content to the outputs are possible, shown as Partial to Source Mapping in [Fig.5]. In these experiments, each output represents a Bark scale frequency band. For the point cloud shown above, the distribution of spectral content is thus neither homogenous nor stationary.

    /images/NIME_2020/dispersion.png
    [Fig.5] Dispersion - routing partials to point sources.

    Back to NIME 2020 Contents

    NIME 2020: Mapping

    Extended DMI Model

    The typical DMI model connects the musical interface with the sound generation through a mapping stage. [Fig.1] shows the extended DMI model for spatial sound synthesis. The joint control of spatial and timbral characteristics offers new possibilities yet makes the mapping and the resulting control more complex.

    /images/NIME_2020/mapping_dmi.png
    [Fig.1] Mapping in the extended DMI model.

    Mapping in Puredata

    We chose Puredata as a graphical interface for mapping controller parameters to sound synthesis and spatialization. Especially in the early stage of development this solution offers maximum flexibility. [Fig.2] shows the mapping GUI as it was used by the participants in the mapping study:

    /images/NIME_2020/patching.png
    [Fig.2] Puredata patch for user-defined mappings.

    Back to NIME 2020 Contents

    NIME 2020: User Study

    User-defined Mappings

    In the first stage of the user study, participants had 30 minutes to create their own mapping, following this basic instruction:

    The objective of this part is to create an enjoyable mapping, which offers the most expressive control over all synthesis and spatialization parameters.

    A set of rules allowed one to many mappings and excluded many to one mappings:

    • Every rendering parameter of synthesis and spatialization must be influenced through the mapping.
    • Control parameters may remain unconnected.
    • A single control parameter may be mapped to multiple synthesizer or spatialization parameters.
    • A synthesis or spatialization parameter must not have more than one control parameter connected to its input.

    Mapping Frequencies

    The mapping matrix shows how some control parameters are preferred for specific tasks, considering the final mappings of all participants:

    /images/NIME_2020/matrix.png
    [Fig.1] Mapping matrix: how often was a control parameter mapped to a specific rendering parameter?

    Back to NIME 2020 Contents

    Physical Modeling: Advanced Models

    More advanced physical models can be designed, based on the principles explained in the previous sections.


    Resonant Bodies & Coupling

    The simple lowpass filter in the example can be replaced by more sophisticated models. For instruments with multiple strings, coupling between strings can be implemented.

    /images/Sound_Synthesis/physical_modeling/plucked-string-instrument.png

    Model of a wind instrument with several waveguides, connected with scattering junctions (de Bruin, 1995):

    /images/Sound_Synthesis/physical_modeling/wind_waveguide.jpg

    References

  • Vesa Välimäki. Discrete-time modeling of acoustic tubes using fractional delay filters. Helsinki University of Technology, 1995.
    [BibTeX▼]
  • Gijs de Bruin and Maarten van Walstijn. Physical models of wind instruments: A generalized excitation coupled with a modular tube simulation platform*. Journal of New Music Research, 24(2):148–163, 1995.
    [BibTeX▼]
  • Matti Karjalainen, Vesa Välimäki, and Zoltán Jánosy. Towards High-Quality Sound Synthesis of the Guitar and String Instruments. In Computer Music Association, 56–63. 1993.
    [BibTeX▼]
  • Julius O Smith. Physical modeling using digital waveguides. Computer music journal, 16(4):74–91, 1992.
    [BibTeX▼]
  • Lejaren Hiller and Pierre Ruiz. Synthesizing musical sounds by solving the wave equation for vibrating objects: part 1. Journal of the Audio Engineering Society, 19(6):462–470, 1971.
    [BibTeX▼]
  • Lejaren Hiller and Pierre Ruiz. Synthesizing musical sounds by solving the wave equation for vibrating objects: part 2. Journal of the Audio Engineering Society, 19(7):542–551, 1971.
    [BibTeX▼]
  • Physical Modeling: Faust Examples

    The functional principle of Faust is very well suited for programming physical models for sound synthesis, since these are usually described in block diagrams. Working with physical modeling in Faust can happen on many levels of complexity, from using ready instruments to basic operations.

    Ready Instruments

    For a quick start, fully functional physical modeling instruments can be used from the physmodels.lib library. These *_ui_MIDI functions just need to be called in the process function:

    import("all.lib");
    
    process = nylonGuitar_ui_MIDI : _;
    

    The same algortithms can also be used on a slightly lower level, combining them with custom control and embedding them into larger models:

    import("all.lib");
    
    process = nylonGuitarModel(3,1,button("trigger")) : _;
    

    Ready Elements

    The physmodels.lib library comes with many building blocks for physical modeling, which can be used to compose instruments. These blocks are instrument-specific, as for example:

    • (pm.)nylonString
    • (pm.)violinBridge
    • (pm.)fluteHead

    Bidirectional Utilities & Basic Elements

    The bidirectional utitlities and basic elements in Faust's physical modeling library offer a more direct way of assembling physical models. This includes waveguides, terminations, excitation and others:

    • (pm.)chain
    • (pm.)waveguide
    • (pm.)lTermination
    • (pm.)rTermination
    • (pm.)in

    From Scratch

    Taking a look at the physmodels.lib library, even the bidirectional utilities and basic elements are made of standard faust functions:

    https://github.com/grame-cncm/faustlibraries/blob/master/physmodels.lib

    chain(A:As) = ((ro.crossnn(1),_',_ : _,A : ro.crossnn(1),_,_ : _,chain(As) : ro.crossnn(1),_,_)) ~ _ : !,_,_,_;
    chain(A) = A;
    

    Karplus-Strong in Faust

    // karplus_strong.dsp
    //
    // Slightly modified version of the
    // Karplus-Strong plucked string algorithm.
    //
    // see: 'Making Virtual Electric Guitars and Associated Effects Using Faust'
    //             (Smith, )
    //
    // - one-pole lowpass in the feedback
    //
    // Henrik von Coler
    // 2020-06-07
    
    import("all.lib");
    
    ////////////////////////////////////////////////////////////////////////////////
    // Control parameters as horizonal sliders:
    ////////////////////////////////////////////////////////////////////////////////
    
    freq = hslider("freq Hz", 50, 20, 1000, 1) : si.smoo; // Hz
    
    // initial filter for the excitation noise
    initial_filter = hslider("initial_filter Hz",1000,10,10000,1) : si.smoo;
    lop = hslider("lop Hz",1000,10,10000,1) : si.smoo;
    
    level = hslider("level", 1, 0, 10, 0.01);
    gate = button("gate");
    gain = hslider("gain",  1, 0, 1, 0.01);
    
    ////////////////////////////////////////////////////////////////////////////////
    // processing elements:
    ////////////////////////////////////////////////////////////////////////////////
    
    diffgtz(x) = (x-x') > 0;
    decay(n,x) = x - (x>0)/n;
    release(n) = + ~ decay(n);
    trigger(n) = diffgtz : release(n) : > (0.0);
    
    
    P = SR/freq;
    
    // Resonator:
    resonator = (+ : delay(4096, P) * gain) ~ si.smooth(1.0-2*(lop/ma.SR));
    
    ////////////////////////////////////////////////////////////////////////////////
    // processing function:
    ////////////////////////////////////////////////////////////////////////////////
    
    
    process = noise : si.smooth(1.0-2*(initial_filter/ma.SR)):*(level)
    : *(gate : trigger(P)): resonator <: _,_;
    

    Waveguide-Strings in Faust

    // waveguide_string.dsp
    //
    // waveguide model of a string
    //
    // - one-pole lowpass termination
    //
    // Henrik von Coler
    // 2020-06-09
    
    
    import("all.lib");
    
    // use '(pm.)l2s' to calculate number of samples
    // from length in meters:
    
    segment(maxLength,length) = waveguide(nMax,n)
    with{
    nMax = maxLength : l2s;
    n = length : l2s/2;
    };
    
    
    
    
    // one lowpass terminator
    fc = hslider("lowpass",1000,10,10000,1);
    rt = rTermination(basicBlock,*(-1) : si.smooth(1.0-2*(fc/ma.SR)));
    
    // one gain terminator with control
    gain = hslider("gain",0.99,0,1,0.01);
    lt = lTermination(*(-1)* gain,basicBlock);
    
    
    idString(length,pos,excite) = endChain(wg)
    with{
    
    nUp   = length*pos;
    
    nDown = length*(1-pos);
    
    wg = chain(lt : segment(6,nUp) : in(excite) : out : segment(6,nDown) : rt); // waveguide chain
    };
    
    length = hslider("length",1,0.1,10,0.01);
    process = idString(length,0.15, button("pluck")) <: _,_;
    

    Physical Modeling: Karplus Strong - Implementation

    plucked_string_444.png


    import numpy as np
    from   numpy import linspace, sin, zeros
    from   math import pi
    %matplotlib notebook
    import matplotlib.pyplot as plt
    from   tikzplotlib import save as tikz_save
    
    from   IPython.display import display, Markdown, clear_output
    import IPython.display as ipd
    import ipywidgets as widgets
    from   ipywidgets import *
    
    
    
    
    fs          = 48000
    L           = 500
    
    
    
    # a function for appending the array again and again
    # arbitrary 300 times ...
    def appender(x):
        y = np.array([])
    
        for i in range(300):
            y = np.append(y,x*0.33)
    
        return y
    
    
    x = np.random.standard_normal(L)
    y = appender(x)
    
    t = np.linspace(0,len(y)/fs,len(y))
    f = np.linspace(0,1,len(y))
    
    fig   = plt.figure()
    ax    = fig.add_subplot(2, 1, 1)
    line, = ax.plot(t,y)
    
    ax2    = fig.add_subplot(2, 1, 2)
    Y = abs(np.fft.fft(y))
    
    Y = Y[0:5000]
    f = f[0:5000]
    
    line2, = ax2.plot(f,Y)
    
    def update(L = widgets.IntSlider(min = 10, max= 1500, step=1, value=500)):
    
        x = np.random.standard_normal(L)
        y = appender(x)
    
        t = np.linspace(0,len(y)/fs,len(y))
        f = np.linspace(0,1,len(y))
    
        Y = abs(np.fft.fft(y))
        Y = Y[0:5000]
        f = f[0:5000]
    
        line.set_ydata(y)
        line2.set_ydata(Y)
    
        fig.canvas.draw_idle()
        ipd.display(ipd.Audio(y, rate=fs))
    
    
    
    interact(update);
    
    <IPython.core.display.Javascript object>
    
    interactive(children=(IntSlider(value=500, description='L', max=1500, min=10), Output()), _dom_classes=('widge…
    

    Karplus-Strong

    Karplus-Strong makes use of the random buffer.

    # this implementation serves for a better
    # understanding and is not efficient
    #
    # - wait for process to be finished in
    #   interactive use
    
    
    fs          = 48000
    L           = 500
    
    # the feedback gain
    gain   = 0.99
    
    # the number of samples used for smoothing
    smooth = 10
    
    
    def karplus_strong(L,gain,smooth):
    
        x = np.random.standard_normal(L)
        y = np.array([])
    
    
        for i in range(96000):
            k   = i%L
            tmp = 0;
    
            for j in range(smooth):
                tmp += x[(k+j) %L]
    
            tmp = tmp/smooth
    
            x[k] = gain*tmp
            y = np.append(y,tmp)
    
        return y
    
    
    y = karplus_strong(L,gain,smooth)
    
    t = np.linspace(0,len(y)/fs,len(y))
    f = np.linspace(0,1,len(y))
    
    fig   = plt.figure()
    ax    = fig.add_subplot(2, 1, 1)
    line, = ax.plot(t,y)
    
    ax2    = fig.add_subplot(2, 1, 2)
    Y = abs(np.fft.fft(y))
    
    Y = Y[0:5000]
    f = f[0:5000]
    
    line2, = ax2.plot(f,Y)
    
    def update(b = widgets.ToggleButtons( options=['Recalculate','Recalculate'],disabled=False),
              L = widgets.IntSlider(min = 10, max= 1500, step=1, value=500),
               gain = widgets.FloatSlider(min = 0.8, max= 1, step=0.01, value=0.99),
              smooth = widgets.IntSlider(min = 1, max= 20, step=1, value=10)):
    
        print(b)
        y = karplus_strong(L,gain,smooth)
    
        t = np.linspace(0,len(y)/fs,len(y))
        f = np.linspace(0,1,len(y))
    
        Y = abs(np.fft.fft(y))
        Y = Y[0:5000]
        f = f[0:5000]
    
        line.set_ydata(y)
        line2.set_ydata(Y)
    
        fig.canvas.draw_idle()
        ipd.display(ipd.Audio(y, rate=fs))
    
    
    
    interact(update);
    
    <IPython.core.display.Javascript object>
    
    interactive(children=(ToggleButtons(description='b', options=('Recalculate', 'Recalculate'), value='Recalculat…
    

    References

  • Vesa Välimäki. Discrete-time modeling of acoustic tubes using fractional delay filters. Helsinki University of Technology, 1995.
    [BibTeX▼]
  • Gijs de Bruin and Maarten van Walstijn. Physical models of wind instruments: A generalized excitation coupled with a modular tube simulation platform*. Journal of New Music Research, 24(2):148–163, 1995.
    [BibTeX▼]
  • Matti Karjalainen, Vesa Välimäki, and Zoltán Jánosy. Towards High-Quality Sound Synthesis of the Guitar and String Instruments. In Computer Music Association, 56–63. 1993.
    [BibTeX▼]
  • Julius O Smith. Physical modeling using digital waveguides. Computer music journal, 16(4):74–91, 1992.
    [BibTeX▼]
  • Lejaren Hiller and Pierre Ruiz. Synthesizing musical sounds by solving the wave equation for vibrating objects: part 1. Journal of the Audio Engineering Society, 19(6):462–470, 1971.
    [BibTeX▼]
  • Lejaren Hiller and Pierre Ruiz. Synthesizing musical sounds by solving the wave equation for vibrating objects: part 2. Journal of the Audio Engineering Society, 19(7):542–551, 1971.
    [BibTeX▼]


  • Contents © Henrik von Coler 2021 - Contact