Real Time MIDI Processing

This tutorial builds on the setup from "Getting Started". You should be comfortable creating Max for Live devices with js objects in them. Before we get into the details of debugging with the Max Console and using the Live API, let's jump into building something.

We will alter MIDI in real time to change the pitch being played based on a simple set of rules. This means we can play a MIDI clip in Live or physically play some notes on MIDI controller hardware and JavaScript will change the pitches being played as they are occurring.

Here's the idea, unimaginatively called octave-switcher.js:

The first time any pitch repeats between consecutive notes, play it one octave up. The second time a pitch repeats (in a row), play it one octave down. The third time the pitch repeats, repeat the pattern by playing the original octave again, and so on: normal, up, down, normal, up, down, etc. And whenever the pitch changes, the octave resets to normal. For example, if we started with this MIDI clip:

example MIDI clip before processing

It will look like this after our JavaScript processes it:

example MIDI clip after processing

Look closely and notice the notes are slightly delayed (check the left edge of the first note in the two screenshots). That's because real time processing adds delay, referred to as latency in this context. As discussed in limitations, JavaScript is relatively slow and will inherently add some latency. I also had the device open in the Max editor when I recorded the processed MIDI in the screenshot above. Actively editing a Live device with the Max editor adds a lot more latency than using the Max device directly inside Live. So keep that in mind: if you have too much latency and you are editing your Max patch, try saving and closing the Max editor, and test again directly in Live.

Real Time Note Events

MIDI note events have a pitch and velocity values (velocity represents how intensely the note is played), and they come in two general flavors: "note on" events representing the start of a note, and "note off" events representing the end of a note. If you've ever handled keyboard events in a web app, this is analogous to keyDown and keyUp events. In a Live MIDI clip, the left edge of a note in the piano roll is the "note on" event and the right edge is the "note off" event.

note on and off events in Live's piano roll

Our JavaScript code will see a separate "note on" and "note off" event for every note in the piano roll.

It's important to understand this because later we'll be working with notes in non-realtime and we can work with note objects that have a start time and duration. The real time situation is completely different: there is no start time because everything is happening now (from the code's perspective), and we can't know a duration because the "note off" event that determines duration hasn't happened yet (when we're processing "note on" events).

How do we know the difference between a "note on" and a "note off" event? By convention, when velocity == 0 it's a "note off", and when velocity > 0, it's a "note on" event. Negative pitches and velocities are invalid and unlikely to be encountered, so you can ignore them.

Intercepting MIDI Notes Events

Ok, enough theory. Let's figure out how to pass all MIDI data through our Live device while intercepting note events. A fresh Max MIDI Effect device starts with a simple MIDI passthrough that connects a midiin object directly to a midiout object:

default MIDI passthrough behavior in a Max MIDI Effect device

To intercept the note events while continuing to pass through everything else, we need to split apart the raw MIDI data stream into the different types of MIDI events. We do that by inserting a midiparse object after the midiin. We then immediate recombine all the separated MIDI events into a single MIDI data stream with a midiformat object that connects all the outlets from midiparse (with the exception of the rightmost) to its corresponding inlets, and continue to use midiout for the final output, like this:

MIDI passthrough with access to note events

Now is a good time to check that MIDI is in fact passing through the device. Do this:

  1. Save the device if you haven't already, for example: octave-switcher.amxd
  2. Add a MIDI clip to the track containing your device
  3. Add some notes to the clip
  4. Add an instrument to the track if you haven't already
    • Instrument selection tips: We will encounter bugs with stuck notes, so it's useful to choose an instrument that plays long sustained notes so the problem can be easily heard. To test things, we'll play clips with lots of short notes, so a fast attack works better than a slow drone/pad sound. A preset for Ableton's built-in Drift instrument called "Synthetic Xylophone" works well.
  5. Play the clip

You should hear the instrument playing the notes:

MIDI passthrough testing with a clip and instrument

All good? If not, debug it before we go any further.

Back to the patch: The leftmost outlet of midiparse outputs the MIDI note events we are interested in. If you hover over it, you should see a tooltip:

This is telling us that outlet sends two-item lists containing pitch and velocity that represents "note on" and "note off" events. Let's take a look: Add a message object (by dragging from the top toolbar or pressing 'm') and connect the midiparse leftmost outlet to the right inlet of the message object. Play the MIDI clip again and you can see the note data, which is two numbers:

That's the pitch and velocity. This screenshot shows "60 100", which is a "note on" event with a pitch of 60 and a velocity of 100.

We intercept this data by routing it through the js object, like this:

That's the complete Max patch structure, but the data in not passing through the js object because we haven't written any code yet. So how do we pass that data from the js inlet to its outlet? Remember, the input is a two-number list containing the pitch and velocity. In js, we define a handler for list input by defining a list() function, and access the arguments with the arguments array. If you open up the js help file, the example script in there shows examples of handling various types of input. For our case, we want to do this:

  
function list() {
  var pitch = arguments[0];
  var velocity = arguments[1];

  post("pitch=" + pitch + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);
}

After we've read the pitch and velocity, we can output them as a two item list with outlet(0, pitch, velocity);, where 0 is the outlet index (we can configure js with multiple outlets, but that's a topic for later). Note: this is equivalent to outlet(0, [pitch, velocity]); but it's unnecessary and inefficient to create an array like that.

Now with the Max patch editor still open, you can open the Max Console and watch the JavaScript code log the pitch and velocity data when you play the MIDI clip again:

pitch=60, velocity=100
pitch=60, velocity=0
pitch=60, velocity=100
pitch=60, velocity=0
pitch=65, velocity=100
pitch=65, velocity=0
pitch=65, velocity=100

Detecting Note On Events and Repeated Pitches

As discussed above, "note on" events have a velocity higher than zero, which is easily checked in code:

  
function list() {
  var pitch = arguments[0];
  var velocity = arguments[1];

  if (velocity > 0) {
    post("Note on\n");
  } else {
    post("Note off\n");
  }

  post("pitch=" + pitch + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);
}

To detect repeat pitches, we can define a top-level variable outside of the function to remember state between different function calls. At the end of each call to our list() handler, we'll store the current pitch in the previousPitch top-level variable. Then, our logic can check the incoming current pitch is the same as previousPitch to detect repeated pitches.

For the purpose of checking the pitch of consecutive notes, it doesn't make sense to check "note off" events, because they represent the end of an existing note instead of a new note. Therefore, we put the logic to check for repeating pitches inside the conditional block for "note on" events:

  
var previousPitch = null;

function list() {
  var pitch = arguments[0];
  var velocity = arguments[1];

  if (velocity > 0) {
    post("Note on\n");
    if (pitch == previousPitch) {
      post("Repeat pitch: " + pitch + "\n");
    }
  } else {
    post("Note off\n");
  }

  post("pitch=" + pitch + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);

  previousPitch = pitch;
}

We're adding all the post() calls so we can confirm things are working as intended in the Max console logs. Does it look like what you expect?

Note on
pitch=60, velocity=100
Note off
pitch=60, velocity=0
Note on
Repeat pitch: 60
pitch=60, velocity=100
Note off
pitch=60, velocity=0
Note on
Repeat pitch: 60
pitch=60, velocity=100

Cycling Octave States

Let's define some top-level constants (still defined using var because this is an old version of JavaScript) to represent the normal pass-through, octave up, and octave down states that we'll cycle through on repeat pitches:

var PASS_THROUGH = 0;
var OCTAVE_UP = 1;
var OCTAVE_DOWN = 2;

And add a top-level state variable to store the current state, which defaults to the normal pass-through:

var state = PASS_THROUGH;

When a repeat pitch is detected, we want to cycle through the states like this:

if (state == PASS_THROUGH) state = OCTAVE_UP;
else if (state == OCTAVE_UP) state = OCTAVE_DOWN;
else state = PASS_THROUGH;

This can be expressed much more succinctly as:

state = (state + 1) % 3;

but I think the more verbose version is much clearer about the intent, so for the purpose of this tutorial, I'll keep using it for clarity.

All those changes together look like:


var PASS_THROUGH = 0;
var OCTAVE_UP = 1;
var OCTAVE_DOWN = 2;

var state = PASS_THROUGH;
var previousPitch = null;

function list() {
  var pitch = arguments[0];
  var velocity = arguments[1];

  if (velocity > 0) {
    post("Note on\n");
    if (pitch == previousPitch) {
      if (state == PASS_THROUGH) state = OCTAVE_UP;
      else if (state == OCTAVE_UP) state = OCTAVE_DOWN;
      else state = PASS_THROUGH;
      post("Repeat pitch " + pitch + ", state → " + state + "\n");
    }
  } else {
    post("Note off\n");
  }

  post("pitch=" + pitch + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);

  previousPitch = pitch;
}

Continuing to test with the logs:

Note on 
pitch=60, velocity=100 
Note off 
pitch=60, velocity=0 
Note on 
Repeat pitch 60, state → 1 
pitch=60, velocity=100 
Note off 
pitch=60, velocity=0 
Note on 
Repeat pitch 60, state → 2 
pitch=60, velocity=100 
Note off 
pitch=60, velocity=0 
Note on 
Repeat pitch 60, state → 0 
pitch=60, velocity=100

Ok! Let's actually change the octave based on the state:


var PASS_THROUGH = 0;
var OCTAVE_UP = 1;
var OCTAVE_DOWN = 2;

var state = PASS_THROUGH;
var previousPitch = null;

function list() {
  var pitch = arguments[0];
  var velocity = arguments[1];

  if (velocity > 0) {
    post("Note on\n");
    if (pitch == previousPitch) {
      if (state == PASS_THROUGH) state = OCTAVE_UP;
      else if (state == OCTAVE_UP) state = OCTAVE_DOWN;
      else state = PASS_THROUGH;
      post("Repeat pitch " + pitch + ", state → " + state + "\n");
    }

    if (state == OCTAVE_UP) pitch += 12;
    else if (state == OCTAVE_DOWN) pitch -= 12;

  } else {
    post("Note off\n");
  }

  post("pitch=" + pitch + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);

  previousPitch = pitch;
}

Handling Note Off Events

Most of the repeat pitch detection and octave-switching logic is in place, but there is a major flaw. Earlier we were discussing how every note comes in a "note on" + "note off" event pair. A key aspect of "note off" events is they always have the same pitch as their corresponding "note on" event. Whenever a "note off" occurs, it ends whatever "note on" was playing at that pitch.

See the problem? We are only changing the pitch for "note on" events, so the notes can no longer end properly.

We can investigate this by creating another MIDI track, setting its input to the track with our Max for Live device, and recording the MIDI output from our device. When you setup your track for recording, be sure to select "Post FX" in the input / "MIDI From" settings. If you select "Pre FX", it will bypass the Max for Live device. Don't forget to arm the track for recording:

The recorded notes should be the same duration as the original MIDI clip. In my case all original notes were short 16th notes. Some of the notes are playing for way too long. The long notes are the ones where we changed the pitch. You may even have stuck notes. If that happens, hit the stop button in Live's transport to send an "all notes off" message to everything.

To fix this, we need to keep track of how we are mapping "note on" pitches from their original pitch to the modified pitch, and apply the same mapping for "note off" events. A basic JavaScript object works as a map data structure:

var noteOffMap = {};

To keep track of whether the pitch is changing, it's convenient to introduce another variable for the incoming pitch that will not change:


function list() {
  var pitch = arguments[0];
  var originalPitch = pitch;

Then, after we might have changed the pitch, we check if we actually changed the pitch for this note. If so, we then keep track of how we mapped the original pitch to the modified pitch in our noteOffMap data structure:


if (state == OCTAVE_UP) pitch += 12;
else if (state == OCTAVE_DOWN) pitch -= 12;

if (pitch != originalPitch) {
  noteOffMap[originalPitch] = pitch;
}

Now we are ready to implement the "note off" behavior. If the pitch was modified for a "note on", make the same modification to the corresponding "note off". We don't want to keep applying the modification, so we also remove this entry from the map:


} else { 
  post("Note off\n");
  var modifiedPitch = noteOffMap[originalPitch];
  if (modifiedPitch != null) {
    pitch = modifiedPitch;
    noteOffMap[originalPitch] = null; // remove the mapping
  }
}

To help test all this new logic, I updated the main log line to show when a pitch is being changed:


post("pitch=" + 
  (pitch == originalPitch ? pitch : originalPitch + " → " + pitch) + 
  ", velocity=" + velocity + "\n");

At this point there's still a bug and this tutorial is getting long, so I'll jump to the solution. I encourage you to experiment and record the output of the device again and make sure you understand why everything is happening the way it is. Add more post() logging if you need to.

Anyway, the remaining bug is we want to detect repeated pitches with respect to the input, but we sometimes change the value of pitch, and then we end our function with previousPitch = pitch;. The problem is pitch is no longer the input pitch if we changed it. The solution is to ensure we use the original input pitch with: previousPitch = originalPitch;

Wrapping Up

Here's all the changes for handling "note off" events properly and the previousPitch bug fix. I cleaned up, removed most of the logging and added a few comments. This is the final version of octave-switcher.js:


var PASS_THROUGH = 0;
var OCTAVE_UP = 1;
var OCTAVE_DOWN = 2;

var state = PASS_THROUGH;
var previousPitch = null;
var noteOffMap = {};

function list() {
  var pitch = arguments[0];
  var originalPitch = pitch;
  var velocity = arguments[1];

  if (velocity > 0) { // handle note on:
    if (pitch == previousPitch) { // cycle state for repeated pitch:
      if (state == PASS_THROUGH) state = OCTAVE_UP;
      else if (state == OCTAVE_UP) state = OCTAVE_DOWN;
      else state = PASS_THROUGH;
    } 
    else { // reset state for changed pitch:
      state = PASS_THROUGH;
    }

    if (state == OCTAVE_UP) pitch += 12;
    else if (state == OCTAVE_DOWN) pitch -= 12;

    if (pitch != originalPitch) {
      noteOffMap[originalPitch] = pitch;
    }
  } 
  else { // handle note off:
    var modifiedPitch = noteOffMap[originalPitch];
    if (modifiedPitch != null) {
      pitch = modifiedPitch;
      noteOffMap[originalPitch] = null;
    }
  }
  // post("pitch=" + (pitch == originalPitch ? pitch : originalPitch + " → " + pitch) + ", velocity=" + velocity + "\n");

  outlet(0, pitch, velocity);

  previousPitch = originalPitch;
}

Next steps

The next tutorial, "The Max Console", shows how to better use post() and the Max console to inspect and debug JavaScript programs running in Max for Live. Tired of ending every post() call with "\n" ? Read on!