A Tale of Two MIDIs

One of the considerations for the class I’m teaching at RobotSpeak is that most people using an Arduino as an USB MIDI controller will want to communicate over USB without having to also connect the controller via MIDI ports.  However I’m a purist, and I want my students to be able to use their controllers for any application, including as a hardware controller in a non-computer based setup.  To do this our serial communication will have to be at 31250 baud.  For the hardware side, a single 220 ohm resistor and a 5-pin din MIDI port are all that is needed to connect the Arduino to a MIDI device.  However on the software side, we will need to run a utility that listens to the Arduino USB serial port, and then sends that data to a virtual MIDI port (OSX supports this natively with the IAC bus in Audio/MIDI setup, PC users can use Maple or MIDI/OX) which can then be read from our DAW software.

This is easier said than done.  A quick check of the Arduino Playground shows several different solutions, however many are either no longer available, or only support MIDI note on/off and CC data.  This may work for my RobotSpeak class as it is just an introduction and basic controller class, but plans for my own controller will include emulation of commercially available controllers that use SysEx for channel names and Pitchwheel data for high resolution 14-bit values (like channel volume).  This is a problem that must be addressed in the future.

For my class I have chosen the application made by Spikenzie Labs.  For windows users, S2MIDI is an option, but the Spikenzie app is written in Java (technically made in Processing with the MIDIBus library) and they do offer versions for both operating systems.

With this choice I ran into two problems, the first being that the OSX version of the application would hang when selecting 31250 baud.  I can only assume this means that the Spikenzie hardware that this utility was written for runs at a faster baud rate, and that MIDI communication in the software world can actually go faster than what I thought was a hard limit in the spec.  (Otherwise they would have caught this, no?)  After some furious Google searching I found that someone had solved this.  Apparently this is a limitation in the RXTX Java Serial library, and by adding the fix from the IRQ5 blog I was able to get the Spikenzie converter to work.  Sort of.  This brings me to my second problem, turning the knobs on my controller I was getting erratic results and useless data.  The MIDI equivalent of a garbled message.

When I developed ColorSynths and coded the entire MIDI spec in Assembly (something I try to mention casually in conversations as often as possible), one of my first oversights was the implementation of running status.  Most MIDI data packets are three bytes: one status byte followed by two data bytes.  The MIDI specification requires us to store the most recent status byte, and if a two byte data packet is received without a status byte, we re-use the last one.  This frees up data to allow us to transmit more information, and while it may seem like a more advanced function of the MIDI spec, it can trip up even the most basic of applications if you aren’t aware of it.

As far as I can tell, the MIDIBUS library that Spikenzie’s application uses (and from the contact page I think it’s possible Spikenzie wrote the library) doesn’t support running status.  This of course wouldn’t be a problem if our Arduino code was directly addressing the serial port and only sending 3 byte data packets.  But I had originally written my code for the class using the Arduino MIDI library, which of course uses running status.  The Spikenzie application had no way of recognizing these two byte data packets that lacked a status byte.

I opened up the included source code to get a better understanding of what was going on.  I had assumed that an application like this would be a simple pass through of data from the serial port to the corresponding MIDI port, byte per byte.  Instead what I found was that the Spikenzie labs app is actually handling and interpereting the data, it’s analyzing each byte and waiting for a Note ON/Off or CC status byte followed by two data bytes, and only transmitting when it receives them in that format.  This wasn’t too tough of a fix, I wrote a routine that recognized the entire MIDI spec, including running status and SysEx, in a few hours (nbd).  But because the MIDIbus library doesn’t support the transmission of MIDI using the runningstatus format, the incoming serial data would overflow the outgoing MIDI data (since it was 33% slower).

My eventual solution was to remove the Arduino MIDI library from my source code and to instead manually create 3 byte only MIDI data packets by addressing the serial port directly in the Arduino code.   This solved all problems (that and replacing the RXTX serial library with the one that supports 31250 baud) and gave us working code, but it’s not optimal and our controller could be 33% faster.

Clearly two features need to be added to the MIDIBus library.  The first is support for transmission and reception of running status.  But the second is a way to handle raw data to and from the MIDI port.  I can see the benefits to having a library that creates the MIDI packets for you (that’s why I preferred the Arduino MIDI library to creating the bytes manually) but this application doesn’t need it.  A simple byte-to-byte pass through of the data would eliminate any errors in implementation of the MIDI standard, and avoid complications like Realtime System events (which are single byte and can occur in the midst of a 3 byte event) or SysEx timing.  SysEx is of particular concern to me, because some commercial controllers will send SysEx data and expect a timely response.  The last thing you’d want is to wait for your SysEx handler in the library to get back to you when your code is supposed to be responding.

I’ve written to Spikenzie Labs and hope they’ll update their app, there are no suitable OSX alternatives listed on the Arduino Playground and eventually the correct solution might be to write my own utility that is a simple pass through.  It’s a shame to have to reinvent the wheel though.