Monday, 14 October 2013

USB controllers in PD

Todays blog post is about a deceptively simple subject which can instantly expand your capabilities in Pure Data, Human Interface Devices or HID.

This was one of the things to excite me about Pd usage in the early days because it introduces what could be termed the 'alchemy' of  computer programming. Simply termed, we can turn a kind of thing with a specific purpose, such as a joypad, into the same thing, with a very different purpose, a musical instrument.

So today I'm working with this rather simple setup:



Now, that's a nice, cheap, third-party USB joypad, I bought for £15, but if you fish around online you can find cheaper, still. Now, almost all game controllers for computers use a set of standards known as HID (Human Interface Device). Which strangely enough is also the name of the object which we can use to get this information into Pd.

Try this patch (remember to plug in a USB controller before opening Pd):

Step 1: Finding your HID device



This is a simple patch to start with, because hid is quite complicated there's a few different messages you can feed into the top. The message 'print' when you haven't told it which device to listen to yet will print out a list of available hid devices in your Pd terminal, looking something like this...


Now, we can see a list of devices, and that hte USB joystick (curiously named MY-POWER when the name ORB is printed on the pad itself) is Device 12. You may also see keyboards, mice, graphics pads etc on this list.

Note that next time you plug in your joypad (or other kind of controller) it may not have the same device number, so you might have to go through this process again when you open a saved patch.

The next step is to open this device and see what information Pd can read from it...

Step 2: Look at the device, having found it



Now, clicking on 'open 12' will tell hid to read the device we found in the previous step, and then the purpose of 'print' has changed. Instead of listing devices Pd can see, it will tell you what parameters it can read from that device. So clicking 'print' will result in a readout like this one...


Now, you can see that we have messages which Pd can get from the controller, 12 btn messages (which will send 1 when a button is pushed, and 0 when it is released) and 7 absolute axes, which are sliding-scale type messages. If, having opened a hid device, we feed hid a 1 (usually from a toggle, but you could feed it in from a message, too) it will start polling (basically, listening to) these events from the controller.

Let's try setting this to poll, then feeding the results out to the terminal via an object called print (not to be confused with print from the message, mentioned above.

Step 3: have a look at what's available 



All we've done here is pipe the output of hid into print, which puts everything it sees onto the terminal, so when I push some buttons on my controller we see something a little bit like this...


Now, this is just a list of messages, but we can see that there's a certain structure there. Firstly there's three things separated by a space, which Pd reads as a list. Firstly the contents of the TYPE column we saw above, then the name of that parameter, then a value of 0 or 1.

There's also a miscellaneous scan parameter, which seems to identify the button being used. I don't really know what this is for, but I know we can safely ignore it.

You'll also notice that every message (1 for on, 0 for off) is sent twice. This is a problem known as 'debouncing' which some controllers are prone to. There are ways around this, but for everything we're going to do here, it won't matter.

The next thing we need to know in order to make this information useful is how to just get the last part of the list, for just one button. For this we're going to use route. All route does is list (without that first item), and everything else out of it's right-hand outlet.

If route has more than one argument, it will gain additional outlets, if the first item in an incoming list matches the first argument, it'll be output via the first outlet. The second argument, the second outlet, and so on...

Step 4: Use route to focus on the controls you're interested in.



So all I've done here is route the lists starting with 'key' to the second route, where it takes the messages btn_0, btn_1, btn_2 and btn_3 and outputs them to four toggles. When I took this screenshot I was holding one of those buttons, so you can see that one toggle is selected.

And that's basically all you need to know about getting (or Mapping) inputs from a hid device into Pd. But we don't want to stop there, what about those joysticks? and the d-pad? Well, with the same process we can route their inputs into our patch, too.

Step 5: A few more mappings...



So there you have it, both of the joysticks output two axes, and moving them around I can easily learn that they range from 0 (at the top, or far left of it's range of motion) to 255 (at the bottom, or far right) and the resting position has both at 128.

The d-pad (direction control on the left hand side of the pad) also outputs two axes of motion, although these are only ever -1 (up and left), 0 (resting) or 1 (down and right).

In the snapshot above I was holding one stick to the bottom-right and also the d-pad.

I think it's time for a little alchemy, let's hide the patch so far in a sub-patch (see Putting Pure Data in a Box) and use some of these controls to make music.

Step 6: Mapping controllers to a simple synth




Now there's a few things here from previous patches, firstly the object 'pd controller' contains the hid patch we've been working on, only I've added a few sends to it (named b1 for button 1 etc. lx for the left stick's x axis etc.). As you can see here:


The patch outside it contains a few things from previous tutorials, an array with 15 numbers (representing two octaves of the major scale), a maxlib/scale which converts the 0 to 255 we have to the 0 to 15 we want. This goes through i to make sure it's an integer (whole number) and to tabread, which reads a number from the array, scale. we add 50 then smooth out the edges with a line~. This is converted to a frequency with mtof~ and feeds an oscillator.

I've then brought in one of my buttons (b1) to turn the volume on or off for the oscillator, multiplied it down a little and fed it to the sound card with dac~

Once again I've described each part of a Pd chain to tell you it's function, but if you haven't read all my other blog posts, this basically turns the left-right motion of the left joystick into a pitch control, in a given scale, and one of our buttons into an on-off switch. This is a simple musical instrument. With a little practise you can play a tune on this.

Now, just for fun, I'm going to use the up-down motion as a volume control and lengthen the envelope on the button control...


So the left stick is taking on the duties of a bit more expression. This also opens the patch up to something of a money shot. Why not have a synth on each of these 4 buttons, an octave apart? One of the strengths of Pd is to quickly duplicate and modify code, so let's try it...

Step 7: A modified joypad quartet



Now, I've done a few things here. Firstly I've moved that volume control to the top of the patch, and sent it to the bottom part of the patch with an audio send (s~ vol), then I've duplicated the bottom part of my patch 4 times. I added 12 to each successive midi note (making it an octave higher), then I used a different button send to turn each one on and off. Hey presto, I can play at a different octave by pressing each of the four buttons.

I also changed all the osc~ objects to phasor~ for a different tone, and lowered the initial + to make the lowest tone a bit lower.

Now is probably the time for a little demo, but first, I should warn of some of the pitfalls of working with cheap and cheerful gaming controllers.

Firstly, you'll find the range of motion on joypads is much narrower than you think. That is, the range which isn't the extreme top or bottom end, or dead centre only takes up a small amount of the space you can move the joystick.

Also, you'll probably trip up over the fact that the top of a stick's motion is 0, and not the bottom.


Well, that range of motion issue has made it a little limited, and it is a very simple instrument, but here it is. My custom joypad instrument and a little exploration of the sounds it's making...



Monday, 30 September 2013

Using Midi, and Pure Data, and mashing up sound

It's been too long, readers, since I posted anything to blogger. I've been busy, and sadly the blog is one of the first things to go, even when I'd just reached the point of putting gifs on a post

Well, this time round I thought we'd have some fun with audio, so let's start with a short clip of sound like this one. I'd download it, it's only two seconds long.


For those interested, this is from the Teabot and it regularly does a good job of informing me and my colleagues that there's a cuppa ready for the drinking.

Let's try analysing the sound the sound a little, it's short, contains three distinct notes, two of which are part of the same syllable, and has quite a hard 'st' sound in the middle of it.

There isn't much more to say about what we're hearing, but what about how the computer makes the sound. As I've said before digital audio is actually a long stream of numbers, each between 1 and -1, which are played at a rate of thousands per second (44.1 thousand, usually). These are interpreted as how far a speaker membrane is pushed or pulled (either side of 0, the resting position) which produces the same pattern as a pressure wave in the air, which you can then hear as sound.

I've also said before that an array in Pure Data holds a list of numbers, which you can then feed into your sound card, to control your speakers.

You can probably see where I'm going with this, if we feed all of the numbers (known as samples) from this sound file into an array in Pure Data, we can then control that sound in a similar way to how we control synthesizers. So, first things first, how do we get it into an array? Try This.

Part 1: Loading sounds in to a Pd array


So, there's an array, named tea, containing my tea sample, but how did it get there?

Once again, let's follow the Pd chain from the top. First is a bang, a button which outputs an activation message, which goes to openpanel. This brings up your usual file opening dialog, when you choose a file in there, it outputs the path to that file (e.g. /home/ajfaraday/Downloads/tea.wav), which is then being passed to a message. As discussed earlier, message boxes send a message down the line, and substitute placeholders (marked with $1, $2, etc.) with whatever is placed in the top, so $1 here will be replaced with the file path. The message itself is fairly simple.

read... resize... the file... the name of the array

this will read the named file, and put it in the array, tea, and re-size it to the size of the file. But only when it is passed to the next object, soundfiler. That's pretty much what soundfiler does, dump sound files into arrays.

So, after clicking the bang and choosing the file, we have it all in an array, it even looks a bit like the waveform you can see on soundcloud (above). Now, what can we do with it?

Step 2: Simply playing a sound


Pure Data can be very simple some times, when we want to play the contents of an array (or table), we can use tabplay~ (as ever, the tilde, ~, means this is an audio object). This, with the name of the array as an argument, plays the sound when it is given a bang. I'm then feeding it into a volume control (*~) and out to my sound card with dac~.

This is very simple, and offers very little control of how the sound is being played, only when.

As I discussed way back in Pure Data and BACON (Part 1), you don't have to keep hitting bangs manually, you can automate them with a metro object and control how regularly that outputs its bang with a number.

Part 3: Automating audio triggers, a simple stutter effect


You'll find what this start's to have an incessant, droning quality, mostly because you're hearing that first (not very well produced) note over and over again. You can have fun with moving that number down to make a very quick repeat (if it's low enough it won't even reach the start of the word), and then moving it back up, in what's known as the 'bouncing ball' effect. This kind of fast audio re-triggering is used by quite a few DJ's, if you listen out for it.

However, this all the time, with no changes, gets difficult to listen to. Let's try taking control of what's going on here with two simple facts, tabread~ can accept a number, which tells it which sample to start on, and any number you can get into Pd, such as MIDI controls, can control anything.

Part 4: Adding position to our stutter


So what're we doing here? Well, I've got the data from the midi keyboard coming in via notein, which gives me the most recently pressed key as a number between 1 and 127. On the two octaves of my small midi keyboard, these are between 48 and 72.

In MIDI Controllers in Pure Data I used spigot to filter out the note-off signals from the keyboard, on this occasion I'm using stripnote, which does the same thing, it's just a little lazier for me to do it that way.

I'm then using maxlib/scale (as I did in How To Make A Musical Instrument in 10 Minutes) to change that range to between 0 and 88,200. Which seems a large, scary, and quite specific number. You might wonder where it's come from. Well, back to the idea of digital audio as a stream of numbers. CD quality audio is usually at 44.1 kHz. Meaning that 44.1 thousand of these numbers are in every second of audio. So, for a two second clip, we need 88.2 thousand, or 88200.

That's a bit of a cheat, not every clip of audio is 2 seconds long. You can find the exact length of our resized array by right-clicking on it and selecting properties. Or, within your patch using arraysize, like so (you may want to do this in future to avoid having to type the length of an array manually):

So we've now taken the range of our keyboards midi numbers and transformed it into the range of samples in the array (give or take a few).

Now, float boxes (f) simply store a number, and if a bang is placed in the left-hand inlet, it will output that number, time and time again. So when we press a key, it changes the number it's storing to a position within the array and whenever the metro fires, the tea sound is played from that point.

When you start the metro, you can press keys and effectively play the different parts of the sound on your keyboard! Like so:



Okay, so it's not the best video, and the screen flickers (this is to do with frame rates, but don't worry about it, just don't, usually, point a camera at a tv or computer screen), but you get the idea. I'm taking direct control of the position of playback from my keyboard.

Well, this is a little limited. But I we can use some of my earlier teaching in Pure Data to take control of other parts of the sound, how loud it's playing, how fast the metro is and when to start and stop the metro direct from the keyboard. Take a look at this...

Part 5: Some more direct controls


I've added comments for which midi controls are entering via which ctlin box (if you don't know what these are, have a look at my earlier post on MIDI) so, from the top-left, lets look at what they're doing.

The first control, labelled 'button', is a button, when it's pushed it alternately outputs 0 or 127, which I've fed into a bang, so every push will be treated the same way. That then feeds a toggle, which will turn the metro on and off. (I may have said before, turning it off is a very valuable feature).

The rotary control is turned, producing figures from 0 to 127, which I've scaled to 0 to 600 which control the speed of the metro re-triggering in milliseconds. Then there's the keys, which I've described in the last part.

Then the mod wheel has been used (via line~ to smooth out the rough edges) to control the volume.

So I can start and stop the sound, control how fast it triggers, where in the sound it starts and how loud it is, all with tactile controls on the keyboard, and I don't have to touch my computer. The only major flaw so far is that the sound won't start when you press a key, as you might expect.

To do this, I'm going to take a send (s) from the key input and send it to the float (f).

Part 6: Triggering with the keys as well...


There's a lot in there, now. It's just starting to get to the point where Pure Data patches get a little difficult to follow. But we should reflect on what's been done here. We've turned a fairly boring sound clip into an instrument we can use to manually control three attributes of sound (the position of playback, a speed we've enforced on it and volume) to make sounds we might not expect to be able to hear from the initial sound.

Again, please excuse some low-quality video, but I think it would be a shame to make this tea-board and not use it to make a little 'teamix'.



I'm exploring some of the sounds it can make here, and not, necessarily attempting to make sounds which are 'commercial', or even musical. But you can see that this is really quite powerful. The important factors here are that every sound I make is repeatable with the same physical actions, every key will always be the same part of the sound.

Next time, I feel, we can probably take this further, notably we haven't changed the speed of playback so far, or tried any effects. There is more we can do with this instrument, but that's for another blog post, I think.


There's a few unfamiliar concepts in this post, and I may not have explained everything very well. If you're confused, feel free to drop me a line at @MarmiteJunction and ask any questions you like. I'm always up for a bit of audio-chatter.

So, until next time, take care, God bless and keep coding.




P.S. As a little extra bonus, here's my second attempt at the 'teamix'


Monday, 12 August 2013

MIDI controllers in Pure Data

Well, it's been ages since I wrote any blogs on Pure Data, and I thought it was time we really got to grips with phyiscal, tactile controllers. For this, we're going to look at a very common way to control music software, MIDI.

MIDI stands for Musical Instrument Digital Interface is a standard for controlling synthesisers and other sound equipment which was invented in the 80's and has been used since with most music technology, in one form or another. Simply put, MIDI is a set of digital signals which are the same coming from any device, and which any synthesizer responds to in the same way. So any MIDI controller can work with any MIDI synth.

Now, MIDI has mostly worked through MIDI cables, but in the last decade or so it's been possible to get USB midi controllers, which use the same system of signals but can be plugged in to a computer, to control music software. Here's an example:


As you an see, it's a keyboard, plugged into a laptop, it's very simple. And now that we have our controller hooked up, it's time to look at the signals we're getting in pure data. Now, different operating systems do this differently, but under the Media menu in Pd, you'll find some midi settings, (in Linux this is either 'default-midi' or 'alsa-midi'. Select this and you'll see a window like this.



Make sure you select a MIDI device as the input device. If you don't see anything here, it might be because you opened Pd before plugging in your midi device, try restarting Pd.

Now, let's see what kind of signals we're getting out of this. First, we're going to try a MIDI object, notein, which, unsurprisingly, takes the signal from a note into your computer. Now, just so we can get a look at it, lets just hook it up with some number boxes.


Simple enough so far, lets see what happens when we press a note on the keyboard.


Incidentally, this is my first attempt at using an animated visual aid.

So, when I press a key, the first number changes depending which key I press. Now, I have explained earlier that midi notes range from 0 to 127 and are one semitone apart. So when I move from a C to an E, the numbers are 4 semitones apart (from 60 to 64).

The second number is what's known as velocity, or how fast the key is pressed. That's a little harder to show with a GIF, so you'll have to take my word for it, or try it yourself. This, too ranges from 0 to 127. And you'll notice that when you take your finger off the key, the velocity is 0, the end of that note.

The third is a little more difficult to explain. It's what's called the midi channel, which is used for when a keyboard or sequencer are sending out notes for more than one instrument, these will be sent on different channels. There are 16 of these. Usually your keyboard will always send signals on channel 1, so for now we're going to ignore this.

Now, besides notes, midi also includes control signals, to get these into Pd we shorten 'control' to 'ctl' and use the object, ctlin.


Now, almost any control which isn't a note will come in via this object. My keyboard has a few rotary controls (otherwise known as knobs), so lets see what happens when we move some of these:


Now, this is simple enough, turning the knob to the right increases the first number from ctlin, turning it to the left reduces this number.

The last number is once again the midi channel, once again we're only using channel 1.

The middle number is very simple, it's which control is the source of the number, there's up to 128 of these (from 0 to 127, sound familiar). And I'll tell you now that there's no real logic to how these are arranged, you just have to move one of the controllers and see which number it is. So the next challenge is to get just the number from one control. Lets try that right hand knob, labelled B4 and on MIDI control number 73.

We're going to use the object called spigot, which has two inlets. If the right hand one is one or higher, whatever goes into the left inlet is passed through, and when the right inlet gets 0, nothing passes through.

We're also going to use an operator, borrowed from the world of mathematics, ==, or double equals. So when the left inlet is equal to it's argument it outputs a 1, for true, and otherwise, it's a 0. So when we connect one to the other...


So, when the control number is 73, the resulting number gets through to the number below it. So that's just the result of that one knob will get to the lowest number, others will not.

Now, we can do the same with our note numbers. So lets replace double equals with more than (>), this follows the same rules. so we can filter just pressing the keys (the 'note on' signal) by filtering it with more than 0.


So this is just a simplified version of midi notes, the number which gets through is just the last key which was pressed.

So, that's about it, getting midi controls into Pure Data as numbers, and as I've said before, any number you can get into pure data can be used to control pretty much anything. So let's try applying it to an earlier pure data post. In Pure Data and Bacon (Part 2), I made this simple synthesiser which uses a random number to make some simple music.


So, I'm just going to build the same chain, only replacing the random number generator with our simple key controller. And I'm going to set the volume (with *~) from that knob.


So, this is a very simple keyboard synthesizer. There are problems with it, firstly the note won't dissipate when you take your finger off the key, you'll have to turn your notes on and off with the midi control. But the important fact here is that you've made it from scratch, and by playing with a physical object (the keyboard) you can make music directly.

This is only to demonstrate how we can use this simple method of getting input to take direct control of just about anything we've done so far in Pure Data. Also, next I'll tell you how to use a keyboard to make boring sounds into interesting sounds with a patch that's similar to the one which saw me looking like this...


Although that's rather a long story. Feel free to get in touch on @MarmiteJunction and ask me about it.

Or for any other reason. So enjoy playing with Pure Data and MIDI, and stay tuned for some audio manipulation.

Wednesday, 26 June 2013

Introducing: The Teabot

Regular readers may notice that I've not written much in my blog recently. Meaning my Pure Data tutorials have dropped off slightly. I've been a little busy with a side-project of mine which has take n a lot of my spare time and thought to produce, and is mostly pointless. Anyway, here's a post telling you all about it, and hopefully I'll manage to explain some programming concepts while I'm at it...

Basically it started with the office teapot, which reflects some basic courtesy in hot beverages at my office. When you make a brew, make one for everyone, and alert folks to the fact that it's there. It's a simple workflow, but there's a few too many variables. I have a few basic issues with it, the teapot readily changes location, people forget (or "forget") to let you know it's there, and there's frankly not enough technology involved. 

This mild daily frustration, and, frankly, the fact that I quite like to play with technology, lead me to produce one of the more useful side-projects in my recent life. The Teabot, the idea was simply this:
  • Read a scale with a computer
  • Show the result on a web page
  • Let people know when the tea's ready to drink
This plan, being quite simple, lead me to explore methods of producing web pages with Ruby (my main programming language) which use much less processing power, memory and hard-drive space than Ruby on Rails (my main web framework). The result I settled on was Sinatra, which does just about nothing as standard except output the result of methods you write over http, the web standard for data transfer.

Then all I had to do was add in the RoR library for embedding ruby to generate files (erb, or embedded ruby) and I could create a front-end to the Teabot that didn't take too much out of the computer. That's exactly what I did, long before I got my hands on some USB readable scales to test it out on. I actually completed a first draft of the application entirely without it. The result looks something like this:


During development (as programmers often do, in the absence of hardware or software they are developing something to integrate with), I used a 'pretend' scale, or mock, which works like this. I know what a scale does, the computer asks it what the weight it's measuring is, and it passes the number back to the computer. This is what's known as a contract, describing how something will work, from the point of view of another application. So as far as my Teabot program's concerned:

  • A request goes to the scale 
  • The scale returns a number
So as long as the mock fulfils the same contract, the Teabot doesn't care whether or not it's a real scale. So I wrote a script that keeps asking me to type a number in, and the Teabot responds to that number in the same way (by saving the number to a file, which is read in various places in the application).

The next thing I had to think about is use cases which is, simply put, the ways in which a program will be used, you can think of these as the success conditions, when you can perform all your use cases the application is finished. Okay, this is a part-truth, but it's usually the right way to go about it. The Teabot is a very simple application, so I can think of around 4 simple use cases (using the standard software personalities, Alice and Bob):
  1. Alice wonders if there is tea, she opens the teabot page to see how much is in the pot, and how long it's been there.
  2. Bob makes a pot of tea, when it's brewed, Alice wants to know about it without constantly checking the page.
  3.  Alice makes a pot of tea, she would like Bob, and other co-workers (lets call them Charles and Deirdre), to know she made it, and what type of tea it is (she made Earl Grey this time).
  4. There are two teapots (used depending on how many people are in the office that day), if the teapot says full, it should mean the pot is full, whichever is being used. 
You can easily expand on these, for instance, a new teapot could be bought for formal occasions, a new 
co-worker could be added to the list, but these are essentially what should be possible. 

Use case one is simple, you can see the result in the screenshot above, users can't see the scale reading and refill-detection algorithm I created with my mocks, but they can see the level indicator in the teapot, and the time since the last refill was detected. 

Use case two means it has to provide users with similar information but in real-time. Now this presents a problem with a basic web interface. HTTP (protocol used to bring websites to your computer, there's loads to read about it online, if you wat to) works like this: Your computer sends a request, asking another computer (identified by a URL) for a particular page, the page is sent back to your browser which then displays it. Most of the time, you have to take some action to make the request, either typing in the address of a site and page, or clicking a link

To fix this, I've introduced polling, in which the Teabot main page automatically sends a request every 10 seconds, the Teabot app then sends back a response containing information about the teapot's current level and whether the other information on the page changes. 

But wait a second, all these people (Alice, Bob, Charles, Deirdre and perhaps someone beginning with E) have the Teabot open, and every ten seconds they're all sending a request, and only a small amount of the information on the page has changed (the headers, links, and footer are the same as well as the teapot image), so why should we re-load all of these?

Actually, the teapot doesn't reload everything every ten seconds, the polling request only returns the working data (who made the current pot, how full it is (in percent), what tea it is, when it was last made, and a few other odds and ends). This is then read (as JSON) and used to update relevant parts of the page are updated. 

There's an opportunity here for some stone soup, I could go in to the story of this term, but instead I could tell you what I mean by it. Stone soup refers to something which doesn't actually serve the purpose of the application, but which intrigues people and makes it seem interesting. In turn the interest generated makes it easier, politically or financially, to continue making it into a better application. 

The stone soup in the Teabot takes the form of an animated teapot! It's actually fairly simple, the outside of the teapot is an image with a transparent space in the centre of it (the outside of this image is the same colour as the background), there's a white space behind it, and a darker brown block between the two which changes it's size to indicate the level of the tea. Using the jQuery 'animate' method the tea moves up and down when the amount of tea in the pot on the scale changes. 

So far, everyone who's seen it working has been fascinated to see this happen, and this has helped me to discuss the future direction of the Teabot with my co-workers, and helped me to spend a little time at work on the Teabot. The little bit of interest generated in it allowed me a little extra resources to make it better, stone soup!

One of the pieces of data returned every ten seconds is whether or not to notify users that tea's ready. This gave me the opportunity to explore the HTML5 audio tag. I'm not going in to detail here, but 5 minutes after a new pot is detected, teabot plays a sample of my own dulcet tones declaring "There's tea!". 

This sounds like stone soup, but it's very directly meeting the conditions for our second use case. When the tea's had time to brew, it only takes one computer open and audible in each room of our office and everyone knows it's ready. In the second day of testing this in my office, there's an almost Pavlovian response, within a minute of this sound the brew-wanting masses descend on the teapot. Usually, a conversation ensues. 

Use cases 3 and 4 are fairly simple, there's a form to add pourers and teas, and a simple page called the "Teapot Station Interface" where, most of the time, users can simply click their name and/or the name of the tea, and change which one's currently in use.


Hidden under the Show List Manager link, there's a simple way to add to these lists.

You'll also notice a list of teapots there, which hints at the fourth use-case. You'll see at the top the button 'Calibrate Scale', this leads to a simple Wizard (multi-step form) which gathers three pieces of data about a teapot...

  • How heavy it is with nothing in
  • How heavy it is when it's got one cup of water in
  • How heavy it is when it's completely full
You can then give it a name, and it' on the list. This allows us to find the percentage of the teapot that's currently full of tea, and also, you guessed it, about how many cupfuls are left. 

That's pretty much what it does, and hopefully I've explained a bit about how I go about making web applications. I've not gone into much detail here, but there is one thing I had HUGE problems with that I've not talked about here. 

I could find nothing at all online which told me how to go about reading data from USB (from the scale) and how to go about interpreting this in ruby. So, mark my words, there will be a post giving you a step-by-step rundown of how I went about making the teabot happen. 

Oh, very last thing, here's what it actually looks like, in development at my flat, and at the office.



Oh, and the scale with my prototype hardware branding...



Also, if you're au fait with github, you can go there and take a look at the Code for the Teabot there. And as ever, I'm always up for discussing my fun coding, so feel free to get in touch on twitter (@MarmiteJunction) and have a chat, ask any questions, look into how to get your own teabot, whatever you fancy. Or just leave me a comment here. 

Tuesday, 11 June 2013

Algorithms of the Street - Part 3: Additive Synthesis

I've been thinking about a talk for conferences for a while now, based on the fact that there are certain ideas, or patterns, which keep coming back when we work with computers. It doesn't matter if we're dealing with web apps, processing large files of formatted data, creating spectacular animations or simply playing around with digital audio, programmers meet a lot of the same problems, with variations on the same solutions.

When we learn the patterns which keep coming back, we find solutions more quickly and can move on to bigger and more complex problems. Furthermore, when we learn about these patterns, how they fit together, and understand them enough to implement them differently, we become better programmers.

Just about every capable programmer will be aware of this concept of patterns, and will use them every day. But as I've said here before, understanding problems from first principals helps us to understand how to solve them (which is why I always try to explain things right from the beginning in my blog posts).

So why wouldn't it be useful to start with a new perspective, say, synthesis in Pure Data, and see some common patterns in action?



Proposal over, this tutorial covers another form of synthesis algorithm, Additive Synthesis. I'm going to explain Additive Synthesis from first principals, but also, because it fits quite well and I'm trying out an idea.

But don't worry, I don't assume my blog posts are necessarily being read by experienced programmers, so I'll be explaining the programming concepts from first principals, too.

Step 1: Getting Set Up 



This one's an old favourite, a random sequence of numbers, converting midi notes to frequencies, into an oscillator then out to the sound card. When you click the toggle at the top there, you'll start to hear a kind of melody. It's all very simple and easy to understand. The trouble is, the old problem, it's only producing a single sine-wave tone, which (take my word for this) only contains a single frequency and makes for a very simple, uninteresting and potentially maddening sound.

The reason this type of sound is so grating, is because in nature (or at least, in sounds produced by actual objects), you never usually hear this pure a tone. The sounds you hear contain a lot of frequencies in different proportions, which gives each sound it's own unique tone, or timbre.

Now, musical sounds tend to have one strong, or fundamental frequency, which defines the pitch you hear, and other strong tones over the top of these, called harmonics. These harmonics follow a sequence of relative frequencies known as the harmonic series, which have been observed by physicists in wave forms of all kinds.

In the harmonic series, the first tone is your fundamental, a part ably played by the oscillator we've already created, then the first harmonic is double that frequency, the third is three times the frequency, and so on.

As I may have said before, synthesis is actually the opposite word to analysis. Now that we've analysed some of the properties of musical tones, we can start to synthesize the same idea. We know how to add oscillators in pure data, we know how to do the arithmetic, so we can add a few more harmonics to our existing patch.

When we're reproducing the harmonic series with a synthesizer, we call each produced tone a partial, because it's a part of one single tone.

Step 2: Adding a few partials...



Now we have 5 partials in the harmonic series, multiplying our frequency by the numbers 1 to 5, but it's still quite an unnatural kind of sound. Mostly because these harmonics are at equal volume. As I mentioned earlier the fundamental tone is the strongest, and the higher partials are weaker. Actually, they tend to get weaker by the same amount each time.

So, in the simplest pattern we can (which just so happens to be called a sawtooth wave), lets divide each partial by it's partial number as well...

Step 3: Successively Weakening Partials


So, each one is multiplied by 1 divided by the number of partial it is. The result is what's called a sawtooth wave, the reason for this is because of how it would look plotted on a graph (or, before digital sound, on an oscilloscope)...

Notice how the wave moves quickly up, then more slowly down (bearing in mind that this is actually happening hundreds, or thousands, of times a second), and looks a little like the teeth of a saw. Actually, it also looks a bit wobbly for a saw tooth. In truth, the more partials we use in our synthesis algorithm, the closer this wave shape will be to being 'mathematically perfect'. So lets try upping it to ten partials. And we'll see the first problem with it.

Step 4: Twice as many partials for twice the accuracy


First things first, some housekeeping. I've decided there was too many wires, so I've replaced the wire at the top with a send called fundamental, carrying the fundamental frequency. Then in each partial it's multiplied by the partial number, fed to an oscillator then the resulting tone is multiplied by one divided by the partial number (which are figures I sat and worked out myself), then sent to something new called throw~.

throw~ is just like send~ (or s~), only it can go from multiple sources to one destination. It's what we call a summing bus, as it sums together (adds up) our tones.

The result is that our wave shape has become closer to mathematically perfect (which would look like it's made up of straight lines)...

Only we're not quite there, and we've run out of space on screen, and even with the sends to tidy it up my partials are overlapping each other, wires going over objects, it's only going to get messier if we try to add more partials with this same method.

Imagine a patch like this with 100 partials. Even if we neatly hide them away in sub-patches or even abstractions (which I'll talk about in a later blog post), there's another problem here. Each oscillator takes up a certain amount of processing power, and any computer only has so much (some more than others, but there's always a limit).

We can get a wave produced with more partials, but we need to start thinking differently about what we've just created.

This patch uses processing power for 10 oscillators (plus 9 control multiplies and 9 audio multiplies) constantly, to produce a sound which is always going to be the same, so why should your computer constantly be working out this waveshape in order to synthesize the tone?

What we need is known in programming terms as an early-binding system, one which does the hard work once then uses a more efficient method of playing back this tone.

What we need in Pure Data is wavetable synthesis. In which a waveform is stored in an array then read from that array at audio speed to produce the tone. The osc~ object actually reads a sine-wave table internally to do it's thing, we'd just be giving it a different lookup, to the result of 100 paritals can be heard while only using the processing power required by one.

Pure Data even has a method called 'sinesum' which will sum up some sinewaves for us, we just need to tell it the levels we want our partials at (so it's still additive synthesis we're doing, just early-binding additive synthesis, instead of the inefficient late-binding additive synthesis we were doing).

Step 5: Lets make a wavetable synth...



What's happening here, then? Well, you'll notice that I've discarded most of the patch, and brought us back to what is in effect the same as the patch we started with. Only instead of osc~ I've got tabosc4~ and the name of an array, waveform. The array, waveform, contains a sine wave, and below it there's a message which is sent to the array via s waveform.

This message is a special message to an array to do additive synthesis with a list of numbers. The first is the length of the array (sinesum will re-size it for us) and to work with tabosc4~ it has to be a power of 2. 32768 is 2 to the power of 15. Then the rest of the list is how loud you want each partial to be. So one partial with an amplitude of 1 is just a sine wave. When this message is clicked Pure Data processes the list of numbers and puts the result in the array for us.

You've probably realised by now that this also sounds exactly like the patch we started with, only to get to the point we were just at, we don't need all those oscillators, we just need to put the list of amplitudes in our message box. (remember, each one is 1 divided by how far along the list it is).

Step 6: Additive synthesis with just a list



I've just expanded my message to have the list of amplitudes I worked out earlier, following those same rules, and added a comma to it (which is the same as sending a different message immediately after) with the word normalize (meaning 'make it fit this amount') and 1. This means that the result will be made to fit between 1 and -1. We can see our sawtooth wave again, and hear it. Only now we're not using all that space on screen or all that processing power. Win!

That's more or less it for Pure Data, but there is one more problem here, and it's not one that's easily solved in Pd. I have however, found a solution which involves using a different tool, the programming language, Ruby. This leads me to another pattern in technology we need to be aware of:

When you have a hammer, everything looks like a nail.

In other words, the tool you're using isn't always the best one to solve the problem.

The problem is that after finding the rule for the partials of a saw wave, I had to sit here and work them out, then write them down. And if I wanted to use a different rule, I'd have to work them out again. Okay, I can use a calculator, but that's still a little tedious. So what if I could write a program to write the list of partials for me. Then we could get 100 partials, without having to work out 100 numbers.

Warning: The following section contains some ruby code, and I might not explain it in depth. Feel free to talk to me in another medium if you want a bit more info.

Step 7: Different kind of nail? Different kind of hammer!




This looks much simpler, but you'll notice that the list of partials isn't there any more. Instead the send to the array, waveform, is fed by an object called netreceive. This simply reads everything that's sent to your computer on that numbered network port. I'm not going to get technical here, but this port is just a destination for your computer to get messages from other computers such as the contents of websites, shared files etc.

Here, however, we're going to pass it information from the same computer, with a Ruby script. Copy the following into a text file (in a plain text editor, not a word processor) ending the file name with .rb:


require 'socket'
connection = TCPSocket.open 'localhost', '4000'

partials = []
100.times do |i|
  i = i+1
  partials << 1.0/i
end

connection.puts "sinesum 32768 #{partials.join(' ')} ;"
connection.puts 'normalize 1;'


So, you might need to know a little ruby to understand this, but it's basically loading the standard Ruby network library, socket. Then using it to make a connection to port 4000 (same as netreceive) on 'localhost' (which is always the computer running this script). Then for the numbers 0 to 99, it adds one (so we're effectively using 1 to 100) then divides 1 by that number and adds it to an array, partials (just like in Pd, this is a list of numbers, although it can be other things in Ruby). Then it sends the message sinesum, that same length, and all the partials it's worked out, separated by spaces.

Incidentally this is the same method of communicating between Ruby and Pure Data is the same as I used in Text to Music.

You'll need ruby installed (usually it is installed by default for Linux and Mac boxes, just don't ask me for help running it on windows). Then you can run this script (in a terminal, 'ruby' then your file name) and it will generate the right amplitudes for 100 partials and pass it in to Pure Data, where it is picked up by netreceive and sent to the array, waveform.

As promised, we can see (and hear) a near-perfect saw wave created by 100 partials...


But Andrew, I hear you say, surely we can use this method to produce other waveforms with this method?

Yes you can, and I have two places for you to look to be able to do this.


  • The Pure Data FLOSS guide (where I initially got the equation I'm using here, and two other commonly used ones.)
and 
  • A github repository I put together to demonstrate this technique. It's also a working example with a few common waveforms.
I've not actually seen any material online about automatically producing partial numbers for additive synthesis in Pure Data. It seems I'm the first one to write about it publicly (feel free to correct me on this). So hopefully it will make a few lives a little easier, a common hope of programmers the world over.

That's it, enjoy playing with Additive Synthesis, and hopefully you've learned a little about those programming patterns. As ever, feel free to leave a comment or contact me on twitter (@MarmiteJunction).

Sunday, 9 June 2013

Putting Pure Data in a box (or: Pure Data Hiding)

Greetings

I've had a couple of weeks off the blogging, so it's time to get started again. You may not know that I'm actually something of a data nerd and tend to get very interested in figures and statistics that are drawn from my life in some way (which is actually quite a good way to go about losing weight, by the way). So I've decided to try and draw a topic from the statistics on this blog. So first, I'm going to share with you the list of search terms google tells me are being used to find it...


Well, the first thing to note is that it's mostly looking for me in some form or another (although I've never been to Toronto), and the most searched thing that's brought people to my blog is this phrase; "Pure Data Hiding"). A quick google search reveals, also, that my blog is several pages in, and while there are various discussions, forum topics, plugins and technical information. It seems people are skipping through it. My guess would be this is because they're not finding the kind of information they want.

So I'm thinking it's time to make good on my recurring promise to blog a little bit about hiding away parts of our pure data patch we're not presently interested in, leaving us free to concentrate on the visible elements of the patch.

In order to show why we would hide parts of Pd patches away, let's try an exercise in "Putting it all together":

Step 1: Pick some components

I often stress that what I'm teaching here are algorithms that can be re-used in different ways, and effectively become parts of a recipe we can vary and put together differently, so, looking back at existing blog posts, let's take this rhythm generator from Pure Data and BACON (Part 1):


The Step sequencer from from the end of Sequencing and Wavetables, Part 1: Arrays in Pure Data



And The complex synth from the end of Algorithms of the Street: Part 2 - Amplitude Modulation:


Note: I would recommend, if you want to understand the patches used in this example, you read the posts they're from first. If not, you can take my word for what they do.

Now, we've got three distinct components here, the first is responsible for the placement of notes in time (rhythm), the second, selects pitches to be produced and the third changes these pitch numbers into a rich synthesized tone. So there's no reason they shouldn't just be chained together to produce a patch which plays the same sequence of notes with the same rich tone with a randomly generated rhythm.

The problem is, I'm using a standard size of patch, to display on my blog, and even if we removed this restriction, the height of this patch wouldn't fit on my screen. We need a way to lock these things away. But, as always, we start with baby steps.

Step 2: A sub-patch

Let's take that first, rhythm generator patch and hide it in a sub-patch:


You can already see that it's not taking up nearly as much of the screen. However, you can also see that you can't see the rhythm generator itself. If you're coding along, you will also have noticed that when you create the object (pd rhythm) it will have opened an empty Pure Data window, called rhythm. This is what's called a sub-patch which is a patch within a patch, it's basically part of our pure data file, but it's not shown in the same window. You may also notice, that it has no inlets or outlets when it's created, making it impossible to connect to the click~. So our first job is to create one...

So, we create an object called outlet, inside the sub-patch, rhythm like so:


And the second we do that, the object (pd rhythm) on the parent patch gains an outlet so it looks like this:
There's no co-incidence here, the outlet inside our sub-patch directly maps to the outlet shown on the pd object. Anything that goes into the outlet object, will go to the outlet of the pd object. So all we need to do now is build our rhythm generator inside the sub-patch and connect it up to the click~, outside it.


If you've read Pure Data and BACON (Part 1), you'll know that this is sending out bangs (the Pd activation message) at a changing set of intervals, based on fractions of 1 second. You'll also know that this needs input from a toggle to start and stop the process. And here, that position is taken up by a receive (r start), which is coming from the send (s start) on the main patch. So we've learned a little more about sub-patches. You can get data in or out via outlet (and, conversely, inlet) or by sends.

Sends, in pure data, actually work between any open pure data patch, as long as they have the same name.

So when we click the toggle in the top right corner, the rhythm generation is done inside our sub-patch, then bangs are passed out of it to click~ and then to our sound card via dac~. It's easy enough to follow through, all we've done is put it in a box, labelled with rhythm, so we know what it does to give us space for the rest of our monster patch.

You can close the window containing the sub-patch and the work it does is still there. Clicking on the sub-patch (pd) object when you're not in edit mode will open the sub-patch window again.

I've also thought ahead a little, here. The patch from Sequencing and Wavetables, Part 1: Arrays in Pure Data starts with a metro, meaning that it's actions are started by bangs. So I've ended my rhythm generation sub-patch just after it's own metro, and put the sound-generation part of it in the main patch. This makes it easier to swap it out for something else that accepts bangs...

Step 3: Back to a sequence generator...



And the content of the sub-patch, sequence:



Now, we've used inlet, to allow us to connect the output of pd rhythm to the pd sequence. Then filled the sub-patch sequence with the middle part of the step-sequencer from Sequencing and Wavetables, Part 1: Arrays in Pure Data. It's the part after the metro (rhythm works a bit like metro, only it's outputting bangs in a more complex sequence) and before it starts producing sound (be cause we know we're going to use it a little later to feed the AM/FM synthesizer from Algorithms of the Street: Part 2 - Amplitude Modulation).

First things first, we've got an object here called loadbang. It's quite simple what this does, it outputs a bang when your file is loaded. In this case, that bang sets the values of the two arrays, scale (which is hidden in the table object, within the sequence sub-patch) and sequence (which is out on the parent patch).

It may seem unusual that I've put the sequence table on the parent, but hidden the numbers setting it away in the sub-patch with the code that's reading the table. But there's a pattern to what we're doing here. I've put things we might want to change, or play with, on the parent, where we can see them, and hidden away the things we might want to change less often, like the code that's playing the sequence and the list of numbers we're going to start with each time.

If everything's working, when you check the toggle in the top right corner, you should see the numbers below pd sequence changing. Now, we can use this to create a melody with our old friend, the oscillator...


This sounds fairly pleasant, but this is an exercise in 'bringing it all together' and I've already written some blog posts about synthesis algorithms, so let's put that into practice...

Step 4: Give the oscillator the boot...



And the contents of the synthesiser sub-patch:


Now, as I explained in Algorithms of the Street, Part 1: Frequency Modulation, anywhere you see an oscillator you can replace it with a more complex synthesis algorithm. The only problem is that this takes up more space on screen (also more processing power, but at this stage it's not likely to be enough to effect your computers performance, so it's not a huge concern). So here I've set up our third sub-patch to respond to numbers in the same way the osc~ object does, accepting a number, and outputting a tone at that frequency.

Also note that instead of using outlet, we're using outlet~. Again, a common pattern in Pure Data, this means it's an audio outlet, as opposed to a signal one. So we're outputting a sound.

With the added bonus that we're controlling the tone from a grid, again on the parent patch, because we may want to change it. And we've achieved what we initially set out to do. The rhythm generation, sequencer and synthesizer hidden away in neat little boxes, and I've done it in just 4 steps. So let's see if we can push this a little further.

Step 5: Solve a problem with a hidden volume control

Try turning off the toggle in the top right corner (start). You'll notice that the sound doesn't stop, it just stops changing it's pitch, and stays maddeningly un-changing. So let's fix this issue with an extra sub-patch...

And the contents of pd volume:


All we've done here is hide our volume controls in another sub-patch, then added a volume envelope which hijacks the start signal, and uses it to control the volume (the 1 and 0 used by toggles multiplies the sound by 1 or 0, leaving it unchanged or silent) with a line generated taking 1 second to fade in and out.

We've fixed an inherent problem with our code without filling up a lot of space on screen with the extra few objects that simply fade in and out.

Another problem with our patch is that the notes are all produced at an identical volume, whereas real sounds, in the real world tend to have what we call a volume envelope, meaning that the volume of the sound changes over time. Think, if you pluck a string, the first part of the sound is loud, then it fades out over time. So let's fix this in another sub-patch...

Step 6: A volume envelope for each note...



And the contents of pd volume-envelope:


The vline~ object is a lot like line~, used in How to make a Musical Instrument in Ten Minutes. Only its messages are a little more advanced. It has groups of arguments, separated by commas. So each one has:

  • Go to this value
  • In this number of milliseconds
  • After waiting this number of milliseconds
Meaning this takes 50 milliseconds to get up to 1, goes down to 0.4 in 50 milliseconds (when the first one is finished) then fades down to 0 in 200 milliseconds when the second is finished. This is a very simple volume envelope but it gives each note a definite start (attack) and end (release). The actual volume envelope, plotted in an array, looks like this:


And because we can write it into an array in Pure Data, we can also read it out of one to perform the same purpose. But more on that in a later post. 


Again, piece by piece we're making this patch more advanced, but not taking up much room, putting things in little boxed labelled with their purpose. Meaning if we want to change something about one or other part of our patch, we can find it fairly easily.

We also know, having put it together, what we expect to go in to each sub-patch, and what we expect to come out of each one. So if we want to debug them, we can just give one sub-patch the input it expects and see if the output is what we expect.

So, for instance, synthesizer expects a number, so we could feed it a number from a number box, and it outputs a tone, so we could just put a dac~ after it and see if it changes when you change the number. That is actually a common feature of programming computers, what we call layers. Different parts of an application which are built on one another, but only rely on the previous layer. So we can test each one individually.

Also, a quick note for the conscientious on sub-patches. Their strength is that they hide code away, but it's also their weakness. If you don't know what a sub-patch does, you won't be able to fix it very easily. Also, in a patch like this you have to open each sub-patch individually to find out exactly what it's doing, which might take quite a lot of time to fully understand if you aren't the one who put it together.

There's a balance to be struck, here. For instance, replacing osc~ with a sub-patch is a fairly common pattern, as it does exactly what osc~ does, but differently. What we tend to call a contract in programming terms. The contract of an osc~ is that it takes a frequency (in herz) and outputs a waveform. Our synthesizer sub-patch does just that, so it's easy to directly replace.

However, to make the patch more readable, almost everything else in it could be on show. Although exactly how much Pure Data you hide in a box is, ultimately, a personal decision. What makes sense to you?

Right, now that the discussion of concept is out of the way, let's plug in one last component. The delay line from How to make a Musical Instrument in Ten Minutes

Step 7: Just for fun, delay.



And the contents of pd delay:


This is just two extra UI elements (a vertical slider, and a vertical radio) feeding in to the classic delay line as used in How to make a Musical Instrument in Ten Minutes (go there if you want to know what this part of the patch is actually doing). And you have control of a part-generative, part sequenced synthesizer and a delay effect.

Have fun messing around with electronic sound...



Well, that's it. Feel free to get in touch via @MarmiteJunction and let me know if you managed to do this (it's about as advanced a Pure Data patch as I've blogged so far) and what you'd like to see explained here.

Until next time, God bless and happy coding.