All posts by Jiffer

Mapping Inputs to Control Sound

Modular-Muse Interface Objects

The modular-muse library includes a set of interface objects which make it easy to connect to the board and read the inputs into Pd. These objects with work with any Arduino compatible device as long as the Arduino is programmed with the modular-muse firmware.

[mm]

The [mm] object is what communicates between the Sound Clippy (or Arduino) board and Pd. Once connected to the [mm] object the Sound Clippy board sends data over a serial port connection about the state of its inputs.

Screen Shot 2015-07-14 at 2.44.33 PM

Tutorial: Connecting to the Sound Clippy board

[mm-digital]

The [mm-digital] object is for reading the state of the digital pins on the Sound Clippy board. The 12 pins can all be read separately and be used to trigger audio file playback, sound envelopes, enable/disable effects among other things.

To read a digital pin create a new object and type

[mm-digital #]

Screen Shot 2015-07-13 at 11.33.51 PM

where the # is the port number you want to read. The digital pins on the Sound Clippys board are on the side of the board that looks a little like a piano:

clippys-digital-piano

 

the digital inputs are labeled with D2-D13.

The [mm-digital] object has two outputs. The left outlet reperesnts the currents state of the pin and outputs a number, a 0 or a 1 depending on if the port is open or connected to ground in hardware.

This value of 1 or 0 can be used directly to turn on and off objects with an on/off state like the metronome object or the [mm-adsr~] object.

While the right outlet sends a “bang” message when the value changes from 0 to 1. This is good for triggering events

Take a look at the digital input examples for more ideas – or read about creating a paper piano hardware controller.

[mm-analog]

The [mm-analog] object is for reading the value at the analog inputs to the Sound Clippy board.  The 8 analog inputs can all be read in separately and used to control different parameters – or even multiple parameters. For more info about interfacing to the analog inputs in hardware see: Analog Inputs for Expressive Control

Mapping Inputs

With an analog input connected its time to map it to an audio parameter.  From the examples, open the patch “analog-filter” and connect the board. Analog port 0 is configured to control the cutoff frequency of the filter between  200 and 2000 Hz.

Screen Shot 2015-07-14 at 3.49.26 PM

This was created by creating a new object and typing:

[mm-analog 0 200 2000]

Creation Arguments

There are three numbers after the object name [mm-analog]. These are called creation arguments. For this object the first is the port number, next is the low value and finally is the high value. These values can be set accordingly depending on what the sensor is going to be used for. In this case a low-pass filter cutoff frequency is being set between 200 and 2000 Hz.

Sensor Calibration

The analog sensor is automatically calibrated to use the range specified in the [mm-analog] object. The object will keep track of the lowest and highest values seen by the analog port and will map the output to be within the specified range.

To calibrate the sensor – click the “calib” button (or send a bang to the “calibrate” inlet) and exercise the sensors full range. e.g. if its a light sensor, cover it for full darkness and shine a light on it to get the upper end.

Now that you know how to use and map analog inputs to any range – think about the elements of sound again – what do you want to be able to control? Try controlling some different audio effects or sound synthesis parameters!

Connecting to the Sound Clippy Board

[mm]

The [mm] object is what communicates between the Sound Clippy (or Arduino) board and Pd. Once connected to the [mm] object the Sound Clippy board sends data over a serial port connection about the state of its inputs.

 

 

 

 

 

Screen Shot 2015-07-14 at 2.44.33 PM

Connecting

To connect the [mm] object to the Sound Clippy board connect a USB cable between the board and the computer. Open an example such as simple-synth.pd from the sound-clippy folder in the modular-muse library or create a new patch and add the [mm] object to it.

Click on the “list” button. This asks Pd to list all the serial ports it sees. The list is printed out to the Pd window. Here is my Pd window after clicking “list”:

Screen Shot 2015-07-14 at 2.47.00 PM

Note: this will look different on different operating systems (this is from a Mac).

The serial port with the “usbserial” in it is the Arduino. In this case it is on port 4. If you’re not sure which one is the Arduino, try unplugging the USB cable from the device, click list, plug it back in, and click list again. You should see a port which is not in the list when unplugged and is in the list when plugged – this is the one you want!

Back in the [mm] object I change the “port” number box to 4. It should connect to the board with a corresponding “opened serial line device #” message in the Pd window.

You should now be able to use the [mm-digital] and [mm-analog] objects to read the inputs from the board. For more on using these see Mapping Inputs.


 

If you are having trouble communicating with the Sound Clippy board, you may need to make sure the correct firmware is installed. Programming the Sound Clippy Firmware

Programming the Arduino

Programming the Arduino

The modular-muse interface objects can be used to read digital and analog inputs from any Arduino compatible device as long as it is programmed with the modular-muse Arduino firmware.

Once it is programmed once it does not need to be programmed again to continue working with the modular-muse Pd library.

To program the Sound Clippy board with the modular-muse firmware – download and install Arduino.  Open the modular-muse Arduino code in the modular-muse library/arduino-code folder. Select the correct board (Nano) and port (probably says something like usbserial and some other numbers and letters, for Windows it will be a comm port) from the Tools menu.

Click the Upload arrow at the top of the code window.

Musical Gestures and Layout

Musical Gestures

Musicians have control over a lot of different aspects of sound that they use to create expressive, dynamic music.  A violinist alters their bowing speed and pressure to effect timbre, a horn player can create a swell in volume, a vocalist varies their vibrato. This is what makes music interesting, it gives it variation over time. With computer generated music, controlling these parameters is the difference between static, lifeless sounds and interesting expressive musical sounds.

Gestures in Digital Music Instruments

With acoustic instruments the connection between the physical gesture and the resulting sound is obvious – its tied directly to the way the instrument is played. With digital music instruments there is no direct physical mapping – the mappings are up to the designer and is done in software.  However, this doesn’t mean you should make arbitrary mappings! In order to create something that is playable, or even intuitive it’s important to consider how a player wil play a new interface.

Some questions to consider:

– What physical gestures do you want to use to play or interact with your instrument?

– What does each hand do? What about other parts of the body?

– Is the instrument to be held, set on a table, around the neck with a strap or some other interaction? Where do the various inputs (buttons, switches and sensors) go?

– What elements of music or sound parameters do the different interactions control?

– Is there a metaphor for this gesture?

– And finally, for each gesture or sound parameter under control, does it need to be a digital or analog input?

Once you have some ideas to explore – its time to try creating an interface and do the mapping in software.

Go to – Mapping Inputs to Sound – for more on how to use the modular-muse interface objects to bring in data from buttons, switches and sensors.

Analog Inputs for Expressive Control

Analog Inputs for Expressive Control

Unlike digital inputs which have 2 states (i.e. on/off, high/low, 1/0), analog inputs allow for continuous* control over a range of values.  This allows for more subtle control of sound parameters than simply turning a sound on or off.

Elements of Music

Music has a lot of richness and variety for us to enjoy. Much of that has to do with the development of instruments over centuries. Strings, woodwinds, brass and percussion comprise the traditional orchestra families. Each produces sound in unique ways and have different playing techniques which contribute to a wide range of timbres and means of expression.

In addition to the variety of instruments and sounds – compositional aspects have an enormous effect on how we experience music.

What are some elements of music?

Here’s my list: pitch, harmony, scale, melody, rhythm, dynamics, timbre

Others can be defined but for now lets use this list to consider how these elements might map to sound parameters – as we might think about controlling them with computers.

Elements of Music Related Sound Parameters
pitch (notes) frequency (rate)
harmony (more than one note) frequency ratios, or series of intervals played together
scale series of intervals, a collection of notes
melody sequence of pitches
rhythm tempo, patterns, loops
dynamics change in volume, articulation envelopes
timbre waveform, synthesis technique, filter

Much of these are enabled by instruments with a wide range of sounds. These include the more obvious things like: low to high,  soft and LOUD. But also enable playing which emulates more expression, and even emotion. For instance can you imagine music being sweet or angry; smooth or bouncy?

Create expressive controllers with Sound Clippys board

Read hardware inputs and map them to sound using the Interface Objects

Digital Inputs – Making a Paper Piano

To create digital inputs for the Sound Clippy board we need to create an electrical switch which can be turned “on” and “off”. In this case, “on” means connected to GND and “off” means disconnected from GND.

Digital Inputs

Each digital input on the board can be triggered individually. Each port has a number printed on the board indicating which port it is.

The [mm-digital] object uses the number after the object name to select the port. For instance, if you want to read port D2, add a new object and type [mm-digital 2] – you should see this:

Screen Shot 2015-07-13 at 11.33.51 PM

With the Sound Clippy board connected, when a digital input is connected to GND through an alligator clip or with another conductor you should see an “X” in the corresponding [mm-digital #] object toggle box.

Alligator clip switch

Open up simple-synth.pd from the modular-muse/examples/sound-clippys folder and connect to the board. (TODO: how to connect)

– Clip an alligator clip to one of the GND  ports (they’re all the same).

– Touch the other end of the alligator clip to one of the digital pins

digital00

When you touch the alligator clip to the digital pin you’re making an electrical connection between the input pin and ground which the board can detect. This is sent to the corresponding [mm-digital] object.

Paper Piano

You can extend the alligator clip wires by connecting to other conductive materials. Copper tape is a good conductor and is easy to work with.

materials:

– construction paper

– copper tape

– Sound Clippys Board

– alligator clips

IMG_2412

To build a successful “piano” we need to be able to connect the digital pins to ground individually. This will be done by cutting individual “keys” into the paper which will become the triggers or buttons.  First its good to decide on the size and cut the basic shape. I’m going to make a basic rectangle piano with 6 keys.

Basic Design

Cut down to size and figure out how the “buttons” will be made. For this simple example I cut the construction paper to a small rectangle about 6″ x 3″, then I folded it in half the long way so it is now 6″x1.5″.

IMG_2413

Cut the individual keys

IMG_2414

Run the ground “wire”

 

All the keys need to be able to connect to ground (GND). Since all the keys are in-line I just need to run one long piece of copper tape which will go under all of the keys.

Keep the tape a little longer than it needs to be so it can be folded around the back. This makes for a better surface to connect to with the alligator clips later.

IMG_2416

Now run copper tape for each individual key. Make sure that the tape is long enough to fold around to the back side of the key where it will make contact with the GND copper tape.

IMG_2418

IMG_2419

Connect

Connect the GND to the ground copper tape which runs under all the keys.

Connect the individual keys to the desired digital inputs.

The example patch “simple-synth” makes each digital input control a different pitch to be played.

The fun has just begun!

Try other examples in the sound clippys example folder to explore different sounds and ways of using digital inputs.

An instrument which never plays a wrong note

Connect the Sound Clippys board with at least 2 analog sensors and one digital switch. Open the example from the modular-muse Pd library called simple-synth in the examples/hardware-inputs folder.

Once you connect to the board you can play this patch with one analog input and one digital input.

With this example it is difficult to play a melody or even a specific note because it can play any pitch between 200 and 800 Hz.

Next open up the patch in the same folder called major-scale. This one uses an array similar to these ChucK examples to specify specific notes which can be played. Now every note you play is part of the specified scale.

Try changing the numbers in the array “scale” to retune the instrument.

ChucK – using arrays to select notes

Installation and Setup

First of all you need to install the midiAudicle, go here: http://chuck.cs.princeton.edu/ and download and install for your operating system.

Now download these example files: chuck-array-exercise

– Open the mini-audicle and make sure your audio device is set up (go to Preferences > Audio).

– Start the “virtual machine”

– Open the first example, 00-random-pitch.ck and click the “+” add button. This will start running the code and you should hear random pitches being played. click the “-” or “x” to stop the program.

Whats going on?

00-random-pitch

This code randomly selects a number between 200 and 1000 to set the frequency in Hz of an oscillator (a sine wave) every 0.2 seconds.

You can change the frequency range to get different selection of notes and you can change the duration between each change but it always sounds somewhat random.

00-random-scale

This example starts to become more musical. Instead of choosing a random number for frequency it chooses from a pre-defined list of notes stored in an array called “scale”. The line of code which stores the notes in the array looks like this:

[0, 2, 4, 5, 7, 9, 11, 12 ] @=> scale;

These numbers represent note values away from the “root” of the scale. The numbers are added to an offset which represents the root (called DO and set to 60 for middle C in this example). In this case a 0 will play a C,  a 2 will play a D (a 1 would play a C#), etc.

So now the pitches are not totally random – they are randomly chosen from this list but they will always be one of these pre-defined notes listed in the array.

What happens if you change the values stored in “scale”? Can you create a minor scale?

02-random-step

This example uses the scale again but instead of randomly selecting pitches – it randomly steps up or down or stays on the current pitch

03-random-skip

This example allows skips to occur (a step is an adjacent note in a scale, and a skip is anything further away than that). This uses an array of possible “skips” which are randomly chosen from. It has more 0 and 1 values than 2 and higher which means its still more likely that it will step to an adjacent note but occasionally will skip a few notes.

Try different values in this array as well as in the scale array to create different melodies.

MIDI Robots

The modular-muse 4-channel driver board can be used to drive solenoids and DC motors. It can be driven with digital output pins from an Arduino.

IMG_1600

Here I’m using it with the Teensy LC to drive the motor driver with MIDI data.

Screen Shot 2015-04-20 at 4.14.18 PM

 

The result is that beats can be programmed from any digital audio workstation (DAW) to trigger motors.

 

 

 

Using the Sound Clippys board

A Paper Piano

For starters here’s an example interface. A paper piano with a distance sensor to control an audio effect. The “tuning” of the piano can be changed to plugging into different ports on the board or by making changes to the program in Pd.

IMG_1598

The Sound Clippys board has two different kinds of inputs, digital and analog.

Digital Inputs

clippys-labels
Sound Clippys board detail – Digital pins on top, Analog pins on the bottom for variable resistors and other analog sensors

There are 12 digital inputs which can have two states – high or low, 1 or 0, connected, or not connected, are different ways to think about it. They use something called a pull-up resistor to make the pin normally HIGH (5 Volts to be exact) and when they are connected to the GND clip (for ground) with a wire or something that conducts electricity they are read as LOW (or 0 Volts).

The 12 digital pins are are lined up along one edge of the Sound Clippys board, staggered a bit to look like a piano.

Connecting to the board

Open Pd and open the patch digital-diatonic.pd which is part of the modular-muse library installation be here:

./modular-muse/examples/02-hardware-inputs/digital-diatonic.pd

From here the [mm] object will be used to identify and connect to the hardware.

1. Before connecting your Arduino, click on “list” – now go to the Pd window. It should have a list of serial ports.  Now connect the board and click the “list” button again. There should be  a port in the list, this is the Arduino. Note which number port it is in the list.

2. Click in the number box and type that number and hit Enter

3. Click Open

If it all worked your are not connected to your board and if you make a connection between any of the Digital pins and Ground you should be able to play the instrument in digital-diatonic.pd.

Making Buttons and Switches

You can play this instrument by clipping an alligator clip to GND and then touching the other end to the digital inputs on the board.

digital00

 

Using conductive material

There are lots of conductive materials available these days which are great to use with this board. Here’s a short list:

Copper tape

Conductive Ink

Conductive Thread

Wires

Here are some examples:

Using conductive paint/ink:

IMG_1272 simone-love brit-tree

 

Tweet Tree:

Using copper tape with paper and wood:

paper-piano

wood-playing wood-side wood

So there are a few ways to using the Sound Clippys digital inputs. Please share yours!

Analog Inputs

analog00

Connect potentiometers or other analog sensors to the headers. These are sensors which should have 3 pins: (1) 5V, (2) signal and (3) ground. Above a small knob potentiometer is plugged in.  Here is a distance sensor:

analog03
Distance Sensor plugged into Sound Clippys analog input

The bottom row of alligator clips are for connecting variable resistors. These can be photocells (change resistance depending on light), force sensitive resistors (change resistance depending on pressure), bend sensor (change resistance depending on amount of bend), etc.

Here is and example with a photocell (light sensor):

analog04-photoresistor