Update!

I watched the recent Kinect SDK launch which gave me some ideas about how to use the Kinect.  One interesting point was that when designing Kinect applications, design them from scratch so don't just take a previous application and implement a Kinect interface.  This lead me to the idea to use kinect as more a live virtual instrument.  Whereas before I was going to have an application similar to a digital audio workstation which would be controlled with Kinect (as the posted image shows).

Now I will concentrate on implementing different ways to interface with a live virtual instrument.  I am going to have it set up as a sever/client application, so that potentially many clients could play an instrument together.  Right now I now I have most of the server done.  The server is designed to accept MIDI events from the clients, so it can pass these into the virtual instruments and play back the sound.

For each VST instrument I am planning to have an XML document associated with it.  This will read when the instrument is loaded in.  It will contain the information needed to know how to manipulate it when it is being used.  For example, when the right hand is moved up it will change the filter cut-off frequency between these values.  I am planning on having different ways to interact with the instruments, so the XML file will show which ways can be used with which instruments and how they will work.

More GUI

I have been working on loading .obj files into my program.  I am doing this so can load 3D objects in which I can use in my interface.  I can use Vector Magic to trace the outline and then use that outline in my interface.

I am also designing my overall program, and how it will fit together.  I am considering what architectural pattern to use.

GUI

Recently I have been looking into the interface design and implementation.  I have been created a storyboard of how the interface might work using PowerPoint.  At the moment I am going to have a augmented reality design.  I will have the interface projected over a simplified camera image of just the user.

I am going to use Windows Presentation Foundation (WPF) as my graphical subsystem.  I am going to create the interface in a 3D environment.  I have been programming simple interfaces that will use the same concepts as the real interface, just to learn how it is down.

I have been playing with the Kinect libraries to see how they work.  I found that Microsoft is releasing a official Kinect SDK very soon.  So I plan to wait till that is released to see if it is worth using.

The Start

I have decided I would like to use the Kinect to allow users to compose music.  I have several ideas how the Kinect could be used, though I will decide on the implementation later.  The two main ideas so far are:

• Allow users to play the instrument itself, hum a tune (use of pitch detection) and use gestures to control arrangements, volumes..
• Users are the music, as their movements would control the sounds of the music.  This would use algorithmic composition.

In the highest level there will be the Kinect/controller system and the audio system. The Kinect system will take the users movements/voice and either translate them into midi notes or a changes to the system.  The audio system will have to load virtual instruments and play the midi notes.

 

So far I have been primarily concerned with audio system as to see how difficult it is to implement.  Two technologies I have been looking at are:

• Steinberg's Virtual Studio Technology (VST) is an interface to connecting virtual instruments and host systems.
• Audio Stream Input/Output (ASIO) is a sound card driver for low-latency digital audio.

Both the SDKs are for C++ applications.  Though I have been able to find ports for both of these in .NET: VST.NET and ASIO.NET.  ASIO.NET seems to be an older port that is not supported anymore, but the functionality is there.  The demo application was not working, and there seemed to be a problem with the DLL on my system.  I have since managed to recompile it and both libraries now work with C#.

 

So far I have managed to load a instrument and send it a midi note.  I then can use the output and input buffers to play the sound.  Though I am not sure if my implementation is the best way to do it.  Both ports are not well documented.  And there are not many examples of how to use them.  Maybe it is better to look at the original libraries they wrap to understand more about them.

 

I will continue attempting this implementation and try optimise how the VST and sound card interact.