Virtual Keyboard Project

This system is presented as the final project for “Engineering Approaches to Music Perception and Cognition” (ISE-575).

The idea behind the project was to create a virtual keyboard to enable the computer users, who do not have access to a midi keyboard, play and record music on their computers using a non-expensive three dimensional input device.

In this project I combined the animating hand that I had designed and implemented for Omni-Grasp project, and Improv, a library for programming with MIDI in C++. As input device I use P5 Glove which at each time provides the three dimensional position and posture of the hand.

User wears a P5 Glove (displayed below) and moves her hand and fingers in the space as she is playing on a keyboard. P5 Glove provides the information regarding the three-dimensional location and direction of the hand for the program. It also sends some information regarding the situation of the fingers which can be interpreted as the angles of the joints in fingers. The price and portability of this device makes it easily accessible for everyone. Unfortunately the data it provides is very noisy and not always accurate. Overall it is a good starting point.

Because the input data can be extremely noisy at times, movement of the hand is being restricted to only horizontal and vertical movements, as well as the normal behavior of the fingers. Therefore, the data regarding the depth and direction of the hand is ignored.

The structure of the system can be roughly divided into three highly correlated units:

·        Graphics and Animation Unit

·        Collision Detection Unit

·        Music unit (Virtual Keyboard)

After analyzing the input data, the Graphics and Animation Unit then processes the information and applies all the required transformations to the base hand model, draws it on the screen and animates the hand when the glove is moving in the real world. Collision Detection Unit detects any contact between the hand and the virtual keyboard object and reports it to the Music Unit or the Virtual Keyboard, which then detects which key is pressed and plays the corresponding note.