Melissa Carrasco - Study in Interaction and Form Matching

The goal of this project is that through the interaction, the participant will become more aware of how they can move and the shapes and forms their body can create. This project uses computer vision and image processing technology to create an interaction that will motivate the viewer to participate, move and fashion themselves in poses that they would not place themselves in their everyday context.

The full person would be tracked and their form projected onto the screen along with a form presented by the system that they are to match. It would be in a space where the background as well as lighting could be controlled. The full interation flow is described in the image below.

The prototype is an implementation of this idea at a smaller scale. The system watches the users hand via a web camera. A form, a hand shadow puppet, is presented to the user and the user must pose their hand so that their form matches. This prototype uses openFrameworks and OpenCv on Windows. openFrameworks is a C++ library and wraps many other API's including OpenCv. OpenCv is a library that provides functions for real time computer vision. Additional information on these two libraries can be found by clicking on the respective links below.

Some challenges encountered when implementing this prototype was that lighting had to be control so that the user's hand did not cast large shadows onto the background. If the shadow was noticable, the system would think that it was either another form and outline its contour or think the shadow was part of the hand causing a distorted contour.

The next step with this project would be to scale this prototype up to track the full body and match against full body forms. I've began testing tracking the full body and found that some parameters and thresholds need to be tweaked to get the desired performance.