Lemonade
Zach Davis, Haewon Kang, Will Wolcott
Lemonade is software that allows a user to interact with live video processing using the XSENS MT9, a device which captures orientation in three dimensions (among other things). Each dimension in orientation is mapped to video processing technique, and the user can thus control the video output in real-time.
The primary goal of this project was to take a first step towards understanding
the ways in which the XSENS MT9 could be used to provide an interactive
experience to the user. Other goals included using real-time video processing
as the basis of interaction and creating a situation where a user with
no prior knowledge could, and would want to, explore the boundaries between
abstract and recognizable .
Development
The Result
The final form of Lemonade creates a 3-dimensional topography
based on color-values of the image and controlled by changing the orientation
of the XSENS MT9. For each frame (15 frames per second), the red, green,
and blue values of each pixel are determined. This information is used
to create a polygon of the same color, and the average intensity of these
color values is used to determine the polygon's "height", or
distance from the background. A polygon is created for each pixel in the
frame, and together they form a 3-D topography of the frame. Pitch, as
captured by the MT9, is used to scale the height of each polygon. Scaling
by zero will thus produce a flat, "normal" image, while scaling
by a large amount will produce a steep topography. Roll is used to control
the size of each polygon. Zero roll will create a polygon for each pixel
in the frame. Increasing the roll will increase the number of pixels each
polygon represents. Yaw is used to change the output frames from full
color to no color. A yaw of zero will produce a normal full-color image.
A yaw of 180 degrees will produce a solid-color image (each polygon is
the same color), but will maintain the 3-D topography or height of each
polygon. Using these three controls , the user can explore the possibilites
offered by the three processing techniques, as well as play with the limitations
of what makes for a recognizable image.
Whie work on Lemonade itself has probably concluded, the hope
is that this was merely the first in a series of projects exploring the
use of the XSENS MT9, or other devices like it, for interaction in an
art or other installation. The idea is that the MT9 would allow precise
measurements of the user's actions to be taken, without seriously imposing
on that user's ability to interact with the work. These measurements could
be extremely useful in integrating scientific or psychological experiment
into an art exhibit, and is something we hope to explore more in the future.
Lemonade.exe v1.0 for Windows XP and 2000
Lemonade Source Code - C++ and Visual Studio .NET project files
Install and Instructions README
rtf file