Qian Liu |Ritesh Lala
Re-Act
Fundamentals of Spatial Computing
Re-Act is a metaphor for a continuously changing, shifting and evolving virtual entity that responds to sensory cues from the outside world and updates its form as a re-action. The form is computed with several iterations of a number of parameters based on the Super-formula proposed by Johan Gielis, first published in the American Journal of Botany in 2003 and described as "A generic geometric transformation that unifies a wide range of natural and abstract shapes". The selection of the form not only made the process computationally chellenging to implement but also artistically stimulating to comprehend. The formula and how it reacts to the audio-visual cues are described in the following sections.
Re-Act from Qian Liu on Vimeo.
Form (Ritesh)
The form of the reacting entity is based on the following parametric equation, and can be computed by its polar co-ordinates.
raux=abs(1/a(1).*abs(cos(n(1)*u/4))).^n(3)+abs(1/a(2).*abs(sin(n(1)*u/4))).^n(4)
r=abs(raux).^(-1/n(2))
Converting the co-ordinate system to cartesian we can obtain x and y positions of the points defining the outline of the contour by the following equations:
x=r.*cos(u)
y=r.*sin(u)
Further the equation was generalized for a 3 dimensional space by obtaining the spherical products of the above formulae for another polar parameter v (where v varies from -pi/2 to pi/2 and u from 0 to 2*pi).
We observed several shapes each visually distinct and abstract for each distinct set of parameters. A continuous increment was added to the parameters to obtain a smooth change in the form as opposed to a jumpy flickering one. A certain range of parameters was selected to conform the entity to a cyclic transition between two states most of which is a visual chaos which converges to a split second of order and then back to chaos again.
Visual Input (Qian || Ritesh)
The visual agents that cause a reaction in the form are colors of objects displayed in front of the camera. OpenCV was used to identify the color of an object situated in the center of the screen and process it to apply on the form vertices. This is done by obtaining the image scanning a certain range of pixel values in it and mapping the brightness values to corresponding RGB values with OpenCV. Although the initial reactions result only in the shade of the form, further complexities can be introduced to change the size, shape and nature of the form as well as the luminance and transperency characterisitcs of the scene. An improvement to the interface could be small square objects with different patterns on all sides to control the visual reactions, since it is also possible to scan and process more colors from a scene.
Auditory Input (Qian || Ritesh)
The auditory agents that cause yet other reactions in the entity are beats detected from the input sound. The detection is based on the BeatDetection class introduced in the Minim library for audio processing in Processing. It functions in two modes: Sound Energy and Frequency Energy modes of which the latter was utilized to detect clear beats of snare, kick/bass drum and high hats.
(Qian)
We tried many different shaders for the right look and feel of the form. The various shaders that were tested are listed below. Finally we used "Gooch Shaders" to experiment with different combinations of warm and cool colors. In the process we overcame many technical difficulties in trying to apply a shader to a form completely described by its polar coordinates.
OpenGL Shading Language(GLSL)
GLSL (OpenGL Shading Language), also known as GLslang, is a high level shading language based on the C programming language. It was created by the OpenGL ARB to give developers more direct control of the graphics pipeline without having to use assembly language or hardware-specific languages.
With the recent advances in graphics cards, new features have been added to allow for increased flexibility in the rendering pipeline at the vertex and fragment level. Programmability at this level is achieved with the use of fragment and vertex shaders.
GLSL shaders themselves are simply a set of strings that are passed to the hardware vendor’s driver for compilation from within an application using the OpenGL API’s entry points. Shaders can be created on the fly from within an application or read in as text files, but must be sent to the driver in the form of a string.
The set of APIs used to compile, link, and pass parameters to GLSL programs are specified in three OpenGL extensions, and became part of core OpenGL as of OpenGL Version 2.0, except for geometry shaders, this became included in the core OpenGL as of OpenGL Version 3.2.
Non-photorealistic rendering
Non-photorealistic rendering (NPR) is an area of computer graphics that focuses on enabling a wide variety of expressive styles for digital art. In contrast to traditional computer graphics, which has focused on photorealism, NPR is inspired by artistic styles such as painting, drawing, technical illustration, and animated cartoons. NPR has appeared in movies and video games in the form of "toon shaders," as well as in architectural illustration and experimental animation.
The term "non-photorealistic rendering" was coined by David Salesin and Georges Winkenbach in a 1994 paper.
Three-dimensional NPR is the style that is most commonly seen in video games and movies. The output from this technique is almost always a 3D model that has been modified from the original input model to portray a new artistic style. In many cases, the geometry of the model is identical to the original geometry, and only the material applied to the surface is modified. With increased availability of programmable GPU's, shaders have allowed NPR effects to be applied to the rasterised image that is be displayed to the screen. The majority of NPR techniques applied to 3D geometry are intended to make the scene appear two-dimensional.
Toon Shader
Toon Shader, (also known as Cel shading or Cel-shaded animation) is a type of non-photorealistic rendering designed to make computer graphics appear to be hand-drawn. Toon-shading is often used to mimic the style of a comic book or cartoon. It is a somewhat recent addition to computer graphics, most commonly turning up in video games. However, the end result of Toon shading has a very simplistic feel like that of hand-drawn animation.
The Toon shader process starts with a typical 3D model. Where Toon shader differs from conventional rendering is in its use of non-photorealistic lighting. Conventional (smooth) lighting values are calculated for each pixel and then mapped to a small number of discrete shades to create the characteristic flat look – where the shadows and highlights appear more like blocks of color rather than mixed in a smooth way.
Hatch Shader
The Hatch shader is Sketch and Toon’s most advanced shader. It places a picture of a stroke (or any other picture you want it to use) multiple times onto the object to create a hatched lines effect. The Hatch shader’s settings give you full control over how these strokes are placed, making it possible for you to create any type of hatched lines effect.
The goal in a hatch shader is to render an object in a way that makes it look hand-drawn, for instance with strokes that look like they may have been drawn with pen and ink. Each stroke contributes to the viewer's ability to comprehend the tone, texture, and shape of the object being viewed. The effect being sought in this shader is that of a woodcut print. In a woodcut, a wooden block carved with small grooves is covered with ink and pressed onto a surface. The image left on the surface is the mirror image of the image carved into the wood block. No ink is left where the grooves are cut, only where the wood is left uncut. Lighting is simulated by varying the width of the grooves according to light intensity.
Combination of Hatch and Toon Shader
Gooch Shader
Gooch shading is a non-photorealistic shading model developed primarily for technical illustration. It outlines the object and uses warm and cool colors to give the sense of depth desired for technical illustrations.
Johan Gielis, "A generic geometric transformation that unifies a wide range of natural and abstract shapes", (American Journal of Botany. 2003;90:333-338.)
M. Brown, "Minim and Beat Detection", Computational Artwork [Online:] http://matthewbrown.net.au/programming/minim-and-beat-detection/
OpenCV for Processing: http://ubaa.net/shared/processing/opencv/
GLSL: http://en.wikipedia.org/wiki/GLSL
Shader Wiki: http://en.wikipedia.org/wiki/Shading
Toon Shader: http://en.wikipedia.org/wiki/Cel-shaded_animation
Gooch Shader: http://www.cycling74.com/docs/max5/tutorials/jit-tut/jitterchapter41.html
Non-photo realistic: http://en.wikipedia.org/wiki/Non-photorealistic_rendering
Download the code here.