Table of Contents (hide)
- Computer-assisted Media Authoring Techniques
- The AlloSphere Research Facility
- Scheinwerfer / The Exploding, Plastic and Inevitable Redux
- ETH Zurich's Digital Art Weeks
- The SQEAK - Real-time Multiuser Interaction using Cellphones
- The Digital Marionette
- Rip my Disk
- Almost Lost with Heinrich Lüber
- Surréance: A River and Five Bridges
- The Pianist's Brain Waves
Computer-assisted Media Authoring Techniques
Ongoing
This is is the primary project I am currently working on. The project is funded by the Swiss National Science Foundation (fellowships for advanced researchers).
Project Description
The volume of digital media content has enormously grown in the recent years: Emerging internet-based applications, such as YouTube or MySpace, are media-rich, emphasise on content sharing, and attract huge communities. As the volume of content further expands, we see an increasing need for not only being able to annotate and retrieve content, but also to produce and edit content in a computer-assisted manner and by employing (semi-) automated methods. However, existing work so far does not satisfy these needs -- both in terms of usable software instruments as well the underlying theoretical frameworks. This viewpoint is also acknowledged by upcoming European Union IST projects: The Framework Programme 7 (FP7) calls for projects addressing Intelligent Content and Semantics and targets at advanced authoring environments for interactive and expressive content [ICT Programme Committee, 2007, Objective ICT-2007.4.2].
This project aims at a systematic classification and implementation of novel content authoring techniques. By "mixed-media content authoring" we understand the interactive editing, or the {\em composition}, of content for multiple modalities (e.g., auditory and visual). The project emphasises on applying computational methods in order to enable a high degree of automatisation of the authoring process. It further aims at techniques that are concerned with expressive human-computer interaction and those that need to operate in real-time, where needed.
Central to the project is the hypothesis that existing traditional composition techniques from individual domains (e.g., music, graphics, video) can be identified, classified, and then be generalised in a foundational framework of interactive media composition techniques that is applicable to automated authoring tasks. We claim that the implementation of these techniques will lead to more effective methods for simultaneous mixed-media authoring. The project starts out by establishing a catalogue of media composition techniques. This includes collecting and classifying existing techniques from specific modalities, identifying common methodologies, and also filling gaps and building bridges with new techniques where needed. Before actually implementing the composition techniques, an appropriate real-time content rendering infrastructure is required. Here, the project benefits from preliminary work carried out at the former Multimedia Laboratory of the University of Zurich and at the Computer Systems Institute of ETH Zurich. As its central part, the project implements relevant composition techniques as part of an interactive media authoring instrument. Special thoughts are given to a human-centred approach in order to prevent results that are stigmatised by software, and in order to provide the highest possible degree of expressiveness to content authors. At its final stage, the project empirically verifies the results by means of concrete application scenarios from media art and entertainment.
The project brings together elements from many different fields, primarily from software engineering, human-computer interaction, and the foundations of multimedia, but also from music composition, film editing theory, and aesthetics. The intellectual merit of the project is to provide systematic foundational work on mixed-media composition techniques by exploiting computational methods and to validate the practicability of the theory in terms of usable software instruments. Besides of dissemination of scientific findings, the broader impacts of the project are expected to be very significant for a wide range of applications that depend on computer-assisted and automated content creation techniques: Among others, example beneficiaries are the entertainment industry or commercial content providers who can benefit from novel expressive authoring instruments, or emerging services beyond IPTV, where automated content composition tasks help supporting whole communities. Collectively, the project aims at advancing the state of the art of interactive media content authoring to a new level.
The AlloSphere Research Facility
Ongoing
The AlloSphere is a large, immersive, multimedia and multimodal instrument for scientific and artistic exploration being built at UC Santa Barbara. Physically, the AlloSphere is a three-story cubic space housing a large perforated metal sphere that serves as a display surface. A bridge cuts through the center of the sphere from the second floor and comfortably holds up to 25 participants. The first generation AlloSphere instrumentation design includes 14 high-resolution stereo video projectors to light up the complete spherical display surface, 256 speakers distributed outside the surface to provide high quality spatial sound, a suite of sensors and interaction devices to enable rich user interaction with the data and simulations, and the computing infrastructure to enable the high-volume computations necessary to provide a rich visual, aural, and interactive experience for the user. When fully equipped, the AlloSphere will be one of the largest scientific instruments in the world; it will also serve as an ongoing research testbed for several important areas of computing, such as scientific visualization, numerical simulations, large scale sensor networks, high-performance computing, data mining, knowledge discovery, multimedia systems, and human-computer interaction. It will be a unique immersive exploration environment, with a full surrounding sphere of high quality stereo video and spatial sound and user tracking for rich interaction. It will support rich interaction and exploration of a single user, small groups, or small classrooms.
The AlloSphere differs from conventional virtual reality environments, such as a CAVE or a hemispherical immersive theater, by its seamless surround-view capabilities and its focus on multiple sensory modalities and interaction. It enables much higher levels of immersion and user/researcher participation than existing immersive environments. The AlloSphere research landscape comprises two general directions: (1) computing research, which includes audio and visual multimedia visualization, human computer interaction, and computer systems research focused largely on the Allosphere itself – i.e., pushing the state of the art in these areas to create the most advanced and effective immersive visualization environment possible; and (2) applications research, which describes the integration of Allosphere technologies to scientific and engineering problems to produce domain-specific applications for analysis and exploration in areas such as nanotechnology, biochemistry, quantum computing, brain imaging, geoscience, and large scale design. These two types of research activities are different yet tightly coupled, feeding back to one another. Computing research produces and improves the enabling technologies for the applications research, which in turn drive, guide, and re-inform the computing research.
Scheinwerfer / The Exploding, Plastic and Inevitable Redux
Ongoing (regular events at different locations)
The project "Scheinwerfer" is one of Corebounce's key activities. In the scope of this project, we create the visual experience at various electronic music events since 2001. Our live-composed and sound-driven visuals are designed to emphasise the theme of the event as well as taking into account the architectural framework. Not coincidentally, Singapore's Centro Club has labelled us as "Club Scientists": Our performances are deeply influenced by the momentary state of the Soundium research software platform. At the same time research is typically induced by specfic artistic performance goals.
Scheinwerfer has performed at some of the coolest locations around the globe, supporting world-class DJs and musicians like Jeff Mills, Rush, Miss Kittin, Dave Clarke, Josh Wink and many more.
The Exploding, Plastic & Inevitable, a recent collaboration of Scheinwerfer with Canadian artist Steve Gibson, is a digital abberation on the scale of its 1960's namesake: A circus of light and sound, a glut of technology, an overload of the senses, an immersion in an audiovisualscape. The EPI has been performed at Digital Art Weeks 2007, Zurich, where it attracted more than 1000 spectators, and at a similar scale at Interactive Future 2007, Victoria, Canada. Stay tuned and don't miss future performances!
→ Corebounce
→ Scheinwerfer
→ The Exploding, Plastic & Inevitable
ETH Zurich's Digital Art Weeks
ETH Zurich, Switzerland, 2005 - 2007
From 2005 to 2007, I was scientific chair of ETH Zurich's Digital Art Weeks. The Digital Art Weeks, an annually recurring festival, are concerned with the application of digital technology in the arts. Consisting of a symposium, workshops and performances, the program offers insight into current research and innovations in art and technology as well as illustrating resulting synergies in a series of performances, making artists aware of impulses in technology and scientists aware of the possibilities of the application of technology in the arts.
The SQEAK - Real-time Multiuser Interaction using Cellphones
ETH Zurich, Switzerland, 2006-2007
This research project explored one approach to providing mobile phone users with a simple low cost real-time user interface allowing them to control highly interactive public space applications involving one user or a large number of simultaneous users. In order to sense accurately the real-time hand movement gestures of mobile phone users, the method uses miniature accelerometers that send the orientation signals over the network’s audio channel to a central computer for signal processing and application delivery. This affords that there is minimal delay, minimal connection protocol incompatibility and minimal mobile phone type or version discrimination. Without the need for mass user compliance, large numbers of users could begin to control public space cultural and entertainment applications using simple gesture movements.
The Digital Marionette
Bellerive Museum, Zurich, Switzerland, 2004; Ars Electronica Center, Linz, Austria, 2006-2008
The interactive installation Digital Marionette impressively shows the audience the look and feel of a puppet the multimedia era: The nicely dressed wooden marionette is replaced by a Lara Croft-like character; the traditional strings attached to puppet controls emerge as inputs to computer controls.
The installation consists of a projection of a digital face, which can be controlled by the visitors. The puppet can be made talking via speech input, and the classical puppet controls serve as controllers for head direction and face emotions, such as joy, anger, or sadness. The whole artistic concept was designed and realised in an interdisciplinary manner, incorporating art historical facts about marionettes, the architectural space, interaction design, and state of the art research results from computer graphics and speech recognition.
Rip my Disk
Interactive Futures 06, Victoria, BC, Canada, 2006; Digital Art Weeks 07, Zurich, Switzerland, 2007
The project "Rip my Disk", presented at Interactive Futures 2006, brings mobile art to the dancefloor by blurring privacy and bringing personal content to the big screen. In an augmented VJ performance, visitors can enter into a dare by letting the Scheinwerfer team "rip" multimedia contents off their personal handphones with the Soundium program to get exposed ("virtually naked") on the dance floor. Others, not so akin to risk taking, can simply enjoy employing their handphones for interactive painting as well as sending media sources such as live video and image material. The sent media is used as personal artifacts, which are continuously adapted and integrated. The result is the personal enhancement of the space around the media owner's location. The project also demonstrates how mobile communication technologies can easily be made accessible to artists or performers.
Almost Lost with Heinrich Lüber
Digital Art Weeks 06, Zurich, Switzerland, 2006
Performative audio-visual installation in collaboration with artists Art Clay and Heinrich Lüber: Streams of spoken words are deformed by interactive technology.
The project was officially supported by Kultur Basel-Stadt and Kulturelles Basel-Land.
Surréance: A River and Five Bridges
Digital Art Weeks 05, Zurich, Switzerland, 2005
This performance was realised in collaboration with Art Clay (Composition) and Anne Faulborn (Cembalo). Art Clay's composition "A River and Five Bridges" is an electronic adaptation of André Bretons "Barrières" and asserts the affinity of rational and irrational. The visual compostion's interpretation, realised by us, absorbs the assertion by implementing a modulated flickering screen. The modulation function is both defined by real-time music analysis and live visual performance. The flicker colors were obtained by two cutouts from Paul Klee's "The Twittering Machine", one in blue tones and one in red tones.
The Pianist's Brain Waves
Musik - Denken - Spielen, School of Music, Drama and Dance, Zurich, Switzerland, 2002
Guerino Mazzola's free jazz improvisation of over a motif of Modes Mussorgsky's "Pictures at an Exhibition" was accompanied by a live visuals interpretation. Instead of music analysis we used an EEG (Electrocncephalogram) to measure Mazzola's brain waves in real-time and use the data for spatial distribution and movement of geometric shapes and different color modulations.