| Instructor: | Karl Yerkes (yerkes@mat.ucsb.edu) |
| TA: | Tim Wood (fishuyo@mat.ucsb.edu) |
| Where: | Elings 2003 (Conference Room) |
| When: | Mondays and Wednesdays, 10:00am to 11:50am. Office hours TBA |
Hands-on introduction to developing multimedia applications. Representation and perception of sound, images, and time. Media computing paradigms including OOP, callbacks, multithreading, OpenGL, distributed computing, algorithmic control, indeterminacy, real-time interactivity, and mapping data between sensory modalities. Students develop a series of audiovisual works (as C++ software) leading to a final project.
Computing with media data, like most interesting topics, has deep aspects and superficial aspects. The goal of the course is to teach the deep aspects: principles, concepts, techniques, and theory. But to do a hands-on project and actually implement something (which is hugely important to me) you must also master many superficial aspects: specific programming languages (in this case C++ along with several libraries including OpenGL, AlloCore, Gamma, and GLV), syntax, alternate names of things, where you need to put your files, etc. I learned programming in the 1990s with a substantially different set of tools from what we use today. Inevitably much of our hard-won superficial knowledge becomes obsolete. But, the deep knowledge is still relevant, useful, and essential to our work. If you understand the deep aspects then you can always easily learn new superficial aspects just by looking them up. But without the deep knowledge you might not even be able to understand the documentation you find.
We see a large part of our role as instructors of this course to be a sort of guide that points out what is deep and what is superficial. I will make every effort to teach the former and I will encourage you to learn the latter on your own, from each other, and from the TA. If you try to learn something superficial and it's really hard then there's probably something deep we need to talk about; please don't be shy with questions.
With that said, I am very excited about the specific state-of-the-art software we will be teaching this quarter: AlloCore. The AlloSphere Research Group, with lessons learned from the first few years of developing content for the AlloSphere, created AlloCore, a C++ library at the center of the shared functionality of the software that creates various real-time interactive, 3D audiovisual immersive virtual worlds. AlloCore is the "core" part of the larger AlloSystem, now hosted on http://github.com, and is still under active development and expansion.
This will be our fifth year teaching 201B using AlloCore. It's an excellent system, possibly the best, for developing general interactive audio and/or visual processing software. Knowing this system will be extremely useful if you want to develop content for the AlloSphere and/or participate in the AlloSphere Research Group. Finally, using it for this course will naturally uncover bugs, generate a demand for useful new features, and expand both AlloCore and its documentation through student contributions. But most importantly, AlloCore will allow you to learn the deep aspects of computing with media data while building a tangible hands-on project.
Many of the details and rough edges of AlloCore involve shallow knowledge that may be especially challenging and/or frustrating; that's often what it's like doing research on "the bleeding edge." Please do not hesitate to ask questions or point out possible bugs.
If you do not have your own computer (or for any other reason) the instructors can set you up with access to various machines on campus.
Development environment: alloproject with its run script
Development cycle: Start with a working example - edit - compile - grok error messages - repeat - run - printf debugging. Answering questions with trivial programs that test hypotheses.
All linking handled through AlloSystem's build system and the "flags.txt" mechanism; no direct configuration of linking parameters. Conceptual understanding of what linking means, dead code elimination.
Control flow (main, procedure definitions and calls, if/for/while, callbacks)
Explanation of some meanings of "static"
Templates: why they exist, how to use them. (Not how to define templates nor template metaprogramming)
User’s view of STL: performance tradeoffs based on required operations, why to prefer STL vectors over C arrays, storing pointers in STL containers rather than objects (to avoid default copy constructor weirdness).
All "struct" (not "class"); no private or protected class variables.
Memory model (memory as array of bytes, pointer as index into this array, procedure call stack, heap, global variables)
Scope of variables
Big-oh notation: what is "N"?, why the most important issue is usually the limit as N is large, nested for loops means polynomial time, dividing problem in half at each stage means log(n).
OOP: classes, instance variables, methods, inheritance, virtual functions, pure virtual functions and abstract base classes. Inner classes. Operator and function overloading.
Callback-based programming (based on App class’ onAnimate(), onDraw(), and onSound(), also Cuttlebone's callbacks).
Understanding where to put code (main(), constructor of App subclass, App callbacks, other classes, non-class functions).
Conceptual introduction to threads: separate audio and video threads, possible issues with synchronization (e.g., deleting data read by the other thread), recommendation of MessageTube (single reader / single writer FIFO) for inter-thread communication. No detailed explanation of other synchronization mechanisms or hands-on experience using threads for multicore parallelism.
Distributed processing, state synchronization, message passing
Installation (via git)
Federation of libraries: Gamma, GLV, OpenGL, AlloCore, AlloUtil, Cuttlebone, etc.
How to use documentation.
Finding examples and unit tests
Tour and demos on first day of class.
Generally not available to students.
Explanation of tracking system and its affordances; opportunity for student projects to use it.
Explanation of surround 3D graphics, how to draw in surround, state synchronization (Cuttlebone)
Color: RGB vs. HSV.
Alpha blending.
OpenGL’s unitless spatial coordinate space.
OpenGL draw modes.
Pushing and Popping OpenGL matrices (without really understanding the content of these matrices or even why they’re "matrices.") Convention of objects drawing themselves with unit size at the origin and then calling Translate, Rotate, Scale
Understanding that AlloCore is a thin wrapper around OpenGL. Why not to use immediate mode (faster and to support OpenGLES); how AlloCore "fakes" immediate mode with a Mesh object.
Sampled sound: bit depth and sampling rate. Aliasing.
Gamma’s use of Generator Pattern (calling unit generator class as procedure with zero arguments that returns the next sample). Assembling unit generator graphs (DAGs) by composing these functions within onDraw(). LFOs, mixing.
Frequency: range of human hearing.
Simple left/right panning (equal amplitude vs. equal power). Use of AlloCore’s AudioScene (Ambisonics under the hood).
Block processing, conceptual model.
How to use filters: filter types (lowpass, highpass, bandpass, notch, allpass) and their parameters (center frequency, resonance)
Handling QWERTY and mouse events. Consuming events or passing them up the GLV tree.
Accepting OSC input.
Demo of the Device Server (Charlie Roberts)
Used as a tool for setting parameters (position, color, etc.) of lots of agents without specifying each individually.
Users’ view of a few random distributions (uniform, Gaussian...).
How to get randomness within a given range.
Various methods for choosing a random color.
Different time sources: Audio callback, Video callback, CPU wall time,CPU timer.
Asking for the current time: MainLoop::now()
Discretization/quantization of time. Why running a n-body simulation with a noninfintessimal time step is not strictly correct.
Why to use a revision control system. Development history of a software project. checkout, update, status, diff, commit, log. Branching and merging.
Submitting class assignments via git.
25% Participation (attendance, contributing to class)
25% Homeworks
50% Final project
| Mailing list: | (Please Sign Up Here!) 201b@lists.mat.ucsb.edu |
| Student interest survey: | http://goo.gl/forms/LeAy6ep79o |
| Class Tumblr: | [to be created] |
| Previous Tumblr 2014: | http://mat201b2014.tumblr.com |
| Previous Tumblr 2013: | http://mat201b2013.tumblr.com |
| Previous Tumblr 2012: | http://mat201b-2012.tumblr.com |