Image Construction - Tracking and Rendering - Challenges
Step four consists of continual tracking to determine the orientation of the head. This is again accomplished via the wall-mounted cameras and back-projection, except that only a horizontal stripe (instead of the whole sphere) need be tracked since the contrast between the face and the hair will provide enough information to determine the orientation of the head. This orientation information is sent to the main server.
The final step is to render three reconstructions of the viewer's head in their central and peripheral vision. Since the main server now has the real-time position and orientation of the individual, the location and orientation of the projected heads can be determined simply by forward-projection and the angles involved (ie. the angle of peripheral vision). The 3D reconstruction created when the person first entered the room is then rendered at the correct locations in the space via multiple projection units.