3-D virtual studio for natural inter-"acting"

Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on "blue screen" technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment.