Since 2015, consumer level capturing and display devices for Virtual and Augmented Reality (VRAR) have been readily available, opening up an emerging new market for VRAR. Of particular interest are 360° videos (360-videos) which can be shown in head mounted displays (HMDs), providing a wide field of regard and immersive viewing experience with realism. Recent advances in hardware has allowed for live streaming 360 videos, providing users a sense of telepresence to the place that the 360 video is being streamed from. However, VR experiences based on 360 videos have limited user interaction, since video is typically consumed through passive observation.

We present a novel technology called MR360 to seamlessly blend virtual objects into live streamed 360 videos in real-time. This provides the VR user the illusion of interacting with the scene objects in the video due to the coherent illumination, shadows, and composition between real and virtual scene computed in real-time. This is conceptually augmented virtuality with 360 videos, and the details of our technology, evaluation, and the applications will be presented and discussed.

Furthermore, other research and practices at Computational Media Innovation Centre (CMIC), a new entrepreneurial research centre at Victoria University of Wellington, New Zealand to link academic research to future commercialisation and startups, will be introduced.

Prof. Taehyun James (TJ) Rhee, is an associate professor (full professor in US system) at Victoria University of Wellington, New Zealand, a founder of Victoria Computer Graphics Degree, founder/director of the Victoria Computer Graphics Research Lab, co-director of Computational Medial Innovation Centre (CMIC), and a founder and leader of the mixed reality startup, DreamFlux, and serve(d/s) for a chair of SIGGRAPH Asia 2018 Virtual and Augmented Reality, and Pacific Graphics 2020.

Before joining Victoria in 2012, he was a principal researcher in the Mixed Reality Group, Future IT Centre at Samsung (2008-2012). Also, he was a senior researcher/researcher of Research Innovation Center at Samsung Electronics (1996-2003). His current research activities are focused on cinematic real-time lighting and rendering, mixed reality, immersive visualization and interaction, and human digital content interaction.