Social Robots and Virtual Humans in 3D TelePresence 

Nadia MAGNENAT-THALMANN
MIRALab, University of Geneva, Switzerland
Institute for Media Innovation
Nanyang Technological University, Singapore
YUAN Junsong
School of Electrical & Electronic Engineering
Nanyang Technological University, Singapore
Daniel THALMANN
École Polytechnique Fédérale, Switzerland
Institute for Media Innovation
Nanyang Technological University, Singapore

Abstract

The objective of this tutorial is to explain the methods to make autonomous virtual humans and social robots communicate with real users in 3D TelePresence applications. The autonomous virtual humans and social robots are supposed to give a partial illusion that they are real. This implies that these autonomous virtual humans and social robots should look the same as the real human, speaks with the same intonation, and be aware of the real situation, the real participants, and the task currently performed. They should show emotions, have a memory, and be able to have a relationship with the user.

The autonomous virtual humans and social robots should react at the right time based on the perception he/she has from the real participants. It implies to evaluate what each real participant is doing. Perception will be obtained by visual and audio input and recognition. In particular, it is of great importance to understand the action and gesture of human beings. In the past few years, there has been a lot of progress in action recognition with conventional video cameras. Effective techniques have been developed to address many challenging issues in real world environments. More recently, the availability of commodity depth cameras has brought a new level of excitement to this field. Rapid progress has been made to address new technical issues in action and gesture recognition with depth cameras. In this tutorial, we introduce the basics for human action and gesture analysis, using both regular and depth cameras.

Each autonomous virtual humans or social robots should also be aware of the other autonomous virtual humans and social robots, which needs to introduce a synthetic perception for the virtual humans. The autonomous virtual humans and social robots react according to the input and its current knowledge. They should also have a visual attention to show their interest in the environment. Their reactions encompass animation (body and facial gestures) and speech synthesis. A common platform for autonomous virtual humans and robots will be presented.

In this tutorial we will address the following topics:

  • Introduction to virtual humans and social robots
  • Recognizing and creating emotions
  • Defining memory process and relationship models
  • Understanding human actions
  • Gesture recognition with regular and depth cameras
  • Perception of the virtual world, virtual senses
  • Generation of motion and animation, visual attention
  • A common platform for autonomous virtual humans and social robots