Multimodal Analytics and Multi-user Natural Interaction

With the explosion of Internet use in every aspect of our lives, and the dramatic expansion of social applications accessed through massively connected smart devices making heavy use of sensors, the amount of data generated grows every day at an unprecedented pace. Together with the increasing complexity and systemic nature of situations humans now face, this poses serious challenges for decision making, especially in situations where users collaborate to address multiple aspects of a problem. Scalable software dealing with big-data storage, retrieval, and analytics are now widely available on the market. However, most can only be operated by specialists or power users with high expertise and lack seamless integration with intuitive and reflective collaboration support tools.

There is a major need to directly target non-expert customers, practitioners and the general public, and provide appropriate representations of full information and intuitive affordances for heterogeneous stakeholders to design solutions, solve problems or make decisions in complex and systemic situations. As a technological response to bridge the gap between data complexity and effective collaboration, multimodal analytics integrates technologies and methodologies to provide interactive and configurable visual, audible, tangible, gesture-based, verbal and non-verbal knowledge representation and systems interaction channels between users and technology-supported complex collaborative task performance in a reflective setting where human-to-human interactions are inherently part of the system.

Our research challenges

Research conducted at the Luxembourg Institute of Science and Technology (LIST) aims at creating and validating technologies that support efficient collaboration in designing artefacts, solving problems and making decisions in complex and systemic situations, based on multimodal knowledge representation and task execution using vision, hearing, speech, touch, or other non-verbal communication channels between and among stakeholders and applications or simulations. A current focus is on the use of Natural User Interactions, and more particularly Tangible Tabletops, to enhance collaboration and decision making through the intuitive manipulation of multi-variable and multi-objective simulations for rapid problem space exploration and generation of what-if scenarios. Applications in logistics, construction, environment, mobility and education have been created.

Research questions that we tackle include:

  • How to create adequate and more natural affordances in human-machine interfaces to foster collaboration in complex situations with heterogeneous stakeholders
  • How to add reflectivity to the system: for example, how to provide feedback to users and to the system about the efficiency of the task performance and the collaboration to trigger system's adaptation
  • How to automatically interpret human-to-human collaboration patterns as part of the system's reflectivity.

Our competencies

The team is made up of computer scientists and user experience specialists, who are supported by software and micro-electronic engineers with digital fabrication skills and competencies ranging from sensor data analysis and image analysis to tangible and multimodal collaborative application development and usability testing including tangible metaphor studies.

Our equipment

Two 50" MultiTaction Multi-touch and tangible tabletops, one 77x122cm and two 48" home-made tangible tabletops based on ReacTIVision, Collaborative table for real-time distant document editing, 3D printers, sensor platform, indoor geolocalisation systems.

Research domains
  • IT

Share this page:

Research Group flyer

Download PDF Flyer