Title : Multi-modal cues for virtual sketching and related tasks

Project Lead : Adam Csapo From : Budapest University of Technology and Economics (None)

Dates : from -- to 2014-09-01 13:35:49

Description :

Motivation and objectives :
The motivation of this proposal is to explore the possibility of creating audio-based CogInfoCom channels in the application area of collaborative virtual sketching. Cognitive Infocommunications (CogInfoCom) is an emerging interdisciplinary research field at the meeting point between the cognitive sciences and infocommunications. My research in the past has dealt with creating a formal definition of CogInfoCom channels -- structured multi-sensory messages which reflect semantic concepts -- and a framework for their design and implementation. One of the main contributions of this framework is that it links syntactic elements with semantic interpretations across sensory modalities. Further, it defines mapping techniques which allow for the user to create a conceptual link between CogInfoCom channels of one sensory modality and CogInfoCom message generated concepts usually encountered through a different modality. Finally, my recent research has dealt with creating interpretable tuning models which allow the user to tune high-dimensional generation parameter spaces used to generate CogInfoCom channels through a lower-dimensional set of tuning parameters. The project is motivated by the results and lessons learned from a previous TNA project carried out at the same institution (Fraunhofer IPK) in December, 2012. By the end of the project, we had developed a working example in which the degree of straightness of simple line-based drawings in a virtual sketching application was mapped onto an audio-based CogInfoCom channel. However, although the application was working, due to a lack of time there was no opportunity to extend the use of the developed audio-based channels to more complex shapes (e.g., for extruded shapes and more complex drawings consisting of more than a single line), or to conduct usability tests and reach any general conclusion on what kind of information is useful to users when complementing the visual feedback of their drawing actions with audio. In this project -- in agreement with the hosting laboratory -- we would like to continue our work along these lines. Following the project, we intend to publish a paper on our results.

Teams :
One of the consequences of research on Internet of Things and 3D Internet is that 3D virtual environments are expected to become interconnectible in the near future, allowing for the shared representation and manipulation of distributed objects and systems, physical and virtual alike. The primary goal in this context is to design, implement and integrate solutions that allow for communication, collaboration and control through such augmented and shared 3D environments. Our research group has an interdisciplinary profile which is comprised of augmented/virtual reality, basic research on cognitive infocommunications and control theoretical approaches. An interdisciplinary combination of these research directions reflects the kinds of interdisciplinary challenges which must be tackled for the seamless integration of physically remote environments with virtual representations for communication, collaboration and control.

Dates :
starting date : 28 October, 2013
ending date : 08 November, 2013

Facilities descriptions :
http://visionair-browser.g-scop.grenoble-inp.fr/visionair/Browser/Catalogs/VPCC.GE.html

Recordings & Results :
During the project, our goal was to augment visual feedback with feedback through audio-based CogInfoCom channels during oversketching operations in a virtual sketching application.Audio-messages were designed based on a model of friction using the Spiral DiscoveryMethod. Besides the shape of the drawing that was being oversketched, the friction model wasalso informed by the speed of the gestures used for oversketching.

Conclusions :
Development of a oversketching algorithm with audio-based feedback. Accomplishment of a user study.




Project Images :





Other project resources :

FinalSummary.pdf