What if a computer knew what you see?
What if a computer knew what draws your attention at the moment?


Takumi Toyama, M.Eng. B.Eng.

Knowledge Management
Trippstadter Strasse 122
D-67663 Kaiserslautern

+49 631 20575 1350


Since the birth of human beings, our attention system has been playing one of the most essential roles for information acquisition in natural environments. We ceaselessly collect a massive amount of information through our sensors, i.e., eyes, ears, skins,... etc. literally we sense a great deal of data in the natural environments; for instance, the resolution of human eyes is typically hundreds times more than that of the current digital camera. Nevertheless, the flow of such a massive amount of information does not overwhelm the capability of our brain system. One of the reasons for this is that our attention system continuously selects the important part from the huge amount of information resources by attending it; Our eye gaze always moves toward an important region in a scene in order to acquire the information required at the moment.

When we look into person's eyes, we could infer what the person is thinking or what the person needs because we know that the attention system controls the eye movements to collect necessary information through them. For instance, when a person is looking at a map in a city, we could infer that the person wants to go somewhere and wants to find the way to get there. When we find such a person, we, as a human can help the person for finding the way to get to the place. In the near future, an attentive computer might be able to help a human when he/she needs a help by inferring human attention in particular contexts.

Recent advances of the eye tracking technology makes this scenario more realistic. By simply wearing an eye tracking glasses, the user can get information when he needs it. We would like to present a couple of prototypes of attentive computers and the future of human-computer interaction.



- Eye Tracking

SMI Eye Tracking GlassesFrom the eye tracker, the computer receives scene images and two eye images (binocular). In the process of eye tracking, the gaze position in the given scene is computed. After computing the gaze position in the scene image, it also analyzes which kind of object is present in the image or in the user gaze. The computer also computes if the user's attention is on the object by analyzing the gaze information. According to the object (or text) presented in the user attention, the system provides the user with information (e.g. shop -> discount items, face -> the name of the person, text -> translation).

Eye Tracking


Here are the sample images of the scene image with the user gaze. Our proposed framework detects the user attention on these objects by analyzing the images.

Gaze on ObjectGaze on FaceGaze on DocumentGaze on Sign


- Augmented Reality

Brother AirscouterIn order to provide the user with visual feedback, we combined the eye tracking glasses with a see-through augmented reality display (head-mounted display). Brother Airscouter is mounted in SMI ETG with a snap-on frame. The user can get visual information on the see-through screen unobtrusively. By calculating the display location in a scene image, the system could also detect gaze on display. Thus, the user can select a button on display by gaze.

Airscouter on ETGAR on Document


Museum Guide 2.0 and TALKING PLACES


Museum Guide 2.0Museum Guide 2.0 is a sophisticated automatic guide system for a museum visitor. It analyzes eye movements of the visitor and recognize to which object he/she pays an attention. When the visitor looks at a particular object for a while, Museum Guide 2.0 supplies him/her with the information about the object with several different forms, such as audio playback, displaying on a smartphone and so on.

This system is realized by combining a mobile eye tracker (We use the SMI Eye tracking Glasses) with an object recognition engine. We also implemented an algorithm for distinguishing attentive gaze from non-attentive gaze in order to detect the viewer's interests on the objects.

Museum Guide 2.0 FigureObject Recognition Framework


Taking Places is a extension of Museum Guide 2.0. Here, the objects are buildings, shops, and signs, any object that exist in ordinary cities. You can get a bargain price of shoe shop, restaurant chef's recommendation, ...etc. by simply looking at them.
Talking Places




You are in a party. A man approaches to you and shakes your hands with a nice smile. "Hello! I wasn't expexting to see you here." You replies, "Me neither, what a coincidnece!" ...But you don't remember his name ...We often have such a problem in our daily life. Our proposed system could perform as your "external brain" to recall the face, the name, and even the personal information of the person. 
Face Recognition FrameworkFace Recognition Example



When we read a document, we often have a problem that the information described the reading document is too less to comprehend what the author(s) would like to convey. We demonstrate how the combination of a wearable eye tracker, a see-through head-mounted display (HMD) and an image based document retrieval engine enhances people’s reading experiences. By using our proposed system, the reader can get supportive information in the see-through HMD when he wants.
AR Document FrameworkAR Document Example 1AR Document Example 2



When you are in a foreign country, every text around you is strange. You would appreciate, if a local is always side by you and tells you the translation of the words that you do not know. Here, you just need to look at the texts (signs) to get the translation. Your attentive assistant helps you to be as if you are a local.

Gaze Translator Example 1Gaze Translator Example 2



Deutschland Land der Ideen: http://www.land-der-ideen.de/ausgezeichnete-orte/preistraeger/virtuelle-stadtf-hrung-talking-places



Video for Museum Guide 2.0



Takumi Toyama; Daniel Sonntag; Markus Weber; Christian Husodo Schulz
Gaze-based Online Face Learning and Recognition in Augmented Reality. In: Proceedings of the IUI 2013 Workshop on Interactive Machine Learning. IUI Workshop on Interactive Machine Learning, Poster Session, located at 18th International Conference on Intelligent User Interfaces, March 19-22, Santa Monica, CA, USA, ACM, 3/2013.

Takumi Toyama; Wakana Suzuki; Andreas Dengel; Koichi Kise
Wearable Reading Assist System: Augmented Reality Document Combining Document Retrieval and Eye Tracking. In: Proceedings of 12th International Conference on Document Analysis and Recognition. (ICDAR-13), Washington, DC, USA, IEEE, 8/2013.

Takuya Kobayashi; Takumi Toyama; Faisal Shafait; Andreas Dengel; Masakazu Iwamura; Koichi Kise
Recognizing Words in Scenes with a Head-Mounted Eye-Tracker. In: IAPR International Workshop on Document Analysis Systems. IAPR International Workshop on Document Analysis Systems (DAS-12), 10th, March 27-29, Gold Coast, Queensland, Australia, IEEE, 3/2012.

Takumi Toyama; Thomas Kieninger; Faisal Shafait; Andreas Dengel
User Gaze Detection on Arbitrary Objects Using Physical Sensors and an Eye Tracker in a Real Environment. In: Asia-Pacific Conference on Computer Human Interaction. Asia-Pacific Conference on Computer Human Interaction (APCHI-12), 10th , August 28-31, Matsue, Japan, ACM, 8/2012.

Takumi Toyama; Thomas Kieninger; Faisal Shafait; Andreas Dengel
Gaze Guided Object Recognition Using a Head-Mounted Eye Tracker. In: The Seventh ACM Symposium on Eye Tracking Research & Applications. Symposium on Eye Tracking Research & Applications (ETRA-2012), March 28-30, Santa Barbara, California, United States, ACM, 2012.

Takumi Toyama; Thomas Kieninger; Faisal Shafait; Andreas Dengel
Museum Guide 2.0 - An Eye-Tracking based Personal Assistant for Museums and Exhibits. In: L. Ciolfi; K. Scott; S. Barbieri (Hrsg.). Re-Thinking Technology in Museums 2011: Emerging Experiences. International Conference on Re-Thinking Technology in Museums, May 26-27, Limerick, Ireland, ISBN 1-905952-31-7 978-1-905952-31-1, University of Limerick, 5/2011.

Takumi Toyama
Object Recognition System Guided by Gaze of the User with a Wearable Eye Tracker. In: Rudolf Mester; Michael Felsberg (Hrsg.). DAGM'11 Proceedings of the 33rd International Conference on Pattern Recognition. Annual Symposium of the German Association for Pattern Recognition (DAGM-11), 33rd, August 30 - September 2, Frankfurt, Germany, ISBN 978-3-642-23122-3, Springer, Berlin, 2011.



We use a mobile eye tracker (SMI Eye Tracking Glasses) produced by SensoMotoric Instruments GmbH (SMI).

SMI Eye Tracking Glasses

SensoMotoric Instruments GmbH (SMI) 

SMI Logo

We conduct a user study in a real museum in cooperation with a local museum in Kaiserslautern.

Museum Pfalzgalerie Kaiserslautern 
mpk Logo
Museumsplatz 1
67657 Kaiserslautern

Telefon: 0631 3647-201
Telefax: 0631 3647-202

E-Mail: info@mpk.bv-pfalz.de