2024 · Interaction Design · Wearable AI

TIMI — Heads-up Context-Aware Multimodal Travel Assistant for AR Glasses

An interaction design portfolio for a wearable AR travel companion that enables heads-up, context-aware information access across four key travel tasks — without pulling users' attention away from their experience.

Roles

Interaction Designer Usability Researcher Product Manager

Tools

Figma · Pupil Labs eye tracker

Duration

4.5 months (CS3240)

Team

William Wang · Siyuan Tong · Xiaoyang Chen · Javier Ng

Situation

TIMI was a 4.5-month interaction design project (CS3240) at NUS. I identified that smartphone travel apps force heads-down interaction that detracts from the travel experience, and saw a perfect use case for wearable heads-up devices given their hands-free interaction, contextual awareness, and wearable nature. I proposed what became the course's first ever heads-up wearable AR interaction design project.

TIMI was a 4.5-month project for CS3240 (Interaction Design) at NUS. The course was traditionally focused on web-based and smartphone-based interaction design — no team in its history had ever proposed a heads-up, wearable AR interaction design project.

Smartphone-based travel apps are widely used by travellers, but their heads-down interaction model constantly pulls attention away from the experience of being somewhere new. I identified this as a meaningful design problem — and saw a perfect use case for wearable heads-up devices, given their hands-free interaction, contextual awareness, and wearable nature. I proposed what became the course's first ever heads-up wearable AR interaction design project.

Task

I independently proposed the project scope, recruited three teammates based on their complementary backgrounds, and led the design of TIMI — a heads-up, context-aware multimodal travel assistant for AR glasses covering four travel tasks: information retrieval, note-taking, photo-taking, and navigation.

My responsibility was to independently propose the project scope, recruit teammates, lead the product vision, and drive the design of TIMI — a heads-up, context-aware multimodal travel assistant for AR glasses that enhances user experience across four key travel tasks:

  1. 01Information retrieval — e.g. looking up an unknown animal species encountered while travelling
  2. 02Note-taking — e.g. recording interesting observations about a museum artifact
  3. 03Photo-taking — e.g. capturing a beautiful photo of nature hands-free
  4. 04Navigation — e.g. finding your way from your hotel to a tourist location

In addition to design leadership, I coordinated research, delegated deliverables across the team, and conducted usability evaluation using mixed methods including eye-tracking.

Actions

I strategically recruited teammates by analysing their pitches for skill fit, led interaction design for the information retrieval task, and coordinated weekly meetings to align design language across all four tasks. Despite having zero prior Figma experience, I learned the tool on the job alongside our interaction designer while applying my HCI knowledge to guide design decisions. I also co-designed a Wizard of Oz evaluation using Figma, Google Meet, and Pupil Labs eye-tracking, assessed through semi-structured interviews, NASA TLX, and SUS questionnaires.

Defined the project scope and led product vision

Motivation AR glasses interaction design was completely unexplored in the course's history — someone needed to make the case for it to the instructor, build a team willing to take the risk, and define what success would look like for an entirely new interaction paradigm
Action
  1. Independently proposed the AR glasses project direction to the course instructor, framing it around the observation that desktop and mobile UI/UX has reached near saturation and wearable UX is the next design frontier
  2. Recruited three teammates willing to work outside the course's conventional project templates
  3. Led product vision, defined design principles, and delegated workstreams across the team throughout the semester
Challenge After pitching the idea during the class's allocated pitch session, only one person approached me to join the team — but a team of four was required for the project to go ahead. Without two more members, the project could not proceed
Solution I reviewed everyone else's pitches carefully and identified two candidates: one with a computer science background and an interest in game design — suggesting familiarity with Unity, an essential tool for mixed reality development — and another with an interaction design background who loved designing for tool-related activities, which aligned well with TIMI's travel assistant concept. I reached out to both individually, met up with each for a quick chat, and successfully recruited them to form the team of four needed to move the project forward

Designed the multimodal interaction system

Motivation AR glasses have no touchscreen — every interaction must be hands-free, context-aware, and low-friction so users can stay present in the environment rather than managing a UI
Action
  1. Broke down the interaction design task by assigning one team member to each of the four tasks, and personally led the interaction design for the information retrieval task
  2. Coordinated weekly team meetings to align design language across all four tasks — particularly between information retrieval and note-taking, which share similar interaction patterns
TIMI multimodal interaction design

TIMI multimodal interaction design for AR glasses

Challenge Despite leading the design effort, I had zero prior background in interaction design and had never used Figma — yet I was responsible for driving the team's design direction
Solution I learned Figma concurrently with the project by working directly alongside our team member with an interaction design background, picking up the tool while actively contributing to design discussions. At the same time, I applied my knowledge in HCI, mixed reality, and heads-up computing to guide how the team should approach designing for a wearable AR context — learning the craft and shaping the vision in parallel

Conducted mixed-method usability evaluation

Motivation We wanted to realistically understand users' experience with the proposed system — but within the time constraints and scope of an interaction design course, building a fully functional prototype was not feasible. We needed a way to simulate real usage without a working implementation
Action
  1. Designed a Wizard of Oz evaluation system using Figma, Google Meet, and the Pupil Labs eye tracker — enabling realistic simulation of the AR interaction experience without a functional prototype
  2. Applied mixed methods to identify usability issues: semi-structured interviews, NASA TLX workload questionnaire, and SUS (System Usability Scale) questionnaire
  3. Iteratively refined the design based on findings from the evaluation for the final submission
TIMI usability evaluation

Wizard of Oz usability evaluation session

Challenge The key challenge was how to realistically simulate users performing all four tasks — information retrieval, note-taking, photo-taking, and navigation — on a system that relied on gaze and voice input, without a working prototype
Solution We designed a Wizard of Oz setup where a team member acting as the "wizard" monitored the participant's real-time gaze patterns via the Pupil Labs software, then manually controlled the mouse cursor on the laptop hosting the Figma prototype to simulate the user's gaze-based selections. This allowed us to faithfully test both the gaze-based and voice-based interactions we had designed, giving participants a realistic sense of the system without any functional code

Results

TIMI became the first AR smartglasses interaction design portfolio in CS3240's course history. The team achieved full marks on all three project milestones, and all four members received a minimum A grade.

Beyond Expectation
  • TIMI was the first AR smartglasses interaction design portfolio in the entire course history of CS3240 at NUS
  • Achieved full marks on all three major project milestones in the course
  • All four team members achieved a minimum of an A grade for the course
Meeting Expectation
  • Delivered a high-fidelity Figma prototype and usability evaluation report for a heads-up multimodal AR travel assistant covering information retrieval, note-taking, photo-taking, and navigation

Key Takeaways

TIMI sharpened my ability to identify emerging design opportunities, lead teams through ambiguous self-directed work, learn new tools on the job, and evaluate non-standard interfaces with mixed methods.

Through this project, I strengthened my ability to:

  • Identify and frame novel design opportunities in emerging technology spaces before they become mainstream
  • Lead a team through ambiguous, self-directed design work with no existing precedent to follow
  • Apply mixed-method evaluation — combining eye-tracking with qualitative methods — to validate interaction design decisions for non-standard interfaces
Official project website
Video demo of high-fidelity prototype