2025 · Wearable AI · Capstone Project

ARTag — Heads-up Multiplayer AR Laser Tag

A fully functional wearable AR laser tag system combining motion-sensing hardware, real-time AI inference on FPGA, and spatial AR visualization into a seamless multiplayer gameplay experience.

Role

XR Developer

Tools

Unity · OpenAI · Python · MQTT · Microsoft HoloLens 2 · FPGA

Duration

4.5 months (Capstone)

Team

William Wang · Shixing Tian · Sebastian · Hongyao Lim · Yizhe Yan

Situation

ARTag was my final-year Computer Engineering capstone (CG4002) at NUS, where a team of five built a laser tag system covering five mandatory components: custom hardware, AR game output, internal communication, external communication, and FPGA-based AI action classification. I volunteered for the AR game output component, becoming the first and only person in the course's history to use Microsoft HoloLens 2 for a heads-up AR display instead of a smartphone.

ARTag was my final-year Computer Engineering capstone project (CG4002) at NUS, undertaken with a team of five. The module required teams to build a laser tag system from scratch, with each team member owning one of five mandatory system components:

  1. 01Custom hardware for game input
  2. 02Custom software for AR game output
  3. 03Internal communication between all system components across hardware and software
  4. 04External communication with the game evaluation server
  5. 05An AI model deployed on FPGA for real-time user action classification across different game attacks

I volunteered to own the AR game output component, and was the first and only person in the course's history to use Microsoft HoloLens 2 for the heads-up software layer responsible for visualizing live match data and special game effects directly in the player's first-person view.

CEG Capstone Project - AR Laser Tag System

CEG Capstone Project — AR Laser Tag System

Task

My scope covered the full AR software layer: a heads-up AR game visualizer on HoloLens 2 showing live health, shields, and attack outcomes in the player's first-person view; the AR image recognition anchor for player tracking; a conversational AI game tutor; and integration of all components with the hardware system via MQTT.

As the owner of the AR game output component, my responsibilities were:

  • Design and develop the heads-up AR game visualizer on Microsoft HoloLens 2 — displaying live player health, shield status, and attack outcomes directly in the player's first-person view
  • Integrate the AR visualizer with the rest of the system — receiving real-time inference results from the FPGA action recognizer and game state from the evaluation server via the MQTT broker

Actions

I self-taught HoloLens 2 development via Microsoft Learn, tested four AR anchor designs to find the optimal head-attached configuration, and built an OpenAI-powered conversational AI tutor in a 3D mixed reality tutorial room. I volunteered to set up MQTT integration after a team ownership gap emerged, contributed action data for AI model training, and stepped in to maintain team morale during a repeated Bluetooth connectivity crisis — ultimately helping the team reach ~90% AI action prediction accuracy.

Developed the heads-up AR game visualizer

Motivation Traditional game UIs sit on flat screens. Translating live match data — health bars, attack effects, shield states — into spatially grounded AR overlays that feel natural in a physical laser tag environment required rethinking game UI from the ground up
Action
  1. Built the HoloLens 2 Unity application with real-time AR overlays showing player health, shield status, and attack outcomes in the player's first-person view
  2. Designed spatial UI elements anchored to the physical world rather than screen-space to maintain immersion during gameplay
  3. Implemented dynamic visual feedback — hit animations, shield break effects — triggered by live game state from the evaluation server
AR Visualizer UI

AR game visualizer UI on Microsoft HoloLens 2

Challenge No one in the cohort — nor in the course's entire history — had used Microsoft HoloLens 2 to build their AR visualizer. There was no existing project to reference, and I needed to quickly familiarise myself with HoloLens 2 development early in the project to avoid falling behind on my component
Solution Completed numerous Microsoft HoloLens 2 tutorials on the Microsoft Learn website — including the HoloLens 2 Fundamentals and Mixed Reality Application Development courses — which equipped me with the technical proficiency needed to programme custom applications for the HoloLens 2 independently

Designed a user-friendly AR image recognition anchor

Motivation For the HoloLens 2 to accurately detect and track the opposing player's position, that player must wear an image recognition anchor — similar to a QR code — that the headset can lock onto. The size, position, and visual appearance of this anchor needed careful design to maximise two dimensions of usability: the HoloLens's ability to consistently detect and track the anchor, and the comfort and satisfaction of the player wearing it
Action Designed and tested multiple AR anchor variants across different sizes (A4 vs A6), placements (head-attached, chest, shoulder, mask-based), and visual designs — evaluating each on tracking reliability and wearer experience
A6 Mask-based Image Target

A6 — Mask-based

A6 Shoulder-top Image Target

A6 — Shoulder-top

A4 Image Tracker on Player Chest

A4 — Player chest

A4 Head-attached Image Target (Final)

A4 — Head-attached ★ Final design

Integrated with the full hardware system

Motivation The AR visualizer was useless in isolation — it needed to receive live data from the FPGA action recognizer, game evaluation server, and wearable sensor hardware in real time for the full gameplay loop to function
Action
  1. Implemented MQTT-based communication connecting the HoloLens 2 to the broker laptop, game evaluation server, and FPGA inference output
  2. Coordinated with teammates responsible for the FPGA pipeline and sensor hardware to align data formats and message schemas
  3. Debugged end-to-end latency across the full system to ensure AR feedback felt responsive during live gameplay
Challenge At the start of the project, the responsibility for setting up communication between HoloLens 2 and the rest of the system was unclear — I assumed it would be handled by the teammate working on internal communication, while that teammate assumed it fell under my AR output component. This created an ownership gap that risked leaving my visualizer disconnected from the rest of the system
Solution After clarifying the misalignment through team communication, and recognising that my teammate had enough on their plate with other components, I volunteered to take ownership of this integration. I independently set up the MQTT connection between HoloLens 2 and the rest of the system by working through YouTube tutorials, successfully bridging my AR visualizer to the game evaluation server and FPGA inference pipeline

Supported team-wide AI training data collection

Motivation The teammate working on the AI component needed action data from all team members to train a model capable of accurately classifying different game inputs — a single person's data would lack the variability needed for the model to generalise across different users. For example, a punching action should be classified as a punch, while a circular hand motion in front of the user should activate the shield. Everyone volunteered their time to contribute action samples for training
Action
  1. Provided repeated action data for all game attacks — including fencing attack, badminton attack, and shield activation — to build a diverse and representative training dataset
  2. Helped troubleshoot hardware issues with the team during data collection sessions, such as inconsistent Bluetooth BLE connection between the wearable sensors and the laptop
Challenge Team morale hit an all-time low midway through the project when the hardware periodically lost its Bluetooth BLE connection with the laptop for no apparent reason — erasing all the data collected in that batch. This happened repeatedly, forcing the team to redo entire sets of actions each time and dragging the data collection process well beyond its intended duration
Solution I stepped in to calm the team down and encouraged everyone to rally around the teammate responsible for hardware and internal communication to debug the issue together, rather than letting frustration pull the team apart. Through collective troubleshooting, we resolved the connectivity issues and collected enough data to train an AI model that achieved approximately 90% action classification accuracy

Implemented a conversational AI game tutor

Motivation For a holistic user experience, I wanted to create an in-world tutorial space where players could learn the game rules in mixed reality before competing — rather than relying on a flat instruction screen that would break immersion
Action
  1. Created a separate mixed reality space that users can enter in HoloLens 2 for the tutorial experience
  2. Integrated the OpenAI API to build an embodied conversational agent that accepts freeform speech input from the user and answers questions about game rules in natural language
Development Process behind Conversational AI Game Tutor

Development process behind the conversational AI game tutor

Challenge No one else in the cohort was building a game tutorial or integrating the OpenAI API in mixed reality — there was no existing project to reference. Debugging was particularly difficult because errors occurring inside the mixed reality environment were hard to surface and translate onto a desktop for investigation
Solution By reading through the OpenAI developer forums and Stack Overflow, I was able to diagnose and resolve all the errors that arose during development, and successfully shipped an embodied conversational agent that could interact with users in mixed reality to explain game rules

Results

Our team was the first and only in the cohort — and in NUS CEG capstone history — to deliver a heads-up AR system using HoloLens 2 instead of a smartphone, achieved the second highest AI action prediction accuracy in the cohort, and secured an A grade for the capstone.

Beyond Expectation
  • The first and only team in the cohort — and in NUS CEG capstone course history — to complete the project with a wearable heads-up AR display using Microsoft HoloLens 2, instead of a smartphone
  • Achieved the second highest AI game action prediction accuracy score in the cohort during the final evaluation
  • The team secured an A grade for the capstone
Meeting Expectation
  • Delivered a fully functional wearable AR laser tag system integrating motion-sensing hardware, FPGA-based real-time action recognition, spatial AR visualization, and a conversational AI game tutor
First and Only Heads-up AR Display CEG Capstone Project in NUS CEG Course History

First and Only Heads-up AR Display CEG Capstone Project in NUS CEG Course History

Key Takeaways

ARTag taught me to own complex technical workstreams in a hardware-software team, learn unfamiliar platforms under time pressure, proactively bridge communication gaps through volunteering, and maintain team cohesion during critical setbacks.

Through this project, I strengthened my ability to:

  • Own end-to-end technical workstreams within a multidisciplinary hardware-software team
  • Work with unfamiliar technologies such as Microsoft HoloLens 2 under time constraints through self-directed learning
  • Hands-on development and design in mixed reality — spanning AR game visualisation, conversational agents, and multiplayer game systems
  • Bridge ownership misunderstandings in team task allocation through proactive communication and volunteering to fill gaps
  • Maintain team unity and morale during unexpected setbacks, keeping the team focused and collaborative despite repeated failures
Full system architecture