Lindsay Atrium VP - Josh Kraft

Project Overview

Led by Dr. Christian Jacob, this project aimed to bring the Lindsay Atrium from Twinmotion into Apple Vision Pro using Unreal Engine. The goal was to leverage the AVP's spatial computing and high-fidelity visuals to create an immersive, educational experience showcasing detailed anatomical models of the human body and other biological structures.

We had to overcome many challenges, but ultimately, we were able to create a functional prototype that allows users to explore and interact with anatomical models on the apple vision pro.

Key Features:

Mixed Reality Passthrough

An (almost) seamless integration of virtual anatomical models with the physical environment using the Vision Pro's passthrough capabilities.

Hand Tracking Controls

Custom gesture controls built from scratch to enable (relatively) intuitive interaction with 3D models using your hands.

High-Fidelity Models

Anatomical models in immersive spatial viewing (getting these to render properly and without dissapearing or crashing the entire app was potentially the most difficult part!)

Spatial Interaction

The ability to walk around, reach through, and view anatomical models in 3D space for comprehensive study.

Unreal Engine to Vision Pro Development Pipeline

Perhaps most compelling, we were among the first people at this time to develop a full pipeline for bringing Unreal Engine 5.3 projects into the Apple Vision Pro while integrating spacial interaction and computing features.

Technical Challenges:

Documentation Gaps: There is (or was at the time) a severe lack of documentation on macOS, VisionOS, and Unreal Engine interaction which necessitated extensive trial-and-error debugging.
Development Suite Configuration Issues: Mismatches and incompatibilities between XCode, MacOS, Unreal Engine, and VisionOS exacerbated these issues, not to mention smaller issues such as Apple Developer Team ID and provisioning bugs.
Model Compatibility: The entire Twinmotion Atrium crashed on AVP even when stripped to simple assets. We had to abandon Twinmotion models and use USDZ models instead, likely due to a disconnect between Unreal's RHI to Apple's Metal API.
Platform Instability: Frequent unexplained launch failures and crashes with no logs necessitated multiple project rebuilds and constant reinstallation cycles.
Hand Tracking: We had to build custom gesture controls from scratch as Unreal lacks native access to the AVP gesture controls and OpenXR VisionOS plugin capabilities were primitive. This required a lot of fine tuning to make the feature intuitive.

Learning Outcomes

The greatest takeaways from this project all revolved around the challenges we faced. Bringing the prototype to life required overcoming many hurdles - ones that lacked documentation, error codes, and in general, clear solutions. I learned that while being an early adopter of new technologies can be frustrating, it also provides some of the realest learning opportunities and allows you to accomplish things that very few people already have.

|