MusicArc
A camera-tracked iPhone game for burn rehab: raise your arm to grow a tree, lower it to water it.
- Year
- 2025–26
- Reading
- 06 min
- Figures
- 13
MusicArc turns the painful, repetitive shoulder stretches of burn recovery into a tree-growing game. The iPhone’s front camera tracks arm height with on-device pose estimation: reaching toward the sun grows a procedurally drawn tree, and resting waters it — so the rest patients usually skip becomes part of the game. A research prototype from the USC Creative Media & Behavioral Health Center, developed with the Burn Unit at Los Angeles General Medical Center.
- Role
- Lead inventor · native iOS development
- Team
- Yiming “Danny” Huang
- Violet Wong
- Caitlyn Guo
- Floyd Scott
- Yiqi “Kiwi” Li
- Yunlei Liu
- Marientina Gotsis
- Disciplines
- iOS
- Games for health
- Computer vision
- Procedural graphics
- Rehabilitation
- Tools
- Swift / SwiftUI Canvas / Vision / AVFoundation / AVAudioEngine / SwiftData / Combine / XcodeGen / Unity
After a burn to the shoulder or armpit, patients must repeat overhead arm-raising and shoulder-extension stretches for weeks to months to restore range of motion and prevent scar contracture.
The exercises are painful and monotonous, and adherence at home is notoriously poor: patients shorten sessions, skip the clinically essential rest between stretches, or abandon the protocol entirely.
The exercise is the controller
MusicArc turns the stretch itself into the game controller. The iPhone’s front camera measures arm height in real time, and that height drives the growth of a procedurally drawn tree. Raising the arm above a threshold makes the tree grow — faster the higher it goes. Lowering it during rest “waters” the tree and earns a growth bonus for the next repetition.
Skipping rest damages the tree’s health, so the recovery periods patients most often rush through become rewarding instead. Raising the arm becomes a voluntary reach toward the sun rather than a clinical exercise to endure, with continuous growth, synthesised music and haptics as positive feedback during an activity otherwise associated only with discomfort.
Rest is part of the game, not downtime.”
Watering the tree
Lowering the arm brings night and rain. A well-rested rep banks a growth multiplier of up to 1.3× for the next stretch; holding the arm up during rest drains the tree’s health. It is the project’s central behavioural intervention: the interval patients most often skip becomes mechanically consequential.
The game also adapts to the patient, not the other way round. A short calibration records each person’s current range of motion and scores everything against it, so someone early in recovery with severely limited mobility gets the same full game as someone nearly recovered — and the clinician can tighten the prescription as range improves.
Fig. 02The sun follows the patient’s hand. Holding it above the dashed line grows the tree — faster the higher it goes. The card at top left shows the camera view with the tracking overlay.
Fig. 03Lowering the arm brings night and rain, filling the water level. A well-watered rest restores tree health and earns a bonus for the next rep.
Fig. 04Calibration records the patient’s current range of motion. In the simulator a silhouette stands in for the camera feed; the tracking overlay is the real one.
1.3×
maximum growth bonus from a well-watered rest
70%
rest compliance needed to restore tree health
30 Hz
game loop driven by live arm height
0
image or sound files shipped with the app
Why existing tools fall short
The team surveyed the tools in use at the LAGMC Burn Unit alongside published literature and available apps. General fitness and physical-therapy apps count reps or play videos; none are designed around the biomechanics of shoulder and axilla burn recovery, and none give feedback during the stretch.
Console and sensor-based exergames have shown comparable or faster range-of-motion recovery with less pain, but they rely on consoles, depth sensors or VR headsets in supervised settings — impractical for patients who already carry a heavy financial burden and exercise alone at home.
Meanwhile no tool tracked or rewarded rest, clinicians had no objective record of what happened between visits, and nothing offered continuous positive feedback to counter the pain and monotony that drive people to quit.
The forest is the record
Every completed session becomes a tree whose size and colour reflect how it went. Patients see their recovery accumulate; at follow-up visits, clinicians see frequency and quality at a glance, without relying on self-report.
The app is clinician-prescribed and patient-operated: behind a PIN, the therapist sets reps, timings and input mode once, and the patient just taps Begin. The front camera is the only sensor, so there is nothing to buy, charge or set up — and everything runs on-device. Camera frames are used in memory for pose inference only; no health data ever leaves the phone.
Fig. 10Every saved session becomes a tree, sized by that session’s growth. Clinicians can export the underlying data.
Fig. 11The clinician sets reps, hold and rest times, the input mode — camera, touch or auto-demo — and which arm to track. Test Run plays it without saving.
Drawn and composed in code
MusicArc ships with no image or sound files. Five tree species — oak, round, bushy, pine and acacia — are drawn in SwiftUI Canvas as continuous functions of growth and health, so a tree can be rendered at any moment between soil and full canopy. The sky, sun, rain, stars and particles are drawn the same way.
Every sound is synthesised at runtime: sine oscillators summed into chords for growth spurts, every 10% of growth, the turn from day to night, and an arpeggio when a session ends. The whole app uses only Apple’s first-party frameworks, with zero third-party dependencies.
Under the hood
- 01
Scale-invariant arm height
Vision’s body-pose request runs on front-camera frames off the main thread. Confidence-gated shoulder, elbow and wrist joints give (wrist − shoulder) ÷ arm length, remapped to 0–1, so the reading doesn’t change with distance from the phone. If a joint drops out it falls back to shoulder–wrist or shoulder–elbow; frames are dropped rather than queued, so the overlay never lags more than one frame.
- 02
Rest as a game mechanic
An @Observable 30 Hz state machine runs over a pre-generated rep timeline. Growth scales with how far the arm is above the sunlight threshold; time below the rest threshold fills a water level, and the next rep’s multiplier is 1 + 0.3 × water. Arm up during rest drains health; above 70% compliance it recovers.
- 03
Per-patient calibration
Raw height is remapped linearly through each patient’s recorded minimum and maximum, so very limited range of motion still gets the full game, and the prescription can tighten over time.
- 04
Five species in Canvas
A TreeRenderer protocol with five implementations of roughly 260–400 lines each draws soil, sprout, trunk, branches, canopy and fruit from growth ∈ [0, 1], and shifts colour with health ∈ [0, 1].
- 05
Audio that never blocks
Chords are written into 44.1 kHz PCM buffers with a linear attack and release. Engine start-up, synthesis and scheduling run on a private queue, because synchronous audio start-up — seconds on Bluetooth — used to freeze the UI.
- 06
Input-agnostic, on-device
Camera, touch-drag and auto-demo inputs share one PoseProvider protocol, so the engine has no input branches and runs in the simulator. Capture rotation comes from RotationCoordinator, fixing joint positions on the iPhone 17’s square front sensor. Sessions persist locally in SwiftData.
From a practicum to the clinic
MusicArc was conceived in CTIN 596, Research Practicum in Interactive Media, at the USC School of Cinematic Arts, under the supervision of the USC Creative Media & Behavioral Health Center. It was catalysed by a collaboration between the Center and the Burn Unit at Los Angeles General Medical Center through a NIDILRR grant led by Dr. Haig Yenikomshian, and occupational therapists from LAGMC’s outpatient unit advised on shoulder and axilla protocols and gave feedback on each iteration.
The idea took shape in September 2025, with a first clinic visit days later. The first playable version, built in Unity, followed in November and was playtested at a second visit in December. After that feedback, development moved to a native iOS app that patients can install on a phone they already own. That build began as a rhythm game, with notes to hit at target arm heights, before being rebuilt around the tree and its rest mechanic in March 2026.
MusicArc is a functional prototype under active development, now focused on usability testing at CMBHC. It has not been evaluated in a clinical study and is not a validated medical device. It is free and open source under GPL-3.0.
