Cutting latency by up to 15% across our full device fleet, using AI to crunch data and adjust algorithms

The "lag" problem

At Hinge Health, our TrueMotion™ computer vision pipeline runs directly on a member's phone or tablet, mapping their body in real time, plotting the position of each joint and body part in both 2D and 3D space. This data powers our 'smart skeleton' visualization, which overlays that mapping directly onto the member's camera feed, as well as other tools like our computer vision motion assessments (CVMA), exercise tracking and corrective guidance.

Under the hood, this pipeline operates across multiple concurrent threads. For this task, we focus on three: one captures frames from the camera, another runs a machine learning inference pipeline to digitize and capture the human in the view, and a third, the interpolator, stitches everything together and estimates missing data.

The interpolator exists primarily to fill in the gaps between real data points, producing a smooth, constant-rate output stream even when device limitations restrict the speed at which we can process the camera's frame rate. Running inference is not instantaneous. As we continuously deploy updates and better-trained models to improve accuracy, the computational cost inevitably rises. Beyond just processing power, device throttling is a major factor, often stronger at the beginning of a session before dropping, along with multiple other unknowns that vary from run to run.

To handle this variance, the interpolator introduces a delay that is constantly updating in real-time. This buffer allows the system to smooth out the jittery arrival of inference results. The challenge is calibrating that delay. It needs to be long enough to ensure enough data is available for interpolation in the "most likely worst case" scenario, but short enough that the member doesn't feel a disconnect between their movement and the on-screen feedback.

We identified algorithmic improvements to optimize this balance, but tuning and fine-tuning the multiple parameters feeding these algorithms presented a massive challenge.

We needed to evaluate hundreds of parameter combinations across our full device fleet, testing against videos with different frame rates and durations to measure performance differences between short user sessions and extended, QA-style runs. In practice, the Hinge Health app is supported on roughly 300 iOS and Android device variants, spanning about 160 distinct phones and tablets. That scale is central to the engineering challenge and the opportunity: unlike MSK solutions built around a single device, TrueMotion has to work well on the device a member already owns, so we have been exploring AI-driven ways to monitor and optimize performance across that broad fleet.

By shifting our focus from writing test cases to harness engineering, we not only compressed the time required but also generated significantly more data points, leading to more robust and higher-confidence results.

The heuristic trap and a dynamic solution

The interpolator's delay is governed by a "most likely" worst-case estimate: how long might it take for pose data to become available for a given frame?

Our original formula was a simple static calculation. While it served us for a time, it failed to account for the dynamic variance of real-world device performance. It struggled to track quick shifts in averages during throttling events and lacked the intelligence to optimize resource usage, often wasting cycles on stale frames just milliseconds before a fresh one arrived.

We replaced the static heuristic with a more intelligent approach. Instead of a fixed multiplier, the new algorithm improves the tracking of historical data to better estimate processing times and analyzes when expected data is set to arrive, allowing the system to adjust dynamically.

Counter-intuitively, the new system occasionally introduces intentional micro-delays when doing so avoids committing to longer, less productive work. This small, tactical trade-off lowers the expected worst-case time for the system as a whole, resulting in a net decrease in latency and a smoother member experience.

The theory was sound, but the parameters were unknown. What works on a high-end device might fail on a low-end one, varying significantly with different video frame rates, operating systems, and platform behaviors. We needed to find a configuration that worked everywhere.

The harness engineering approach

Finding values that work for a single scenario on a single device is already a challenge. Finding values that hold across our entire supported fleet, spanning multiple platforms, OS versions, thermal states, and video frame rates, introduces a problem of an exponential scale.

Rather than manually scripting these permutations, we used AI coding assistants to architect a testing "harness", a scaffold that automates the tuning, deployment, execution, and data aggregation of the experiments, iterating through the cycle and analyzing the gathered data to pinpoint the best results.

Phase 1: Implementation

We used AI to implement the algorithmic variations we wanted to validate. The primary focus was making key parameters runtime-configurable so values could be adjusted dynamically without rebuilding. We also designed a robust telemetry system to capture granular performance statistics, carefully accounting for our concurrent threading model to ensure clean data logging.

Phase 2: Infrastructure & execution

We built the infrastructure to execute these tests remotely on mobile devices. The harness deployed the build, iterated through multiple parameter combinations and tuning scenarios, and ran them against a suite of test videos to capture data across different scenarios. It gathered all the data into a standardized, well-formatted structure, splitting the workload: critical metrics were calculated on-device, while the harness performed the final statistical aggregation after receiving the results. Crucially, the harness also defined a priority order, establishing which configurations to test first and how to fall back to safe baselines if aggressive settings failed.

Phase 3: Analysis

With the data collected, we tasked the system with directly analyzing the results against our specific thresholds. It filtered through the thousands of data points, comparing availability against baselines and ranking configurations based on worst-case performance to identify the optimal setup that met our strict acceptance criteria.

Phase 4: Reporting

Finally, we generated a comprehensive report. This document detailed the exact recommended values backed by the data, provided a clear rationale for the selection, and defined the precise steps for the rollout.

The results

This extensive tuning process enabled us to find the best combination, improving latency on all devices by a significant factor. We saw an average gain of 5% across our test device fleet, scaling up to a 15% gain on lower-end devices, which was the main target of this improvement. That mattered because the goal was not to optimize for a single benchmark phone, but to make TrueMotion feel faster and more reliable across the hundreds of iOS and Android device variants our members actually use.

Crucially, we achieved this with a negligible impact on interpolation availability. The system remains robust, maintaining smooth output while making the experience feel significantly more responsive and "snappier" for the user.

Looking ahead

These improvements are shipping in our next release, bringing a more responsive experience to members on older devices. However, the lasting legacy of this project is the validation of the methodology itself. We have found that Harness Engineering is a force multiplier, and we plan on leveraging it more heavily.

As agentic programming advances and models become capable of learning and executing increasingly complex tasks, we intend to expand in this direction. We aim to extend this approach beyond parameter tuning, using these tools to assist in implementing and validating algorithms themselves and fundamentally improving our existing mechanisms.