State-of-the-art pose estimation models were trained on general human motion. But athletes move differently — faster, more asymmetrically, under physical load. CalTennis is the first large-scale sports AI benchmark built specifically to expose that gap. The sharpest thing about it is the framing: the generalization of pose estimation models to sports has been assumed rather than measured. This post (summarised in the video above) walks through what the benchmark found, and why naming a failure matters more than reporting a score.
Table of Contents
You can’t improve what you can’t measure
Generic pose models are trained on everyday motion. Athletes move differently, and until now the assumption that those models carry over to sport was exactly that — an assumption. CalTennis provides an empirical basis for the claim that standard benchmarks do not predict performance on athletic motion, and it begins to isolate which performance dimensions are specifically degraded. That isolation is the prerequisite for targeted methodological progress: you cannot fix what you have not localised.
CalTennis — the dataset and the benchmark
CalTennis (Caltech) is a large multi-view tennis video dataset and benchmark for monocular-to-3D pose estimation: 40 players (professional and amateur) and over 11 million synchronized multi-camera frames, with 3D ground truth obtained by triangulation — precise skeleton coordinates for every frame. It is public under CC BY 4.0, which is what makes it a foundation others can build on rather than a one-off result.
The headline finding is a dissociation. Relative joint angles: current models do reasonably well. Absolute depth accuracy and foot contact timing: they fail systematically. The paper also argues that existing metrics such as MPJPE are inadequate for sport, and proposes new sport-specific ones.
That pattern is diagnostic, not just disappointing. It is consistent with the hypothesis that models trained on quasi-static general motion learn appearance-based shortcuts for joint configuration — shortcuts that do not transfer to the temporal and depth demands of fast athletic action. Whether sports-specific training data alone would close the gap, or whether it reflects a structural limitation requiring architectural change, is not resolvable from benchmark performance alone.
Why depth and foot contact failing together matters
This is the part worth dwelling on. Depth estimation and ground contact timing are mechanically coupled in many athletic tasks, because contact forces depend on both. So a model that fails both simultaneously is likely to produce biomechanically implausible estimates precisely during the high-load events most relevant to injury risk.
In other words, these are not two independent bugs that happen to co-occur — they are the two inputs to the quantity you most want, failing exactly where you most need it. A model can look competent on joint angles and still be useless for the landing, the cut, or the serve.
The benchmark as a compass
CalTennis does not just expose weaknesses — it names exactly what to fix: depth estimation and ground contact timing. The benchmark’s real contribution is making this failure mode testable and reproducible, which is the precondition for eventually resolving it. As the video puts it: a good benchmark does not measure progress, it defines it.
For how today’s models score on general motion, see our breakdown of pose estimation accuracy across six models. The depth problem CalTennis isolates is the same one that separates the setups in our overview of 3D markerless motion capture approaches — and it is the shared ceiling running through the four papers putting the biomechanics lab in your pocket.
Frequently asked questions
What is CalTennis?
A large multi-view tennis video dataset and benchmark from Caltech for monocular-to-3D pose estimation: 40 professional and amateur players, over 11 million synchronized multi-camera frames, and 3D ground truth via triangulation. It is released publicly under CC BY 4.0.
Where do pose estimation models fail on athletic motion?
Relative joint angles hold up reasonably well. Absolute depth and foot contact timing fail systematically. The likely reason is that models trained on quasi-static everyday motion learn appearance-based shortcuts that do not transfer to fast, loaded action.
Is MPJPE a good metric for sports?
Not on its own. CalTennis argues that standard metrics like MPJPE are inadequate for sport — they can look acceptable while the depth and contact information that sports biomechanics actually depends on is wrong — and proposes new sport-specific metrics instead.
Reference
[1] Demler, I., Xie, X., Werner, B., Szczuka, A., & Perona, P. (2026). CalTennis: Large Multi-View Tennis Video Dataset and Benchmark of Monocular-to-3D Pose Estimation. Caltech. arXiv:2606.20542 (CC BY 4.0)
Takashi Fukushima — Sports Science & Pose Estimation.
▶ Subscribe on YouTube · Website · ORCID · Contact