Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Biderman, Dan, Whiteway, Matthew R, Hurwitz, Cole, Greenspan, Nicholas, Lee, Robert S, Vishnubhotla, Ankit, Warren, Richard, Pedraja, Federico, Noone, Dillon, Schartner, Michael, Huntenburg, Julia M, Khanal, Anup, Meijer, Guido T, Noel, Jean-Paul, Pan-Vazquez, Alejandro, Socha, Karolina Z, Urai, Anne E, Cunningham, John P, Sawtell, Nathaniel B, Paninski, Liam (April 2023) Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools. bioRxiv. ISSN 2692-8205 (Submitted)

[thumbnail of 10.1101.2023.04.28.538703.pdf] PDF
10.1101.2023.04.28.538703.pdf - Submitted Version
Available under License Creative Commons Attribution No Derivatives.
Download (23MB)

Abstract

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Item Type: Paper
Subjects: organism description > animal behavior
neurobiology
neurobiology > neuroscience
CSHL Authors:
Communities: CSHL labs > Churchland lab
CSHL labs > Engel lab
CSHL labs > Zador lab
SWORD Depositor: CSHL Elements
Depositing User: CSHL Elements
Date: 28 April 2023
Date Deposited: 26 Nov 2024 14:50
Last Modified: 26 Nov 2024 14:50
PMCID: PMC10168383
Related URLs:
URI: https://repository.cshl.edu/id/eprint/41742

Actions (login required)

Administrator's edit/view item Administrator's edit/view item