IEEE TMI · Minor Revision

Endoscopic tissue point tracking

Track tissue through what the camera cannot trust.

Endo-TTAP combines motion, semantics, and uncertainty with a two-stage flow-point supervision strategy—keeping trajectories reliable through deformation, occlusion, jitter, reflection, and smoke.

Compare tracking Read preprint Code

Rulin Zhou^*, Wenlong He^*, An Wang^*, Qiqi Yao, Haijun Hu, Jiankun Wang, Xi Zhang, Hongliang Ren^†

REN Lab and collaborators · Shenzhen, Hong Kong

^* Equal contribution ^† Corresponding author

Endoscopic sequence showing ground-truth, MFT, and Endo-TTAP tracked points

2-stagehybrid supervision

250video segments

5clinical stress tests

3evaluation datasets

The tracking gap

A point can disappear. The anatomy does not.

Surgical tissue rarely behaves like an ordinary video benchmark. It stretches, leaves the field of view, hides behind instruments, and changes appearance in seconds. Dense manual trajectories are also costly to annotate.

Non-rigid motion

The same tissue point moves with deformation—not as a rigid object.

Visual interruption

Instruments, glare, and smoke can erase reliable appearance cues.

Sparse supervision

Expert trajectory labels are valuable, but too scarce for direct scaling.

Our approach

Learn motion first. Adapt to surgery second.

Endo-TTAP turns complementary weak signals into one robust tracking system.

Stage 1

Synthetic initialization

Dense optical flow teaches motion.

Supervised flow ground truth builds a stable motion representation before entering the surgical domain.

Stage 2

Surgical adaptation

Real videos teach anatomy.

Flow consistency and pseudo-point trajectories reduce dependence on sparse manual annotations.

ACA

Curriculum adaptation

An exponential schedule progressively bridges synthetic flow and real surgical appearance.

MFGA

Multi-facet attention

Fuses multi-scale flow, DINOv2 semantics, and motion patterns to predict position, visibility, and uncertainty.

PLG

Pseudo-label generation

Combines segmentation, feature anchors, two trackers, and trajectory filtering to create dense supervision.

Endo-TTAP overview: hybrid dataset, main framework, multi-facet guided attention, and pseudo-label generator — **System overview.** Hybrid data and hybrid supervision feed a motion–semantic tracking model with explicit occlusion and uncertainty reasoning.

Endo-TTAPC5

A stress test built around surgical failure modes.

250 endoscopic video segments organized around five clinically meaningful challenges—not just average motion.

Challenge-aware evaluation makes it easier to see where a tracker fails and why.

Tissue deformation tracking example — 01Tissue deformation

Benchmark snapshot

Reliable across public and surgical datasets.

Selected results below are reported in the public arXiv v1 preprint. The IEEE TMI manuscript is currently under minor revision.

SurgT3D robustness

0.946

Rob_3D with 0.9 ± 0.8 mm 3D tracking error.

STIRendpoint error ↓

13.324

MFTIQ 15.659Endo-TTAP

Lower endpoint error with 0.784 2D accuracy.

Endo-TAPC5^v12D accuracy

0.742

Performance on the challenge-oriented preprint benchmark.

^v1 The preprint uses “Endo-TAPC5”; the revised project uses “Endo-TTAPC5”.

Visual comparison

Switch the challenge. Watch the trajectories.

Non-rigid tissue motion tests whether tracked points follow anatomy rather than a rigid image pattern.

BaselineMFT

OursEndo-TTAP

01 / 05 Tissue deformation

Blue: baseline trajectory · Red: Endo-TTAP trajectory

Explore the work

Paper, code, and citation.

The public preprint and website code are available now. Journal review is ongoing.

Preprint PDF GitHub repository REN Lab

BibTeX · preprint

@article{zhou2025endottap,
  title   = {Endo-TTAP: Robust Endoscopic Tissue
             Tracking via Multi-Facet Guided Attention
             and Hybrid Flow-point Supervision},
  author  = {Zhou, Rulin and He, Wenlong and Wang, An
             and Yao, Qiqi and Hu, Haijun and Wang,
             Jiankun and Zhang, Xi and Ren, Hongliang},
  journal = {arXiv preprint arXiv:2503.22394},
  year    = {2025}
}