AAAI 2026 · Oral Presentation

VISION × LANGUAGE × SURGERY

Tracking a point is easier when we understand what happened to it.

VL-SurgPT pairs surgical point trajectories with textual descriptions such as pulled, reflected, or occluded—then uses that context to make tracking more robust.

Read the paper Watch tracking

PDF arXiv Dataset Website source

Four surgical scenes with tracked tissue points and semantic status labels

semantic tracking view

Point 03 · Pulled

Point 07 · Clear view

AAAI-26 ORAL Computer Vision 18

Rulin Zhou^*, Wenlong He^*, An Wang^*, Jianhang Zhang, Xuanhui Zeng, Xi Zhang, Chaowei Zhu, Haijun Hu^†, Hongliang Ren^†

The Chinese University of Hong Kong · Shenzhen People’s Hospital · The University of Hong Kong · Shenzhen University · CUHK Shenzhen Research Institute

^* Equal contribution ^† Corresponding authors

01 · THE PROBLEM

The same point can look completely different one second later.

Surgical scenes break the assumptions of ordinary tracking. Tissue bends, tools cover targets, the camera moves, wet surfaces reflect light, and cautery fills the view with smoke.

Tissue deformation

A physical point moves on a non-rigid surface with no stable shape.

Instrument occlusion

A tool can hide the target while the tissue continues moving underneath.

Camera jitter

Rapid viewpoint changes create blur and misleading global motion.

Surface reflection

Highlights move independently and resemble visual landmarks.

Cauterization smoke

Contrast disappears as boundaries become hazy or temporarily invisible.

The missing signal

Coordinates tell us where. Language tells us why.

(688, 502)

TYPE · Tissue STATUS · Smoke obscuration

02 · THE DATASET

VL-SurgPT adds semantic context to real in vivo motion.

Data were collected from da Vinci Xi robotic procedures. ICG fluorescence anchors the ground truth; two clinicians then label intermediate frames at 1 fps with both coordinates and point status.

908in vivo clips

754tissue clips

154instrument clips

17,171visible tissue points

9visual status labels

8benchmark trackers

VL-SurgPT collection, annotation, point status, instrument, and scenario taxonomy — **From surgery to synchronized multimodal labels.** Fluorescent anchors, expert annotation, 9 point states, 7 instrument types, and 5 challenging tissue scenarios.

ANNOTATION WORKFLOW

Ground truth with clinical supervision

ICG markers provide reliable endpoints. Clinicians use EVA to trace points and assign semantic labels through each clip.

DATASET VIEW

Every point carries a visual state

The label travels with the point, describing whether it is clear, pulled, reflected, obscured, or outside the field of view.

03 · THE METHOD

TG-SurgPT turns predicted status into a tracking cue.

The method extends Track-On with an attribute prediction head and cross-modal attention. At inference time, it predicts its own semantic labels—no manual text is required.

SEE

Track visual features

Track-On extracts query, dense frame, and coarse matching features.

DESCRIBE

Predict point attributes

A status head estimates point type and current visual condition.

REFINE

Guide the trajectory

Text–vision attention produces an offset that corrects the initial position.

TG-SurgPT architecture with Track-On visual features, a CLIP text encoder, attribute prediction, and text-guided attention — **Training:** ground-truth descriptions supervise semantic understanding. **Inference:** predicted attributes automatically guide coordinate refinement.

04 · THE RESULTS

Semantic guidance improves both tissue and instrument tracking.

TG-SurgPT outperforms eight vision-only baselines across the reported benchmark metrics while retaining practical processing speed.

Tissue · Average Jaccard 62.88

+4.33 vs Track-On

Tissue · End Point Error 11.02 px

−2.77 px vs Track-On

Instrument · Average Jaccard 49.52

+2.55 vs Track-On

Instrument · Occlusion Accuracy 89.79

+4.72 vs Track-On

Inference speed 9.72 fps

accuracy without giving up practicality

QUALITATIVE TRACKING

Follow the blue points.

TG-SurgPT Track-On MFT

Tissue tracking across adverse scenarios. TG-SurgPT is blue, Track-On is red, and MFT is green.

05 · RESOURCES

Explore the paper, data sample, and project materials.

VL-SurgPT opens a path from purely geometric tracking toward context-aware surgical perception. Use the links below to inspect the official paper and available data.

Official publicationAAAI Proceedings ↗ PreprintarXiv ↗ Data sampleGitHub ↗ Annotation softwareEVA ↗

CITATION

@inproceedings{zhou2026vlsurgpt,
  title     = {Bridging Vision and Language for Robust
               Context-Aware Surgical Point Tracking:
               The VL-SurgPT Dataset and Benchmark},
  author    = {Zhou, Rulin and He, Wenlong and Wang, An
               and Zhang, Jianhang and Zeng, Xuanhui
               and Zhang, Xi and Zhu, Chaowei and
               Hu, Haijun and Ren, Hongliang},
  booktitle = {AAAI},
  year      = {2026},
  pages     = {13755--13763},
  doi       = {10.1609/aaai.v40i16.38383}
}