Real-time surgical video understanding

SurgStream

A Workflow-Anchored Proactive Streaming Framework
and Benchmark for Real-Time Surgical Video Understanding

Ruipeng Ma1,2Qing Xu3Jie Liang2,✉Zhen Chen1,✉
1 The Hong Kong Polytechnic University2 Eastern Institute of Technology, Ningbo3 University of Nottingham, UK

✉ Corresponding authors

Updates

arXiv and publication updates will appear here when available.

01 / Meet SurgStreamReal-time surgical video interaction
SurgStreamObserve · Remember · RespondResearch film

Perception, memory
and proactive response.

Jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass.

Explore the recorded examples
02 / Overview and methodWorkflow memory and response timing

What to remember
and when to respond

SurgStream combines workflow-structured memory with response supervision within valid procedural windows.

WAMWhat to remember

Workflow-Anchored Memory

WAM writes timestamped Observe records at stable instrument–action–target changes. At each confirmed phase boundary, it replaces that phase’s records with a compact Summary, retaining the rest of memory.

Observe recordsPhase Summary
See a memory example
DTLWhen to respond

Deadline-aware Transition Learning

DTL redraws each target response time inside its evidence-to-deadline window in every training epoch, favouring earlier valid times. Sparse state supervision teaches when to wait and when to respond.

EvidenceResponseDeadline
See a timed response
03 / In the streamSelected recorded examples

Real-time surgical
video interaction

Explore five temporal modes. Replay the source footage and follow when evidence becomes available and a response is recorded.

CholecT50 · VID7307:32 — 07:44
Source video
Continuous source footage, 07:32–07:44.
Real-time perception Recorded output

What is happening now?

Answer questions about the visible surgical scene as frames arrive.

Recorded replay · ready to play
    Question

    A selected question will appear here.

    SurgStream

    A recorded model response will appear here.

    Selected recorded research outputs, not an accuracy estimate. Each website video excerpt is continuous; timestamps refer to the original recordings. The main film shows the FP request as pending from an earlier time. CM summaries are model-generated.

    Example records
    04 / SurgStream-BenchResponse timing grounded in expert annotations

    SurgStream-Bench

    11,667 streaming interactions across various tasks evaluate what a model says and when it responds. Expert annotations ground the questions, evidence times and deadlines.

    Built from Cholec80 and CholecT50, with shared recordings grouped by procedure to prevent split leakage.

    5temporal modes
    5clinical scenarios
    Clinical scenarios
    WorkflowInstrument and actionAnatomyDocumentationNarration
    05 / Research resourcesRead, explore, build

    Research resources

    ManuscriptFramework, benchmark and evaluation protocolDemonstration recordsSelected questions, responses and source timestamps

    Code and benchmark release links will be added here.

    Method diagram