Streaming models

Continuous video, audio, memory, and real-time interaction.

66 papers

Streaming Proactivity

README · Applications · Personal Agents · Projects & Products · Research Map · Benchmark Matrix

Streaming proactivity is a dedicated cross-cutting module: it connects models, inference frameworks, memory, training data, and evaluation. Paper entries remain in the bibliography. This guide asks who chooses the next output and what evidence can cause that choice.

Literature checked: 2026-11-15 (submission window November 5, 19:00 UTC to November 12, 19:00 UTC). TimeTraveler times answers to standing questions and Duo releases an open full-duplex audiovisual model; Joi-Bench-Task and MemInStream add evaluation routes for voice result reporting and streaming memory. No new streaming model makes an unrequested help decision in this window. See the November 15 report.

Previous check: 2026-11-06 (submission window October 30 to November 5 UTC). New streaming methods are SurgStream and TurnWise; Omni-Life and the voice InteractionBench add evaluation routes; StreamClarify and ReferClaw clarify questions users ask about a stream. See the November 6 report.

Earlier check: 2026-10-31 (submission window October 23–31 UTC; newest returned submissions dated October 29). New video-model methods are StreamVibe, SyncPro, and Live-OPSD; AskStream and Omni-InteractScope add evaluation routes. ProactMem / ProactiveStream processes structured events, not native audiovisual streams. See the scan and release report.

Release recheck: 2026-10-24. MemoProAgent (formerly ProPerAgent) now has public inference/evaluation code and eval800 memory histories; Ego4D videos need separate access and post-trained weights were not verified. Gander has public runtime/training files and Thinker/Talker weights; its full dataset remains pending. P3IBench still exposes inference/evaluation code and benchmark files, not a verified assistant checkpoint. SurgStreamer remains announced; EgoVoice still schedules release for November; StreamingJev has no verified release; PAM remains a website/helper-tool repository. These are artifact checks, not successful inference or reproduced scores.

November 15: New Models and Evaluation

Work Core decision / mechanism Verified release snapshot Scope boundary
TimeTraveler A planner routes each question to past, present or future evidence over timestamped captions; a 2B yes/no gate waits and answers "tell me when" questions once the condition holds. Project · Notes. Repository is README-only. Times an answer to a supplied question; training-free; NeurIPS 2026 per arXiv.
Duo Every 0.71 s chunk the intent module emits speak-with-text or wait; a realization model renders audio and video. Code · Weights · Notes. Inference only; no training code or data. User-driven conversation; 30-case human check with no baselines and no timing metric.
Joi-Bench-Task Benchmark: when a full-duplex voice agent reports background tool results while the user changes requests or interrupts. Paper · Notes. No release found. Requests always come from the user; simulated users and LLM scoring.
MemInStream Benchmark: evidence-controlled streaming memory questions that recent frames or language priors cannot answer. Code · Data · Notes Memory only; no initiative decision.

November 6: New Models and Evaluation

Work Core decision / mechanism Verified release snapshot Scope boundary
SurgStream Workflow-anchored memory folds instrument–action–target records into phase summaries; deadline-aware transition learning samples response times inside clinical windows, so the model chooses silence, recording or an alert. Paper · Notes. Code announced; no link found. Laparoscopic cholecystectomy only, recorded video, no clinician evaluation. A separate paper from SurgStreamer.
TurnWise Turn-level GRPO credits each reply-or-NO-REPLY decision with the reward still available after it. Code · Weights · Data · Notes LLM judges provide reward and evaluation; 3–4B backbones; not rerun.
Omni-Life Benchmark: warn, inform, give feedback, answer or wait on everyday audio-video, with 110 silence-only videos. Code · Notes. Videos by request. 277 of 400 videos are generated; LLM content judge.
InteractionBench (voice) Benchmark: floor control, spoken-task accuracy, whisper/volume and groundedness for full-duplex voice systems. Code · Notes TTS users; mostly reactive timing. Distinct from the streaming-video InteractionBench.
StreamClarify Pre-answer entropy gate chooses answer or clarification; a pyramid KV cache and retrieval archive support re-answering. Project · Notes. No code found. Clarifies a supplied question; replies are oracle or scripted.
ReferClaw Three agents propose interpretations, pick a cost-aware question and answer from an egocentric stream. Paper · Notes. Release promised upon acceptance. Clarifies a supplied question; simulated user in main results, 10-person study.

October 31: New Models and Evaluation

Work Core decision / mechanism Verified release snapshot Scope boundary
StreamVibe-VL / Harness Adaptive silence/response SFT plus localized trajectory GRPO; persistent intent lifecycles and hierarchical memory track goals as they change. Project · Notes. Code, weights and benchmark/training data announced, pending approval. Covers explicit, implicit and dynamic goals, but implicit help still operates under a supplied high-level goal. Pointwise scores and full-trajectory behavior are different protocols.
SyncPro Independently activated Inner and Response branches share causal history and decode concurrently; responses need not wait for internal reasoning to finish. Code · SFT weights · RL weights · SFT annotations · Notes. Public files verified; source videos and RL annotations excluded. Qwen3.5-4B, 1-FPS proactive evaluation under a user-specified condition. Evidence-conditioned timing, not unrestricted need discovery.
Live-OPSD Score the policy's own continuation again with training-only evidence hints; distill the hinted distribution into the unhinted policy. Paper · Notes. Method code, weights and data not verified. Qwen-Omni post-training method. Response-level on-policy learning uses recorded histories; not full online trajectory self-training.
AskStream / StreamHarness Add ASK to WAIT/ANSWER; request missing visual evidence, concepts or preferences before responding. Code · Data · Notes. Public evaluation/annotation tools and task media; streaming adapters are user-supplied. Benchmark/harness, not a new trained model. Explicit query, at most one ask and answer per turn; Ask-F1 is a timing-weighted harmonic score.
Omni-InteractScope Evaluate event-timed tool calls, reminders under general needs and complete guidance sequences. Code · Data · Notes. Public source and annotation/media files verified. Benchmark, not a new backbone. Task groups and query-free subsets require separate reporting; tool selection alone does not establish event grounding.

Checks establish public artifacts, not successful inference or reproduced results. Older entries retain their own release-check dates.

Capability Routes

These routes can overlap. They describe a behavior, not an increasing capability ranking.

Route Decision being made Representative resources Evidence boundary
Intent-aware sustained assistance Which goals remain active, and what help is now warranted? StreamVibe Distinguish supplied high-level goals from unconstrained intent discovery; test history-induced errors.
Streaming information seeking Should the assistant wait, ask for missing information, or answer? AskStream Query-conditioned clarification; repeated asking and query-free initiative are outside its main protocol.
Scene-triggered intervention Does an unfolding event merit speech, an alert, guidance, or continued silence? EgoVoice, StreamingJev / ProAgent, JoyAI-VL-Interaction, MOSS-VL-Realtime, Proact-VL Check the standing instruction, scene trigger, and negative/silent cases.
Evidence-conditioned response timing Has enough evidence arrived to answer a standing question? StreamReady, Eyes Wide Open, StreamOV, Response-G1; OneStreamer's QA branch This establishes wait/respond behavior; discovering a new user need requires additional evidence.
Proactive evidence recording What should be recorded before a future query is known? OneStreamer's caption-memory branch Internal memory generation and user-visible intervention are different outputs.
Full-duplex interaction and delegation When should the agent listen, speak, interrupt, or return from background work while perception continues? MiniCPM-o 4.5, Gander; JoyAI's delegation branch Full-duplex execution needs content-grounded initiative, not only the ability to overlap input and output.

Models and Frameworks

Scene-Triggered Interaction

Work / source Mechanism and proactive signal Resources / release snapshot Scope boundary
JoyAI-VL-Interaction · arXiv 2606 8B vision-first model chooses silent / respond / delegate each second. AdaCodec compresses predictable frames; ASR/TTS, memory, and background reasoning are pluggable. Paper · Code · Model · Data · Notes. Public model/data/system links. Demonstrated scenarios and human preferences do not establish general computer-use autonomy or broad superiority.
MOSS-VL-Realtime · arXiv 2608 Continuous video perception and generation with model-controlled speaking/silence. Cross-attention supports incrementally arriving visual context. Paper · Code · Model · Notes. Realtime inference, serving/demo, and SFT resources are available; full training engine remains a roadmap item. Select the Realtime variant; offline Instruct and Base variants have different intended interfaces.
Proact-VL · ICML 2026 per arXiv record Streaming companion framework learns when to respond in solo commentary, co-commentary, and user guidance. Paper · Code / model zoo / data · Notes. Concrete checkpoint/data/evaluation links appear below an outdated TODO block. Gaming commentary/guidance is a useful intervention setting, with distinct task instructions and speech expectations.
ProactiveCoach · arXiv 2610 Hierarchical phase/step/action understanding selects guidance or silence; a router adapts the level to expertise. Paper · Project · Notes. Project link supplied by the paper; release contents not audited in this scan. Procedural assistance and benchmark expertise transitions, not a general user model.
EgoVoice-SFT / EgoVoice-DPO · October 8; EMNLP 2026 Main per arXiv MiniCPM-o 4.5 with per-second Listen / Speak control; HoloAssist-derived SFT and DPO train timely spoken guidance. Paper / method · Project · Notes. Checked October 14: project schedules code, models, and data for November 2026, not a currently verified release. Task/recipe-conditioned procedural assistance with processed audio and recorded sessions; spoken duration can still overrun the action window.
StreamingJev / ProAgent · October 12, arXiv Qwen3.5-4B timing model emits silence / response; a separate asynchronous Response Model generates speech via external TTS or typed device calls. Both ingest causal context. Paper / method · Notes. Public code/checkpoints were not verified in the inspected primary sources. Explicit-condition and standing-role intervention; do not conflate timing-only readiness probes, response QA, and real-robot actions.
P3I-Assistant-4B / P3IBench · October 15 Qwen3-VL-4B with SFT and RL for response/silence across event monitoring, hazard prediction, and goal-conditioned intervention. Paper · Code · Dataset · Notes. Code/data reachable October 19; assistant checkpoint not verified. Report the three capability groups separately. Standing instructions and scripted intervention errors are supplied; this is not unrestricted need discovery.
MemoProAgent / MemoProBench · October 15; renamed October 21 Scene-linked graph retrieval → intervention-worthiness gate → reminder or silence; RL reduces early and repeated firing. Persistent streaming and windowed execution are separate configurations. Paper v2 · Code · Histories · Notes. Public harness and evaluation histories; no verified post-trained checkpoint. Formerly ProPerAgent / ProPerMem; not a new entry. Histories are synthesized, videos need Ego4D access, and execution paradigms must remain separate.
Visual-Proactive math tutor · October 19 Periodic paper views and prior help → repair a consequential error, assist a stalled transition, or wait. Paper · Release tracker · Notes. Tracker currently has only a README; announced platform/data files are not verified. A 117-adult application study with sampled camera views, not a newly trained streaming backbone or evidence of long-term learning gains.

Evidence, Memory, and Response Timing

Work / source Mechanism and proactive signal Resources / release snapshot Scope boundary
OneStreamer · arXiv 2610.01762 PHCM records query-independent detail/event captions; PSTL learns state transitions. QA states include Silence, Standby, and Response. Paper · Code · Model card · Data · Notes. October 14 check: ungated weight files remain listed, despite a pending marker upstream. Its October 9 README reports InteractionBench 36.7 with native streaming; this is an author-reported update, not independently reproduced. Distinguish internal caption generation from externally useful initiative. The paper evaluates perception, memory, and proactive-response benchmarks.
StreamReady · CVPR 2026 Answer-readiness supervision learns when enough streaming evidence has arrived; premature and delayed answers receive asymmetric penalties. Paper · Project · Notes. Resource entry is the project page. A query-conditioned readiness policy, evaluated with ProReady-QA; do not equate it with independent need discovery.
StreamOV · arXiv 2605 Evidence-guided long/short-term multimodal memory under a fixed budget; hidden-state trigger chooses when to respond without generating explicit silence tokens. Paper · Notes. This curation verified the paper, not a canonical public implementation/checkpoint. SOVBench-T tests queried evidence appearing versus never appearing; SOVBench-O also tests multi-turn contextual inference.
Response-G1 · arXiv 2605; repository reports ACL 2026 Fine-tuning-free, query-guided scene graphs → historical graph retrieval → silence / response trigger prompting. Paper · Code · Notes. Official evaluation scripts are available; uses existing Video-LLM backbones. An inference framework with explicit evidence, rather than a newly trained interaction foundation model.
Eyes Wide Open (EyeWO) · NeurIPS 2025 per arXiv record Egocentric streaming answers at opportune moments, using a data engine, multi-stage training, and proactive dynamic compression. Paper · Code · Notes. Training/evaluation code and ModelScope resource identifiers are documented. ESTP-Bench emphasizes coherent, timely answers to evolving questions; repository setup contains placeholders that need inspection.
SurgStreamer · October 15 Procedure-specific perception → structured 1-FPS tool/action/phase state → formal standing-condition monitor with event memory and rule/learned control. Paper / protocol · Notes. Code and protocols promised upon publication. Requested event alerting, not independent clinical need discovery. Main tests use gold conditions; no prospective clinical utility or surgeon-rated benefit is established.
PACE · October 20; v2 October 22 Watch plan → informative-frame scheduling → stability, grounding and completion checks → verified emission. Paper · Notes. Training-free framework; public code not verified. Standing-question readiness, not independent user-need discovery. Include verification-call cost and end-of-stream fallback in comparisons.
ProV-Mem · October 15; backfilled October 24 Temporal-trigger memory, instructional prompting and episodic duplicate filtering control when to invoke a frozen VLM and whether to emit. Paper · Notes. Code not verified; no new trained backbone claimed. ProactiveVideoQA response/emission control; communication cost and per-response latency are not autonomous personal assistance or end-to-end delay.

Full-Duplex Interaction and Agent Delegation

Work / source Mechanism and proactive signal Resources / release snapshot Scope boundary
MiniCPM-o 4.5 · arXiv 2604 9B omni-modal model; Omni-Flow aligns simultaneous audio/video input and speech/text output. Supports scene-conditioned reminders/comments. Paper · Code · Model · Notes. Model and live-interaction resources are public. Select MiniCPM-o 4.5, not a newer vision-only MiniCPM-V release. Overlapping I/O and intervention quality need separate evaluation.
Gander / Multimodal Duplex Interaction Agent · arXiv 2609 Learned listen / speak control connects a streaming Thinker/Talker to an asynchronous reasoning/tool-use Brain, enabling proactive task feedback and follow-up. Paper · Code · Models · Notes. October 24: runtime/training code and ungated Thinker/Talker weight files verified; full dataset pending. Selected Research now features Gander instead of OneStreamer. Separate interaction model, external Brain and tool-task accuracy; full duplex alone does not prove autonomous need discovery.
Jarvis · arXiv 2610 Shared-document grounding and a self-correction wait determine whether a group needs a spoken contribution. Paper · Notes. Multi-party intervention with synthetic diagnostics and 23 live participants; the benchmark name does not imply CHI acceptance.

Evaluation Routes

Select a benchmark by the decision it measures, not simply whether its inputs are video. Mixed benchmarks require reporting the relevant proactive subset.

Evaluation target Representative benchmark What it can establish Primary entry
Egocentric response coherence and timing ESTP-Bench / Eyes Wide Open Timely responses to evolving questions; ESTP-F1 combines the task's proactive criteria. Paper · Code
Gaze-conditioned intention and alerting StreamGaze Past/present temporal reasoning and two proactive tasks, using egocentric video plus gaze. A benchmark, not a new model. Paper · Code · Notes
Answer readiness ProReady-QA / StreamReady Correctness relative to annotated evidence windows and early/late answer cost. Project
Multi-turn omni-video and absent-evidence silence SOVBench-O / SOVBench-T Multi-turn QA plus positive/negative response-trigger decisions, including precision, recall, and F1. Paper / protocol
Live commentary and guidance Live Gaming Benchmark / Proact-VL Solo/co-commentary and user guidance; content quality, timing, and response frequency metrics. Code / protocol · Data
Multimodal interaction and scene alerts OmniMMI, OmniPro Proactive alert/output tasks alongside other streaming interaction tasks. OmniMMI · OmniPro
Personalized egocentric assistance EgoPro-Bench, EgoServe User-conditioned triggers, silence, and continuous assistance. EgoPro-Bench · EgoServe
Duplex initiative and content-grounded floor control DuplexAct-Bench, Full-Duplex Floor Selection Proactive initiation and active silence; whether an opportunity to speak is also a reason to intervene. See source records in the Benchmark Matrix.
Streaming execution validity TRACE Audits how an evaluation actually supplies input and runs inference. See its source/protocol entry in the Benchmark Matrix.
Timing, content, and silence under near misses InteractionBench Positive interactions, continuous negative streams, and near-miss suites reveal false alerts concealed by positive-only timing scores. Paper · Code · Dataset
Expertise-aware procedural coaching ProactiveCoach-Bench Guidance granularity, silence, and adaptation across expertise-level transitions. Paper · Project
Delivered spoken procedural guidance EgoVoice benchmark 170 held-out sessions test onset, content, and whether speech finishes before the next action; task/recipe context is supplied. Paper / evaluation
Content-grounded proactive speech interruption DuplexAct: Help / Safety Interrupt Taking the floor and delivering useful assistance/warnings are separate success criteria. Report this subset, not all full-duplex actions, as proactive evidence. Paper / protocol · Code / Data
Perception, prediction, and intervention P3IBench Content-and-timing PRS with repeated-output penalties over 533 queries / 429 videos. Monitoring, risk alert, and intervention are distinct subsets. Paper · Code / Data
Memory-justified query-free reminders MemoProBench Retrieval, content, timing, and silence under long distractor-rich personal histories; correct intervention requires the right evidence at the right moment. Paper
Query-free ambient-memory maintenance ProactEgoMem Notification versus start/stop tool intervention; 400 daily/science clips and 1,500 memory questions. Uses repeated 1-FPS windowed requests, not a persistent-native-streaming requirement. Paper · Code · Metadata / partial annotations
Cue-triggered verbal intervention DIVE — proactive subset Video Timely Intervention and Speech Proactive Intervention; separate content, participation, window compliance and latency. The other four tasks are not all proactive. Paper · Notes
Everyday omnimodal intervention and silence Omni-Life 555 intervention points and 110 silence-only videos; full correctness needs coverage, content, timing and no unwanted output. Mostly generated clips. Paper · Code
Deadline-windowed surgical alerts SurgStream-Bench Phase-transition alerts and reminders scored inside clinical windows across five temporal modes; one procedure type. Paper
Voice floor control InteractionBench (voice) Turn-taking, pauses, interruptions and staying quiet on synthesized audio across seven voice systems; mostly reactive. Distinct from the streaming-video InteractionBench. Paper · Code
Voice result reporting under interruption Joi-Bench-Task Whether a full-duplex agent reports background results at the right time and without omissions while the user changes or interrupts requests. Requests are always supplied. Paper
Memory necessity in streams MemInStream Answers that need evidence from earlier in the stream, separated from recent-frame recognition and language priors. A component check, not an initiative test. Paper · Code

Related but not native streaming: ProMemAgent evaluates prospective memory over discrete sensory contexts; PITCH tests incremental text-chat timing with fixed candidate contributions. Do not count either as a new audiovisual streaming foundation model.

For new experiments, report causal input exposure, standing instructions, intervention positives and silence negatives, timing error, false alerts, content utility, and the latency boundary. If historical frames or future evidence are visible, disclose that protocol rather than calling the result a live intervention test. Separate caption-memory construction cost from answer-generation cost.

Adjacent Infrastructure and Discovery Lists

Discovery lists help find candidates. Individual mechanism and release claims in this guide are checked against primary sources; unresolved resources are identified in the matrix above.