Original figure · paper authors ↗Streaming models
Continuous video, audio, memory, and real-time interaction.
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Paper Figure 4 · authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Original figure · paper authors ↗
Paper Figure 1 · authors ↗
Paper Figure 1 · authors ↗
Paper Figure 2 · authors ↗
Original figure · paper authors ↗Streaming Proactivity
README · Applications · Personal Agents · Projects & Products · Research Map · Benchmark Matrix
Streaming proactivity is a dedicated cross-cutting module: it connects models, inference frameworks, memory, training data, and evaluation. Paper entries remain in the bibliography. This guide asks who chooses the next output and what evidence can cause that choice.
Literature checked: 2026-11-15 (submission window November 5, 19:00 UTC to November 12, 19:00 UTC). TimeTraveler times answers to standing questions and Duo releases an open full-duplex audiovisual model; Joi-Bench-Task and MemInStream add evaluation routes for voice result reporting and streaming memory. No new streaming model makes an unrequested help decision in this window. See the November 15 report.
Previous check: 2026-11-06 (submission window October 30 to November 5 UTC). New streaming methods are SurgStream and TurnWise; Omni-Life and the voice InteractionBench add evaluation routes; StreamClarify and ReferClaw clarify questions users ask about a stream. See the November 6 report.
Earlier check: 2026-10-31 (submission window October 23–31 UTC; newest returned submissions dated October 29). New video-model methods are StreamVibe, SyncPro, and Live-OPSD; AskStream and Omni-InteractScope add evaluation routes. ProactMem / ProactiveStream processes structured events, not native audiovisual streams. See the scan and release report.
Release recheck: 2026-10-24. MemoProAgent (formerly ProPerAgent) now has public inference/evaluation code and eval800 memory histories; Ego4D videos need separate access and post-trained weights were not verified. Gander has public runtime/training files and Thinker/Talker weights; its full dataset remains pending. P3IBench still exposes inference/evaluation code and benchmark files, not a verified assistant checkpoint. SurgStreamer remains announced; EgoVoice still schedules release for November; StreamingJev has no verified release; PAM remains a website/helper-tool repository. These are artifact checks, not successful inference or reproduced scores.
November 15: New Models and Evaluation
| Work | Core decision / mechanism | Verified release snapshot | Scope boundary |
|---|---|---|---|
| TimeTraveler | A planner routes each question to past, present or future evidence over timestamped captions; a 2B yes/no gate waits and answers "tell me when" questions once the condition holds. | Project · Notes. Repository is README-only. | Times an answer to a supplied question; training-free; NeurIPS 2026 per arXiv. |
| Duo | Every 0.71 s chunk the intent module emits speak-with-text or wait; a realization model renders audio and video. | Code · Weights · Notes. Inference only; no training code or data. | User-driven conversation; 30-case human check with no baselines and no timing metric. |
| Joi-Bench-Task | Benchmark: when a full-duplex voice agent reports background tool results while the user changes requests or interrupts. | Paper · Notes. No release found. | Requests always come from the user; simulated users and LLM scoring. |
| MemInStream | Benchmark: evidence-controlled streaming memory questions that recent frames or language priors cannot answer. | Code · Data · Notes | Memory only; no initiative decision. |
November 6: New Models and Evaluation
| Work | Core decision / mechanism | Verified release snapshot | Scope boundary |
|---|---|---|---|
| SurgStream | Workflow-anchored memory folds instrument–action–target records into phase summaries; deadline-aware transition learning samples response times inside clinical windows, so the model chooses silence, recording or an alert. | Paper · Notes. Code announced; no link found. | Laparoscopic cholecystectomy only, recorded video, no clinician evaluation. A separate paper from SurgStreamer. |
| TurnWise | Turn-level GRPO credits each reply-or-NO-REPLY decision with the reward still available after it. | Code · Weights · Data · Notes | LLM judges provide reward and evaluation; 3–4B backbones; not rerun. |
| Omni-Life | Benchmark: warn, inform, give feedback, answer or wait on everyday audio-video, with 110 silence-only videos. | Code · Notes. Videos by request. | 277 of 400 videos are generated; LLM content judge. |
| InteractionBench (voice) | Benchmark: floor control, spoken-task accuracy, whisper/volume and groundedness for full-duplex voice systems. | Code · Notes | TTS users; mostly reactive timing. Distinct from the streaming-video InteractionBench. |
| StreamClarify | Pre-answer entropy gate chooses answer or clarification; a pyramid KV cache and retrieval archive support re-answering. | Project · Notes. No code found. | Clarifies a supplied question; replies are oracle or scripted. |
| ReferClaw | Three agents propose interpretations, pick a cost-aware question and answer from an egocentric stream. | Paper · Notes. Release promised upon acceptance. | Clarifies a supplied question; simulated user in main results, 10-person study. |
October 31: New Models and Evaluation
| Work | Core decision / mechanism | Verified release snapshot | Scope boundary |
|---|---|---|---|
| StreamVibe-VL / Harness | Adaptive silence/response SFT plus localized trajectory GRPO; persistent intent lifecycles and hierarchical memory track goals as they change. | Project · Notes. Code, weights and benchmark/training data announced, pending approval. | Covers explicit, implicit and dynamic goals, but implicit help still operates under a supplied high-level goal. Pointwise scores and full-trajectory behavior are different protocols. |
| SyncPro | Independently activated Inner and Response branches share causal history and decode concurrently; responses need not wait for internal reasoning to finish. | Code · SFT weights · RL weights · SFT annotations · Notes. Public files verified; source videos and RL annotations excluded. | Qwen3.5-4B, 1-FPS proactive evaluation under a user-specified condition. Evidence-conditioned timing, not unrestricted need discovery. |
| Live-OPSD | Score the policy's own continuation again with training-only evidence hints; distill the hinted distribution into the unhinted policy. | Paper · Notes. Method code, weights and data not verified. | Qwen-Omni post-training method. Response-level on-policy learning uses recorded histories; not full online trajectory self-training. |
| AskStream / StreamHarness | Add ASK to WAIT/ANSWER; request missing visual evidence, concepts or preferences before responding. | Code · Data · Notes. Public evaluation/annotation tools and task media; streaming adapters are user-supplied. | Benchmark/harness, not a new trained model. Explicit query, at most one ask and answer per turn; Ask-F1 is a timing-weighted harmonic score. |
| Omni-InteractScope | Evaluate event-timed tool calls, reminders under general needs and complete guidance sequences. | Code · Data · Notes. Public source and annotation/media files verified. | Benchmark, not a new backbone. Task groups and query-free subsets require separate reporting; tool selection alone does not establish event grounding. |
Checks establish public artifacts, not successful inference or reproduced results. Older entries retain their own release-check dates.
Capability Routes
These routes can overlap. They describe a behavior, not an increasing capability ranking.
| Route | Decision being made | Representative resources | Evidence boundary |
|---|---|---|---|
| Intent-aware sustained assistance | Which goals remain active, and what help is now warranted? | StreamVibe | Distinguish supplied high-level goals from unconstrained intent discovery; test history-induced errors. |
| Streaming information seeking | Should the assistant wait, ask for missing information, or answer? | AskStream | Query-conditioned clarification; repeated asking and query-free initiative are outside its main protocol. |
| Scene-triggered intervention | Does an unfolding event merit speech, an alert, guidance, or continued silence? | EgoVoice, StreamingJev / ProAgent, JoyAI-VL-Interaction, MOSS-VL-Realtime, Proact-VL | Check the standing instruction, scene trigger, and negative/silent cases. |
| Evidence-conditioned response timing | Has enough evidence arrived to answer a standing question? | StreamReady, Eyes Wide Open, StreamOV, Response-G1; OneStreamer's QA branch | This establishes wait/respond behavior; discovering a new user need requires additional evidence. |
| Proactive evidence recording | What should be recorded before a future query is known? | OneStreamer's caption-memory branch | Internal memory generation and user-visible intervention are different outputs. |
| Full-duplex interaction and delegation | When should the agent listen, speak, interrupt, or return from background work while perception continues? | MiniCPM-o 4.5, Gander; JoyAI's delegation branch | Full-duplex execution needs content-grounded initiative, not only the ability to overlap input and output. |
Models and Frameworks
Scene-Triggered Interaction
| Work / source | Mechanism and proactive signal | Resources / release snapshot | Scope boundary |
|---|---|---|---|
| JoyAI-VL-Interaction · arXiv 2606 | 8B vision-first model chooses silent / respond / delegate each second. AdaCodec compresses predictable frames; ASR/TTS, memory, and background reasoning are pluggable. | Paper · Code · Model · Data · Notes. Public model/data/system links. | Demonstrated scenarios and human preferences do not establish general computer-use autonomy or broad superiority. |
| MOSS-VL-Realtime · arXiv 2608 | Continuous video perception and generation with model-controlled speaking/silence. Cross-attention supports incrementally arriving visual context. | Paper · Code · Model · Notes. Realtime inference, serving/demo, and SFT resources are available; full training engine remains a roadmap item. | Select the Realtime variant; offline Instruct and Base variants have different intended interfaces. |
| Proact-VL · ICML 2026 per arXiv record | Streaming companion framework learns when to respond in solo commentary, co-commentary, and user guidance. | Paper · Code / model zoo / data · Notes. Concrete checkpoint/data/evaluation links appear below an outdated TODO block. | Gaming commentary/guidance is a useful intervention setting, with distinct task instructions and speech expectations. |
| ProactiveCoach · arXiv 2610 | Hierarchical phase/step/action understanding selects guidance or silence; a router adapts the level to expertise. | Paper · Project · Notes. Project link supplied by the paper; release contents not audited in this scan. | Procedural assistance and benchmark expertise transitions, not a general user model. |
| EgoVoice-SFT / EgoVoice-DPO · October 8; EMNLP 2026 Main per arXiv | MiniCPM-o 4.5 with per-second Listen / Speak control; HoloAssist-derived SFT and DPO train timely spoken guidance. | Paper / method · Project · Notes. Checked October 14: project schedules code, models, and data for November 2026, not a currently verified release. | Task/recipe-conditioned procedural assistance with processed audio and recorded sessions; spoken duration can still overrun the action window. |
| StreamingJev / ProAgent · October 12, arXiv | Qwen3.5-4B timing model emits silence / response; a separate asynchronous Response Model generates speech via external TTS or typed device calls. Both ingest causal context. | Paper / method · Notes. Public code/checkpoints were not verified in the inspected primary sources. | Explicit-condition and standing-role intervention; do not conflate timing-only readiness probes, response QA, and real-robot actions. |
| P3I-Assistant-4B / P3IBench · October 15 | Qwen3-VL-4B with SFT and RL for response/silence across event monitoring, hazard prediction, and goal-conditioned intervention. | Paper · Code · Dataset · Notes. Code/data reachable October 19; assistant checkpoint not verified. | Report the three capability groups separately. Standing instructions and scripted intervention errors are supplied; this is not unrestricted need discovery. |
| MemoProAgent / MemoProBench · October 15; renamed October 21 | Scene-linked graph retrieval → intervention-worthiness gate → reminder or silence; RL reduces early and repeated firing. Persistent streaming and windowed execution are separate configurations. | Paper v2 · Code · Histories · Notes. Public harness and evaluation histories; no verified post-trained checkpoint. | Formerly ProPerAgent / ProPerMem; not a new entry. Histories are synthesized, videos need Ego4D access, and execution paradigms must remain separate. |
| Visual-Proactive math tutor · October 19 | Periodic paper views and prior help → repair a consequential error, assist a stalled transition, or wait. | Paper · Release tracker · Notes. Tracker currently has only a README; announced platform/data files are not verified. | A 117-adult application study with sampled camera views, not a newly trained streaming backbone or evidence of long-term learning gains. |
Evidence, Memory, and Response Timing
| Work / source | Mechanism and proactive signal | Resources / release snapshot | Scope boundary |
|---|---|---|---|
| OneStreamer · arXiv 2610.01762 | PHCM records query-independent detail/event captions; PSTL learns state transitions. QA states include Silence, Standby, and Response. |
Paper · Code · Model card · Data · Notes. October 14 check: ungated weight files remain listed, despite a pending marker upstream. Its October 9 README reports InteractionBench 36.7 with native streaming; this is an author-reported update, not independently reproduced. | Distinguish internal caption generation from externally useful initiative. The paper evaluates perception, memory, and proactive-response benchmarks. |
| StreamReady · CVPR 2026 | Answer-readiness supervision learns when enough streaming evidence has arrived; premature and delayed answers receive asymmetric penalties. | Paper · Project · Notes. Resource entry is the project page. | A query-conditioned readiness policy, evaluated with ProReady-QA; do not equate it with independent need discovery. |
| StreamOV · arXiv 2605 | Evidence-guided long/short-term multimodal memory under a fixed budget; hidden-state trigger chooses when to respond without generating explicit silence tokens. | Paper · Notes. This curation verified the paper, not a canonical public implementation/checkpoint. | SOVBench-T tests queried evidence appearing versus never appearing; SOVBench-O also tests multi-turn contextual inference. |
| Response-G1 · arXiv 2605; repository reports ACL 2026 | Fine-tuning-free, query-guided scene graphs → historical graph retrieval → silence / response trigger prompting. | Paper · Code · Notes. Official evaluation scripts are available; uses existing Video-LLM backbones. | An inference framework with explicit evidence, rather than a newly trained interaction foundation model. |
| Eyes Wide Open (EyeWO) · NeurIPS 2025 per arXiv record | Egocentric streaming answers at opportune moments, using a data engine, multi-stage training, and proactive dynamic compression. | Paper · Code · Notes. Training/evaluation code and ModelScope resource identifiers are documented. | ESTP-Bench emphasizes coherent, timely answers to evolving questions; repository setup contains placeholders that need inspection. |
| SurgStreamer · October 15 | Procedure-specific perception → structured 1-FPS tool/action/phase state → formal standing-condition monitor with event memory and rule/learned control. | Paper / protocol · Notes. Code and protocols promised upon publication. | Requested event alerting, not independent clinical need discovery. Main tests use gold conditions; no prospective clinical utility or surgeon-rated benefit is established. |
| PACE · October 20; v2 October 22 | Watch plan → informative-frame scheduling → stability, grounding and completion checks → verified emission. | Paper · Notes. Training-free framework; public code not verified. | Standing-question readiness, not independent user-need discovery. Include verification-call cost and end-of-stream fallback in comparisons. |
| ProV-Mem · October 15; backfilled October 24 | Temporal-trigger memory, instructional prompting and episodic duplicate filtering control when to invoke a frozen VLM and whether to emit. | Paper · Notes. Code not verified; no new trained backbone claimed. | ProactiveVideoQA response/emission control; communication cost and per-response latency are not autonomous personal assistance or end-to-end delay. |
Full-Duplex Interaction and Agent Delegation
| Work / source | Mechanism and proactive signal | Resources / release snapshot | Scope boundary |
|---|---|---|---|
| MiniCPM-o 4.5 · arXiv 2604 | 9B omni-modal model; Omni-Flow aligns simultaneous audio/video input and speech/text output. Supports scene-conditioned reminders/comments. | Paper · Code · Model · Notes. Model and live-interaction resources are public. | Select MiniCPM-o 4.5, not a newer vision-only MiniCPM-V release. Overlapping I/O and intervention quality need separate evaluation. |
| Gander / Multimodal Duplex Interaction Agent · arXiv 2609 | Learned listen / speak control connects a streaming Thinker/Talker to an asynchronous reasoning/tool-use Brain, enabling proactive task feedback and follow-up. | Paper · Code · Models · Notes. October 24: runtime/training code and ungated Thinker/Talker weight files verified; full dataset pending. | Selected Research now features Gander instead of OneStreamer. Separate interaction model, external Brain and tool-task accuracy; full duplex alone does not prove autonomous need discovery. |
| Jarvis · arXiv 2610 | Shared-document grounding and a self-correction wait determine whether a group needs a spoken contribution. | Paper · Notes. | Multi-party intervention with synthetic diagnostics and 23 live participants; the benchmark name does not imply CHI acceptance. |
Evaluation Routes
Select a benchmark by the decision it measures, not simply whether its inputs are video. Mixed benchmarks require reporting the relevant proactive subset.
| Evaluation target | Representative benchmark | What it can establish | Primary entry |
|---|---|---|---|
| Egocentric response coherence and timing | ESTP-Bench / Eyes Wide Open | Timely responses to evolving questions; ESTP-F1 combines the task's proactive criteria. | Paper · Code |
| Gaze-conditioned intention and alerting | StreamGaze | Past/present temporal reasoning and two proactive tasks, using egocentric video plus gaze. A benchmark, not a new model. | Paper · Code · Notes |
| Answer readiness | ProReady-QA / StreamReady | Correctness relative to annotated evidence windows and early/late answer cost. | Project |
| Multi-turn omni-video and absent-evidence silence | SOVBench-O / SOVBench-T | Multi-turn QA plus positive/negative response-trigger decisions, including precision, recall, and F1. | Paper / protocol |
| Live commentary and guidance | Live Gaming Benchmark / Proact-VL | Solo/co-commentary and user guidance; content quality, timing, and response frequency metrics. | Code / protocol · Data |
| Multimodal interaction and scene alerts | OmniMMI, OmniPro | Proactive alert/output tasks alongside other streaming interaction tasks. | OmniMMI · OmniPro |
| Personalized egocentric assistance | EgoPro-Bench, EgoServe | User-conditioned triggers, silence, and continuous assistance. | EgoPro-Bench · EgoServe |
| Duplex initiative and content-grounded floor control | DuplexAct-Bench, Full-Duplex Floor Selection | Proactive initiation and active silence; whether an opportunity to speak is also a reason to intervene. | See source records in the Benchmark Matrix. |
| Streaming execution validity | TRACE | Audits how an evaluation actually supplies input and runs inference. | See its source/protocol entry in the Benchmark Matrix. |
| Timing, content, and silence under near misses | InteractionBench | Positive interactions, continuous negative streams, and near-miss suites reveal false alerts concealed by positive-only timing scores. | Paper · Code · Dataset |
| Expertise-aware procedural coaching | ProactiveCoach-Bench | Guidance granularity, silence, and adaptation across expertise-level transitions. | Paper · Project |
| Delivered spoken procedural guidance | EgoVoice benchmark | 170 held-out sessions test onset, content, and whether speech finishes before the next action; task/recipe context is supplied. | Paper / evaluation |
| Content-grounded proactive speech interruption | DuplexAct: Help / Safety Interrupt | Taking the floor and delivering useful assistance/warnings are separate success criteria. Report this subset, not all full-duplex actions, as proactive evidence. | Paper / protocol · Code / Data |
| Perception, prediction, and intervention | P3IBench | Content-and-timing PRS with repeated-output penalties over 533 queries / 429 videos. Monitoring, risk alert, and intervention are distinct subsets. | Paper · Code / Data |
| Memory-justified query-free reminders | MemoProBench | Retrieval, content, timing, and silence under long distractor-rich personal histories; correct intervention requires the right evidence at the right moment. | Paper |
| Query-free ambient-memory maintenance | ProactEgoMem | Notification versus start/stop tool intervention; 400 daily/science clips and 1,500 memory questions. Uses repeated 1-FPS windowed requests, not a persistent-native-streaming requirement. | Paper · Code · Metadata / partial annotations |
| Cue-triggered verbal intervention | DIVE — proactive subset | Video Timely Intervention and Speech Proactive Intervention; separate content, participation, window compliance and latency. The other four tasks are not all proactive. | Paper · Notes |
| Everyday omnimodal intervention and silence | Omni-Life | 555 intervention points and 110 silence-only videos; full correctness needs coverage, content, timing and no unwanted output. Mostly generated clips. | Paper · Code |
| Deadline-windowed surgical alerts | SurgStream-Bench | Phase-transition alerts and reminders scored inside clinical windows across five temporal modes; one procedure type. | Paper |
| Voice floor control | InteractionBench (voice) | Turn-taking, pauses, interruptions and staying quiet on synthesized audio across seven voice systems; mostly reactive. Distinct from the streaming-video InteractionBench. | Paper · Code |
| Voice result reporting under interruption | Joi-Bench-Task | Whether a full-duplex agent reports background results at the right time and without omissions while the user changes or interrupts requests. Requests are always supplied. | Paper |
| Memory necessity in streams | MemInStream | Answers that need evidence from earlier in the stream, separated from recent-frame recognition and language priors. A component check, not an initiative test. | Paper · Code |
Related but not native streaming: ProMemAgent evaluates prospective memory over discrete sensory contexts; PITCH tests incremental text-chat timing with fixed candidate contributions. Do not count either as a new audiovisual streaming foundation model.
For new experiments, report causal input exposure, standing instructions, intervention positives and silence negatives, timing error, false alerts, content utility, and the latency boundary. If historical frames or future evidence are visible, disclose that protocol rather than calling the result a live intervention test. Separate caption-memory construction cost from answer-generation cost.
Adjacent Infrastructure and Discovery Lists
- Live VLM WebUI: camera/serving interface; proactive policy depends on the connected backend.
- Awesome VLM Streaming Video: discovery by streaming interaction, memory, models, and evaluation.
- Awesome Streaming Video Understanding: model/paper/dataset/benchmark discovery.
- Awesome Streaming Agents: continuous-input and active-activation perspective.
Discovery lists help find candidates. Individual mechanism and release claims in this guide are checked against primary sources; unresolved resources are identified in the matrix above.