400 everyday audio-video clips test whether a live assistant warns, informs or stays silent at the right moment. The best system fully handles at most 4.48% of the clips that need help.
Editorial summary · not the author abstract. Read the linked paper for the complete abstract and claims.Figure 1: real-life use cases for omnimodal real-time AI assistants. Original figure · paper authors ↗
Release evidence
The repository has the evaluation package, annotations, prompts and reference results. Videos are available by email request, and the data license allows evaluation only.