TL;DR: Any AI that watches your screen is a privacy decision before it is a feature. The questions that matter are: is it off by default, can you scope it to specific apps, does a screenshot ever leave the machine, and does it admit when it does not actually know? In MetaWhisp the answers are yes, yes, only behind a separate opt-in, and yes — the agent will not claim anything it could not verify from what it read.
Why this category makes people nervous
"AI that watches your screen" describes both the most useful assistant you could build and the most invasive. The difference is not the model. It is the set of things the software refuses to do — and whether those refusals are defaults or buried settings.
Your screen is the worst possible data source from a privacy standpoint. It contains everything: the document you are writing, the message you have not sent, the bank balance in the other tab, the credential your password manager just filled in. A tool that reads all of it and sends it somewhere has effectively read your whole working life.
So the useful way to evaluate one of these is backwards. Not "what can it do?" but "what has it been built to decline?"
The four questions worth asking
1
Is it off by default? Screen reading should never be something you discover is already running. If the first you hear about it is a notification, the default was wrong.
2
Can you scope it to specific apps? A single global on/off switch is not enough. You want to say "read my editor and my browser, and nothing else" — and you want password managers excluded whether or not you remembered to exclude them.
3
Does an image of your screen leave the machine? Reading text is one thing. Sending a picture of your window to a vision model is a different thing, and consenting to the first is not consenting to the second.
4
Does it admit what it cannot see? A model given only text will happily assert that a button is greyed out. It cannot know that. An assistant that guesses about your screen is worse than one that stays quiet.
How MetaWhisp's Screen Agent answers them
1. Off, and dependent on something else that is also off
Screen Context is off by default. The Screen Agent runs on top of it, so with Screen Context off the agent cannot run at all — the settings panel says so explicitly rather than failing silently. Two deliberate actions stand between a fresh install and anything reading your screen.
2. Blocklist or allowlist, and password managers win either way
You choose one of two modes. Blocklist mode reads everything except the apps you exclude. Allowlist mode reads only the apps you name and nothing else.
The detail that matters is what happens when the two collide. Password managers are merged into the blocklist automatically, and the blocklist beats the allowlist. If you allowlist your password manager — by accident or on purpose — it still cannot be read. The safe outcome does not depend on you configuring it correctly.
3. Visual mode is its own consent, never bundled
With Visual mode off, MetaWhisp works from recognised text only. With it on, one downscaled image of the allowed, focused window may be sent to a cloud vision model — and only when text alone cannot answer. The image is never stored and never logged.
This is a separate switch from cloud text processing on purpose. Agreeing that recognised text may be processed in the cloud is not agreeing that pictures of your screen may be. Bundling those two into one toggle is the design mistake this avoids.
4. It will not claim what it could not verify
In text-only mode the agent is explicitly constrained: it will not assert things only eyes can confirm — that a button is disabled, that a field is highlighted, that something is visually out of place. It has the recognised text and it stays inside it.
Every card it does show cites where it read the thing it is telling you. If it cannot point at evidence, it does not speak.
Quiet by default, and nothing gets lost
The second failure mode of screen-aware assistants is not privacy — it is noise. An agent that comments on everything gets switched off within a day.
1
Pacing. Three settings — Quiet, Balanced, Frequent — plus a cooldown, so you control how often it is allowed to speak at all.
2
Pause for now. Stops the interruptions without turning the feature off. Held comments keep collecting while you are heads-down.
3
The Inbox. Cards on screen are short-lived by design. Every comment lands in the Inbox anyway — including the ones the agent decided to hold back rather than show you. ⌘⌥O opens it, and the menu bar carries an unread count so a missed card is not a lost one.
4
Colour means something. It is reserved for something actually being wrong, not used to make ordinary comments look urgent.
A comment is a starting point, not a dead end
Two things you can do with any card:
- Continue it. Press Return and the comment carries into the conversation with its source context intact — you are not re-explaining what you were looking at.
- Reject it. Mark it not helpful and say what was wrong with it. The feedback is the point; an agent nobody can contradict drifts.
As of version 1.3.27 the agent also reads your open tasks and confirmed facts before it looks at your screen, rather than treating each glance as a fresh universe. It had the ability to look those up for a while and simply never used it.
What to check before you enable any of this
| Question | Bad answer | What you want |
|---|
| Default state | On, or on after an update | Off, and it says so |
| App scoping | Global on/off only | Allowlist or blocklist, with password managers force-excluded |
| Screenshots | Bundled with text consent | Its own switch, off by default |
| Retention | Vague, or "for quality" | Named: what is kept, for how long |
| Claims | Confident about visual state from text | Refuses to assert what it did not read |
| Missed notifications | Gone | Collected somewhere you can go back to |
Apply that list to anything in this category, ours included. If a tool cannot answer one of these rows plainly, that is the answer.
Frequently asked questions
Is screen-watching on by default?
No. Screen Context is off by default, and the Screen Agent depends on it — with Screen Context off, the agent cannot run. Visual mode is a third, separate switch, also off.
Can I limit which apps it sees?
Yes — blocklist mode (everything except what you exclude) or allowlist mode (only what you name). Password managers are force-excluded and the blocklist beats the allowlist, so allowlisting one cannot expose it.
Does a screenshot go to the cloud?
Only with Visual mode on, only one downscaled image of the allowed focused window, and only when text alone cannot answer. Never stored, never logged. With Visual mode off, no image leaves the machine.
What stops it inventing things about my screen?
In text-only mode it will not claim anything only eyes could verify — a disabled button, a highlighted field. Every card cites the evidence it read.
How do I make it stop without switching it off?
Pause for now. Interruptions stop, the feature stays on, held comments keep collecting in the Inbox (⌘⌥O).
Do I lose a comment if I miss the popup?
No. Cards are deliberately short-lived on screen, but everything lands in the Inbox — including comments the agent held back — with an unread count in the menu bar.
Does it need a subscription?
It needs access to an LLM: a local model, your own API key, or Pro. The screen reading itself happens on your Mac.
Related reading
