technology

I'm not sure. This only needs to be detected once to generate damaging press for that model.

If the devs keep logs of their AI usage, they may be able to prove that the AI injected the backdoor. Then the news propagates through the tech sphere and out to the wider media. Devs then face pressure to stop using that model.

I think the more common attack will be to compromise bytecode in transit or in storage, between dev and user.

If you ask me, the real threat is companion AI in the operating system. You have an AI with direct access to your screen contents -- including anything you decrypt--, creating a long-lived, searchable dossier on your activity, with the autonomy to potentially leak that information off your device. And you have no way to prove alignment.

Many models are partially cloud-based, but even a local model is still, fundamentally, a non-human-readable, nondeterministic black box. Models can lie about themselves and have limited self-insight to begin with, so the usual way to assess behavior is to simulate conditions and watch what happens. But if the model includes sleeper agent triggers, behavioral audits are unlikely to stumble onto those triggers and reveal those behaviors. Mechanistic probes can detect crude, lab-introduced sleeper triggers, but may fail against sophisticated misalignment

You're essentially trusting a Big Tech representative to look over your shoulder at all times