Ok, I interpreted it to mean that the VMs were being created as-needed and was keyed to your key specifically (which would be the most secure scenario, I think) and couldn't figure out what that could possibly work economically. But it makes more sense if just a separately encrypted host is decrypting your request and encrypting your reply along with everyone else's.

The random nature of LLM applications makes Verifiable Computing non-trivial and I’m not sure what the state-of-art is there.

Running inference is only pseudorandom, the output is then treated as a distribution and a pseudorandom selection is made according to the distribution. The heavy compute parts are all deterministic, the bit at the end that adds chatbot flavor is only pseudorandom.

As long as they share entropy sources, given the same seed they will always produce the same output. This is a trick that's exploited in image generation applications, using cached execution keyed on the seed means that alterations only need to be calculated at the part of the pipeline that needs to be changed (saving all of the previous steps).

LLMs don’t have to be random AFAIK: if you turn down the temperature parameter and send the same seed every time you get the same result

https://dylancastillo.co/posts/seed-temperature-llms.html

for most people this isn’t exactly what you want because “temperature” is sometimes short-handed as “creativity”. it controls how out of left field the result can be

The articles are light on detail but the code’s all there. The approach makes sense if the VMs are not cryptographically signed with the user’s key, but are just signed against another key to verify authenticity. I read it as if each VM was created on the fly for a user and signed with that users’s key, but that seems unlikely after re reading it.

I wonder if we can really trust the TEEs. Isn't it their hardware where they are quite free to do what they want? Also it looks very vulnerable to the side-channel attacks.

For both MobileCoin and Signal the TEE are not relied upon for the security of the general application but only for some convenience feature that are required for mass adoption but that you can go without using them.

Also, while TEE are not bulletproof, in such a server situation, it means that getting user data means much more invasive compromise than just querying the database. It's an imperfect solution, but an improvement nonetheless.

This is still crypto, yo

He, more than others, has earned rights to move freely between technology industries without scrutiny.

Not really a lie, it just highly relies on the guarantees these chips can provide. In the past there have been multiple issues with the design or implementation of these things: https://sgx.fail/ https://tee.fail/

I think it's a pragmatic approach to a difficult problem. You're still trusting that Proton is doing what they claim to be doing by not logging any of the data, and it lacks the verifiable trust chain of the VM that this Conifer system has (which theoretically would let you audit the code to confirm that there is no logging going on, and then check the crypto hash of the VM running your LLM conversation to confirm that it is in fact running that code), but if you trust Proton this step isn't as important. Otherwise the approaches look fairly the same.Proton is using PGP for the inflight encryption to the LLM while Conifer is using... maybe PGP too, I'm not sure, but a similar approach. And as others have said here, LLMs are stateless so if you can trust that the platform isn't logging the requests, there should be no record of your discussions.

What other cryptographic practical applications are more worthy of his attention right now?