this post was submitted on 03 Jun 2026
815 points (99.6% liked)
People Twitter
10036 readers
379 users here now
People tweeting stuff. We allow tweets from anyone.
RULES:
- Mark NSFW content.
- No doxxing people.
- Must be a pic of the tweet or similar. No direct links to the tweet.
- No bullying or international politcs
- Be excellent to each other.
- Provide an archived link to the tweet (or similar) being shown if it's a major figure or a politician. Archive.is the best way.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Tell that to all the Github users that are screaming about the new token based billing. In reality inference on these massive models with big context windows is expensive, but was subsidized so hard, that nobody has an accurate feeling for the cost.
No, it is cheap and efficient. It is relative, and the comparison is to model training. But yeah, its not free
Sure it's much much cheaper than training, but importantly those companies are not recouping anything with inference because it is still more expensive than what they are selling it for.
They are double bankrupting themselves.
At work we run inference for a research project with an open weights model in the public cloud another part of my company provides and we pay around 25$ a day for a VM with a single L40s. It's both slow - despite not even serving concurrent users - and kind of bad in its outputs.
Edit: Interference -> Inference, arguing on the internet after waking up first thing in the morning might not have been the best idea