TechLich

the LLM is not doing machine learning while users are using it

This is a small terminology misconception. The LLM is not doing "training" during inference. It's still a "machine learning" system.

In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That's why some systems seem to be able to "remember" things from previous "conversations". Your message is vectorised and then that vector used to look up similar past interactions. The model isn't fine tuning on that, so it's not "long term" memory, but the model can take it into account for future interactions.

AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.

to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub

They're a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn't be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it's dangerous to underestimate them.

running the LLM, which is done with a random number generator and a massive matrix of probable next words.

Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy's old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn't have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).

If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.

The stuff about human learning also isn't quite right. There are different types of "learning" and different kinds of memory.

Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).

The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It's not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.

Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.

This isn't to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.

While this advice is true for all models, when it comes to agentic tasks (add this small feature/write this test harness/find bugs/suggest improvements), open source models are still way behind, vibe code or not.

Claude Fable or even Opus in an editor like Zed have a 1 million token context window and will "think" through the goals of the application, test their changes, work through debugging processes the way a programmer would, stop to ask for clarification, check diagnostic tools and linters, prompt to run test code, etc.

Llama, Gemma and Qwen etc. Do lack a lot of the world knowledge to get the goals of the application, but they also just don't have the debugging skills, won't test their code, don't always tool call correctly, get confused as the context increases and nobody has enough vram to run on large context sizes locally.

They can do autocomplete on small functions but aren't really there for more complex tasks.

On top of that, the biggest problem is that the best open source models are trained and released by the same giant tech conglomerates that have an interest in not competing with their own products. Qwen is Alibaba, Llama is Meta, gpt-oss is OpenAI. Even the more "independent" ones, kimi (Moonshot) and GLM (z.ai) are mostly funded by Alibaba and Tencent. They're released for research and marketing purposes and to please their corporate backers with inflated stock. Almost nobody has the resources to train new models from scratch. People make lots of merges and fine tunes but AI is not democratised the way that traditional programming tools have been.

Maybe some day there will be enough cheap compute for open source communities to pool together resources to build competing models but they're not really there yet :(

I don't think the "companies making ready-to-drink coffee products at industrial scale" that article is talking about this being designed for are running a 1500 watt appliance for 15 minutes a day.

They're thinking more that factories won't need to use traditional extractors which generally need to heat stuff to high temperatures to make coffee milk drinks and soluble instant coffee, etc.

They don't need to have one.

You can report it here: https://cveform.mitre.org/

Use the CNA-LR since I don't think they have a CNA.

You were probably trying to do the right thing disclosing, just know that there is a better process for it (even if you think the devs are asshats, it's good to do it like that for the community who aren't).

Even if it only affects admins, that includes admins of forks etc.

I'm sure there's probably more vulnerabilities to find.

This, assumes the vendor acts in good faith

Responsible disclosure does not assume the vendor acts in good faith. Usually the disclosure period is around 90 days before the vulnerability is released, fixed or not (although this is negotiable with a good faith vendor).

Forks etc. could have been informed privately first too if possible.

amateurs now have access to tools they should not, and WILL forgo proper standardized communication channels to disclose issues

This is not a good argument. Undisclosed zero days in the wild have always been part of the threat model. Amateurs with LLMs or not, a large percentage of vulnerabilities are not disclosed responsibly and are only fixed after damage has been done. Putting people and their personal information at risk because you want to make a point about the dangers of zero days (which everyone is already aware of) is woefully unethical.

Not everyone is privileged enough to afford security courses, and standardized education.

That doesn't mean we should abandon these things. The vendor can report the CVE too. Or anyone else with an interest in it. It doesn't have to be the untrained amateur grey hat asking Claude for vulns. A malicious threat actor exploiting a system doesn't report it either. The community benefits from skilled people handling things properly. Pretending that it doesn't because most people don't have those skills is silly.

Public disclosure is good, but responsible disclosure usually involves informing the dev first, giving them a period of time to push out a patch and then publicly disclosing for the community to learn from.

If writing a lot of bash scripts, I really recommend shellcheck. It's a linter for bash that gives a lot of good advice and points out common issues/inefficiencies and errors. There's plugins for most editors or you can just run it in a terminal. I also like that it has good documentation that tells you why something might be wrong or inadvisable.

https://github.com/koalaman/shellcheck

Yeah. Wikipedia calls it "link aggregation" and the standard is IEEE 802.1AX which also calls it that and the protocol LACP. I think the real reason for so many names is that the standard wasn't developed until later so everyone built their own competing incompatible implementations with different names and it was a mess for years.

Linux implemented it with the Linux bonding driver and switch manufactures made up their own proprietary extensions for it but the standard didn't become a thing until like 2000. Seems like "teaming" is one of the most popular names for it.

Why does this have so many names?

Some stuff calls it bonded, sometimes it's teamed, sometimes LAGed or aggregated or bundled or link channelled or ethertrunked or smartgrouped or Multi-link trunked etc. etc.

I want to know what the 3 minutes of mind blowing entertainment on Mel Croucher's Computer Fun Line was.

Also "Thou mayest blame" and "Canst thou say"

Hurts my brain a little.