Ah shit, I thought it had reverb but it doesn't seem to :(, my bad.

The issue here is that you are describing the goal of LLMs, not how they actually work. The goal of an LLM is to pick the next most likely token. However, it cannot achieve this via rudimentary statistics alone because the model simply does not have enough parameters to memorize which token is more likely to go next in all cases. So yes, the model "builds up statistics of which tokens it sees in which contexts" but it does so by building it's own internal data structures and organization systems which are complete black boxes.

Also, going "one token at a time" is only a "limitation" because LLMs are not accurate enough. If LLMs were more accurate, then generating "one token at a time" would not be an issue because the LLM would never need to backtrack.

And this limitation only exists because there isn't much research into LLMs backtracking yet! For example, you could give LLMs a "backspace" token: https://news.ycombinator.com/item?id=36425375

Have you tried that when it’s correct too? And in that case you mention it has a clean break and then start anew with token generation, allowing it to go a different path. You can see it more clearly experimenting with local LLM’s that have fewer layers to maintain the illusion.

If it's correct, then it gives a variety of responses. The space token effectively just makes it reflect on the conversation.

We’re trying to make a flying machine by improving pogo sticks. No matter how well you design the pogo stick and the spring, it will not be a flying machine.

To be clear, I do not believe LLMs are the future. But I do believe that they show us that AI research is on the right track.

Building a pogo stick is essential to building a flying machine. By building a pogo stick, you learn so much about physics. Over time, you replace the spring with some gunpowder to get a mortar. You shape the gunpowder into a tube to get a model rocket and discover the pendulum rocket fallacy. And finally, instead of gunpowder, you use liquid fuel and you get a rocket that can go into space.

Whoops, meant to say: "In many cases, they can accurately (critique their own work)". Thanks for correcting me!

Have you even read the article?

IMO it does not do a good job of disproving that "humans are stochastic parrots".

The example with the octopus isn't really about stochastic parrots. It's more about how LLMs are not multi-modal.

it just predicts the next word out of likely candidates based on the previous words

An entity that can consistently predict the next word of any conversation, book, news article with extremely high accuracy is quite literally a god because it can effectively predict the future. So it is not surprising to me that GPT's performance is not consistent.

It won't even know it's written itself into a corner

It many cases it does. For example, if GPT gives you a wrong answer, you can often just send an empty message (single space) and GPT will say something like: "Looks like my previous answer was incorrect, let me try again: blah blah blah".

And until we get a new approach to LLM's, we can only improve it by adding more training data and more layers allowing it to pick out more subtle patterns in larger amounts of data.

This says nothing. You are effectively saying: "Until we can find a new approach, we can only expand on the existing approach" which is obvious.

But new approaches come all the time! Advances in tokenization come all the time. Every week there is a new paper with a new model architecture. We are not stuck in some sort of hole.

Looks very useful! BTW the GitHub badge/link in the README is broken.

What are you talking about? The issue to bring back captchas was only opened 4 days ago!

Captchas were only removed 2 weeks ago, no one spoke up then: https://github.com/LemmyNet/lemmy/issues/2922

The developers have nothing against captchas. They were the ones who originally built and added the feature: https://github.com/LemmyNet/lemmy/pull/1027

Github: https://github.com/facebookresearch/ijepa

Paper: https://arxiv.org/pdf/2301.08243.pdf

I would recommend you ditch the second nginx layer. It's a waste of resources and it can cause a multitude of issues if the configuration isn't done correctly.

• If you are hosting multiple domains on the same server, disable the nginx container in the docker-compose.yml file and copy Lemmy's nginx config into your system's nginx config (e.g. /etc/nginx/).
  • If you go this route you should also delete the lemmyexternalproxy network, delete internal: true on the lemmyinternal network (required to enable port forwarding) and add port forwards to the lemmy and lemmy-ui docker services. Here's what that would look like: https://www.diffchecker.com/vjfEFuz6/
• If you are not hosting multiple domains on the same server, simply edit the port forwards in the docker-compose.yml file for the proxy service to bind to whatever your external facing IP is.

The default Lemmy nginx config should handle websockets properly. Are you putting it behind a second nginx layer?

I believe all comments on all communities you interact with are saved locally.

I see the potential but Lemmy in its current state is very buggy. There needs to be a huge uptick in dev activity to iron out all the bugs and usability issues before June 30th hits. Otherwise, I see little hope of adoption.

The performance issues also need to fixed ASAP. Sure, you could just "use a different instance" but you can't even federate with overloaded instances!

EDIT: Looks like there are a lot of fixes coming in this PR: https://github.com/LemmyNet/lemmy-ui/pull/1081