No Stupid Questions

I think you may be mixing a couple of things together, but I'll take a crack at this.

When you get an Ai generated response from a search engine, this is usually a modified RAG (retrieval augmented generation) approach. How this works is that the content from web pages are already pre-processed into embeddings (numerical representations of the text). When you perform a search, your search text is turned into an embedding and compared (numerical similarity) to the websites to get the most related content for your search. That means that the LLM only parses and processes a very small subset of the returned websites to generate its response.

Another element you might be asking about is how can these agentic AI systems handle larger tasks (things like OpenClaw). That is a bit more complicated and dependent on the systems design, but basically boils down to two things. The first is the "reasoning models" first break concepts into smaller tasks meaning the LLM only has to worry about a subset of a larger task. Secondly, a lot of these systems will periodically merge all past context into a compressed state that the LLM can handle (basically summaries of summaries) or add them to a database for future/faster reference.

At the end of the day, your understanding of the limits of LLM are correct, all the progress we've really seen with LLMs (over the past couple of years) has been the creation of systems to work around their limitations. The base technology isn't getting much better, but the support around it is.

I don't have the answer, but hopefully this post does, is a step by step guide to create your own LLM: https://karpathy.github.io/2026/02/12/microgpt/

The LLMs will just predict probabilities for the single next token based on all previous tokens in the context window (it's own and the ones entered by the user, system prompt or tool calls). The inference engine / runtime decides which token will be selected, usually one with high probably but that's configurable.

The LLM can also generate (predict) special tokens like "end of imaginary dialogue" to end it's turn (the runtime will give the user a chance to reply) or to call tools (the runtime will call the tool and add the result to the context window).

The LLM does not really care about if the stuff in the context was put there by a user, the system prompt, a tool or whatnot. It just predicts the next token probabilities. If you configure the runtime accordingly it will happily "play" the role of the user or of a tool (you usually don't want that).

Some of the tool calls are e.g. web searches etc. and the search results will be added to the context window. The LLM can decide to do more calls for further research, save data in "memory" that can be accessed by later "sessions" or call other tools (new tools pop up daily).

Models tend to get larger context windows with every update (right now it's usually between 250K - 2M tokens but the model performance usually gets worse with more filled context windows (needle in a hay stack).

To keep the window small agentic tools often "compact" the context window by summarizing it and then starting a new session with the compacted context.

Sometimes a task is split into multiple sessions (agents) that each have their own context window. E.g. one extra session for a long context subtask like analysis of a long document with a specific task and the result is then sent to an orchestrator agent in charge of the big picture.

The fact that everything in the context window regardless of the origin is used to predict the next token is also the reason why it's so difficult to avoid prompt injection. It all "looks" the same for the LLM and there is no "hard coded" way from excluding anything.

Others have explained it well; splitting calls up into parallel subjobs, and programatic prompt engineering.

And what is the non theortical limit of AI?

Shrug.

But practically, transformer models are kinda hitting an "innovation" wall. Big companies aren't taking risks to try and fix (say) the necessity of temperature to literally randomize outputs, or splitting instructions/context/output, or self correction (like an undo token), adaptation on the fly, anything.

All this has been explored in research papers, yet they aren't even trying it at larger scales. They're simply scaling up what they have, or (in the case of the Chinese labs) focusing on lowering resource usage.

Basically, corporate LLM development is far more conservative than you've been lead to believe, and that's the wall LLMs are smacking into.

An LLM reads the previous prompts and replies, plus any base prompts. This is considered the context window. Don't ask me why its not infinite.

The machine will then generate text following the previous text that continues the spirit and intent of the previous text, based on other texts previously digested into weights.

Its the same thing as your phones autocomplete but with a few gigabytes of weights instead of a few kilobytes.

If the data its working with is larger than the context, it will lose it. Theres a chance it'll halucinate anyway because the text generator later in the text is non-deterministic. Say you're working with insurance data. Maybe your data is familiar enough to data it previously injested data. So now it starts using wrong data, but it "feels" right as far as the LLM is concerned, because its a text generator, not a truth checker.

You can ask it to look again but its just generating fresh tokens while the context gets more polluted.

Just start looking at the volumes of non-trivial psuedo-information it generates and just try to verify some of the facts it states about your data.

The "agents" and "agentic" stuff works by wrapping the core innovation (the LLM) in layers of simple code and other LLMs. Let's try to imaging building a system that can handle a request like "find where I can buy a video card today. Make a table of the sites, the available cards, their prices, and how they compare on a benchmark." We could solve this if we had some code like

search_prompt = llm(f"make a list of google web search terms that will help answer this user's question. present the result in a json list with one item per search. <request>{user_prompt}</request>")
results_index = []
for s in json.parse(search_prompt):
  results_index.extend(google_search(s))
results = [fetch_url(url) for url in results_index]
summarized_results = [llm(f"summarize this webpage, fetching info on card prices and benchmark comparisons <page>{r}</page>") for r in results]

return llm(f"answer the user's original prompt using the following context: <context>{summarized_results}</context> <request>{user_prompt}</request>")

It's pretty simple code, and LLMs can write that, so we can even have our LLM write the code that will tell the system what to do! (I've omitted all the work to try to make things sane in terms of sandboxing and dealing with output from the various internal LLMs).

The important thing we've done here is instead of one LLM that gets too much context and stops working well, we're making a bunch of discrete LLM calls where each one has a limited context. That's the innovation of all the "agent" stuff. There's an old Computer Science truism that any problem can be solved by adding another layer of indirection and this is yet another instance of that.

Trying to define a "limit" for this is not something I have a good grasp on. I guess I'd say that the limit here is the same: max tokens in the context. It's just that we can use sub-tasks to help manage context, because everything that happens inside a sub-task doesn't impact the calling context. To trivialize things: imagine that the max context is 1 paragraph. We could try to summarize my post by summarizing each paragraph into one sentence and then summarizing the paragraph made out of those sentences. It won't be as good as if we could stick everything into the context, but it will be much better than if we tried to stick the whole post into a window that was too small and truncated it.

Some tasks will work impressively well with this framework: web pages tend to be a TON of tokens but maybe we're looking for very limited info in that stack, so spawning a sub-LLM to find the needle and bring it back is extremely effective. OTOH tasks that actually need a ton of context (maybe writing a book/movie/play) will perform poorly because the sub-agent for chapter 1 may describe a loaded gun but not include it in its output summary for the next agent. (But maybe there are more ways of slicing up the task that would allow this to work.)

Disclaimer: I am honestly a layman in this field. I may get a bunch of stuff wrong, but am happy to learn from experts. Feel free to point mistakes out and destroy me in the replies.

Simplifying and phrasing my understanding, an LLM works like - Given a prompt: Write a program to check if input is an odd number (converts the prompt to embedding), then the LLM plays a dice game/probability game of: given prompt, then generate a set of new tokens.

This feels like an oversimplification. Unfortunately, I can't think of a good analogy without anthromorphosising LLMs.

IMO this anime scene works well enough as an analogy at a super high level: anime_irl

"Comprehending what other people is saying is one step" - encoder

"Thinking about how to answer is one more step" - working with the feature representation

"Putting the things that popped into my mind into words is another step" - decoder

Now my question is, how are the current LLM's are able to parse through a bunch of search results and play the above dice game?

By current LLMs, I am going to assume that you are not referring to the raw models, but platforms like ChatGPT, Perplexity, etc with UIs for you to interact with the underlying models.

There are fundamentally two different problems here: searching the web for answers, and putting the answers into words.

Like at times it reads through say 10 URLs and generate results, how are they able to achieve this?

If I ask you: "What is the colour of fire engines?", I imagine you would answer "Red", sometimes "Yellow", off the top of your head.

What if I ask you "What are the 10 longest rivers in the world"? I believe you won't be able to give me an answer right away. What you can do is a web search, find the answer, then present the results to me. You can give it to me in 10 short bullets points, or you can come up with an essay with paragraphs describing each river.

You probably got my point by now, but to make it explicit: finding an answer and putting it into words are two different processes. They are independent of each other, so the final text output can be as long or as short as need be.

For these LLM platforms, when the model "doesn't know" the answer, they probably have a subroutine that searches the web, then feed the answer to the underlying model. The model then packages the search results into readable form - in words instead of vectors - to you.

What's the engineering behind generating such huge verbose of texts?

Sorry but I can't think of a good answer to this at the moment; leaving it to others for now - unless I managed to think of something good.

Cause I always argue about the theoretical limitations of LLM, but now that these "agents" are able to manage huge verbose of text I dont seem to have a good argument. So what exactly is happening? And what is the ~~limit of AI~~ non theortical limit of AI?

Same for this question.

Hope the partial answer helps; tried my best to ELI5.