37
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 09 Sep 2024
37 points (97.4% liked)
Programming
17583 readers
93 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 2 years ago
MODERATORS
Nah, LLMs have severe context window limitations. It starts to get wackier after ~1000 LOC.
Yeah this has been my experience too. LLMs don't handle project specific code styles too well either. Or when there are several ways of doing things.
Actually, earlier today I was asking a mixtral 8x7b about some bash ideas. I kept getting suggestions to use find and sed commands which I find unreadable and inflexible for my evolving scripts. They are fine for some specific task need, but I'll move to Python before I want to fuss with either.
Anyways, I changed the starting prompt to something like 'Common sense questions and answers with Richard Stallman's AI assistant.' The results were remarkable and interesting on many levels. From the way the answers always terminated without continuing with another question/answer, to a short footnote about the static nature of LLM learning and capabilities, along with much better quality responses in general, the LLM knew how to respond on a much higher level than normal in this specific context. I think it is the combination of Stallman's AI background and bash scripting that are powerful momentum builders here. I tried it on a whim, but it paid dividends and is a keeper of a prompting strategy.
Overall, the way my scripts are collecting relationships in the source code would probably result in a productive chunking strategy for a RAG agent. I don't think an AI would be good at what I'm doing at this stage, but it could use that info. It might even be possible to integrate the scripts as a pseudo database in the LLM model loader code for further prompting.
Gemini has a 1 million token limit. Also instead of just giving it the entire source you can give it a list of files and the ability to query them (e.g. to read an entire file, or search for usages/definitions of terms etc.).
In my experience, token limits mean nothing on larger context windows. 1 million tokens can easily be taken up by a very small amount of complex files. It also doesn’t do great traversing a tree to selectively find context which seems to be the most limiting factor I’ve run against trying to incorporate LLMs into complex and unknown (to me) projects. By the time I’ve sufficiently hunted down and provided the context, I’ve read enough of the codebase to answer most questions I was going to ask.
Right but presumably you can let the AI do that hunting.
Haven't tried Gemini; may work. But, in my experience with other LLMs, even if text doesn't exceed the token limit, LLMs start making more mistakes and sometimes behave strangely more often as the size of context grows.