157
NVIDIA’s new AI chatbot runs locally on your PC
(www.engadget.com)
This is a most excellent place for technology news and articles.
Pretty much every LLM you can download already has CUDA support via PyTorch.
However, some of the easier to use frontends don't use GPU acceleration because it's a bit of a pain to configure across a wide range of hardware models and driver versions. IIRC GPT4All does not use GPU acceleration yet (might need outdated; I haven't checked in a while).
If this makes local LLMs more accessible to people who are not familiar with setting up a CUDA development environment or Python venvs, that's great news.
I'd hope that this uses the hardware better than Pytorch. Otherwise, why the specific hardware demands? Well, it can always be marketing.
There are several alternatives that offer 1-click installers. EG in this thread:
AGPL-3.0 license: https://jan.ai/
MIT license: https://ollama.com/
MIT license: https://gpt4all.io/index.html
(There's more.)
Ollama with Ollama WebUI is the best combo from my experience.
Gpt4all somehow uses Gpu acceleration on my rx 6600xt
Ooh nice. Looking at the change logs, looks like they added Vulkan acceleration back in September. Probably not as good as CUDA/Metal on supported hardware though.
getting around 44 iterations/s (or whatever that means) on my gpu