Thoughts on the ollama runner (infosec.pub)

submitted 11 months ago by sapient_cogbag@infosec.pub to c/fosai@lemmy.world

4 comments fedilink hide all child comments

Currently I'm using the ollama runner for messing around with the mistral 7b models (only on CPU, I have no discrete gpu >.<) - I like that it has a very simple CLI and fairly minimal configuration (the Arch Linux package even comes with a systemd service, it's pretty neat).

However, I don't know how sustainable it is. It hosts a database of models on it's own here, but I don't know how dependent the code is on a central online repository.

Ideally, I'd love if we had an AI runner (including with the ability to use LoRA modules) that can natively pull from torrentfiles or something with similar p2p architecture. I imagine this would be better for long-term sustainability and hosting/download costs of the projects ^.^

Thoughts on this, and any other suggestions/comparisons/etc?

you are viewing a single comment's thread
view the rest of the comments

[-] rufus@discuss.tchncs.de 2 points 11 months ago* (last edited 11 months ago)

What does ollama add on top of llama.cpp?

I use KoboldCPP that also works very well on CPU.

And Oobabooga's UI (with llama.cpp as a CPU backend) is also easy to set up.