LocalLLaMA

3125 readers

31 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago

MODERATORS

SkySyrup@sh.itjust.works

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

Don't overlook llama.cpp's rpc-server feature. (sh.itjust.works)

submitted 1 week ago by ThreeJawedChuck@sh.itjust.works to c/localllama@sh.itjust.works

6 comments fedilink hide all child comments

Hey everybody, brand new to running local LLMs, so I'm learning as I go. Also brand new to lemmy.

I have a 16 GB VRAM card, and I was running some models that would overflow 16GB by using the CPU+RAM to run some of the layers. It worked, but was very slow, even for only a few layers.

Well I noticed llama.cpp has an rpc-server feature, so I tried it. It was very easy to use. Lin here, but probably similar on Win or Mac. I had an older gaming rig sitting around with a GTX 1080 in it. Much slower than my 4080, but using it to run a few layers is still FAR faster than using the CPU. Night and day almost.

The main drawbacks I've experienced so far are,

By default it tries to split the model evenly between machines. That's fine if you have the same card in all of them, but I wanted to put as much of the model as possible on the fastest card. You can do that using the --tensor-split parameter, but it requires some experimenting to get it right.
It loads the rpc machine's part of the model across the network every time you start the server, which can be slow on 1 gigabit network. I didn't see any way to tell rpc-server to load the model from a local copy. It makes my startups go from 1-2 seconds, up to like 30-50 sec.
Q8 quantized KV cache works, but Q4 does not.

Lots of people may not be able to run 2 or 3 GPUs in one PC, but might have another PC they can add over the network. Worth a try, I'd say, if you want more VRAM space.

you are viewing a single comment's thread
view the rest of the comments

[–] afk_strats@lemmy.world 4 points 1 week ago* (last edited 1 week ago) (2 children)

What's the advantage over Ollama (on Linux)? I've tried a few different combinations of Nvidia cards and each time, they all got used to their full potential (it seemed)

Im in the process of working out how to combine an AMD card with an Nvidia

[–] ThreeJawedChuck@sh.itjust.works 4 points 1 week ago (1 children)

What’s the advantage over Ollama?

I'm very new to this so someone more knowledgeable should probably answer this for real.

My impression was that ollama somehow uses the llama.cpp source internally, but wraps it up to provide features like auto-downloading of models. I didn't care about that, but I liked the very tiny dependency footprint of llama.cpp. I haven't tried ollama for network inference.

There are other backends too which support network inference, and some posts allege they are better for that than llama.cpp is. vllm and ... exllama or something like that? I haven't looked into either of them. I'm running on inertia so far with llama.cpp, since it was so easy to get going and I'm kinda lazy.

[–] mutual_ayed@sh.itjust.works 3 points 1 week ago

That's a pretty good summation unless you want a more technical breakdown like how it handles weights, or how it runs connectors for multi node setups.

Check out mozillas project for LLMs. It's a nice halfway point between llama.cpp and openwebui.

https://github.com/Mozilla-Ocho/llamafile

[–] ThreeJawedChuck@sh.itjust.works 3 points 1 week ago* (last edited 1 week ago) (1 children)

To add to my lame noob answer, I found this, which has a better rundown of ollama vs llama.cpp. I don't know if it's considered bad form to link to ##ddit on lemmy, so ~~I'll just put the title here and you can search for it on there if you want~~ link added per comment from mutual_ayed below. There are a couple informative posts which are upvoted. "There is a big difference between use LM-Studio, Ollama, LLama.cpp?"

[–] mutual_ayed@sh.itjust.works 3 points 1 week ago

Feel free to link anything you think is relevant.