AI

6427 readers

20 users here now

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen.

founded 5 years ago

Run Qwen3.6 MTP GGUFs locally ~1.4–2.2× faster with no accuracy loss and with only 18gb VRAM (huggingface.co)

submitted 1 day ago by yogthos@lemmy.ml to c/artificial_intel@lemmy.ml

6 comments fedilink hide all child comments

The change is a result of MTP support landing in llama.cpp. The Qwen3.6 Unsloth GGUFs are now out of experimental mode, with llama.cpp has merged many PRs, and MTP is now properly supported in Unsloth.

https://unsloth.ai/docs/models/qwen3.6#mtp-guide

you are viewing a single comment's thread
view the rest of the comments

[–] yogthos@lemmy.ml 4 points 1 day ago* (last edited 1 day ago) (1 children)

@davel@lemmy.ml the requirements for running Qwen just got significantly lower, it's basically the best local model at the moment

[–] davel@lemmy.ml 3 points 1 day ago (1 children)

Thanks. I haven’t bought hardware to run things locally yet. I did buy some DeepSeek tokens this weekend to play around with. Maybe I should rent until the bubble pops and then buy a supercomputer at fire sale prices.

[–] yogthos@lemmy.ml 4 points 1 day ago

Oh yeah, that's definitely the best approach if you don't already have the hardware since DeepSeek is just absurdly cheap to use. Eventually, hardware prices are going to come down, and local models are going to keep getting more efficient too. So, dumping a few grand on a rig right now doesn't really make much sense.