261
Self-Hosted AI is pretty darn cool
(lemmy.world)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 09 Aug 2024
261 points (93.6% liked)
Selfhosted
40716 readers
462 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
What kinds of specs do you need to run it well? I've got a laptop with a 3070.
You probably want 48gb of vram or more to run the good stuff. I recommend renting GPU time instead of using your own hardware, via AWS or other vendors - runpod.io is pretty good.
Kinda defeats the purpose of doing it private and local.
I wouldn't trust any claims a 3rd party service makes with regards to being private.
Llama3 8b can be run at 6gb vram, and it's fairly competent. Gemma has a 9b I think, which would also be worth looking into.
IDK, looks like 48GB cloud pricing would be 0.35/hr => $255/month. Used 3090s go for $700. Two 3090s would give you 48GB of VRAM, and cost $1400 (I'm assuming you can do "model-parallel" will Llama; never tried running an LLM, but it should be possible and work well). So, the break-even point would be <6 months. Hmm, but if Severless works well, that could be pretty cheap. Would probably take a few minutes to process and load a ~48GB model every cold start though?
Assuming they already own a PC, if someone buys two 3090 for it they'll probably also have to upgrade their PSU so that might be worth including in the budget. But it's definitely a relatively low cost way to get more VRAM, there are people who run 3 or 4 RTX3090 too.