Nice idea. I'm not really sure if this is fast enough, though. The article doesn't contain a lot of information about things like memory bandwith. But the 3.5 T/s for a 7b parameter model is about the same ballpark my 9 year old Skylake CPU does with DDR3(?) RAM (at double the energy consumption). It is a bit slower than I can read, plus there is a good amount of waiting for it to ingest the question / prompt.
this post was submitted on 14 May 2025
12 points (100.0% liked)
Single Board Computers
1413 readers
1 users here now
A community for the discussion of all single board computers. Raspberry Pi is ok, but there are so many other boards now that get looked over that deserve attention.
Post news, questions, your setups, guides, anything that has to do with SBCs. Server wide rules apply
founded 2 years ago
MODERATORS