I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.
Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s
The only genuine use is accessibility, but it’s purely audio/text recognition.
On that note:
LocalTranslate is an offline translation application that uses Firefox’s neural translation models (from mozilla/firefox-translations-models) to perform high-quality translations locally on your device.
Wild conclusion
The answer to your question is, it depends on your expectations. The tradeoff of a typical no VRAM machine is latency and quality.
My progression:
- I decided to try ollama and ran a small model on 8G RAM on a VPS. Slow but it worked. Ran an 3B parameter models – slowly. Struggled with 8B models. Problems with hallucinations from these small models. This was my induction into self hosting llms.
- Installed similar setup on a SFF desktop with 16G RAM. Ran 8B models slowly. More capability, but still below the threshold I was looking for. Open WebUI + ollama worked but still too slow.
- Acquired a 16G VRAM card and installed in an old desktop with 16G RAM. This runs 12B models and crosses the threshold, in my opinion, of usability as an assistant. This setup runs Open WebUI, again reinforcing usability.
- Am I satisfied? No. I’d like bigger, faster meaning more VRAM, on a newer platform, but content enough now to continue experimenting with what I have.
In sum: Don’t expect low laitency multi-step reasoning/planning with a low resource / no VRAM machine. Anything close to what you can get online at reasonable speeds requires ample VRAM.
For a reality check … I did a quick run on those three machines as they are still active. What follows has been synthesized through a cloud based model from real data:
same prompt, three machines
Prompt: “Explain how a hash table works, including collisions, in about 250 words.”
Run once on each machine withollama run <model> --verbose, all Q4_K_M quantization.Machine Model Processor Speed (tokens/s) Total time Quality (/20) VPS, 8 GB RAM gemma3n:latest (6.9B) CPU only 5.3 1m 11s 16 SFF desktop, 16 GB RAM gemma3:4b (3.9B) CPU only 6.8 51s 13 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking off GPU 31.2 11.5s 18 Desktop, 16 GB VRAM gemma4:12b (11.9B), thinking on GPU 30.9 38.5s 18 What stood out
- The GPU box generated about 4-5x faster than the SFF while running a model roughly 3x bigger.
- Both CPU-only machines landed at 5-7 t/s. Usable, but slow for interactive chat.
- Turning thinking off on the 12B cut total time from 38s to 11.5s with no change in speed (~31 t/s) and no drop in answer quality on this prompt. Thinking would likely matter more for math, code, or multi-step problems, which this test doesn’t cover.
- The 12B had no factual slips. The smaller models made small conceptual errors (blurring hash code vs. index, calling the whole array a “bucket”) and one skipped lookup entirely.
- All of them left out worst-case O(n) behavior and resizing.
Caveats
- Different models on each machine, so this shows what each box can run well, not identical hardware running identical software.
- The thinking-on run generated about 1,180 tokens versus about 350 with thinking off, so compare tokens/sec rather than total time across those two rows.
- Quality scores are my rough 1-5 ratings on accuracy, length, completeness and cleanliness, with an LLM’s help, from one run each at temperature 1. Treat them as anecdotal, not a benchmark.
- gemma3n is listed by Ollama as 6.9B, but it uses an “effective parameter” design (Google calls it E4B, roughly 4B in practice), so it’s lighter than a dense 7B. That’s why it ran on the 8 GB VPS where a standard 8B struggled.
I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.
It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.
You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.
You can get the TernaryQuench version of Qwen3.8:27b to run great on a GPU with just 16GB of VRAM. I started using it last week and it’s been amazing.
I have a 4060 Ti 16GB and was using the 14b version for a long while but that ternary conversion tech is just enough to squeeze the full 27b version into my card with plenty of room for context 👍
Is that by penkia on hugging face? (How safe is running random models?)
Note that you would need to use Llama.cpp for the Mix of Expert type models like the Qwen mentioned, Ollama doesn’t have the flexibility of commands.
Ollama can run MoE. It just decides for you where to place things. Also the Qwen model mentioned is a sense model.
That wasn’t my experience, but maybe I just didn’t run across the right instructions, or it was the particular model. Llama lets you adjust parameters far more. Two sides of the same coin, really.
Ollama is just llamacpp under the hood. Ollama is just an opinionated version that automatically sets things up that will work for most people. Llamacpp definitely has more of those “nerd knobs” that you can configure for better performance. The flipside is that it is far more complicated. Granted using another LLM to help you makes it not so bad.
Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.
Most of the world’s food comes from small local farms. Maybe join a CSA if you live in an industrialized capitalist country without a proper farmer’s market nearby
Yes. LMstudio runs on most recent systems with a 4-12B token model available to load (there’s lots to choose from) and can answer questions and do editing/scripting tasks without any changes.
I run it on a M1 laptop and it does what I want, fairly quickly. See if it does what you’re looking for.
99% of AI models available are based on stolen work, so even running it locally is immoral…
“Copyright” is immoral and exists to protect capitalists. Intellectual property as a concept needs to die, and knowledge needs to be accessable to everybody.
You should try it out. Ollama is what you’re looking for and it’s easy to set up.
Totally: You can run stuff on an even worse specced potato computer! It’ll just be slower, possibly to the point where you don’t really find it to be worth it, but you could still run local models. You won’t really be slowing it down, it’ll be the hardware limitations, but there are plenty of models that can work. There are specific model suggestions in the other comments but maybe give folks some more info on what exactly you want to do and you might get some more useful info.
Maybe write some scripts for task automation, I am not really sure. As I said I haven’t really used it before and don’t really buy into how awesome it is. So I just kind of want to play around with it, without giving all my data away or spending money to upgrade my computer for something I likely won’t use often or at all after testing it out.
I’d put it like this: If you already know what you’re doing code-wise, “AI” (read: LLMs) can potentially be helpful but you should vet absolutely everything it does and assume (probably correctly) that it has fucked up in some way.
If you do not know how to code, it’s probably not a great idea to trust it with anything important, but if you’re just doing it as a learning experience, that’s OK. Just don’t fall into the trap of letting it think for you: It is not a replacement for learning how to write code.
It’s something that has been wildly oversold in certain ways because the people running it have a financial incentive to hype it up, but used correctly and with caution, it can be potentially useful in the hands of someone who already knows what they’re doing.
Yeah it would be 100% for fun and or learning. Coding is not my profession, and I have imposter syndrome in my actual profession… Lol. I certainly wouldn’t trust any code I or an LLM wrote to provide it to someone else or contribute to anything.
Use open weight models either small enough to run yourself or via your diverse online host of choice.
Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B
If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.
The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.
