The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.

I donate my compute to https://aihorde.net/ once in a while. got to play around with some cool AI bullshit there. Its fun to play around with.

Its all volunteer run. I got my steam deck pushing small updates to peoples like dnd rollplay kinda stuff. koboldcpp makes it easy to set up and you can use hugging face models or your own.

Theres some really excellent open models out there if you care to look! Most are hyper specialized better than say GPT and such.

Use open weight models either small enough to run yourself or via your diverse online host of choice.

I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data

Download Ollama. Download one of the smaller Qwen models. Have at it.

Try qwen3.5:9b (a 7.6Gb model) to start.

Lumo and some on duck.ai seem fine for the free usage I do. Here’s a fun one that appears accurate, enjoy! duck.ai/share/01cee404-a240-4c14-9a83-584d406c5a68#Ch22BaAoEAejJEeRNZYh-MfEYLFoDCsVZsPd23IGU5w

Depends what you’re wanting to use that AI for. Is it figuring out protein chains that might help make a vaccine? Sure. Is it making media at the expense of creators? No thanks. It’s hard to justify the costs of large AI rollout and honestly, from what I’ve read and heard, it’s hamstrung from the power grid.

Death to clankers.

To be honest, I’m not sure how it all works exactly, but I downloaded some models that run locally and i can choose an agent that can run models that will work with my (and your) limited graphics memory. It’s all local and you don’t have any limits or anything and it’s private. No tokens. However, it will run very slowly depending on what you are doing.

I haven’t used it. I tried a couple of times but I was trying to generate an image and apparently it was too much for it and I never tried it again. But maybe that somehow supports rich assholes. I don’t know. I didn’t pay anything and I was under the impression it was open source.

If you want to try, you can go on YouTube and search under something like “running AI locally.” And you should find some tutorials. Good luck.

Personally I have a love/hate relationship with AI. It’s simultaneously amazing and horrifying.

Use it all you want. Don’t listen to people here. They’re fucking stupid. These are the same people who told everyone not to get into bitcoin. Meanwhile all the assholes are millionaires and these people here gave up every space the had to the investors and are still trying to moral police others.

It’s here and isn’t going away. You’re not a hero for not using it. Learn it like any other tool. Get good with it. Use it to better your life unlike people here who will continue to lose power, territory, success and try to drag you down with them.

Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.

If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…

You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.

Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s

The answer to your question is, it depends on your expectations. The tradeoff of a typical no VRAM machine is latency and quality.

My progression:

• I decided to try ollama and ran a small model on 8G RAM on a VPS. Slow but it worked. Ran an 3B parameter models – slowly. Struggled with 8B models. Problems with hallucinations from these small models. This was my induction into self hosting llms.
• Installed similar setup on a SFF desktop with 16G RAM. Ran 8B models slowly. More capability, but still below the threshold I was looking for. Open WebUI + ollama worked but still too slow.
• Acquired a 16G VRAM card and installed in an old desktop with 16G RAM. This runs 12B models and crosses the threshold, in my opinion, of usability as an assistant. This setup runs Open WebUI, again reinforcing usability.
• Am I satisfied? No. I’d like bigger, faster meaning more VRAM, on a newer platform, but content enough now to continue experimenting with what I have.

In sum: Don’t expect low laitency multi-step reasoning/planning with a low resource / no VRAM machine. Anything close to what you can get online at reasonable speeds requires ample VRAM.

For a reality check … I did a quick run on those three machines as they are still active. What follows has been synthesized through a cloud based model from real data:

same prompt, three machines

Prompt: “Explain how a hash table works, including collisions, in about 250 words.”
Run once on each machine with ollama run <model> --verbose, all Q4_K_M quantization.

What stood out

• The GPU box generated about 4-5x faster than the SFF while running a model roughly 3x bigger.
• Both CPU-only machines landed at 5-7 t/s. Usable, but slow for interactive chat.
• Turning thinking off on the 12B cut total time from 38s to 11.5s with no change in speed (~31 t/s) and no drop in answer quality on this prompt. Thinking would likely matter more for math, code, or multi-step problems, which this test doesn’t cover.
• The 12B had no factual slips. The smaller models made small conceptual errors (blurring hash code vs. index, calling the whole array a “bucket”) and one skipped lookup entirely.
• All of them left out worst-case O(n) behavior and resizing.

Caveats

• Different models on each machine, so this shows what each box can run well, not identical hardware running identical software.
• The thinking-on run generated about 1,180 tokens versus about 350 with thinking off, so compare tokens/sec rather than total time across those two rows.
• Quality scores are my rough 1-5 ratings on accuracy, length, completeness and cleanliness, with an LLM’s help, from one run each at temperature 1. Treat them as anecdotal, not a benchmark.
• gemma3n is listed by Ollama as 6.9B, but it uses an “effective parameter” design (Google calls it E4B, roughly 4B in practice), so it’s lighter than a dense 7B. That’s why it ran on the 8 GB VPS where a standard 8B struggled.

I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.

It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.

You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.