What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
You must log in or register to comment.
The anti-AI sentiment here is ridiculous. They were asking a legitimate question!
I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷♀️
AI effectively raped and pillaged all human information for the benefit of a handful of corporations. It’s use has been shown to reduce the skills of school children and even deskill those with established knowledge. Then there’s the environmental considerations in a time where the world is decades behind appropriate climate change management… Id say it’s pretty understandable that it evokes passionate responses.
I’m sure OPs RTX 3060 will have a tolerable effect on the school children and the rainforest.
@FauxLiving @YeahToast “I’m sure my driving to work will doom the world” same mentality.
Also, it’s not so much use of a model as the creation of the model. Training is expensive in terms of resources and energy. Using the model and normalizing AI usage leads to more AI usage and more AI normalization, which leads to new models being trained…
Put it into perspective:
- Everyone doing their laundry uses more water than all data centers combined. Golf courses also use vastly more water.
- Data centers use about 1-2% of global electricity.
People driving cars pollutes more as does global shipping, regular old consumerism, etc. The environmental impact of data centers is negligible in the grand scheme of things.
Now that AI is making open source software the most secure software ever written (the hard way, hehe), not using it to scan your stuff is now considered worst practices and basically idiotic negligence. No company that uses software or anything with firmware can afford to ignore it anymore.
There’s no way out of it. Accept the new AI reality and push for good regulations. Example: If we made it so that all AI data centers had to be powered via renewable energy on site that would allay most of the energy use/environmental concerns.
I don’t buy it. If Datacentres were the powersipping butterflies you suggest, why all the new fossil power plants everywhere? I would probably not have given a shit about the datacentres, if not for all the new fossil pollution that is part and parcel of the construction.
Lies, damn lies, and statistics.
Here’s the thing, datacenters could be powersipping butterflies if they built solar farms and ground source heat pumps when they build out the data centers. But the Epstein class are too cheap, greedy and selfish to do that without being forced to. They are accustomed to having the government, directly or indirectly, lower their costs and guarantee them profits.
@riskable Justify using an unethical technology however you want, but it’s still the same basic argument that drivers often make (which is what I was pointing out). There is always some worse impact to the environment, and that is why we don’t need to change our behavior.
Ethics are for suckers, I guess?
AI isn’t inherently unethical. The very premise of saying such a thing is flawed.
Is it unethical to use AI to find new drugs and medical treatments? To use it for physics research? What about regular research? Writing code?
Where do you draw the line?
Also, AI was used to design the current crop of fusion reactors and the full potential of that is only just getting started. If it turns out that AI solves all the issues and generates a working fusion reactor design, it’ll make every one of your arguments moot. Because that one achievement will be worth everything. It would “save the world” so to speak (from an energy/emissions standpoint).
Chatting with an LLM is less impact than driving a car. Think about that for a second before you say using it is unethical. It’s literally more negative for the world for someone to drive a car every day to work than it is to ask an AI to find something on the Internet or write a program in Python.
Or perhaps your underlying complaint is more about capitalism than it is about AI?
How AI was trained is unethical. There was no consent, limited renumeration to individuals for using their work. If I drive to work, I provide value to people . Having AI scrape and summarize searches (incorrectly at times) has no benefit. Forcing all work meetings to use AI, has no benefit.
AI effectively raped and pillaged all human information
Copying information is not theft. Learning from copied information is not theft.
Theft is when you take something and deprive the owner of what you took. If I make a copy of your car, I am not stealing your car, you still have it and can drive it, wreck it, whatever.
Theft is when you take something and deprive the owner of what you took
So google ai summary now taking content from the web and putting at the top of their search, thus generally stopping people clicking on websites and taking their revenue -> theft.
So people should get free referrals to their site from google? Is that a right now?
Free referrals to their site? What are you talking about? Google is a search engine, an indexer, its role is to point people to websites based on their search terms. It now scrapes all of the websites information and people no longer visit those websites???
Whoosh!
php server and a local certificate authority.
Currently, I’m using Hermes and OpenCode mostly, backed by Ollama running a couple of different models. Usually I’m running the Gemma4 or Gwen3.5 models.
Memory management is done through Obsidian as every harness seems to have really good skills for it.
Mind elaborating how you route them through obsidian?
My Hermes agent and OpenCode both can see the same Obsidian vault. I just started working with OpenCode and haven’t quite figured it out, however my Hermes agent has learned that everything I work on needs a file in Obsidian and that if it needs to look something up that it should look through Obsidian first. Prompts usually tell them to “look in the vault” or “write up what we’ve been doing in the vault, suitable for another agent to understand what is going on with no prior context”.
It’s not currently all that reliable, but I’ve only been working with Hermes for a few months and started playing with OpenCode a few weeks ago. I need to work a bit more on orginization, though. Last time I looked in there, I backed out quickly as it was a god awful mess.
Ollama on an N150 mini PC, 16GB, no GPU, Open WebUI on top. Being honest, 1-3B models are fine for searching my offline Kiwix stuff and 8B works if you don’t mind waiting. I’m turning this into a small box (heyeddi.com/projects/doomsday-box) and I keep going back and forth on whether people even want AI on it, or just the media and offline archives. Would you?
vLLM to host and openwebui for a frontend that I really only use to demo models to friends.
harness
Opencode running with full access inside a rootless podman container with only the directories I want mounted (usually a single project, sometimes read only).
So far though, I’ve only used it to document and report on project repos because Qwen is not that good at actually coding (although it can give pretty decent conf file examples, but not always optimal).
Also useful for parsing pita data dump type files that I’m too lazy to construct multiline awk commands for.
I just use llama.cpp with Zoo Code, the VSCode extension
join us !localllama@sh.itjust.works
Openwebui pointing to llamacpp running on an old ass GPU. It’s good enough for what it is.
What’s stuck for me after trying too many things:
- Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
- Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
- LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
- SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it
Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.
This community won’t be much helpful but there is one specific for local LLMs. I think it was ai_ and some other…
Llama-swap running llama.cpp and strata. For the tools I use oh my pi and nvim with plugins
- Pi - plain and simple - run inside firejail.
- lamma.cpp w/ vulkan backend
That’s it.
Pi Coding Agent on the laptop, openwebui as a self hosted service.
I don’t host LLMs myself. My best hardware has 16GB of RAM, I could run a 4B model or so, but what’s the point? So I buy compute with ZDR turned on.
It depends on how complex you want to make it.
The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.
Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.
You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.
Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).
If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.
Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.
I’m running it through Llama.cpp but it will likely be available via LM Studio as well.
https://duck.ai/. I also don’t let LLM touch code.
