So I actually just joined this community a few days ago to get some ideas. I live in a very old home, about 140-150 years old. It’s been a really great house, but it has its quarks.
I initially came here to get some ideas for home improvements. So I guess I’ll just ask. What are some extremely easy upgrades that you recommend someone with an old house make? They can be a little expensive, it doesn’t matter.
This is the point everyone downvoting me seems to be missing. OP wanted something comparable to the responsiveness of chat.chatgpt.com… Which is simply not possible without insane hardware. Like sure, if you don’t care about token generation you can install an LLM on incredibly underpowered hardware and it technically works, but that’s not at all what OP was asking for. They wanted a comparable experience. Which requires a lot of money.
Nice! Glad you got it working. There shouldn’t be any significant performance increase, but there’s at least a tiny one. lmao.
What kind of hardware do you need to run with comparable responsiveness to chatgpt?
Generally you need between $8-10,000 worth of equipment to get a relative responsiveness from a self-hosted LLM.
Anyone downvoting clearly doesn’t understand the hardware requirements to be able to run an LLM with a significant model that rivals ChatGPT. ChatGPT is a multi-billion dollar AI cluster…
OP specifically asked what kind of hardware you need to run a similar AI model with the same relative responsiveness, and GPT4 has 1.8 trillion parameters… Why would you lie and pretend like you can run a model like that on a fucking raspberry pi? You’re living in a dream world… Offline models like that require 128 GB of RAM which is $900-1200 in RAM alone…
It’s pretty great. The only thing you have to remember is that the caddy instance and the container you’re proxifying have to be within the same docker network. So you’ll definitely want to use the caddy2 container if this is the setup you want to pursue.
If not then you can just use IP addresses inside or outside of a container it doesn’t matter.
Then I found out my services would work better with Caddy
Exceptional idea. Cloudflare is nice, but Caddy will always win IMO. Additionally, considering you were able to get Caddy working, that simply drives home that unfortunately your reverse_proxy didn’t work because it was somehow misconfigured. Caddy is also a reverse_proxy.
My comment is pretty much what I said. You have an extremely complex environment that you’re not fully making use of. For example, you’re having issues with a reverse_proxy, but you had Tailscale presumably the whole time. Why not just use your VPN to reverse_proxy your requests if you were having issues?
Also using Caddy + Cloudflare is fine if you want to use cloudflare for DNS, however, Caddy handles all certificates itself. So you have Caddy, which can handle all the SSL certs itself, but you put Cloudflare on top of it to manage SSL certs. It’s just convoluted.
It’s a good environment, but a little overkill.
I really love home labs but this sounds incredibly over engineered for its purpose… I would expect a set up like this for an enterprise environment…
I very highly recommend that you take the time and just switch. Caddy is simply fabulous. It’s designed to work (assuming it’s compiled with the module) with containers and use docker networks for routing. It makes it easy to spin up containers and directly reference the container names instead of remembering IP addresses and particularly comes in handy when your entire environment is containerized.
You can pull the caddy image and run it in docker and as long as your environment is configured correctly you can simply reverse_proxy @container and you’re done. Caddy pulls all the relevant port information directly from the container API.
I get such a nerd boner thinking about it.
AI for form processing and/or paperwork would be great if done right.
Immagine you submit an application and either it’s verified almost immediately, or is elevated to a real person because there’s an issue. Could be awesome.