Author: Kalin
I have a hobby Go project where I have a few use cases that could really benefit from LLM. I am running Ollama at my home server but that's not accessible to my production environment which is running on places such as https://fly.io, https://render.com or a VM somewhere in the public clouds.
I explored several options:
That's amazing but expensive so I don't want to run my production with it.
That's actually a nice option but I would rather not have to deal with a lot of setup or depend on offerings that might require payment eventuall if I manage to scale.
That adds some attack surface to my application and since that worker will do a lot of updates to my data (database) it's not a great option without investing heavily in security which I don't have the time for in this hobby project.
OpenZiti helps setting up a secure overlay network between my home environment and the production environment where I run my application. It's easy to use (thanks to Zrok). Also my application is written in Go so potentially I can add some pieces from Zrok into my own code to make the integration process easier. Like automatically set the token by having a small binary at my home server that stores the token on secure secret sharing server or even send it directly to the application to establish the connection.
Here's how I did it manually using the zrok cli:
zrok share private http://192.168.88.11:3000
zrok access private 6jbvz44pqj5o
If I configure that endpoint in my application it just works like if it is running locally against http://192.168.88.11:3000 in my local network. That's really great because I did zero code changes to integrate this.
Now I can go further into the development of my app without having to pay for LLMs which helps me keep my cost low while I am still starting up.