Connecting Hobby Project to LLM

Author: Kalin

2024-06-10 16:34:00 +0200 CEST

I have a hobby Go project where I have a few use cases that could really benefit from LLM. I am running Ollama at my home server but that's not accessible to my production environment which is running on places such as https://fly.io, https://render.com or a VM somewhere in the public clouds.

I explored several options:

That's amazing but expensive so I don't want to run my production with it.

That's actually a nice option but I would rather not have to deal with a lot of setup or depend on offerings that might require payment eventuall if I manage to scale.

That adds some attack surface to my application and since that worker will do a lot of updates to my data (database) it's not a great option without investing heavily in security which I don't have the time for in this hobby project.

OpenZiti helps setting up a secure overlay network between my home environment and the production environment where I run my application. It's easy to use (thanks to Zrok). Also my application is written in Go so potentially I can add some pieces from Zrok into my own code to make the integration process easier. Like automatically set the token by having a small binary at my home server that stores the token on secure secret sharing server or even send it directly to the application to establish the connection.

Here's how I did it manually using the zrok cli:

  1. Share the resource from within my local network:
zrok share private http://192.168.88.11:3000
  1. Use the command that I got on the remove VM:
zrok access private 6jbvz44pqj5o
  1. Now I can access http://127.0.0.1:9191 on the remote VM.

If I configure that endpoint in my application it just works like if it is running locally against http://192.168.88.11:3000 in my local network. That's really great because I did zero code changes to integrate this.

Now I can go further into the development of my app without having to pay for LLMs which helps me keep my cost low while I am still starting up.