Finished proxy - but I can’t use it

I finished writing the proxy. I had to:

1. Retrieve the provided containers internal IP
2. Generate a random IPv6 address within the given prefix and add it using netlink
3. Start a proxy, listening at the generated IPv6 for each port, and proxying it to the containers IP.

This works, but unfortunately I can’t use it!

My server provider, datalix, gives a /64 subnet for each VPS, but what they don’t tell you is that you can only have 10 routed IPv6 addresses! This is not enough, and I have to manually add the addresses anyways. This means that I can’t listen on the generated IP address, and also that I can’t give each container an IP. This is a big problem, because some applications (like a minecraft server) require a plain TCP connection, which does not work behind a reverse proxy since there is no hostname passed.

My only option is to rent a dedicated server, which I can’t afford right now.

So now I will continue work on other parts of the project, like making it viable and easy to deploy different services, offer one click apps/database and potentially work on a way provision servers from the cloud.

Commits:

1. Retrieve container IPv4 address on expose command
2. Finish proxy

Finished proxy communication

Wrote a library to make receiving and sending messages really simple. Uses unix sockets so they are pretty fast.

The server is started in the background, then different commands are received through channels.

On the client, the connection is started in the background but c.Start() waits for the connection to change to connected.

Then messages can be sent, but the connection has to be sustained using c.Wait() which waits until the server disconnects.

Now I can start work on querying the docker container for its internal ip address and proxying it.

Commit

Started work on the proxy API

Not much to show, but I started writing code to create a unix socket to listen for messages. I’ll need to pass a container name, and array of ports to publish. I’ll also have to figure out how to pick new ipv6 addresses

long time no see

I kinda hit a roadblock and waiting till I got past it until posting a devlog, but that never happened.

That roadblock was trying to give each container a public ipv6 address. I was trying to get docker to expose ports on a specific ipv6 address on each container, but no matter what I did there still ended up being port conflicts.

I found a better alternative now though, which is writing a proxy service that binds to an ipv6 address, and forwards the traffic to the containers internal ipv4 address. This works, but now I need to write an API so my virtual server can specify a container name and ports to forward to a public address, and dynamically create these addresses

I still worked on other stuff though, so here’s what I have changed: (Sorry for massive devlog)

1. The docker virtual server now syncs products automatically to the platform. Before it had to be done through a cURL command.
2. Add a default postgres url to configuration
3. Add a Key-Value input (UI library)
4. Add support support for Key Value inputs to platform
5. Allow users to configure volumes
6. Simplify virtual server configuration by removing suboptions
7. Allow regex validation for text inputs
8. Fixes for KV Input: 1, 2
9. Add a README
10. Disable email verification by default
11. Style scrollbar on chrome
12. Make error message more detailed
13. Allow users to specify port protocol with regex validation
14. Add a database utility to makefile

Lotta Features!: Toasts, Container Recreation, and Ability to reconfigure services

1. Added toast messages - needed a better way to show error messages so I implemented toasts. These were implemented using a MutationObserver, which watches an element for changes, in this case for new nodes being added to it. This lets me automatically remove toasts after a certain time and fade them out. Using a MutationObserver allows toasts to be added using HATEOS (HTMX) instead of only being able to create toasts from JavaScript.
2. Added the ability to reconfigure services. Before only the value was shown, but now you can update configuration and save it. Added an API endpoint so virtual servers could request new service configuration. Also reworked code to use toasts, and generally improved code quality by moving code into separate functions
3. Added the option to recreate containers. This is needed to update the container image, and/or refresh configuration, for example new ports or even a different image. This depended on the changes in 2.

1. Toasts
2. Reconfigure services
3. Recreate containers

Feature: Service Actions

Virtual servers can now list actions for the user.

This is fully programmatic, so when the virtual server sends statistics and links, it also responds with a list of actions like:

{
  "label": "Stop",
  "code":  "stop_container",
  "type": ui.ButtonTypeDanger,
  "confirm": {
    "title": "Are you sure you want to stop this container?",
    "description": "Application data may be lost.",
    "button_label": "Confirm"
  }
}

Each action has an optional and configurable confirmation dialog. Doing it like this allows the virtual server to have different actions based on the status of the container for example, like start if its stopped and stop/restart if its running.

I also added a loading indicator to all buttons, which shows whenever a HTMX request is being made.

1. Sort links by name (to prevent them shuffling)
2. Add service actions

Feature: Service overview page

Shows the status and utilisation of the container, and also the links.

The links are found by parsing the containers labels and extracting the traefik rules.

Fixed a few issues with the docker manager:

1. Only one manager was started but the channel width was 5, meaning only one service was handled at a time. Now I start 5 managers, so 5 containers are handled at once.
2. Made it so loading services at start runs in the background as well, so the API is available a lot quicker
3. Each port uses a different service name in traefik, not sure if this was an issue but could of potentially caused problems when a service has multiple ports

Also added loading indicator and status dot components.

Next I’ll implement manual start/stop of containers (general action buttons), then service deletion and then volumes

1. Service overview page
2. Loading indicator/status dot

Feature: Fully automatic reverse proxy

I added support for configuring ports, then implemented the Traefik reverse proxy giving each port a unique (non random - based on service id and port) subdomain. The service is given automatic TLS (error due to self signed cert - when I deploy I will have a domain and use letsencrypt)

Next I need to add a better display, so the link shows in app

Commit

Feature: Parse and store multi-input values

1. I allowed configuration of multi inputs, to set the min and max values/length of number/text inputs
2. I parsed and stored the values on the backend, so they are now available to the virtual server

I need to:

1. Allow updating configuration after creation
2. Figure out how to implement a reverse proxy with variable subdomains to expose ports

1. Configure multiinputs
2. Add parsing + storage

Feature: Multi-Inputs

I added an input to my ui library that lets you add infinite entries. Supports text/number types. Also fixed an issue where modals could overflow the screen.

Built this so I can have better configuration for my docker service, i.e to set ports and volumes.

Haven’t implemented the backend storage for these but the components work

Commit

Feature: Tabs

The service page looked pretty bad so I added tabs to my UI library.

Commit

Feature: View all your services

Doesn’t show much rn

Had a nightmare figuring out why the service list was overflowing the page. Solution was somehow to but min-width: 0 on the container 😭

Commit

Feature: Load existing on services on start + async container manager

I added an endpoint to the platform API (GET /api/services) that returns all of the clients services.

On start, the virtual server will request services to run, and start them one concurrently using golangs channel features (containers to run are added to a queue where up to five containers are handled at a time)

1. Add async container manager
2. Add API endpoint and start containers on virtual server start

Feature: Run docker images

I implemented the very basics of running a docker container, and added a logs endpoint

Definitely need to improve the UI for the service page, and make a tabs component. Also need to make stuff async so I don’t wait for the image to pull

Commit

Feature: Configuration groups

I added the ability to set configuration groups. These are purely visual. A virtual server may set whether the options are visible by default. For example to hide addons or advanced config options

Commit

Feature: Order services

I implemented the backend for ordering a service.

Diagram explaining flow
Text scored with red has not been implemented yet

It works by

1. Storing the service and configuration in a database
2. Sending a request to the virtual server to run the service (signing the message so the virtual server can trust it)

This setup is pretty complicated but it allows for virtual servers to run virtually anything and have configuration options for all of it.

I have yet to implement:

1. Per addon pricing (e.g +1 credit for +1 GB RAM)
2. Configuration groups, which would allow for sections of configuration (e.g simple, advanced, and addons)
3. Advanced options (required, selects etc)
4. A proper virtual server that actually runs stuff
5. A good service page

1. Implement service orders
2. Disable spinners on number inputs

WIP Feature: Get service modal

I implemented a modal that loads on click, and has all the addons/options of the service.

I need to style number inputs better, and make it so servers can set min/max values and lengths

Commit

Feature: Programmatic services

You can now add services through an API, using a generated client id and secret.

A server would call the API, passing a list of products to sync. You pass a name, description, unique code, a price and a pricing schedule (per hour, day, or month). You can also add addons with options, currently only text or numbers (e.g for additional RAM).

I also added a page to view all available services.

Next I’ll implement “getting” the service

1. Add services API + page
2. Show credit price

Feature: Disable email verification

Added an env variable (MAIL_ENABLE) that will disable any email sending and functions like verification. With this enabled, signing up will automatically mark the account as verified and sign them in, skipping the verify step.

Did this so I can develop entirely locally and without internet, I’m on holiday in the Outer Hebrides where internet isn’t great. Writing this on a 7 hour ferry 🫩

Also changed the flags field from bigint -> integer to save space, I won’t need 64 bits)

1. Disable Email Verification
2. Fix silliness

Feature: API client secret regeneration

You can now regenerate the client secret.

Mostly copied code from the confirm deletion, tweaked to show the new secret after.

Also improved error handling so now it should show an error message if something fails instead of sending the error into the void

Commit