Vault for Secret Management

Hello, Nice to see you again! 👋😊

🧵 This is the part 05 of the Home Lab 2.0 series.

Secret Management 🔐 is always a challenging exercise and like many of you, I have seen my fair share of credentials on plain text config files. Leaked and forgotten long living credentials in many shapes and forms are not a rare sight in todays IT system labyrinths. All the leading cloud providers have their own secret management solutions that tightly integrate with the rest of their eco-system. However they fall short in a multi-cloud environments.

Hashicorp Vault shines as an independent centralised highly capable secret management platform that can be integrated with many cloud providers as well as other existing solutions. Though I have used Vault in a few of my previous work places (including the current work setup) I didn’t have my own environment to tinker with as I wish. Therefore I decided to deploy my own instance of Hashicorp Vault to manage secrets in my Home Lab setup. Of course, I will get a few more gray hair in the process but thats a small price to pay 🤩

Host Selection 🏛️🏠

My initial plan was to use a VM to deploy Vault as a service or use a bunch of containers with Docker Compose. However during this Home Lab 2.0 journey, it was apparent that LXC Containers are perfect choice for most of my applications. They are super efficient (easy on my poor mans hardware 🫠) and first class citizens in a Proxmox setup. Though LXC containers don’t offer a similar level of isolation as a virtual machine, running them as unprivileged with a minimal up-to-date Linux destro template offers a balanced outcome for a Home Lab setup. Therefore I opted for an Ubuntu based LXC Container as my Vault host.

Vault LXC Container

I could easily reuse my Terraform templates that I used for Keycloak and Traefik deployments to get the host up and running. I have a separate VLAN for the Home Lab and that makes my life easier when it comes to allocate a static IP to the hosts.

📣

By the way, the code for my Home Lab 2.0 is on Github and you will be able to easily adopt it for your own setup if needed.

Install and Configure Vault 🛠️🧑‍💻

This is the first time that I ever tried to setup my own Vault. Just running a Vault docker container in my Mac to poke around doesn’t really count, right?😊. Therefore, I took some time and did a bit of a digging to figure out the best way to set this up. Usual AI overlords (Gemini, Claude, ChatGPT) provided me some useful configuration help (most were riddled with errors) but I went through a lot of iterations to get things working with Ansible.

I decided to install Vault using apt as that was the most straightforward way to install Vault in Ubuntu. Since Hashicorp recommends using Vault’s Integrated Storage (Raft) for most use cases, I didn’t bother to mess with anything else. Also it was the simplest for my need. As you may have noticed, I separated host setup (Terraform) and application installation and configuration (Ansible, Bash) as two distinct actions for my setup. This reduced the complexity of the Home Lab 2.0 code. The seal/unseal process of Vault after each restart was a separate piece of challenge. After looking into the available options I ended up with file based auto unseal (duct tape-ish) solution as an acceptable workaround ¯\_(ツ)_/¯ for my Home Lab 2.0 setup. This approach stores the single shard unseal key in a file but if anyone/anything is able to get to that far into my network, I’m cooked without a doubt and will have a lot more things to worry about 😂.

Maybe I will think about auto-unseal Vault using AWS Secrets Manager in future, as an improvement to the current setup 🤞.

Integration with Traefik and Keycloak 🔗🤝

The main Ansible role uses two separate roles for the Vault init and Keycloak setup. Once this got executed, I dropped in a new config file into dynamic folder of the Traefik, added a vault-headers section into middlewares config and re-run the Traefik playbook. Also I had to update my DNS records in AdGuard and pfSense. My initial setup seems to paid off as the integration effort with Traefik and Keycloak became a quick and easy process than I expected 😍.

Keycloak client configuration

# traefik/ansible/roles/traefik/files/dynamic/vault.yml
# make sure to add the middlewares as well.
http:
  routers:
    to-vault:
      entryPoints:
        - websecure
      rule: "Host(`vault.jayforweb.com`)"
      service: vault
      tls:
        certResolver: dnsresolver
      middlewares:
        - vault-headers

  services:
    vault:
      loadBalancer:
        servers:
          - url: "http://192.168.193.80:8200"
        healthCheck:
          path: "/v1/sys/health"
          interval: "30s"
          timeout: "5s"

# traefik/ansible/roles/traefik/files/dynamic/middlewares.yml
http:
  middlewares:
    redirect-to-https:
      redirectScheme:
        scheme: https
        permanent: true

    vault-headers:
      headers:
        customRequestHeaders:
          X-Forwarded-Proto: "https"
          X-Forwarded-Port: "443"
        customResponseHeaders:
          X-Frame-Options: "DENY"
          X-Content-Type-Options: "nosniff"
          Referrer-Policy: "strict-origin-when-cross-origin"

Vault is Alive 🎉🍾

As I mentioned above, it took me many iterations to get everything working and I think there will be other challenges when I try to use the Vault with GitLab and the Kubernetes cluster. So far, I’m happy about how everything is coming into place. The Home Lab 2.0 is already serving me well as it gives me the opportunity to experiment with all these amazing tools, in my own terms.

I sincerely hope that my effort in writing these things and sharing the not-so-perfect code will help at least one more person to figure out a thing or two.

Next, onwards to setup GitLab!

Happy tinkering! 😊

𓇢𓆸🧘🏻‍♂️𝓟𝓮𝓪𝓬𝓮