Hello, Nice to see you again! ๐๐
๐งต This is the part 05 of the Home Lab 2.0 series.
Secret Management ๐ is always a challenging exercise and like many of you, I have seen my fair share of credentials on plain text config files. Leaked and forgotten long living credentials in many shapes and forms are not a rare sight in todays IT system labyrinths. All the leading cloud providers have their own secret management solutions that tightly integrate with the rest of their eco-system. However they fall short in a multi-cloud environments.
Hashicorp Vault shines as an independent centralised highly capable secret management platform that can be integrated with many cloud providers as well as other existing solutions. Though I have used Vault in a few of my previous work places (including the current work setup) I didnโt have my own environment to tinker with as I wish. Therefore I decided to deploy my own instance of Hashicorp Vault to manage secrets in my Home Lab setup. Of course, I will get a few more gray hair in the process but thats a small price to pay ๐คฉ
Host Selection ๐๏ธ๐
My initial plan was to use a VM to deploy Vault as a service or use a bunch of containers with Docker Compose. However during this Home Lab 2.0 journey, it was apparent that LXC Containers are perfect choice for most of my applications. They are super efficient (easy on my poor mans hardware ๐ซ ) and first class citizens in a Proxmox setup. Though LXC containers donโt offer a similar level of isolation as a virtual machine, running them as unprivileged with a minimal up-to-date Linux destro template offers a balanced outcome for a Home Lab setup. Therefore I opted for an Ubuntu based LXC Container as my Vault host.
I could easily reuse my Terraform templates that I used for Keycloak and Traefik deployments to get the host up and running. I have a separate VLAN for the Home Lab and that makes my life easier when it comes to allocate a static IP to the hosts.
This is the first time that I ever tried to setup my own Vault. Just running a Vault docker container in my Mac to poke around doesnโt really count, right?๐. Therefore, I took some time and did a bit of a digging to figure out the best way to set this up. Usual AI overlords (Gemini, Claude, ChatGPT) provided me some useful configuration help (most were riddled with errors) but I went through a lot of iterations to get things working with Ansible.
I decided to install Vault using apt as that was the most straightforward way to install Vault in Ubuntu. Since Hashicorp recommends using Vaultโs Integrated Storage (Raft) for most use cases, I didnโt bother to mess with anything else. Also it was the simplest for my need. As you may have noticed, I separated host setup (Terraform) and application installation and configuration (Ansible, Bash) as two distinct actions for my setup. This reduced the complexity of the Home Lab 2.0 code. The seal/unseal process of Vault after each restart was a separate piece of challenge. After looking into the available options I ended up with file based auto unseal (duct tape-ish) solution as an acceptable workaround ยฏ\_(ใ)_/ยฏ for my Home Lab 2.0 setup. This approach stores the single shard unseal key in a file but if anyone/anything is able to get to that far into my network, Iโm cooked without a doubt and will have a lot more things to worry about ๐.
Maybe I will think about auto-unseal Vault using AWS Secrets Manager in future, as an improvement to the current setup ๐ค.
Integration with Traefik and Keycloak ๐๐ค
The main Ansible role uses two separate roles for the Vault init and Keycloak setup. Once this got executed, I dropped in a new config file into dynamic folder of the Traefik, added a vault-headers section into middlewares config and re-run the Traefik playbook. Also I had to update my DNS records in AdGuard and pfSense. My initial setup seems to paid off as the integration effort with Traefik and Keycloak became a quick and easy process than I expected ๐.
http:
routers:
to-vault:
entryPoints:
- websecure
rule: "Host(`vault.jayforweb.com`)"
service: vault
tls:
certResolver: dnsresolver
middlewares:
- vault-headers
services:
vault:
loadBalancer:
servers:
- url: "http://192.168.193.80:8200"
healthCheck:
path: "/v1/sys/health"
interval: "30s"
timeout: "5s"
http:
middlewares:
redirect-to-https:
redirectScheme:
scheme: https
permanent: true
vault-headers:
headers:
customRequestHeaders:
X-Forwarded-Proto: "https"
X-Forwarded-Port: "443"
customResponseHeaders:
X-Frame-Options: "DENY"
X-Content-Type-Options: "nosniff"
Referrer-Policy: "strict-origin-when-cross-origin"
Vault is Alive ๐๐พ
As I mentioned above, it took me many iterations to get everything working and I think there will be other challenges when I try to use the Vault with GitLab and the Kubernetes cluster. So far, Iโm happy about how everything is coming into place. The Home Lab 2.0 is already serving me well as it gives me the opportunity to experiment with all these amazing tools, in my own terms.
I sincerely hope that my effort in writing these things and sharing the not-so-perfect code will help at least one more person to figure out a thing or two.
Next, onwards to setup GitLab!
Happy tinkering! ๐
๐ข๐ธ๐ง๐ปโโ๏ธ๐๐ฎ๐ช๐ฌ๐ฎ
Reference ๐๐
Vault Installation - Official documentation
Vault Seal/Unseal process
Vault unseal options comparison
