Overhaul PKI & Secrets management
Our current PKI kinda sucks. Step CA needs human intervention for the first time issuing, or anytime it fails to renew before the certificate expires.
The client also isn't supported by an official Nix module - we have our own here. Currently, Promtail (which pushes email logs) is the only thing using it.
Additionally, our current setup for secrets means that the whole file is encrypted, so part of our config can't be public. This also complicates setting up testbeds, etc. a lot.
One option could be hashicorp vault, which allows for both PKI and secrets management. As well as issuing certificates, it can automatically set and rotate passwords for LDAP users, Keycloak services, and probably more. I also made a plugin that lets it manage Kerberos principal passwords, although it needs some polish. Machines can authenticate in different ways - probably either kerberos or token authentication.
We can then use the official vault agent, or the unofficial vault-cli project which doesn't need to run as a daemon.
This would complicate our setup more, so we should evaluate to what extent its worth it, and make a plan for migration.