Certificate Renewal and Monitoring

Fuzzball deployments rely on several internal TLS certificates. On Kubernetes deployments these certificates are issued by cert-manager and renewed automatically before they expire. This page describes what renews on its own, what to expect when a renewal happens, and how to monitor certificate expiry.

cert-manager renews each internal certificate 30 days before it expires. This applies whether the Fuzzball operator manages its own cert-manager installation or the deployment uses an external cert-manager.

A customer-provided substrate CA (configured inline through spec.fuzzball.substrate.mtls) is stored as-is and is not renewed by cert-manager. You own its lifecycle — record its expiry date and replace it before it lapses.

Deployments that use the in-cluster PostgreSQL database receive a server certificate that is valid for one year and renewed automatically 30 days before expiry. When the certificate is renewed, a Reloader instance deployed alongside the database restarts the PostgreSQL pod so the server picks up the new certificate. This causes a brief database interruption, roughly once a year; Fuzzball services reconnect automatically.

Deployments backed by a cloud-managed database service (for example AWS RDS or GCP Cloud SQL) are not affected — the cloud provider manages those certificates.

To control when the restart happens, force the renewal during a planned maintenance window instead of waiting for the automatic renewal to land at an arbitrary time. With cmctl installed:

$ cmctl renew database-server-cert -n fuzzball-database
$ kubectl -n fuzzball-database wait --for=condition=Ready certificate/database-server-cert --timeout=120s

Forcing a renewal drives the same cert-manager renewal flow as the automatic schedule, and the automatic renewal remains in place as a backstop if a maintenance window is missed.

Compute nodes provisioned dynamically per workflow need no action here: nodes within a workflow are provisioned together and receive the current CA at startup. This section applies to long-lived compute nodes — static or on-premises fleets that keep running across a CA rotation.

Fuzzball Orchestrate detects a renewed substrate mTLS CA and distributes it to compute nodes automatically. The Fuzzball extension service on each long-lived compute node does not adopt the rotated CA on its own, though — running workloads hold their TLS material in memory, so the swap happens only when you signal that the time is right. Until then, each affected node logs the warning (visible with journalctl -u fuzzball-substrate on bare-metal nodes, or in the substrate container’s logs on Kubernetes-hosted nodes):

substrate CA was rotated but this extension still uses the previous CA

The 30-day warning alert from Monitoring Certificate Expiry is the cue to schedule a maintenance window. During that window, raise the apply generation in the fuzzball namespace:

$ kubectl -n fuzzball create configmap substrate-mtls-apply --from-literal=generation=1

On subsequent rotations, increment the value instead:

$ kubectl -n fuzzball patch configmap substrate-mtls-apply --type=merge -p '{"data":{"generation":"2"}}'

Within about a minute, every long-lived compute node mints a certificate from the rotated CA and applies it live — no restarts. Each node logs applied rotated substrate CA without restart, and its substrate-extension/substrate-mtls-ca expiry metric switches to the new CA’s expiry.

The generation must strictly increase, and each value triggers at most one apply attempt per node — raise it again to retry after a failure. Raising the generation while no rotation is pending does nothing, so a stale signal cannot apply a future rotation at an uncontrolled time.

Schedule the apply when in-flight workloads are drained or their interruption is acceptable: running job containers hold the previous CA in memory and cannot pick up the new one. Both CAs remain valid during the roughly 30-day renewal overlap, so there is time to schedule the window — but converge every node inside it.

Compute nodes whose substrate runtime predates live certificate update (substrate versions older than v2.6.0), or whose apply attempt failed, keep logging the rotation warning; restart the substrate service on those nodes instead. On bare-metal or VM nodes:

$ systemctl restart fuzzball-substrate

On Kubernetes-hosted substrate nodes, delete the substrate pod on that node — the replacement pod starts on the current CA.

Every Fuzzball component exports a certificate expiry metric for the certificates it actually holds in memory — not just what is stored in Kubernetes secrets. A renewed certificate that fails to reach a running component therefore still raises an alert, because that component keeps reporting the old expiry.

The metric is named fuzzball_certificate_expiry_seconds in Prometheus-family backends (fuzzball.certificate.expiry in OpenTelemetry/Datadog). Its value is the number of seconds until the certificate expires, and it goes negative once the certificate has expired. It carries two labels:

The metric is exported over OTLP to whatever collector the deployment is configured with (for example a Datadog agent or a site-provided OpenTelemetry collector). Fuzzball does not ship an alerting service — wire alerts into your own monitoring backend. Recommended thresholds:

• Under 30 days (warning) — renewal should already have happened; investigate before it becomes urgent.
• Under 7 days (critical) — renewal or propagation has failed.
• At or below 0 (critical) — the certificate has expired and connections using it are failing.

Also alert on the metric’s absence (for example absent(fuzzball_certificate_expiry_seconds) in Prometheus) — a series that never arrives fires no threshold alerts.

Docker Compose deployments do not include cert-manager, so certificates are not renewed automatically. The deployment’s wildcard TLS leaf certificate is valid for one year and is renewed manually with the --tls-renew flag of the fuzzball cluster docker-compose update command — see Other maintenance operations. Schedule the renewal before the certificate’s one-year expiry.