CertBot cron jobs are failing - Leaving certificates to expire.

[WadeBarnes]

We've had a significant number of CertBot cron job failures recently. These failures are leaving site certificates to expire.

The issue appears to be one of timing. It seems that in some cases the containers are failing to start in a timely fashion (throttling messages can be seen in the logs) while other times the route(s) generated by the container are not ready in time for the authentication callbacks.

Manually triggering the job(s) after updating the configuration with settings that slow down the process of creating the routes and triggering the authentication callbacks allow the process to complete successfully.

Latest failures:

• a99fd4-dev
• a99fd4-test
• ca7f8f-dev
• ca7f8f-prod
• ca7f8f-test
• e79518-dev
• e79518-prod
• e79518-test

[WadeBarnes]

startingDeadlineSeconds is set to 60 (1 minute) seconds
activeDeadlineSeconds is set to 300 (5 minutes) seconds

The startup active deadline appears to be the issue: Job was active longer than specified deadline

So not the issue first thought.

[WadeBarnes]

Throttling issues and therefore exceeding the active deadline could be due to the low CPU limit set on the cron jobs:

resources:
  limits:
    cpu: 250m
  requests:
    cpu: 50m

[WadeBarnes]

Updating resources to:

resources:
  requests:
    cpu: 100m

[WadeBarnes]

Now that all jobs are updated, we'll monitor for a few runs to see if that resolves the issues.

[WadeBarnes]

Jobs failed in:

• a99fd4-dev
• ca7f8f-dev
• ca7f8f-test
• ca7f8f-prod
• e79518-prod

Increased activeDeadlineSeconds to 600 seconds (10 minutes). Updated the remaining environments to match.

[WadeBarnes]

Most of the jobs failed again yesterday. Updated startingDeadlineSeconds to 300 seconds (5 minutes), to see if that helps.

[WadeBarnes]

All jobs for a99fd4 completed successfully on the 19th.
All jobs for ca7f8f completed successfully on the 19th.
All jobs for e79518 completed successfully on the 19th.

However the run in e79518-test failed to renew a certificate on the 19th, however a follow-p manual run the following day renewed the certificate without issue.

It appears the route was not ready - query resulted in a 404.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
networkpolicy.networking.k8s.io/certbot-allow-ingress created
service/certbot created
route.route.openshift.io/acme-challenge-tails-test.vonx.io created
...
Renewing an existing certificate for tails-test.vonx.io
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: tails-test.vonx.io
Type: unauthorized
Detail: 142.34.194.118: Invalid response from https://tails-test.vonx.io/.well-known/acme-challenge/<redacted>: 404
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 8080. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.
Failed to renew certificate tails-test.vonx.io with error: Some challenges have failed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[WadeBarnes]

Failed jobs - Exceeded active deadline (which is 10 minutes now):

• e79518-dev
• e79518-test
• e79518-prod
• a99fd4-dev
• a99fd4-test
• ca7f8f-test

Other environments succeeded, and there were now certificates to update.

[WadeBarnes]

While running the failed jobs manually I was able to capture what a script hang looks like. The key piece being error: no objects passed to create.

I0423 12:30:18.575724      14 request.go:665] Waited for 1.01540256s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/monitoring.rhobs/v1?timeout=32s
CERTBOT_DEBUG = false
Deleting ACME challenge resources ...
I0423 12:30:21.679155      56 request.go:665] Waited for 1.001373351s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/performance.openshift.io/v1alpha1?timeout=32s
I0423 12:30:31.697117      56 request.go:665] Waited for 1.218326596s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/quota.openshift.io/v1?timeout=32s
No resources found
networkpolicy.networking.k8s.io/certbot-allow-ingress created
service/certbot created
error: no objects passed to create
Manage individual certificates for each unique host.
No new certificate(s) have been issued
Deleting ACME challenge resources ...
I0423 12:30:43.442442     184 request.go:665] Waited for 1.0195646s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/postgres-operator.crunchydata.com/v1beta1?timeout=32s
I0423 12:30:53.460872     184 request.go:665] Waited for 1.237014989s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/pipelinesascode.tekton.dev/v1alpha1?timeout=32s
service "certbot" deleted
networkpolicy.networking.k8s.io "certbot-allow-ingress" deleted

https://console.apps.silver.devops.gov.bc.ca/k8s/ns/a99fd4-dev/cronjobs/certbot/jobs

[WadeBarnes]

I'll integrate a fix for such issues into the work here; bcgov/certbot#38

[WadeBarnes]

Same thing is happening with the job in https://console.apps.silver.devops.gov.bc.ca/k8s/ns/a99fd4-test/cronjobs/certbot/

Seems to be an issue mostly when there is nothing to do. i.e. There are no routes that are being maintained by certbot. The script hangs on the cleanup phase. Which makes sense as to why it runs when CERTBOT_DELETE_ACME_ROUTES is set to false.