Fix OpenShift “certificate has expired or is not yet valid” error

Introduction

In the following post, we will see how Replacing the ingress certificate in OpenShift helps this error. When operating an OpenShift cluster you will run into several certificate issues. One of the common issues we see with customers is the ingress certificate expiring with the following error:

Unable to connect to the server: x509: certificate has expired or is not yet valid

For example:

$ oc get nodes
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-09-08T14:43:53+03:00 is after 2023-09-04T11:41:44Z

Our commands are on a cluster where a certificate was already been replaced in the past using Replace OpenShift default ingress certificate procedure.

Procedure

Let’s create a new Certificate Signing Request(csr) called wildcard_2023.apps.ocp.k8s.co.il.csr based on the previous csr_answers.txt file.

$ openssl req -new -key wildcard_2023.apps.ocp.k8s.co.il.key  -out wildcard.apps.ocp.k8s.co.il.csr -config <(cat csr_answers.txt)

Then, we’ll create the certificate wildcard_2023.apps.ocp.k8s.co.il.crt based on the csr and CA files.

$ openssl x509 -req -in wildcard_2023.apps.ocp.k8s.co.il.csr -CA myCA.crt -CAkey myCA.key -CAcreateserial -days 365 -sha256 -out wildcard_2023.apps.ocp.k8s.co.il.crt -extfile wildcard.ext -extensions req_ext

Patch OpenShift

Now that we have the certificates, let’s put them in OpenShift.

First, make sure you have the full chain in your pem file:

$ cat wildcard_2023.apps.ocp.k8s.co.il.crt /path/to/myCA.crt > fullchainwildcard-2023.apps.ocp.k8s.co.il.pem

Then, let’s create a secret to hold our certificate

$ oc create secret tls fullchainwildcard-2023.apps.ocp.k8s.co.il \
  --cert=/path/to/fullchainwildcard-2023.apps.ocp.k8s.co.il.pem \
  --key=/path/to/wildcard.apps.ocp.k8s.co.il.key \
  -n openshift-ingress --insecure-skip-tls-verify=true

Finally, patch the ingress controller

$ oc patch ingresscontroller.operator default \
  --type=merge -p \
  '{"spec":{"defaultCertificate": {"name": "fullchainwildcard-2023.apps.ocp.k8s.co.il"}}}' \
   -n openshift-ingress-operator --insecure-skip-tls-verify=true

Check the replacing of the pods:

$ watch -d -n1 oc get pods -n openshift-ingress --insecure-skip-tls-verify=true

Patch the OpenShift API endpoint

We need to also replace the API endpoint certificate. In our case the *.ocp.k8s.co.il is also in the certificate. So we can use the same certificate to both ingress and api.

Create the secret for the API certificate:

$ oc create secret tls fullchainwildcard-2023-api.apps.ocp.k8s.co.il \
  --cert=/path/to/fullchainwildcard-2023.apps.ocp.k8s.co.il.pem \
  --key=/path/to/wildcard.apps.ocp.k8s.co.il.key \
     -n openshift-config --insecure-skip-tls-verify=true

Patch the API server with the certificate.

$ oc patch apiserver cluster \
     --type=merge -p \
     '{"spec":{"servingCerts": {"namedCertificates":
     [{"names": ["api.ocp.k8s.co.il"], 
     "servingCertificate": {"name": "fullchainwildcard-2023.apps.ocp.k8s.co.il"}}]}}}' --insecure-skip-tls-verify=true

Follow the replacement process with this command:

$ watch -d -n1 oc get clusteroperators kube-apiserver --insecure-skip-tls-verify=true

Test

We can make sure we have Fix OpenShift “certificate has expired or is not yet valid” error with the following commands for example:

$ oc get nodes
$ oc get co

To see the current certificate details, use the following command:

$ echo | openssl s_client -connect api.ocp.k8s.co.il:6443  | openssl x509 -noout -text | less

Summary

Working with OpenShift can occasionally lead to certificate-related roadblocks. In explaining how to fix the “certificate has expired or is not yet valid” error by replacing the ingress certificate. We hope our insights help you to tackle it with confidence. Stay proactive, and ensure your OpenShift deployments run seamlessly.

Additional hardening and improvements for OpenShift and Kubernetes can be found at our Security Blogs Category.

Enjoy.

References

Recovering from expired Openshift-ingress certificates (OCP4.x)

kube-apiserver pod does not come up