OpenShift and IPA Series

This post is part of a collection of blog posts related to OpenShift and FreeIPA (aka idM).

Introduction

I’m rebuilding my lab with a focus on OpenShift 4 and during this rebuild, I’m working on “modernizing” several aspects of my configuration. This includes attempting (again) to integrate in FreeIPA for managing hosts, certificates and other items.

OpenShift 4’s default node operating system, CoreOS, does not support being directly attached to FreeIPA in the way a traditional Linux host would be. But, OpenShift 4 still requires certain certificates as well as can integrate with an LDAP provider for authentication and authorization.

This post describes creating the certificates for the ingress controller and API endpoints for OpenShift. This isn’t nearly as straightforward as it would seem.

For this example, I’m using the version of FreeIPA included with RHEL 8, which has an official product name of Identity Management (idM). For this article, I’ll be calling everything FreeIPA to keep it generic.

Environment Description

First, let’s look at the environment. This process requires only a few pieces of information:

  • FreeIPA base domain: private.opequon.net
  • OpenShift base domain: ocp4.private.opequon.net
  • API and Ingress Controller IP Address: 172.31.0.120

Creating the Certificates

Creating DNS Zones and Host Entries

First create the ocp4.private.opequon.net zone. This will be the base name of my cluster.

ipa dnszone-add ocp4.private.opequon.net --admin-email=admin@private.opequon.net

Next create entries for each required OpenShift hostname. apps must be created, even though only the wildcard is used, because it’s used to create the certificate later on.

ipa dnsrecord-add ocp4.private.opequon.net api --a-rec=172.31.0.120
ipa dnsrecord-add ocp4.private.opequon.net api-int --a-rec=172.31.0.120
ipa dnsrecord-add ocp4.private.opequon.net apps --a-rec=172.31.0.120
ipa dnsrecord-add ocp4.private.opequon.net *.apps --a-rec=172.31.0.120

Notice, I’m not creating a reverse entries here as it is not specifically required and since 172.31.0.120 is shared between the API and Ingress Controller, I’m not sure what I’d want it to be. In a real production situation, I would likely want reverse entries for everything.

Finally, create the host principal.

ipa host-add apps.ocp4.private.opequon.net
ipa host-add api.ocp4.private.opequon.net

Wildcard Profile

With FreeIPA, certificates are generated based on a profile and the default profile is acceptable for most normal certificates, like our api certificate.

By default, FreeIPA does not include a profile for creating wildcard certificates and much of the documentation has warnings around wildcard certificates being deprecated. While this may be true in some sense, wildcard certificates are still very much a part of the OpenShift installation.

In order to support wildcard certificates, we need to create a new certprofile, which defines how these certificates should be created. This profile will take care of prefixing the certificate’s Common Name and Subject Alternate Names (SANs) with *..

To define a new profile, first, we extract the default certprofile:

ipa certprofile-show caIPAserviceCert --out wildcard.cfg

Next we need to modify the cert profile to automatically prefix the Subject and SAN fields with *.. The instructions for how to do this were found on Fraser Tweedale’s blog and are an extension of Documenation on the FreeIPA wiki.

The diff below shows the changes needed to the wildcard.cfg file:

[root@ipa ocpcerts]# diff wildcard.cfg wildcard.cfg.orig 
19c19
< policyset.serverCertSet.1.default.params.name=CN=*.$request.req_subject_name.cn$, O=PRIVATE.OPEQUON.NET
---
> policyset.serverCertSet.1.default.params.name=CN=$request.req_subject_name.cn$, O=PRIVATE.OPEQUON.NET
32,40c32,33
< policyset.serverCertSet.12.default.class_id=subjectAltNameExtDefaultImpl
< policyset.serverCertSet.12.default.name=Subject Alternative Name Extension Default
< policyset.serverCertSet.12.default.params.subjAltNameNumGNs=2
< policyset.serverCertSet.12.default.params.subjAltExtGNEnable_0=true
< policyset.serverCertSet.12.default.params.subjAltExtType_0=DNSName
< policyset.serverCertSet.12.default.params.subjAltExtPattern_0=*.$request.req_subject_name.cn$
< policyset.serverCertSet.12.default.params.subjAltExtGNEnable_1=true
< policyset.serverCertSet.12.default.params.subjAltExtType_1=DNSName
< policyset.serverCertSet.12.default.params.subjAltExtPattern_1=$request.req_subject_name.cn$
---
> policyset.serverCertSet.12.default.class_id=commonNameToSANDefaultImpl
> policyset.serverCertSet.12.default.name=Copy Common Name to Subject Alternative Name
119c112
< profileId=wildcard
---
> profileId=caIPAserviceCert

After updating the wildcard.cfg file, it needs to be imported into IPA, a profile created, and appropriate hosts (in this case apps.ocp4.private.opequon.net) associated to that profile.

ipa certprofile-import wildcard --file ./wildcard.cfg --desc 'Wildcard certificates' --store 1

ipa caacl-add-profile wildcard-hosts --certprofiles wildcard

ipa caacl-add-host wildcard-hosts --hosts apps.ocp4.private.opequon.net

Referenced in the documentation is the need to add the ipa certificate authority to that profile. I’m not sure if this is actually needed.

ipa caacl-add-ca wildcard-hosts --cas ipa

Generating CSRs

Once the wildcard profile is in place, a certificate signing request (CSR) must be created for each domain.

In the CSR for the *.apps.private.opequon.net, it’s important to note that the CSR’s Common Name does not include the *. portion, just apps. The wildcard profile created in the previous section will prefix *. to the certificate it generates based on the CSR.

# openssl req -newkey rsa:4096 -keyout apps.key -out apps.csr

Generating a RSA private key
.........................................................................................................................................................++++
.............................................................++++
writing new private key to 'apps.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:VA
Locality Name (eg, city) [Default City]:Reston
Organization Name (eg, company) [Default Company Ltd]:Opequon Networks
Organizational Unit Name (eg, section) []:OpenShift
Common Name (eg, your name or your server's hostname) []:apps.ocp4.private.opequon.net
Email Address []:admin@private.opequon.net

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

This creates a csr file and an encrypted key, which we will later decrypt for loading into OpenShift.

I generally rename the keyfiles to indicate that they are encrypted:

mv -iv apps.key apps.enc.key

We will want generate a CSR for both apps.ocp4.private.opequon.net and api.ocp4.private.opequon.net

Generating Certificates

The csr file is input to the ipa cert-request command. For the *.apps domain we specify the --profile wildcard option to generate the certificate with our wildcard profile.

# ipa cert-request apps.csr --principal host/apps.ocp4.private.opequon.net --profile wildcard
  Issuing CA: ipa
  Certificate: MIIFvTCC<TRUNCATED>
  Subject: CN=*.apps.ocp4.private.opequon.net,O=PRIVATE.OPEQUON.NET
  Issuer: CN=Certificate Authority,O=PRIVATE.OPEQUON.NET
  Not Before: Mon Nov 02 17:55:28 2020 UTC
  Not After: Thu Nov 03 16:55:28 2022 UTC

The api certificate is generated in the same fashion, but without the --profile wildcard flag.

ipa cert-request api.csr --principal host/api.ocp4.private.opequon.net

I have not found a good way to actually extract the certificate PEM file from IPA, so I just copy the output of ipa cert-request into a textfile (e.g. apps.crt, but the name doesn’t really matter) in the normal PEM format:

-----BEGIN CERTIFICATE-----
MIIFvTCC<TRUNCATED>
-----END CERTIFICATE-----

Finally, when creating the CSRs, openssl requires the key file be encrypted with a password. OpenShift (and most other users of SSL) expect the key file to be unencrypted. So to decrypt the key, use

openssl rsa -in apps.enc.key -out apps.key

A final item to gather is the IPA certificate, which can be found in /etc/ipa/ca.crt on any host enrolled to the FreeIPA server.

At the conclusion of this process, I should have a set of certificates, key files, CSR files, and the IPA Certificate Authority.

[root@yavanna ssl]# ls -la
-rw-r--r--.  1 root root 2071 Nov  2 13:47 api.crt
-rw-r--r--.  1 root root 1789 Nov  2 13:46 api.csr
-rw-------.  1 root root 3414 Nov  2 13:45 api.enc.key
-rw-------.  1 root root 3243 Nov  2 13:50 api.key
-rw-r--r--.  1 root root 2119 Nov  2 13:44 apps.crt
-rw-r--r--.  1 root root 1793 Nov  2 13:43 apps.csr
-rw-------.  1 root root 3414 Nov  2 13:42 apps.enc.key
-rw-------.  1 root root 3243 Nov  2 13:50 apps.key
-rw-r--r--.  1 root root 1667 Nov  2 13:51 ca.crt

Loading New Certificates into OpenShift

Now that we have gathered all the materials, we can load these into an OpenShift cluster. There are three primary steps.

Loading the Certificate Authority

Since my FreeIPA Certificate Authority is self generated, it’s not included in the default CoreOS certificate bundle. In order to ensure that all CoreOS nodes trust my Certificate Authority, it must be loaded into the cluster-wide proxy configuration.

Confusingly enough, this step is included with the documentation for Replacing the default ingress certificate although it’s not directly related to that action.

First create a configmap containing the root CA which we gathered in the previous section:

oc create configmap custom-ca \
     --from-file=ca-bundle.crt=</path/to/example-ca.crt> \
     -n openshift-config

The name of this configmap is custom-ca. The name of the configmap can be changed but it must match the the next patch command!

Next, patch the cluster-wide proxy to include this certificate authority.

oc patch proxy/cluster \
     --type=merge \
     --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

Again, if the ConfigMap was created with a different name than custom-ca then this patch command must be updated to match the name of the ConfigMap!

After applying this patch, the MachineConfigOperator will update all nodes in the cluster. Certain nodes may be down, and it may be desirable to wait to peform the next steps until all nodes are back up and the ClusterOperators all are in good state.

Loading the Ingress Controller Certificate

See Replacing the default ingress certificate for the official documentation.

To load the Ingress Controller, first we create a certificate chain file which contains the *.apps.ocp4.private.opequon.net certificate, then any intermediate certificates and finally the certificate authority. My simple FreeIPA set up does not have any intermediates, so we can just concatenate the apps.crt and ca.crt files together.

cat apps.crt ca.crt > appsChain.crt

Do be careful about line endings. It’s important that when certificates are contatenated together that each certificate begins on it’s own line. For example:

GOOD:

-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----

BAD:
-----END CERTIFICATE----------BEGIN CERTIFICATE-----

Once the chain file is created, then we can use it with the key file to create a secret. This example names the secret custom-ingress-secret. As in the prior section, the name of this secret can be changed, but it must be consistent between all commands!

oc create secret tls custom-ingress-secret \
     --cert=</path/to/appsChain.crt> \
     --key=</path/to/apps.key> \
     -n openshift-ingress

With the secret in place, then we can patch the IngressController object:

oc patch ingresscontroller.operator default \
     --type=merge -p \
     '{"spec":{"defaultCertificate": {"name": "custom-ingress-secret"}}}' \
     -n openshift-ingress-operator

Again, it’s very important that the name of the secret match in the patch command!

This should cause Ingress Controller pods to be recreated in the openshift-ingress project. Once these new pods are ready, the endpoint should be using this new certificate.

Loading the API Certificate

See Adding API server certificates for the official documentation.

Loading the API certificate is the most dangerous part of this activity. If done incorrectly, then the apiservers can become unavailable. Luckily, if this is the case, then the OpenShift web console will still be available via the ingress controller.

The overall process is very similar to updating the Ingress Controller certificate.

To start, we again need to create a certificate chain:

cat api.crt ca.crt > apiChain.crt

Next we need to create a secret again:

oc create secret tls custom-api-certificate \
     --cert=</path/to/apiChain.crt> \
     --key=</path/to/api.key> \
     -n openshift-config

Finally, we need to patch the APIServer object:

oc patch apiserver cluster \
     --type=merge -p \
     '{"spec":{"servingCerts": {"namedCertificates":
     [{"names": ["api.ocp4.private.opequon.net"], 
     "servingCertificate": {"name": "custom-api-certificate"}}]}}}'

Again, the name of the secret must match what is put into the above patch. Additionally, notice that we must specify the hostname associated with this certificate.

This will cause the API pods on the OpenShift masters to restart with the new certificate. During such time, there may be small outages of the API depending on the load balancer setup.

Important to note that after applying this change, the kubeconfig file generated during installation may not work or may throw certificate errors. The kubeadmin username and password should continue to work.

Next Steps

After loading certificates, the next step would be to integrate FreeIPA’s LDAP as an authentication provider. This topic is covered in the next part.