Documentation

Certificates in VEBA

In project VMware Event Broker Appliance, both client certificates and TLS (Transport Layer Security) certificates play essential roles in securing communication between various components within the cluster. Client certificates are generated by kubeadm during the Kubernetes cluster installation and are vital for authentication and authorization of users and services.

In order to encrypt the communication channels between Kubernetes components, the VMware Event Broker Appliance generates a self-signed TLS certificate. This TLS certificate is also used to support different web endpoints running on the appliance such as Stats (/stats), Status (/status), Logs (/bootstrap) and Events (/events). This will cause browsers to show the certificate as untrusted.

This sections includes documentaiton on how to e.g. update or renew the client as well as TLS certificates in VEBA.

Table of Contents

Updating the TLS Certificate on VEBA

For organizations that require the use of a TLS certificate from a trusted authority, the VMware Event Broker Appliance provides an option for users to provide their certificate information during the OVF property configuration when deploying the virtual appliance.

In order to use a certificates from a trusted authority, please follow the steps outlined below.

Assumptions

  • Certificates from a trusted authority pre-downloaded onto your local desktop
    • The public/private key pair must exist before hand. The public key certificate must be .PEM encoded and match the given private key.

Steps

In the example, the private key file is named privateKey.key and certificate file is named certificate.crt

  • Step 1: Encode both the private key and the certificate file using base64 encoding.

Microsoft Windows (PowerShell)

$privateKeyContent = Get-Content -Raw privateKey.key
$privateKeybase64 = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($privateKeyContent))
Write-Host "Encoded Private Key:`n$privateKeybase64`n"

$certContent = Get-Content -Raw certificate.crt
$certbase64 = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($certContent))
Write-Host "Encoded Certificate:`n$certbase64`n"

Encoded Private Key:
LS0tLS1CRUd......==


Encoded Certificate Key:
LS0tLS1CRUe......==

MacOS/Linux

cat privateKey.key | base64

LS0tLS1CRUd......==

cat certificate.crt | base64

LS0tLS1CRUd......==

  • Step 2: Using the output from the previous step, the base64 content can now be provided in Custom TLS Certificate Configuration section of the OVF property during the deployment of the VMware Event Broker Appliance.

  • Custom VMware Event Broker Appliance TLS Certificate Private Key (Base64)

  • Custom VMware Event Broker Appliance TLS Certificate Authority Certificate (Base64)

  • Step 3: Power on the VMware Event Broker Appliance and ensure that the provided TLS certificate is now used instead of the auto-generated self-sign TLS certificate by opening a browser to one of the VMware Event Broker Appliance endpoints such as /status (admin/VMware1!).

Replacing an existing Certificate on VEBA

If you need to replace the default self-signed certificate, or replace an expired certificate, follow the steps outlined below.

Assumptions

  • Your SSL certificate must match the FQDN that you picked when you deployed VEBA
  • Pre-download your public and private key from your Certificate Authority, which must be .PEM encoded.

Steps

  • Step 1: Transfer or copy the contents of the root certificate to the VMware Event Broker Appliance. If you need a free, publicly signed certificate, see the Let’s Encrypt section below.

  • Step 2: Delete the existing Event Router TLS secret
kubectl -n vmware-system delete secret eventrouter-tls
  • Step 3: Create a new Event Router TLS secret with the new certificate keypair.
kubectl -n vmware-system create secret tls eventrouter-tls --key privkey.pem --cert pubkey.pem
  • Step 4: Delete the existing Contour TLS secret
kubectl -n contour-external delete secret default-cert
  • Step 5: Create a new Contour TLS secret with the new certificate keypair.
kubectl -n contour-external create secret tls default-cert --key privkey.pem --cert pubkey.pem
  • Step 6: Check for valid HTTPproxy status
kubectl -n vmware-system get httpproxy

Expected output:

NAME           FQDN               TLS SECRET        STATUS   STATUS DESCRIPTION
event-router   veba.domain.com    eventrouter-tls   valid    Valid HTTPProxy

Obtaining a Signed SSL Certificate from Let’s Encrypt

This section demonstrates installation of the Let’s Encrypt Certbot Docker image onto the VEBA appliance, then uses DNS validation to verify domain ownership.

Steps

  • Step 1: Start the Docker daemon. VEBA uses containerd for its container runtime - the Docker daemon is disabled by default.
systemctl start docker
  • Step 2: Pull the Certbot Docker image
docker pull certbot/certbot
  • Step 3: Run certbot. For the -d (domain) switch, use your VEBA FQDN. You will be prompted for an e-mail address as well as some yes/no questions.
docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" `
-v "/var/lib/letsencrypt:/var/lib/letsencrypt" `
-v "/var/log/letsencrypt:/var/log/letsencrypt" `
certbot/certbot certonly --manual -d veba02.vmweventbroker.io --preferred-challenges dns

Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): certificates@vmweventbroker.io

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Requesting a certificate for veba02.vmweventbroker.io

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:

_acme-challenge.veba02.vmweventbroker.io.

with the following value:

KfqRahey6wChxY_cZgNbRAlRpS34KQhjvQvTXnzRXgo

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.veba02.vmweventbroker.io.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you've just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

  • Step 4: Using your public DNS provider’s tools, configure the required TXT record as prompted in Step 2.

  • Step 5: Press Enter to continue. If you have configured DNS properly, the certificate PEM files will be saved in the location specified.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/veba02.vmweventbroker.io/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/veba02.vmweventbroker.io/privkey.pem
This certificate expires on 2022-04-12.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate's expiry date.

  • Step 6: Install the certificate - follow the instructions starting with step 2 of Replacing an Existing Cert on VEBA. Note from the output above that the public key file is named fullchain.pem - you will need to pass this value for the --cert argument when creating the Kubernetes TLS certificates.

  • Step 7: Stop the Docker daemon

systemctl stop docker
  • Step 8: (optional) - If you want to automate renewals, this is an excellent blog on configuring automated certificate renewals using DNS validation.

Renew Kubeadm generated Certificates

The VMware Event Broker Appliance is using kubeadm to setup Kubernetes. By default, kubeadm generates all the certificates needed for a cluster to run. The client certificates expire after 1 year by default. See also the official section in the Kubernetes documentation: Certificate Management with kubeadm.

Normally, certificates will be renewed during a Kubernetes version upgrade. It’s not intended to run in-place updates for a VEBA instance. Users would just redeploy a new VEBA version. If a VEBA instance runs longer than 365 days, all Kubernetes client certificates will expire.

This section describes how to renew the respective certificates.

  • Step 1: Check the expiration of all certificates generated by kubeadm.
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 11, 2025 20:45 UTC   359d            ca                      no
apiserver                  Jan 11, 2025 20:45 UTC   359d            ca                      no
apiserver-etcd-client      Jan 11, 2025 20:45 UTC   359d            etcd-ca                 no
apiserver-kubelet-client   Jan 11, 2025 20:45 UTC   359d            ca                      no
controller-manager.conf    Jan 11, 2025 20:45 UTC   359d            ca                      no
etcd-healthcheck-client    Jan 11, 2025 20:45 UTC   359d            etcd-ca                 no
etcd-peer                  Jan 11, 2025 20:45 UTC   359d            etcd-ca                 no
etcd-server                Jan 11, 2025 20:45 UTC   359d            etcd-ca                 no
front-proxy-client         Jan 11, 2025 20:45 UTC   359d            front-proxy-ca          no
scheduler.conf             Jan 11, 2025 20:45 UTC   359d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 09, 2034 20:45 UTC   9y              no
etcd-ca                 Jan 09, 2034 20:45 UTC   9y              no
front-proxy-ca          Jan 09, 2034 20:45 UTC   9y              no
  • Step 2: Initiate the certificate renewal process by executing kubeadm certs renew all.
kubeadm certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

  • Step 3: Reboot the appliance.

  • Step 4: Since the client certificates has changed, the old kubeconfig is not valid anymore. Copy the newly created kubeconfig to /root/.kube/.

Backup the old config file:

cp /root/.kube/config /root/.kube/old-$(date --iso)-config

ls -rtl /root/.kube/

total 12
-rw------- 1 root root 5639 Jan 12 20:45 old-2024-01-18-config
drwxr-x--- 4 root root 4096 Jan 13 09:08 cache

Copy the new config file.

cp /etc/kubernetes/admin.conf /root/.kube/config

ls -rtl /root/.kube/

total 20
-rw------- 1 root root 5639 Jan 12 20:45 old-2024-01-18-config
drwxr-x--- 4 root root 4096 Jan 13 09:08 cache
-rw------- 1 root root 5635 Jan 18 08:26 config

Validate functionality.

kubectl get deploy -A

NAMESPACE            NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager         cert-manager                       1/1     1            1           5d11h
cert-manager         cert-manager-cainjector            1/1     1            1           5d11h
cert-manager         cert-manager-webhook               1/1     1            1           5d11h
contour-external     contour                            2/2     2            2           5d11h
contour-internal     contour                            2/2     2            2           5d11h
knative-eventing     eventing-controller                1/1     1            1           5d11h
knative-eventing     eventing-webhook                   1/1     1            1           5d11h
knative-eventing     pingsource-mt-adapter              0/0     0            0           5d11h
knative-eventing     rabbitmq-broker-controller         1/1     1            1           5d11h
knative-eventing     rabbitmq-broker-webhook            1/1     1            1           5d11h
knative-serving      activator                          1/1     1            1           5d11h
knative-serving      autoscaler                         1/1     1            1           5d11h
knative-serving      controller                         1/1     1            1           5d11h
knative-serving      domain-mapping                     1/1     1            1           5d11h
knative-serving      domainmapping-webhook              1/1     1            1           5d11h
knative-serving      net-contour-controller             1/1     1            1           5d11h
knative-serving      webhook                            1/1     1            1           5d11h
kube-system          antrea-controller                  1/1     1            1           5d11h
kube-system          coredns                            2/2     2            2           5d11h
local-path-storage   local-path-provisioner             1/1     1            1           5d11h
rabbitmq-system      messaging-topology-operator        1/1     1            1           5d11h
rabbitmq-system      rabbitmq-cluster-operator          1/1     1            1           5d11h
vmware-functions     default-broker-ingress             1/1     1            1           5d11h
vmware-functions     kn-pcli-tag-00001-deployment       1/1     1            1           21h
vmware-functions     sockeye                            1/1     1            1           5d11h
vmware-functions     sockeye-trigger-dispatcher         1/1     1            1           5d11h
vmware-functions     vcsa-source-adapter                1/1     1            1           5d11h
vmware-functions     veba-pcli-tag-trigger-dispatcher   1/1     1            1           21h
vmware-sources       horizon-source-controller          1/1     1            1           5d11h
vmware-sources       horizon-source-webhook             1/1     1            1           5d11h
vmware-sources       vsphere-source-webhook             1/1     1            1           5d11h
vmware-system        tinywww                            1/1     1            1           5d11h
vmware-system        veba-ui                            1/1     1            1           5d11h
vmware-system        vmware-event-router-webhook        1/1     1            1           5d11h

Check the new expiration dates.

kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 17, 2025 08:11 UTC   364d            ca                      no
apiserver                  Jan 17, 2025 08:11 UTC   364d            ca                      no
apiserver-etcd-client      Jan 17, 2025 08:11 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 17, 2025 08:11 UTC   364d            ca                      no
controller-manager.conf    Jan 17, 2025 08:11 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 17, 2025 08:11 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 17, 2025 08:11 UTC   364d            etcd-ca                 no
etcd-server                Jan 17, 2025 08:11 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 17, 2025 08:11 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 17, 2025 08:11 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 09, 2034 20:45 UTC   9y              no
etcd-ca                 Jan 09, 2034 20:45 UTC   9y              no
front-proxy-ca          Jan 09, 2034 20:45 UTC   9y              no

Replacing the default self-signed TLS certificate in VMware Event Broker Appliance.