Documentation

Using private Container Registry with VEBA

By default, the VMware Event Broker Appliance can integrate with any Open Container Initiative (OCI) compliant container registry for hosting and deploying container images that uses a TLS certificate from a trusted authority such as Docker Hub or Amazon Elastic Container Registry (ECR) as an example.

For organizations that require the use of a private container registry and uses a self-signed TLS certificate, an additional post-deployment configuration is required within the VMware Event Broker Appliance. Please follow the steps outlined below.

Assumptions

  • VMware Event Broker Appliance v0.7.5 or later
  • Root CA Certificates from a trusted authority has been pre-downloaded onto your local desktop
  • Un-deploy any functions that has been attempted using private registry prior to instructions below

Note: For those using the Harbor registry, the root CA certificate is located in : /etc/docker/certs.d/[FQDN]/ca.crt

Steps

In this example, the root CA certificate key file is named ca.crt and is located in /root

  • Step 1: Copy the root CA certificate from your private registry to VMware Event Broker Appliance Appliance. If SSH has not been enabled, go ahead and start it up by logging into the VM Console and running the following command:
systemctl start sshd
  • Step 2: SSH to the VMware Event Broker Appliance and make a backup of the original containerd configuration file
cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
  • Step 3: Edit the ‘/etc/containerd/config.toml’ file using VI and locate the following section [plugins."io.containerd.grpc.v1.cri".registry.mirrors] within the configuration file.

Append the following two lines below this section and replace the REPLACE_ME_FQDN value with FQDN of the private registry and REPLACE_ME_PATH_TO_ROOT_CA_CERT value the full path to the root CA certificate located on the VMware Event Broker Appliance

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
   [plugins."io.containerd.grpc.v1.cri".registry.configs."REPLACE_ME_FQDN".tls]
   ca_file = "REPLACE_ME_PATH_TO_ROOT_CA_CERT"
  • Step 4: Restart the containerd service for the change to go into effect/.
systemctl restart containderd
  • Step 5: Verify containerd is successfully running before proceeding to the next step
systemctl status containerd

● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2022-03-28 19:35:06 UTC; 1 day 3h ago
       Docs: https://containerd.io
    Process: 30072 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 30073 (containerd)
      Tasks: 545
     Memory: 1.2G
     CGroup: /system.slice/containerd.service
  • Step 6: Create a kubernetes secret in the knative-serving namespace that points to full path of the root CA certificate of private registry which should reside within the VMware Event Broker Appliance
kubectl -n knative-serving create secret generic customca --from-file=ca.crt=/root/ca.crt
  • Step 7: Retrieve the current Knative Serving controller deployment and save it to a file named knative-serving-controller.yaml
kubectl -n knative-serving get deploy/controller -o yaml > knative-serving-controller.yaml
  • Step 8: Create the following YTT overlay which will be used to patch the Knative Serving controller to reference the root CA certificate from the private registry
cat > overlay.yaml <<EOF
#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind":"Deployment", "metadata": {"name": "controller", "namespace": "knative-serving"}})
---
spec:
  template:
    spec:
      containers:
      #@overlay/match by=overlay.subset({"name": "controller"})
      -
         env:
             #@overlay/append
             - name: SSL_CERT_DIR
               value: /etc/customca
         #@overlay/match missing_ok=True
         volumeMounts:
             - name: customca
               mountPath: /etc/customca
      #@overlay/match missing_ok=True
      volumes:
        - name: customca
          secret:
            secretName: customca
EOF
  • Step 9: Apply the YTT transformation to create the new Knative Serving controller YAML file named new-knative-serving-controller.yaml
ytt -f overlay.yaml -f knative-serving-controller.yaml > new-knative-serving-controller.yaml
  • Step 10: Apply the new Knative Serving controller configuration
kubectl apply -f new-knative-serving-controller.yaml

It can take a couple of minutes for the previous Knative Serving controller to terminate and spawn the new configuration. You can monitor the progress using the following commmand and ensure the READY state shows 1/1

kubectl -n knative-serving get deployment/controller -w

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
controller   1/1     1            1           29h

Note: If for some reason the deployment is not re-deploying, you can run kubectl -n knative-serving delete deployment/controller and then perform the apply operation with the new Knative Serving YAML.

Using private container registry with the VMware Event Broker Appliance.