Agent-Based Red Hat OpenShift Cluster Install
With OpenShift 4.11, Red Hat introduced a new agent-based installer for OpenShift to the official OpenShift installer. The aim is to provide the flexibility of user-provided infrastructure (UPI) installs with the ease of use that the OpenShift Assisted Installer offers for connected environments while in fully disconnected or air-gapped environments. Currently, it is only supported on bare-metal and VMware deployments.
With agent-based install, you create a bootable image with the OpenShift-install command to deploy the cluster. There is also an in-place bootstrap, so no extra node is required. It works in fully disconnected deployments with a mirrored local registry. Additionally, it supports single-node OpenShift (SNO), compact 3-node clusters, and highly available topologies.
Let us familiarize ourselves with the Agent-Based Installer workflow. Unlike the bare metal IPI OpenShift installation, there is no need for a provisioning host. This is because one of the nodes runs the assisted service early in the boot process and eventually becomes the bootstrap host. The assisted service validates and confirms all hosts checking in meet the requirements and triggers a cluster deployment. Once the cluster deployment kicks off, all the nodes get their RHCOS image written to disk, but only the non-bootstrap nodes reboot and begin to instantiate a cluster. Once they come up, the original bootstrap node reboots and comes up from the disk to join the cluster. The bootstrapping is complete at that point, and the cluster comes up just like any other installation method until it is finalized.
Pre-Requisites –
- Access to VMware virtual or Bare Metal machines
- A bastion host with RHEL 9 or over
- Admin access to RHEL 9, VMware virtual machines, or bare metal machines
- Access the Red Hat portal to download the pull secret, OpenShift Command line (OC), and OpenShift Installer utility running on the bastion host.
- Access is needed to add DNS A or CNAME records to private DNS.
- Base domain address
Assumption –
- Profound familiarity with Red Hat Linux commands, facilitating effective navigation and administration of the Red Hat Linux operating system.
- A sound command of Red Hat OpenShift command utilities, including but not limited to OpenShift-Install and OC, to facilitate the deployment and management of the OpenShift cluster.
- Domain resolution requires a working knowledge of DNS servers and the ability to create and manage CNAME records.
- Following this procedure, you will deploy the Red Hat OpenShift cluster with 3 master and 2 worker nodes.
Step-by-Step Instructions –
- Download Red Hat Pull Secret from here
- Run the following command to download and untar OpenShift command line (OC) and OpenShift-install on the bastion host
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.14.18/openshift-client-linux-4.14.18.tar.gz
tar -xvf openshift-client-linux-4.14.18.tar.gz
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.14.18/openshift-install-linux-4.14.18.tar.gz
tar -xvf openshift-install-linux-4.14.18.tar.gz
mv oc /usr/bin/
mv openshift-install /usr/bin/- Install JQ on the bastion host, update the Red Hat Pull secret, and move it under /root/.docker/
yum install jq -y
cat pull-secret.txt | jq > pull-secret.json
mkdir /root/.docker/
cp pull-secret.json /root/.docker/auth.json- Login to VMware Vcenter and edit the VM configuration by following the below steps
- Right-click on VM and click edit settings
- Click VM Options — Click Advanced — Click Edit Configuration — Add new configuration parameter.
- Enter disk.EnableUUID as a key and set value as TRUE - Register DNS A or CNAME records and DNS PTR records pointing to ELB or HA Proxy VM.
api.<cluster_name>.<base_domain>.
api-int.<cluster_name>.<base_domain>.
*.apps.<cluster_name>.<base_domain>.
<master><n>.<cluster_name>.<base_domain>.
<worker><n>.<cluster_name>.<base_domain>.- Install HA Proxy on the bastion host and configure the proxy.cfg file
yum install haproxy -y
cp / etc/haproxy/haproxy.cfg / etc/haproxy/haproxy_orginial.cfg
vi /etc/haproxy/haproxy.cfg- Replace the existing content of the haproxy.cfg with the following.
global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen api-server-6443
bind *:6443
mode tcp
server master0 master0.ocp4.example.com:6443 check inter 1s
server master1 master1.ocp4.example.com:6443 check inter 1s
server master2 master2.ocp4.example.com:6443 check inter 1s
listen machine-config-server-22623
bind *:22623
mode tcp
server master0 master0.ocp4.example.com:22623 check inter 1s
server master1 master1.ocp4.example.com:22623 check inter 1s
server master2 master2.ocp4.example.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server worker0 worker0.ocp4.example.com:443 check inter 1s
server worker1 worker1.ocp4.example.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server worker0 worker0.ocp4.example.com:80 check inter 1s
server worker1 worker1.ocp4.example.com:80 check inter 1s- Update node name with the actual IP address or FQDN registered in the DNS
- Update the master/worker<n>.ocp.example.com with your cluster name and base domain. For example, master/worker<n>.cluster_name.base_domain
- Install nmstate dependency by running the following command.
sudo dnf install /usr/bin/nmstatectl -y- Run the following commands and update the bastion host’s firewall.
sudo firewall-cmd --add-port=6443/tcp --permanent # kube-api-server on control plane
sudo firewall-cmd --add-port=22623/tcp --permanent # machine-config server
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --add-port=9000/tcp --permanent # haproxy stats
sudo firewall-cmd --reload
sudo setsebool -P haproxy_connect_any 1
sudo systemctl enable haproxy
sudo systemctl start haproxy
sudo systemctl status haproxy --no-pager- Create install-config.yaml file and update it as instructed
apiVersion: v1
baseDomain: example.com #Update your base domain here
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform: {}
replicas: 2
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: ocp #Update your openshift cluster name here
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 192.168.0.1/24 #Update your node network CIDR here
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
ImageDigestMirrorSet:
mirrors:
<Paste your private repo address here>
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
mirrors:
<Paste your private repo address here>
source: registry.ci.openshift.org/ocp/release
#Paste the private repo cert below under additionalTrustBundle
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
-----END CERTIFICATE-----
publish: External
pullSecret: '' #paste your pullSecret here
sshKey: <Paste your public ssh key here>- Create a copy of install-config file
cp install-config.yaml install-config.org.yaml- Create agent-config.yaml file and update it as instructed.
apiVersion: v1alpha1
kind: AgentConfig
metadata:
name: ocp # Update your cluster name here, ensure it is same as you provided in install-config.yaml
rendezvousIP: <Enter the IP of master-1 node>
hosts:
- hostname: master-1
role: master
interfaces:
- name: ens192
macAddress: <enter the mac address>
networkConfig:
interfaces:
- name: ens192
type: ethernet
state: up
mac-address: <enter the mac address>
ipv4:
enabled: true
address:
- ip: <enter the IP address>
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- <enter the DNS server IP address>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <Enter the gateway IP address>
next-hop-interface: ens192
table-id: 254
- hostname: master-2
role: master
interfaces:
- name: ens192
macAddress: <enter the mac address>
networkConfig:
interfaces:
- name: ens192
type: ethernet
state: up
mac-address: <enter the mac address>
ipv4:
enabled: true
address:
- ip: <enter the IP address>
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- <enter the DNS server IP address>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <Enter the gateway IP address>
next-hop-interface: ens192
table-id: 254
- hostname: master-3
role: master
interfaces:
- name: ens192
macAddress: <enter the mac address>
networkConfig:
interfaces:
- name: ens192
type: ethernet
state: up
mac-address: <enter the mac address>
ipv4:
enabled: true
address:
- ip: <enter the IP address>
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- <enter the DNS server IP address>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <Enter the gateway IP address>
next-hop-interface: ens192
table-id: 254
- hostname: worker-1
role: worker
interfaces:
- name: ens192
macAddress: <enter the mac address>
networkConfig:
interfaces:
- name: ens192
type: ethernet
state: up
mac-address: <enter the mac address>
ipv4:
enabled: true
address:
- ip: <enter the IP address>
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- <enter the DNS server IP address>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <Enter the gateway IP address>
next-hop-interface: ens192
table-id: 254
- hostname: worker-2
role: worker
interfaces:
- name: ens192
macAddress: <enter the mac address>
networkConfig:
interfaces:
- name: ens192
type: ethernet
state: up
mac-address: <enter the mac address>
ipv4:
enabled: true
address:
- ip: <enter the IP address>
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- <enter the DNS server IP address>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <Enter the gateway IP address>
next-hop-interface: ens192
table-id: 254- Create a copy of agent-config.yaml file
cp agent-config.yaml agent-config.org.yaml- Create an ISO image by running the following command
openshift-install - dir=. agent create image- Copy the ISO image to the VMware datastore and boot your VMs
- First, boot the master-1 node and rest after every few seconds.
After you boot the agent image and make network services available to the host, the agent console application performs a pull check to verify that the current host can retrieve release images.
If the primary pull check passes, you can quit the application to continue with the installation. If the pull check fails, the application performs additional checks, as seen in the Additional Checks section of the TUI, to help you troubleshoot the problem. Failure to perform additional checks is not necessarily critical as long as the primary pull check succeeds.
If host network configuration issues cause an installation to fail, you can use the console application to adjust your network configurations.
- Tracking and verifying installation progress
openshift-install - dir <install_directory> agent wait-for bootstrap-complete - log-level=info- To track the progress and verify successful installation, run the following command.
openshift-install - dir <install_directory> agent wait-for install-completeOnce you see the following message, log in to your OpenShift cluster and verify all nodes, pods, services, routes, etc, are in healthy condition and working as expected.
...................................................................
...................................................................
INFO Cluster is installed
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run
INFO export KUBECONFIG=/home/core/installer/auth/kubeconfig
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sno-cluster.test.example.comI hope this installation procedure saves you time and effort when deploying the openshift cluster on-prem in a bare-metal or VMware environment.
