Deploying Red Hat OpenShift on AWS Without Route53: A UPI Method Guide
Sufficient online documentation is readily accessible for installing Red Hat OpenShift on the AWS platform. Nevertheless, it is noteworthy that most of these resources advocate using AWS Route53 for domain management. The official documentation explicitly states that a failure to utilize Route53 may result in installation issues. However, when dealing with enterprise clients, a multitude of complexities often arise, including their specific requirements for domain name services. Therefore, the question naturally arises: does the inability to use Route53 preclude the deployment of a Red Hat OpenShift cluster within a customer’s AWS account? The definitive answer is “No.” It is indeed possible to execute the installation without Route53, provided the customer opts for the “User-Provided Infrastructure (UPI)” approach.
In this blog post, I will meticulously outline a step-by-step guide for installing Red Hat OpenShift using the UPI method, thereby enabling users to bypass the reliance on Route53. When opting for a cluster with user-provisioned infrastructure, it is imperative to acknowledge that deploying all essential infrastructure components and machines necessitates manual intervention.
Pre-Requisites –
- AWS account access with the permission to create EC2 instances, ELB, Target groups, Security Groups, DHCP Options set, Endpoints, IAM Policies, and Roles.
- AWS S3 bucket created and publicly accessible
- DNS server access and permissions to create CNAME records
- Base domain registered in the customer DNS server
- Red Hat Openshift pull secret for the UPI install on AWS.
- Red Hat Linux Bastion host running and can access Red Hat OpenShift nodes (ec2 instances) via SSH.
- Openshift-Install, Openshift command (OC) line utility and AWS CLI installed on the bastion host.
Assumption –
The individual responsible for the installation of the Red Hat OpenShift cluster should possess practical expertise in the following key areas:
- Proficiency in creating, configuring, and managing AWS services through the AWS Command Line Interface (CLI) and the web console.
- Profound familiarity with Red Hat Linux commands, facilitating effective navigation and administration of the Red Hat Linux operating system.
- A sound command of Red Hat OpenShift command utilities, including but not limited to Openshift-Install and OC, to facilitate the deployment and management of the OpenShift cluster.
- A working knowledge of DNS servers and the ability to create and manage CNAME records are essential for domain resolution.
Furthermore, the following prerequisites must be in place for a successful installation:
- The Virtual Private Cloud (VPC) and Subnet configuration should already exist, encompassing both public and private subnets. These subnets must have internet access facilitated through AWS Internet Gateway and NAT Gateway or Proxy for seamless communication.
- The OpenShift cluster will be situated within a Private Subnet, with access to the DNS server, while internet connectivity for the cluster will be maintained through Proxy, Internet Gateway (IGW), or NAT Gateway.
- It is essential to note that this is a Mint installation, for which access keys and secret keys will be provided, and it does not involve the Security Token Service (STS), which would typically require the use of ccoctl.
In summary, successfully installing the Red Hat OpenShift cluster necessitates a comprehensive skill set spanning AWS service management, Linux proficiency, OpenShift utilities, DNS management, VPC/Subnet setup, and understanding the deployment environment’s connectivity requirements.
Architecture Diagram —
The architectural representation below outlines the configuration of our Red Hat installation within the AWS environment. In this scenario, our deployment strategy involves siting both master and worker nodes within private subnets. Connectivity to the Internet will be achieved by utilizing NAT Gateway and Internet Gateway. Specifically, we intend to implement a Network Load Balancer (NLB) for the master nodes, while a distinct Application Elastic Load Balancer (ELB) will be provisioned to serve the worker nodes. To ensure seamless accessibility to these load balancers, we will incorporate CNAME entries within the customer’s Private DNS server, thereby establishing a clear and resilient domain name resolution mechanism.
Step-By-Step Instructions –
- Run the following command to clone the required files from the git hub repository.
git clone https://github.com/kapilrajyaguru/Deploying-Red-Hat-OpenShift-on-AWS-Without-Route53-A-UPI-Method-Guide.git2. Store VPCId, SubnetId, and other parameters in variables
vpcid=$(aws ec2 describe-vpcs | jq -r '.Vpcs[] | select(.Tags[].Value="<Tag name of your vpc>")? | .VpcId')
privatesubnets=$(aws ec2 describe-subnets --filter="Name=vpc-id,Values=$vpcid" | jq -r '.Subnets[] | select(.Tags[].Value=="<Tag name of your private subnet>") | .SubnetId' | paste -s -d" ")
cidr=$(aws ec2 describe-vpcs | jq -r '.Vpcs[] | select(.Tags[].Value="<Tag name of your vpc>")? | .CidrBlock')3. Change the Directory where you stored the Git Clone files and run the following AWS CLI command to create an openshift IAM Policy
aws iam create-policy --policy-name openshift_policy --policy-document file://openshift_policy.json4. Create an EC2 role by running the below command
aws iam create-role --role-name openshift --assume-role-policy-document file://ec2_role.json5. Run the following command to get ARNs of the policies created so far
aws iam list-policies | grep openshift_policy6. Run the following commands to attach policies to the openshift role created earlier; update your arn from the above command.
aws iam attach-role-policy --policy-arn arn:aws:iam::<Openshift policy arn>:policy/openshift_policy --role-name openshift7. Run the following command to create an instance profile and add a role to it.
aws iam create-instance-profile --instance-profile-name openshift
aws iam add-role-to-instance-profile --instance-profile-name openshift --role-name openshift8. Run the following command to create a security group.
aws ec2 create-security-group --group-name openshiftSG --description “Security group for OpenShift Cluster” --vpc-id $vpcid
sgid=$(aws ec2 describe-security-groups --filter="Name=group-name,Values=openshiftSG" | jq -r '.SecurityGroups[] | .GroupId' | paste -s)9. Please execute the following command to incorporate the necessary ingress rules into the security group. It is important to note that the security group rule being added in this instance represents a simplified version, permitting access exclusively from the bastion host. Depending on the specific requirements of your application, additional rules may be warranted to enable the opening of ports as needed.
aws ec2 authorize-security-group-ingress --group-id $sgid --ip-permissions IpProtocol=tcp,FromPort=22,ToPort=22,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=19531,ToPort=19531,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=22623,ToPort=22623,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=6443,ToPort=6443,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=2379,ToPort=2380,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=6641,ToPort=6642,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=4789,ToPort=4789,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=6081,ToPort=6081,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=500,ToPort=500,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=4500,ToPort=4500,IpRanges="[{CidrIp=$cidr}]" IpProtocol=50,FromPort=0,ToPort=65553,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=9000,ToPort=9999,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=9000,ToPort=9999,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=30000,ToPort=32767,IpRanges="[{CidrIp=$cidr}]" IpProtocol=udp,FromPort=30000,ToPort=32767,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=10250,ToPort=10250,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=10257,ToPort=10257,IpRanges="[{CidrIp=$cidr}]" IpProtocol=tcp,FromPort=10259,ToPort=10259,IpRanges="[{CidrIp=$cidr}]"10. Create a DHCP option set by running the following command
aws ec2 create-dhcp-options --dhcp-configuration “Key=domain-name-servers, Values=<Your DNS Server IP>” “Key=domain-name, Values=<Your Base domain>”11. Run the following commands to associate your DHCP option set to your VPC.
aws ec2 describe-dhcp-options | grep DhcpOptionsId
aws ec2 associate-dhcp-options --dhcp-options-id <your dhcp-option-set id> --vpc-id $vpcid12. Run the following command to create EC2, S3 and Network Load Balancer Endpoints in your VPC
aws ec2 create-vpc-endpoint --vpc-id $vpcid --vpc-endpoint-type Interface --service-name com.amazonaws.us-west-2.elasticloadbalancing
aws ec2 create-vpc-endpoint --vpc-id $vpcid --vpc-endpoint-type Interface --service-name com.amazonaws.us-west-2.S3
aws ec2 create-vpc-endpoint --vpc-id $vpcid --vpc-endpoint-type Interface --service-name com.amazonaws.us-west-2.ec213. Create a Network load balancer for Bootstrap and master nodes; enter all the private subnet IDs where the master nodes will get deployed.
aws elbv2 create-load-balancer --name openshift-master-elb --type network --scheme internal --subnets <Enter your private subnet ids> --security-groups $sgid --tags Key=kubernetes.io/role/internal-elb,Value=114. For the master to find the bootstrap node, add api.clustername.basedomain and api-int.clustername.basedomain CNAME entry into your DNS server pointing to master Loadbalencer DNS name
- Create target groups for the port 6443 and 22623
aws elbv2 create-target-group --name master-targets-6443 --protocol TCP --port 6443 --vpc-id $vpcid
aws elbv2 create-target-group --name master-targets-22623 --protocol TCP --port 22623 --vpc-id $vpcid- Create a listener for ports 6443 and 22623
aws elbv2 create-listener --load-balancer-arn <loadbalancer-arn of master-elb> --protocol TCP --port 6443 --default-actions Type=forward,TargetGroupArn=<targetgroup-arn of worker-targets-6443>
aws elbv2 create-listener --load-balancer-arn <loadbalancer-arn of master-elb> --protocol TCP --port 22623 --default-actions Type=forward,TargetGroupArn=<targetgroup-arn of worker-targets-22623>15. Run the following command to create ssh key pair, which you will use to log in to nodes with username ‘core’.
ssh-keygen -t ed25519 -N ‘’ -f ~/.ssh/id_openshift
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_openshift16. Run the following command to edit the install-config with your environment variables.
cp install-config-original.yaml install-config.yaml
Vi install-config.yaml- Update your base domain, your vpc cidrblock, RedHat pull secret and SSH key created in the previous step
- If you plan to use Proxy for internet access, update the proxy details in the install-config and uncomment those lines.
- If you are installing the Red Hat Openshift cluster in a region other than US-West-2, please also update the region.
- I have included only two worker nodes for this experiment. If you want to use this cluster for other predefined use cases and need more worker nodes, please appropriately increase the worker node count.
17. Run the following command to create manifests files
openshift-install create manifests --dir=<path to your install-config>
rm -f <installarmtion_directory>/openshift/99_openshift-cluster-api_master-machines-*.yaml
rm -f <installation_directory>/openshift/99_openshift-cluster-api_worker-machineset-*.yaml - When removing the master and worker machine set, it’s important to note that the number of workers specified in the install-config becomes irrelevant. This is due to the fact that workers will be manually created, as opposed to being provisioned through the machine set. The decision to bypass the machine set is primarily rooted in the complexities associated with DNS management. To mitigate potential challenges in this regard, it is advisable to consider the creation of infrastructure nodes (infra nodes) as an alternative solution.
18. Check that the mastersSchedulable parameter in the <installation_directory>/manifests/cluster-scheduler-02-config.yml Kubernetes manifest file is set to false
vi cluster-scheduler-02-config.yml- Locate the mastersSchedulable parameter and ensure that it is set to false.
- Save and exit the file.
19. Remove the private zone and public zone sections from the <installation_directory>/manifests/cluster-dns-02-config.yml DNS configuration file
vi cluster-dns-02-config.yml- The file should look as follows.
20. You will find the metadata.json file created in your Directory. Open the file, copy the value of “infraID,” and save it. You will need this value to insert to the bootstrap, master, and worker node tags.
21. To create the Ignition configuration files, run the following command from the Directory that contains the installation program
./openshift-install create ignition-configs --dir=<installation_directory>22. Copy the Ignition files to AWS S3
aws s3 cp bootstrap.ign s3://<Your S3 bucket>/bootstrap.ign
aws s3 cp master.ign s3://<Your S3 bucket>/master.ign
aws s3 cp worker.ign s3://<Your S3 bucket>/worker.ign23. Print the current x86_64 or aarch64 AMI for an AWS region, such as us-west-2:
ami=$(openshift-install coreos print-stream-json | jq -r ‘.architectures.x86_64.images.aws.regions[“us-west-2”].image’)24. Update bootstrap_userdata.txt, master_userdata.txt and worker_userdata.txt with your S3 location details
25. Launch your bootstrap node using he following command.
aws ec2 run-instances --image-id ami-04e29ab892209d108 --instance-type t2.xlarge --count 1 --subnet-id <Your Subnet ID> --key-name openshift_key --security-group-ids $sgid --iam-instance-profile Name=openshift --user-data file://bootstrap_userdata.txt --tag-specifications ‘ResourceType=instance,Tags=[{Key= kubernetes.io/cluster/<your infraid from metadata.json file>,Value=shared}, {Key= Name,Value=bootstrap}]’ --block-device-mappings file://mapping.json26. Go to target groups for ports 6443 and 22623 and register the bootstrap node as a target
27. Wait for the target group to report the bootstrap node healthy on ports 6443 and 22623
28. Export the kubeadmin credentials:
export KUBECONFIG=<installation_directory>/auth/kubeconfig29. Run the following command to spin up three master nodes
aws ec2 run-instances --image-id ami-04e29ab892209d108 --instance-type t2.xlarge --count 3 --subnet-id <Your Subnet ID> --key-name openshift_key --security-group-ids $sgid --iam-instance-profile Name=openshift --user-data file://master_userdata.txt --tag-specifications ‘ResourceType=instance,Tags=[{Key= kubernetes.io/cluster/<your infraid from metadata.json file>,Value=shared}, {Key= Name,Value=master}]’ --block-device-mappings file://mapping.json30. Go to target groups for ports 6443 and 22623 and register master nodes as target
31. Wait for the target group to report master nodes healthy on port 6443 and 22623
32. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:
oc get csr- To approve all pending CSRs, run the following command:
oc get csr -o go-template=’{{range .items}}{{if not .status}}{{.metadata.name}}{{“\n”}}{{end}}{{end}}’ | xargs --no-run-if-empty oc adm certificate approve33. Change to the Directory that contains the installation program and start the bootstrap process that initializes the OpenShift Container Platform control plane:
./openshift-install wait-for bootstrap-complete --dir <installation_directory> --log-level=info34. Once the bootstrap process is complete then, remove the bootstrap node from the target group and delete the bootstrap ec2 before starting the worker nodes
- Run the following command to spin up two worker nodes
aws ec2 run-instances --image-id ami-04e29ab892209d108 --instance-type t2.large --count 2 --subnet-id <Your Subnet ID> --key-name openshift_key --security-group-ids $sgid --iam-instance-profile Name=openshift --user-data file://worker_userdata.txt --tag-specifications ‘ResourceType=instance,Tags=[{Key= kubernetes.io/cluster/<your infraid from metadata.json file>,Value=shared}, {Key= Name,Value=worker}]’ --block-device-mappings file://mapping.json- After a few minutes, you will see an additional ELB created. Copy the DNS name of this ELB and continue to the next step.
35. Add *.apps.clustername.basedomain CNAME entry into your DNS server pointing to the DNS address you copied in the previous step.
- Until you update the above record, your cluster operators’ “authentication,” “console,” and “ingress” will continue to fail.
36. Also, ensure you update the new load balancer’s security group with the security group you created earlier.
37. Update the tag value of the new load balancer to share for the key “kubernetes.io/cluster/<your infraid from metadata.json file>”
38. Run the following command to see the status of node and cluster operators
oc get nodes
oc get co- Watch the cluster components come online
watch -n5 oc get clusteroperators39. Run the following command delete control-plane-machine-set operator
oc delete controlplanemachineset.machine.openshift.io cluster -n openshift-machine-api40. From the Directory that contains the installation program, complete the cluster installation:
./openshift-install --dir <installation_directory> wait-for install-complete41. Obtain the password for the kubeadmin user from the kubeadmin-password file on the installation host:
cat <installation_directory>/auth/kubeadmin-password42. List the OpenShift Container Platform web console route:
oc get routes -n openshift-console | grep ‘console-openshift’I trust that this blog proves to be a valuable resource in assisting you with the deployment of a Red Hat OpenShift cluster within the AWS environment while seamlessly integrating your private DNS setup. Your feedback and engagement are greatly appreciated, and I encourage you to express your thoughts by liking, sharing, and providing comments on this blog. Your insights and contributions are a source of inspiration, and I eagerly anticipate hearing from you.
Additional Resources
