This blog post is about common interview questions that you will probably get when interviewing for a devops position.
I have created a list of questions in multiple categories like linux, cloud, containerization, version control, ci cd etc which might be helpful in your devops interview preparation. I will also keep updating the list as required.
Devops is a challenging field and there can be many other topics and questions which you might encounter. Interview questions will vary depending on the company to which you’re applying.
And remember, not all interviews are the same for everyone even for the same position in the same company.
Technology will always change, and knowledge becomes obsolete, but you need to show that you learn fast and keep current.
I wish you success !!!
]]>Kubeadm is a tool to bootstrap Kubernetes clusters. It performs necessary actions to get a minimum viable cluster up and running.
In this post, I have explained how to create a highly available Kubernetes cluster on AWS using Kubeadm.
We will create the following AWS resources using terraform.
Refer to the above diagram for a better understanding of AWS architecture.
I have used us-east-1 region with 3 availability zones (us-east-1a, us-east-1b, us-east-1c) to setup the cluster.
To prevent direct attacks on the cluster and the application, both master and worker nodes will be created in private subnets so that the instances can never be reached from the internet.
Master and worker nodes can reach the internet using NAT Gateway.
Each node should have at least 2 CPU/vCPU cores, 2GB RAM, and 10GB disk space, so I have used t2-medium instances for both master and worker nodes.
All the AWS resources required by kubernetes cluster have a tag with key "kubernetes.io/cluster/<cluster-name>"
. Kubernetes itself will also use this tag on things that it creates, and it will use a value of “owned”. This value does not need to be used on resources that Kubernetes itself did not create.
Master Node Security Group Inbound Rules:
Port range | Source | Description |
---|---|---|
22 | Bastion Host SG | allows bastion host to ssh into master nodes. |
6443 | 0.0.0.0/0 | allows NLB to reach the api server on port 6443. As we are using NLB with target type as instance-id so the client IP of incoming traffic is preserved due to this the target instances will see the actual Client IP address as the source address, hence the source value is 0.0.0.0/0. |
2379-2380 | Master node SG | etcd server is accessible on these ports. |
10259 | Master node SG | kube-scheduler is accessible on port 10259 port. |
10257 | Master node SG | kube-controller-manager is accessible on port 10257 port. |
10250 | Master node SG | kubelet is accessible on port 10250 port. |
Master Node Security Group Outbound Rules:
Port range | Destination | Description |
---|---|---|
80 | 0.0.0.0/0 | allows master nodes to reach internet on port 80 (HTTP). |
443 | 0.0.0.0/0 | allows master nodes to reach internet on port 443 (HTTPS). |
6443 | 0.0.0.0/0 | allows master nodes to reach control plane via NLB. |
10250 | VPC CIDR | allows kube api server to reach kubelet. |
All | VPC CIDR | allows communication between different control plane components. Ideally, the destination should be the security group id of the master node but since terraform doesn’t support destination as another security group so I added CIDR range of VPC. |
Worker Node Security Group Inbound Rules:
Port range | Source | Description |
---|---|---|
22 | Bastion Host SG | allows bastion host to ssh into master nodes. |
10250 | Master node SG | allows api server to access kubelet on port 10250. |
All | Worker node SG | allows communication between worker nodes on all protocols. |
Worker Node Security Group Outbound Rules:
Port range | Destination | Description |
---|---|---|
80 | 0.0.0.0/0 | allows worker nodes to reach internet on port 80 (HTTP). |
443 | 0.0.0.0/0 | allows worker nodes to reach internet on port 443 (HTTPS). |
6443 | 0.0.0.0/0 | allows worker nodes to reach control plane via NLB. |
All | VPC CIDR | allows communication between worker nodes on all protocol. |
To create a cluster using kubedm, we need to install kubectl, kubeadm, kubelet, docker, disable swap, configure iptables, etc on each EC2 instance. Click here to learn more.
Since we will be deploying multiple nodes in the cluster, installing these tools manually on each one would not be feasible. Packer will be used to resolve this problem.
Packer is a tool used to create AMIs. It takes a base image (in our case Ubuntu-bionic-18.04), installs all the dependencies, and creates a new image which contains the packages and configurations required to run the Kubernetes cluster.
By using a pre-baked image, you can drastically reduce the time it takes to scale up the nodes.
To keep things simple we will use the same AMI for both master and worker nodes.
The packer tool takes a json template file with a series of declarations and commands for it to follow. The template tells Packer what plugins(builders, provisioners) to use, how to configure the plugins, and in what order to run them.
Open a command prompt or terminal, clone the git repo and navigate to the packer directory. It has a json template(“k8sami.json”) and a shell script(commands to install required packages and configure AMI). Modify variables and other parameters in json template if required.
I have used a shell provisioner in json template which will execute the shell script to configure the instance into the desired state.
Run the below command on the local machine. It will take around 30-40 minutes to create the AMI.
1 | AWS_PROFILE=<profilename> packer build k8sami.json |
Follow the below steps to execute the terraform scripts:
Step 1 - Create a S3 bucket to store the terraform state. Assign the bucket name to backend config in “config.tf” file.
Step 2 - Create an EC2 key pair either through AWS console or CLI and assign it to cluster.k8s_ssh_key_name variable in terraform.tfvars file. This key pair would be required to ssh into the bastion, master, and worker nodes.
Step 3 - Set the value of ami variable to the newly created ami name, aws_account_no to the AWS account number in terraform.tfvars file, modify other variables(For eg:- cluster_name, region, vpc_cidr, etc) as required.
Step 4 - Navigate to terraform directory, and run the below command to initialize terraform working directory.
1 | AWS_PROFILE=<profilename> terraform init |
Step 5 - Run the below command to preview the changes that terraform will make to infrastructure.
1 | AWS_PROFILE=<profilename> terraform plan |
Step 6 - To create the infrastructure for the Kubernetes cluster run terraform scripts using the below command
1 | AWS_PROFILE=<profilename> terraform apply |
Step 7 (optional) - To delete the infrastructure run the below command
1 | AWS_PROFILE=<profilename> terraform destroy |
After the Terraform scripts are executed successfully, we can start with the cluster set up.
We will use below kubeadm configuration to initialize the control plane.
In order for the kubernetes cluster to function properly, the API server, controller manager, and every kubelet on master/worker node must have the flag “cloud-provider: aws”. It has been passed as extraArgs in the configuration.
1 | apiVersion: kubeadm.k8s.io/v1beta2 |
SSH into one of the master node via bastion host, create a file with above configuration, set the value of clusterName(use same cluster name which was used in terraform scripts), assign <NLB url:6443>
to controlPlaneEndpoint, VPC CIDR to podSubnet and save the file with name “cluster-config.yaml”.
Run the below command to initialize the Kubernetes cluster.
1 | sudo kubeadm init --config cluster-config.yaml --upload-certs |
During kubeadm init, kubeadm uploads the ClusterConfiguration object to your cluster in a ConfigMap called kubeadm-config in the kube-system namespace. This configuration is then read during kubeadm join, kubeadm reset and kubeadm upgrade.
If the kubeadm init command is executed successfully, it will print the below message about how to add additional master(control plane) nodes and worker(data plane) nodes.
To make kubectl command work for your non-root user, run these commands, which are also part of the kubeadm init output:
1 | mkdir -p $HOME/.kube |
If the kubeadm init command fails, you can debug the errors using the below command on master nodes.
1 | journalctl -a -xeu kubelet -f |
To delete all the configurations that kubeadm has done so that you can start your configuration from scratch again, run the below command.
1 | kubeadm reset |
To join other master nodes we will use following configuration:
1 | apiVersion: kubeadm.k8s.io/v1beta2 |
SSH into other master node and create a file “master-join-config.yaml” with above configuration.
We will use the same values for token, caCertHashes & certificateKey from “kubeadm join” command which was printed as output of “kubeadm init” command.
Set the value of controlPlane.localAPIEndpoint.advertiseAddress to ip address of the master node, apiServerEndpoint to <NLB url:6443>
, and nodeRegistration.name to hostname of the instance(EC2 private dns entry of the instance).
Run the below command to join the master node to the cluster:
1 | sudo kubeadm join --config master-join-config.yaml |
When the above command is executed successfully, it will print the following message.
If you run kubectl get nodes command on master node you would find that the node status is “not ready”.
To fix this we need to install CNI (Container Network Interface) in the cluster.
Kubernetes cluster uses some overlay network inside for the pods and other services so we need to deploy a pod network so that Kubernetes resources can communicate with each other internally.
I have used weave-net pod network for the cluster. It can be installed using a single yaml configuration file which can be downloaded from here or you can use the same yaml file “weave-daemonset-k8s-1.11.yaml” which I have used. It is in the CNI directory of the cloned repo on your local machine, copy that file to master node and run the below command.
1 | kubectl apply -f weave-daemonset-k8s-1.11.yaml |
Run kubectl get nodes command again and this time you would see the status of nodes as ready.
SSH into worker node and create a file “worker-join-config.yaml” with below configuration.
1 | apiVersion: kubeadm.k8s.io/v1beta2 |
Set the values of token, apiServerEndpoint, caCertHashes, nodeRegistration.name in the same way as we did for master node to join the cluster in the previous section.
Run the below command to add worker nodes in the cluster.
1 | sudo kubeadm join --config worker-join-config.yaml |
The following message will be printed after the command is successfully run.
So far we have run all the commands from master and worker nodes, but this is good only for the first time when we configure the cluster. We should be able to access the cluster from the laptop also.
To access the cluster from your local machine, follow these steps:
After that, run the below command to test if you can access the cluster from your local machine, and it should print the number of nodes in the cluster.
1 | kubectl get nodes |
Alternatively, you can also specify other kubeconfig files by setting “–kubeconfig” flag with kubectl commands.
1 | kubectl get nodes --kubeconfig admin.conf |
If you scale up the worker nodes, you have to run the “kubeadm join …” command manually every time which isn’t very convenient.
I have automated this process using a script (“worker-userdata.sh.tpl”) which is passed as user-data to worker node’s launch configuration during terraform script execution.
Whenever a new worker node is launched, the script gets executed and the node automatically joins the cluster. Please note that it only works when the control plane is already bootstrapped.
This script uses AWS CLI and AWS System Manager run command to execute commands remotely on a control plane node.
Steps performed by the script:
1 | kubeadm token generate |
1 | kubeadm token create <token> --print-join-command |
1 | kubectl -n kube-system get cm kubeadm-config --kubeconfig /home/ubuntu/.kube/config -o yaml | grep controlPlaneEndpoint |
1 | openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout | |
1 | curl -s http://169.254.169.254/latest/meta-data/local-hostname |
1 | cat <<EOF > /home/ubuntu/worker-join-config.yaml |
1 | kubeadm join --config /home/ubuntu/worker-join-config.yaml |
1 | kubeadm token delete <token> |
To test the auto-join functionality, manually scale up the worker nodes using AWS Autoscaling Group.
It might take a few minutes for the worker node to join the cluster, run the below command from your local machine to check if the new worker node has already joined the cluster.
1 | kubectl get nodes |
In case if worker node couldn’t join the cluster, login into the worker node and run the below command to check the logs for any errors that might have occurred during user-data script execution.
1 | cat /var/log/cloud-init-output.log |
To test if everything is working fine, we can deploy a sample application.
Use the below command to deploy the nginx web server from your local machine.
1 | helm install mywebserver bitnami/nginx |
To verify if pod object was successfully deployed, we can run the following command:
1 | kubectl get pods -l app.kubernetes.io/name=nginx |
To get the complete URL of this service, run:
1 | kubectl get service mywebserver-nginx -o wide |
Copy the value for EXTERNAL-IP, open a new tab in your web browser, and paste it in, you should see a welcome message, similar to:
As a part of the high availability test, we will simulate master node failure by scaling it down to one instance using AWS Autoscaling Group.
After that, we can run the below command from the local machine to test if the Kubernetes cluster is still accessible.
1 | kubectl get po |
If the command is executed successfully, it means that the cluster is still accessible and you can create more pods, deployment, services etc.
If you loose the join commands which were printed after initializing the control plane using “kubeadm init” command then you can regenerate those commands.
SSH into already working master node and run the below commands:
To print the join command:
1 | kubeadm token create --print-join-command |
To generate a new certificate key, re upload certs in the master node:
1 | sudo kubeadm init phase upload-certs --upload-certs --config cluster-config.yaml |
You can then join the master and worker nodes using the same steps described in the previous sections.
Please let me know if you have any questions or comments.
]]>This post explains how to use AWS Cloudformation stack to automate infrastructure deployment to host a static website. The cloudformation stack also creates a CI/CD pipeline for automated deployment of code and it also setups a Elasticsearch service for S3 access logs analysis and visualization .
Static website using Cloudformation Git repo
Step 1 - As the lamdba function in the main template uses a S3 bucket to store the code, Create a S3 bucket first using this template either from AWS console or by below AWS CLI command.
1 | aws cloudformation create-stack --stack-name {stackname} --template-body file://{path_to_template_file} |
Step 2 - Upload all the lambda zip code files to the bucket created in above step using following AWS CLI command.
1 | aws s3 cp {folder path}/ s3://{bucketname}/lambdas/ --recursive |
Step 3 - After that main stack can be created using this template. It might take around 40 to 50 minutes to create all the resources.
1 | aws cloudformation create-stack --stack-name {stackname} --template-body file://{path_to_template_file} --capabilities CAPABILITY_IAM --parameters ParameterKey=DomainName, ParameterValue={basedomain} ParameterKey=PreExistingHostedZoneDomain, ParameterValue={hosted zone} ParameterKey=PreExistingHostedZoneId, ParameterValue={hosted zone id} ParameterKey=ProjectName, ParameterValue={project name} |
Use following commands to track the stack events and output.
1 | aws cloudformation describe-stacks --stack-name {stackname} |
Step 4 - Once the stack is created, the output of stack will provide the site bucket name, Kibana url, CodeCommit url, Cognito user pool id.
1 | aws cognito-idp admin-create-user --user-pool-id {userpoolid} --username {username} --user-attributes Name=email_verified,Value=true,Name=email,Value={emailid} --region {region} --temporary-password {temp password} |
As soon as the code is pushed to AWS Codecommit, AWS Codepipeline is triggered by the push , then it uses AWS CodeBuild to build the code & finally AWS CodeBuild copies the files to S3 site bucket.
The template fully automates the provisioning of certificate including validation. It uses DNS validation method to validate the certificate. In order to use ACM Certificate with Cloudfront distribution, the certificate will be created in us-east-1 region only.
The template has been tested in us-east-1(North Virginia) and ap-south-1(Mumbai) regions. It should work in other regions as well.
Please leave your comments below or if you have any questions.
]]>