k8scenario.github.io
k8scenario/k8scenario is an open source tool for running Kubernetes Application Troubleshooting Workshops.
k8scenario/k8scenario contains
- the sources for a Go binary k8scenario
- the sources for scripts which implement the scenarii to be troubleshooted.
The k8scenario/k8scenario tool is a command-line application
- with a minimal textual menu to allow users to see what scenarii are available to run
- downloads a selected scenario
- deletes/recreates the k8scenario namespace
- installs the selected scenario into the k8scenario namespace
- provides some minimal hint of the task to perform/problem to fix
- loops until the “problem” is fixed
The k8scenario/k8scenario tool automatically downloads available scenarii as zip files from the https://k8scenario.github.io site.
The Go binary is also available from the https://k8scenario.github.io site.
1. Taking part in a Troubleshooting Lab
To run the lab you will be provided with a Virtual Machine in the Cloud.
Your instructor will have provided login credentials.
Note: AFTER THE WORKSHOP you are free to run this workshop on your own laptop with a Kubernetes cluster installed - please don’t do this during the workshop as the download of images will kill the conference WiFi … for everyone!!
Once logged in to your VM you will need to
- Create a ~/bin directory and place it in your PATH
- Download some tools (kubectl, kind, k8scenario) to ~/bin
- Launch k8scenario
Create a local directory ~/bin and place it in your PATH as follows:
mkdir ~/bin
echo 'export PATH=~/bin:$PATH' >> ~/.bashrc
. ~/.bashrc
1.1 Installing Docker
sudo bash -i
apt-get update -y
apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
docker run hello-world
usermod -aG docker ubuntu
docker version
Now logout and login as user ubuntu and verify
docker version
1.1.1 Verifying Docker
Verify Docker
Verify that Docker runs correctly on your system: ```bash docker version ``` should show output similar to: ``` Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:29:52 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:28:22 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683 ``` Reference: https://docs.docker.com/install/linux/docker-ce/ubuntu/1.2 Downloading kubectl
Install kubectl
The tool can be downloaded using the command: ```bash wget -O ~/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl ``` **NOTE**: or in case of copy/paste problems do this as 2 separate commands: ```bash RELEASE=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt) wget -O ~/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/$RELEASE/bin/linux/amd64/kubectl ``` Then check that kubectl is available: ```bash chmod +x ~/bin/kubectl kubectl version ``` Reference: https://kubernetes.io/docs/tasks/tools/install-kubectl/1.3 Downloading KIND
Install KIND
The tool can be downloaded from: https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64 Download the kind executable and place it in your ~/bin directory ```bash wget -O ~/bin/kind https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64 chmod +x ~/bin/kind kind version ```1.5 Downloading k8scenario
show
The tool can be downloaded from: https://mjbright.github.io/static/bin/k8scenario Note: The open source version of this repository is available at the URL below but has less scenarii available: https://k8scenario.github.io/static/bin/k8scenarioNote: The tool operates on the k8scenario namespace which it creates/deletes for each scenario. The tool - as written - requires full cluster permissions, so will likely not run on your production cluster - and it shouldn't !
```bash wget -O ~/bin/k8scenario https://mjbright.github.io/static/bin/k8scenario chmod +x ~/bin/k8scenario k8scenario --version ```
2. Creating a KIND Kubernetes cluster
Create a file called ‘kind_2.yaml’ with the following content:
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
disableDefaultCNI: true
podSubnet: 192.168.0.0/16
nodes:
- role: control-plane
- role: worker
Note: We are disabling the default CNI, kindnet. We will install Calico in a later step.
2.1 Deleting any existing cluster:
If you already have a cluster which you wish to replace, first delete that cluster.
You can list current clusters with the
kind get clusters
``` command.
Supposing that the cluster is named '*kind*', delete it as follows:
```bash
kind delete cluster --name kind
2.2 Create the 2-node cluster:
Create the 2-node cluster as shown, and check that the cluster is created.
kind create cluster --config kind_2.yaml
kind get clusters
Calico
Note: In case of a problem to create the cluster, delete the cluster and try again.
Verify that you have a 2-node cluster:
kubectl get nodes
You should see output similar to
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.17.0) 🖼�
✓ Preparing nodes 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹��️️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a nice day! 👋
Verify that you can access the cluster
kubectl get nodes
You should see output similar to:
NAME STATUS ROLES AGE VERSION
kind-control-plane NotReady master 3m18s v1.17.0
kind-worker NotReady <none> 2m41s v1.17.0
Note that the nodes are ‘NotReady’ because we have not yet installed the Pod network - for this we need to install the Calico CNI networking extension.
2.3 Installing the Calico CNI
Download the Calico manifest
We will install the latest 3.11 release
See: https://docs.projectcalico.org/v3.11/getting-started/kubernetes/installation/calico
Perform the following steps to install Calico
wget -O calico_v3.11.yaml \
https://docs.projectcalico.org/v3.11/manifests/calico.yaml
kubectl apply -f calico_v3.11.yaml
kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=true
You should now see several calico Pods running in the kube-system namespace.
> kubectl get pods -n kube-system | grep calico
calico-kube-controllers-5b644bc49c-8h6pw 1/1 Running 0 125m
calico-node-f6j49 1/1 Running 0 114m
calico-node-zjxl9 1/1 Running 0 114m
Once those Pods are all in the Running state, you should see your nodes in the Ready state as shown below.
Note: It is possible that we need to restart the Pods if the Nodes do not go to the Ready state - we describe this below.
After a few seconds, but wait up to a minute, you should see output similar to:
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready master 3m18s v1.17.0
kind-worker Ready <none> 2m41s v1.17.0
You may now move on to section 3 to start the tool
If your Nodes don’t go to the Ready state (or calico Pods are not Running)
In this case we need to restart the ‘calico-node-’ Calico Pods.
You can do this in one command as follows:
kubectl -n kube-system delete $(kubectl get deploy,pods --no-headers -A
-l k8s-app=calico-node | awk '{ print $2; }'
Check now that your Nodes are all Ready.
2.4 Allowing Pods to be scheduled to the Master Node
By default Kubernetes installers configure Master nodes to disallow user (non-system) Pods.
Verify that this is the case for the Master node (kind-control-plane) but not for the Worker node (kind-worker):
> kubectl describe node kind-control-plane | grep -i taint
Taints: node-role.kubernetes.io/master:NoSchedule
> kubectl describe node kind-worker | grep -i taint
Taints: <none>
We will remove the taint on the Master to allow us to schedule Pods across all nodes Note: This should not be done on production clusters
> kubectl taint node kind-control-plane node-role.kubernetes.io/master-
node/kind-control-plane untainted
> kubectl describe node kind-control-plane | grep -i taint
Taints: <none>
3. Debugging resources
show
An excellent resource to introduce you to the process of debugging applications running on Kubernetes is the "Visual guide on troubleshooting Kubernetes deployments" available on the @learnk8s blog at https://learnk8s.io/troubleshooting-deployments .
Follow @learnk8s on twitter at https://twitter.com/learnk8s for some excellent Kubernetes resources.
4. Using k8scenario
k8scenario is a command line tool.
When the tool is launched, it downloads a list of available scenarii and presents the user with a very basic menu of available scenarii:
Launching the tool
./bin/k8scenario
Version: 2019-Dec-28_20h33m29
serverUrl: https://k8scenario.github.io/static/k8scenarii
Downloading index.list
Available scenarii: 0 1 2 20 21 3 40
Selecting a scenario:
To run scenario0, enter 0 and press the <ENTER> key and you will see that the tool
- first deletes the k8scenario namespace if it already exists
- creates the k8scenario namespace in which the task will be performed
- installs the ‘problem scenario’ into the namespace
- outputs a hint message which may be rather obscure …
- loops until the problem is fixed
- Note: There may be some transient Pods which are used to detect if the problem is fixed or not
select scenario>>> 0
Installing scenario ... into namespace k8scenario
namespace/k8scenario created
namespace/k8scenario created
k8scenario Active 0s
k8scenario Active 0s
# hint1:
Just launch a Pod (and/or a Deployment/ReplicaSet) with a name starting with 'basictest', e.g. basictest-xxxx in the namespace called k8scenario (created for you)
# subjects:
- pods, namespace
- exit status: 1
[scenario0]/1 - fix incomplete - Sleeping <10> secs ...
Solve the problem
Did you see the hint?
# hint1:
Just launch a Pod (and/or a Deployment/ReplicaSet) with a name starting with 'basictest', e.g. basictest-xxxx in the namespace called k8scenario (created for you)
# subjects:
- pods, namespace
In the above example, it is sufficient to create a Pod with an appropriate name, and the simplest way to do this might be to create a Deployment as follows:
kubectl -n k8scenario create deploy basictest --image=mjbright/ckad-demo:1
Note:
- It is necessary to specify the namespace with the ‘-n’ option, see change context/namespace below for an alternative
- For this particular task, there is no constraint on the image, so you can choose any available/accessible image
Change context/namespace
show
Note: WARNING - the below assumes you are on a vanilla cluster, not a production cluster !Note that to eliminate the need to set the namespace on each command, it is possible to modify the namespace of the current context
kubectl config set-context $(kubectl config current-context) --namespace k8scenarioAlternatively, you can install the kubectx/kubens tool to change the namespace of the current context as ```bash kubens k8scenario ``` You can install the tool from https://github.com/ahmetb/kubectx **Note:** You can simply git clone that repository and copy the kubens script into your PATH at ~/bin/
You can check the result as:
kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kind-kind_2 kind-kind_2 kind-kind_2
After use you should reset using
kubectl config set-context $(kubectl config current-context) --namespace ""
Wait for the tool to detect that the problem is fixed
The tool loops, checking periodically to see if the task is complete/problem is fixed.
When it detects completion it will report the fact and drop back to the selection menu as show below:
- exit status: 1
[scenario0] - cluster broken - Sleeping <10> secs ... - exit status: 1
[scenario0] - cluster broken - Sleeping <10> secs ... - exit status: 1
[scenario0] - cluster broken - Sleeping <10> secs ... - exit status: 1
[scenario0] - cluster broken - Sleeping <10> secs ...
---- [scenario0] WELL DONE !!!! - The scenario appears to be fixed !!!!
Available scenarii: 0 1 2 20 21 3 40
select scenario>>>
Instructions for creating scenarii for the k8scenario tool
Refer to Creating Scenarii to learn how to create and test your own scenarii.