INDEX
########################################################### 2024-01-03 13:30 ########################################################### Just as a separate note, I'm looking at a python video that mentions the "rich" module from rich import print, pretty, inspect pretty.install() inspect(OBJECT, methods=True) # Shows all the methods of an object (with docs)
Devops Journey... Kubernetes Tutorial for Beginners https://www.youtube.com/watch?v=1Lu1F94exhU Kubernetes master nodes control worker nodes and allocate tasks - with a "manfiest" file Pods are deplpoyable objects that contain 1 or more containers Master server: API server, scheduler, controllers and shared config storage (etcd) Worker nodes: Runtime (container software), kubelet (pod manager) and kube-proxy (networking) Pod = containers, shared volumes and running spec - controlled by manifest files # Pod-manifest.yaml apiVersion: v1 # Specifies kubernetes API version kind: Pod metadata: # Uniquely identify object (namespaces, labels etc.) name: nginx spec: # Specify desired state containers: - name: nginx image: nginx:1.23.1 ports: - containerPort: 80 Pods are not deployed directly - use deployments (allows replicas, rollouts etc.) ReplicaSet defines the template (the pod to use) and replicas (number of pods to run) Services expose pods as network services, balancing connections between them Select a label criteria and it will group all pods with that label A typical setup is "AWS ELB" (load balance) -> Service -> Pods # Services-manifest.yaml apiVersion: v1 kind: Service metadata: name: mywebapp labels: app: mywebapp spec: ports: # Specify protocol/ports to send traffic on - port: 80 protocol: TCP name: flask selector: # Which pods to forward traffic to app: mywebapp # Label "app=mywebapp" tier: frontend # Label "tier=frontend" type: LoadBalancer # Allows external clients to access A volume is persistent storage for a pod ConfigMaps = plaintext key-values. Secrets = encrypted key-value Minikube is a basic kubernetes program to run locally # Download and install minikube-linux minikube config set driver docker # Start minikube with docker minikube start minikube status # Check if running properly minikube dashboard --url # Starts a dashboard website # Install kubectrl - for managing kubernets - aliased to "k" k version # Check if installed properly k config view # Shows ~/.kube/config k cluster-info # Gives info about kube cluster, relevant IPs/URLs etc. k get nodes # Lists running nodes k descripe node minikube # Gives all spec info on "minikube" node k cordon minikube # Turns off scheduling for new containers k get nodes # Now shows "minikube" as "Ready,SchedulingDisabled" k uncordon minikube # Turns scheduling back on Namespaces break clusters into virtual clusters/envs - can set resource quotas k get namespace # Lists all namespaces k create namespace prod # Similarly "delete namespace" # namespace-prod.yaml apiVersion: v1 kind: Namespace metadata: name: prod labels: name: prod k create -f namespace-prod.yaml # Creates the namespace k describe namespace prod # Gives info on that namespace Can deploy a basic app with kubectl k get pods # Lists all pods k get pods -n prod # Checks "prod" namespace for pods - also use "--all-namespaces" k config set-context --current --namespace=dev # Sets default namespace to use # This is a copy-paste command to deploy basic hello world app within "dev" namespace k create deployment hello-node --images=k8s.gcr.io/echoserver:1.4 -n dev k get events -n dev # Shows the log of kubernetes events (creating, scaling etc.) # Need to create a service to access the pod k expose deployment hello-node --type=LoadBalancer --port=8080 -n dev k get services -n dev # Can see this node is running # Now need to forward connections to load balancer (e.g. use Elastic Load Balancer) minikube service hello-node -n dev # Exposes port to localhost Now you can make a deployment script (usually copied from elsewhere) # deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mydeployment labels: # These manage how objects are organised - use labels app: mywebapp tier: frontend spec: replicas: 1 # How many replicas to run selector: matchLabels: # Apply to objects with these labels app: mywebapp tier: frontend template: metadata: labels: app: mywebapp tier: frontend spec: containers: - name: mycontainer # defines a single container image: devopsjourney1/mywebapp ports: - containerPort: 80 k apply -f deployment.yaml # Runs the deployment from that file k get deployments.apps # List apps running in deployments k get replicaset # Lists groups of replicas (1 per app in this case) k get pods # Lists each individual pod k describe pod POD_ID # Get in depth info about that pod k describe deployment mydeployment # Similarly, gives info on that deployment k delete -f deployment.yaml # Shuts down this deployment Can extend a yaml file (treat 1 file as multiple) by separating with a "---" This makes a load balancer service for this deployment # (extending) deployment.yaml apiVersion: v1 kind: Service metadata: name: mywebapp labels: app: mywebapp spec: ports: - ports: 80 protocol: TCP name: flask selector: app: mywebapp tier: frontend type: LoadBalancer --- # ... DEPLOYMENT part goes here k apply -f deployment.yaml k get services # Shows webapp is now running minikube service mywebapp # Exposes the webapp to localhost Now you want to consider scaling up pods to run multiple # (extending) deployment.yaml # ... SERVICE part goes here --- apiVersion: v1 kind: ConfigMap # Defines the environment variables to pass in metadata: name: myconfigmapv1.0 data: BG_COLOR: '#12181b' FONT_COLOR: '#FFFFFF' CUSTOM_HEADER: 'Customized with a configmap!' --- # ... DEPLOYMENT part goes here (slightly changed as below) spec: replicas: 5 ... template: ... spec: containers: - name: mycontainer ... envFrom: - configMapRef: name: myconfigmapv1.0 # Don't change configmap as running pods don't know to update when this is changed # Instead, write a new configmap and point (in the spec) to the new version, to update it k apply -f deployment.yaml Now add in resource constraints to limit usage on the server # (extending) deployment.yaml # ... All the same except for the deployment spec spec: ... template: ... spec: containers: - name: ... resources: requests: # Minimum allocated usage memory: "16Mi" cpu: "50m" # 50 milli CPUs = 1/2 CPU limits: # Maximum usage memory: "128Mi" cpu: "100m" # 100 milli CPUs = 1 CPU k logs -l app=mywebapp # Gets logs from a specific app (-f to follow) k rollout restart deployment mydeployment # Restarts all pods k drain minikube --ignore-daemonsets=true --force # Turns off all pods from a deployment ... --delete-emptydir-data # Also wipes the data k uncordon minikube # Reruns the pods
Devops Journey... Kustomize https://www.youtube.com/watch?v=spCdNeNCuFU Kustomize makes manifest files simpler - using patching and overlays in plain YAML # Folder structure base: # This is configuring the defaults for all environments - deployment.yaml # This is the MAIN manifest file - kustomization.yaml - services.yaml # Seperate file for the kubernetes services (not relevant) overlays: # these folders "patch" the base config for different environments dev: - config.properties # For environment variables - kustomization.yaml # Tells kustomization where to inherit config from - replicas.yaml # The manifest data that overrides "base" prod: - config.properties - kustomization.yaml - replicas.yaml test: - config.properties - kustomization.yaml - replicas.yaml # base/kustomization.yaml resources: # defines which files to apply this on (e.g. overwrite labels) - deployment.yaml - services.yaml commonLabels: # Keep labels consistent app: mykustomapp commonAnnotations: app: mykustom-annotations namePrefix: kustom- nameSuffix: # So now app name = "kustom-mywebapp-v1" -v1 Kubectl has a built-in support for kustomize kubectl kustomize # Generates a new manifest file from all of this config (as string output) kubectl apply -k . # Applies kubernetes changes WITH kustomise changes Now add support for different environments (repeat for all environments) # overlays/dev/kustomization.yaml bases: - ../../base # Inherits the base folder namespace: dev # Make sure this namespace exists before running kubectl patches: # Use the data in these files to override the main manifest - replicas.yaml configMapGenerator: - name: mykustom-map env: config.properties # overlays/dev/replicas.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mywebapp spec: replicas: 2 # overlays/dev/config.properties BG_COLOR=#12181b FONT_COLOR=#FFFFFF CUSTOM_HEADER=Welcome to DEV environment kubectl apply -k overlays/dev # Creates dev deployments - can similarly do for prod/test
########################################################### 2024-01-04 17:30 ########################################################### Devops Toolkit... OpenFunction: Run Serverless functions on Kubernetes https://www.youtube.com/watch?v=UGysOX84v2c Can use many different services and write your own "glue" to connect them Openfunction lets you run serverless functions inside containers # function.yaml apiVersion: core.openfunction.io/v1beta2 kind: Function metadata: name: openfunction-demo spec: version: "v2.0.0" image: "c8n.io/vfarcic/openfunction-demo:v0.0.1" imageCredentials: name: push-secret build: # Decides how to build the serverless function builder: openfunction/builder.go:v2.4.0.1-17 # Tool to build go code env: FUNC_NAME: SillyDemo FUNC_CLEAR_SOURCE: "true" srcRepo: # Point to the code to be built url: "https://github.com/devopsparadox/openfunction-demo.git" revision: "main" serving: template: containers: - name: function imagePullPolicy: IfNotPresent env: # Misc env variables ... triggers: http: port: 8080 This allows you to build an image, the function will run and it will autoscale kubectl --namespace a-team apply --filename function.yaml # Run in kubernetes kubectl --namespace a-team get functions -w # WATCH the list of functions made # Get addresses of functions - gives an internal and external URL kubectl --namespace a-team get function openfunction-demo \ --output jsonpath="{.status.addresses}" | jq . # Open a pod with curl to check what's running kubectl --namespace a-team run curl --image=radia/busyboxplug:curl -i --tty curl INTERNAL_URL # This works, and gives the expected output curl EXTERNAL_URL # This fails ("NO") as gateway is not setup correctly kubectl --namespace a-team get routes # Gets the actual external url that works curl -X POST "EXTERNAL_URL/video?..." # Some curl command to the api to make a video curl "EXTERNAL_URL/videos" | jq . # Command to lits all the videos kubectl --namespace a-team get pods # Scales down to 0 when unused (after ~2mins) # Comes back as soon as you curl it again - more request = scaled higher How can you do this without writing functions? Can it run normal applications? # app-no-build.yaml apiVersion: core.openfunction.io/v1beta2 kind: Function metadata: name: silly-demo spec: # No "build" spec so no need to build it - just run version: "v1.0.0" image: "c8n.io/vfarcic/silly-demo:1.4.58" serving: template: containers: - name: function imagePullPolicy: IfNotPresent triggers: http: port: 8080 kubectl --namespace a-team apply --filename app.yaml kubectl --namespace a-team get functions -w
Devops Toolkit... What are Kubernetes Resources https://www.youtube.com/watch?v=aM2Y9m2Kazk Kubernetes is just an extensible API - extend with CRDs (Custom resource definitions) CRDs are just event publishers - need to have controllers to react to events Therefore kubernetes can be used to manage anything Operators are patterns on how to create controllers - simplified they are the same Kubernetes API requires yaml and json - Helm just creates these yaml requests Helm is a client side kubernetes tool - CRs are internal tools, within the cluster
########################################################### 2024-01-05 10:00 ########################################################### Devops Toolkit... Kaniko - Container images without Docker https://www.youtube.com/watch?v=EgwVQN6GNJg Docker is not good to use in Kubernetes as it cannot itself run inside containers Kaniko lets you build images from a Dockerfile inside a container How would you do this normally? Build an image manually like this export GH_ORG=devopsparadox # Where the repo is hosted git clone https://github.com/$GH_ORG/kaniko-demo.git; cd kaniko-demo docker image build --tag devops-toolkit . # Dockerfile FROM klakegg/hugo:0.78-2-alpine AS build # Build code RUN apk add -U git COPY . /src RUN make init RUN make build FROM nginx:1.19-4.alpine # Host code RUN mv /usr/share/nginx/html/index.html /usr/share/nginx/html/old-index.html COPY --from=build /src/public /usr/share/nginx/html EXPOSE 80 What about inside a cluster container? # docker.yaml apiVersion: v1 kind: Pod metadata: name: docker spec: containers: - name: docker image: docker args: ["sleep", "10000"] restartPolicy: Never minikube start kubectl apply --filename docker.yaml # Applies changes to kubernetes kubectl wait --for condition=containersready pod docker # waits till container is running kubectl exec -it docker -- sh # Enter terminal inside docker container # From inside the container apk add -U git git clone ...; cd kaniko-demo docker image build --tag devops-toolkit . # Error: Cannot connect to Docker daemon kubectl delete --filename docker.yaml Now try to mount sockets for docker to get it to work inside the container # docker-socket.yaml apiVersion: v1 kind: Pod metadata: name: docker spec: containers: - name: docker image: docker args: ["sleep", "10000"] volumeMounts: - mountPath: /var/run/docker.sock name: docker-socket restartPolicy: Never volumes: - name: docker-socket hostPath: path: /var/run/docker.sock kubectl apply --filename docker-socket.yaml # Repeat commands as before docker image build --tag devops-toolkit . # Now works The reason this works is it uses Docker running on the node, not inside the container This opens up a large vulnerability as exposes node controls inside the cluster So how do you use Kaniko for this? # kaniko-git.yaml apiVersion: v1 kind: Pod metadata: name: kaniko spec: containers: - name: kaniko image: gcr.io/kaniko-project/executor:debug args: ["--context=git://github.com/vfarcic/kaniko-demo", # Pull from any code storage "--destination=vfarcic/devops-toolkit:1.0.0"] # Where to push image to (Dockerhub) volumeMounts: - name: kaniko-secret mouintPath: /kaniko/.docker restartPolicy: Never volumes: - name: kaniko-secret secret: # Provides credentials for pushing images secretName: regcred items: - key: .dockerconfigjson path: config.json # Create secret regsitry with secret credentials export REGISTRY_SERVER=https://index.docker.io/v1/ export REGISTRY_USER=...; export REGISTRY_PASS=...; export REGISTRY_EMAIL=... kubectl create secret docker-registry regcred \ --docker-server=$REGISTRY_SERVER --docker-username=$REGISTRY_USER \ --docker-password=$REGISTRY_PASSWORD --docker-email=$REGISTRY_EMAIL kubectl apply --filename kaniko-git.yaml kubectl logs kaniko --follow # Watch the logs as everything builds # Should be building the image and pushing it to dockerhub # Could alt set "--context=dir://workspace" to save it locally
Devops Toolkit... Bitnami Sealed Secrets https://www.youtube.com/watch?v=xd2QoV6GJlc How can we include secrets in git? Get kube cluster, kubeseal and bitnami controller # Generate a manifest which stores secrets in base64 encoding kubectl --namespace default create secret generic mysecret --dry-run=client \ --from-literal foo=bar --output json # Now seal the file ... | kubeseal | tee mysecret.yaml # Now encrypts the values of the key-value pair kubectl create --filename mysecret.yaml # kubeseal can decrypt this in this cluster kubectl get secret mysecret --output yaml # Outputs the decrypted (base64) form Essentially this stores a pub-priv keypair in the controller's secret store So encryption is controller-dependent kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key \ -o yaml > main.key # Backup key from system kubectl apply -f main.key # Restore key to system
Andrew Malkov... Install Kubernetes on Raspberry Pi https://www.youtube.com/watch?v=rOXkutK8ANc Make a master and worker node - use raspberry pi imager to format the PIs This tutorial uses k3s as it's a fairly lightweight kubernetes # Select "OS lite 64bit", with ssh, user-pass and set hostname # Connect nodes to router and get IP from router interface (or set static) # Set up master node: ssh USER@MASTER_NODE sudo apt update; sudo apt upgrade # Update the base OS sudo vim /boot/cmdline.txt # Append "cgroup_memory=1 cgroup_enable=memory" # Reboot node curl -sfL https://get.k3s.io | sh - sudo kubectl get nodes # Should have control-plane running # Need the node-token to add worker nodes to the cluster sudo cat /var/lib/rancher/k3s/server/node-token # Set up worker nodes: # Repeat: ssh ... apt ... vim ... reboot ... curl -sfL https://get.k3s.io | K3S_URL=https://MASTER_NODE:6443 \ K3S_TOKEN=... sh - # Sets up worker node linked to master node sudo kubectl get nodes # On the master, should show new nodes sudo cat /etc/rancher/k3s/k3s.yaml # Paste this in your kubeconfig file # This is useful for managing the master node remotely - set contexts
Devops Toolkit... Kubernetes Troubleshooting with Komodor https://www.youtube.com/watch?v=kZPOtz85_zI Komodor is a troubleshooting tool that can do monitoring and alerts kubectl --namespace production apply --filename k8s # Run some folder of apps # In Komodor (website), filter to Services - has eventlog and a state timeline # Can also tell you best practices to apply within manifest files # Filter to Monitors - add rule, trigger if number of replicas less than 2 # Can set timeouts, namespace to apply to and where to send alerts (slack?) # This creates a custom "best practice" warning
########################################################### 2024-01-06 21:30 ########################################################### That Devops Guy... Kubectl Basics for Beginners https://www.youtube.com/watch?v=feLpGydQVio kubectl is the main tool for configuring Kubernetes # .kube/config apiVersion: v1 clusters: # Clusters in system (endpoint running on) - cluster: certificate-authority-data: ... server: https://kubernetes.docker.internal:6443 name: docker-desktop # As in this case kubernetes is running in docker desktop contexts: # Configuration pointing to a cluster (cluster + user) - context: cluster: docker-desktop user: docker-desktop name: docker-desktop current-context: docker-desktop kind: Config preferences: {} users: # Users (credentials) for accessing clusters - name: docker-desktop user: client-certificate-data: ... client-key-data: ... kubectl config # Sets basic kubectl config ($KUBECONFIG=$HOME/.kube/config) # Can merge configs by setting KUBECONFIG to multiple comma separated locations kubectl config --kubeconfig="..." # Point to another file The main thing "config" does is set and get the context (rest configured in file) kubectl config current-context # Gets the context being used kubectl config get-contexts # Active one has a * kubectl config use-context CONTEXT_NAME kubectl version # Gives version of system (important with multuple clusters) Namespaces are like project/resource groups - used to isolate resources kubectl get namespaces # There are a few pre-made ones kubectl create namespace test # Creates a "test" namespace # The "get" command is for listing resources: pods, deployments, services, secrets etc. # Kubernetes uses the "default" namespace when -n is not specified kubectl get pods -n test # Lists all "pods" within the "test" namespace kubectl describe pod PODNAME -n NAMESPACE # Gives all info about a resource
That Devops Guy... Kubernetes Deployments for Beginners https://www.youtube.com/watch?v=DMpEZEakYVc A deployment is a desired state for a group of pods/replica sets # deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: example-deploy labels: app: example-app spec: selector: matchLabels: app: example-app replicas: 2 # Run a minimum of 2 instances strategy: type: RollingUpdate # Gracefully roll out changes rollingUpdate: maxSurge: 1 maxUnavailable: 0 template: metadata: labels: app: example-app spec: # This is the pod spec containers: - name: example-app # Can have multiple containers in 1 pod image: aimvector/golang:1.0.0 # This is what you are running imagePullPolicy: Always ports: - containerPort: 5000 # Port to expose for the application livenessProbe: # Kubernetes monitors and restarts if there are failures httpGet: path: /status port: 5000 initialDelaySeconds: 3 periodSeconds: 3 resources: # Set limits on CPU and memory for this container requests: memory: "64Mi" cpu: "50m" limits: memory: "256Mi" cpu: "500m" A pod is like a VM - can have multiple containers inside (e.g. db, caching, routing) kubectl apply -f deployment.yaml # Takes desired state and applies it kubectl get deployment # Shows active deployments kubectl describe deployments NAME # Shows all metrics about deployment kubectl logs POD # Can troubleshoot pod issues
That Devops Guy... Config Management https://www.youtube.com/watch?v=o-gXx7r7Rz4 Programs should have external config files (e.g. config.json) This means you can separately mount a config file when running docker docker run -it -v ${PWD}/golang/configs/:/configs aimvector/golang:1.0.0 In kubernetes you define configmaps and then point deployments to these maps # configmap.yaml apiversion: v1 kind: ConfigMap metadata: name: example-config dat: config.json: { "environment": "dev" } kubectl apply -f configmap.yaml # Apply a written configmap yaml file kubectl create configmap example-config --from-file config.json # Does not use configmap.yaml kubectl get configmaps kubectl get configmap CONFIG_NAME -o yaml # View a specific config file You should also add this to your deployment to make sure config is applied # (extends) deployment.yaml ... spec: ... template: ... spec: containers: - name: example-app ... volumeMounts: - name: config-volume mountPath: /configs/ volumes: - name: config-volume configMap: # Mounts the configmap from that file name: example-config kubectl apply -f deployment.yaml # Now also deploys configmap
That Devops Guy... Kubernetes Secret Management https://www.youtube.com/watch?v=o36yTfGDmZ0 In kubernetes, secrets are stored in kubernetes and are referenced by name # secrets.yaml apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque stringData: secrets.json: { "api_key": "somesecretgoeshere" } kubectl apply -f secret.yaml kubectl create secret generic mysecret --from-file secrets.json This is separate from the deployment spec so secrets are independent You add these in the same exact way as configMaps to deployment.yaml # (part of) deployments.yaml volumes: - name: secret-volume secret: name: mysecret Secret values are not encrypted - just stored with access permissions Kubernetes just lets you decouple the secrets from the application
That Devops Guy... Load balancing and service discovery https://www.youtube.com/watch?v=xhva6DeKqVU Traditionally, apps and services communicate over IPs but IPs change (use DNS) Point DNS to loadbalancers and then NAT back to applications - lots of port management etc. In kubernetes, you label pods (e.g. "app=foo") and make a service that points to these Services do load balancing between pods for you and let you manage ports and endpoints Inside the namespace there is a DNS service running so pods can communicate Container runtimes make network bridges to link pods You define subnets for the pods on each machine and kubernetes manages the DNS between pods "kubeproxy" controls iptables inside the kernel to route traffic properly kubectl get pods --show-labels # Shows what labels are applied to each pod # service.yaml apiVersion: v1 kind: Service metadata: name: example-service spec: selector: app: example-app # Match to label "app = example-app" ports: - protocol: TCP port: 80 # Where we accept traffic targetPort: 5000 # Where traffic is sent within the containers kubectl apply -f services.yaml # Creates the service kubectl describe service SERVICE_NAME # Show endpoints for pods
That Devops Guy... Understanding StatefulSets in Kubernetes https://www.youtube.com/watch?v=zj6r_EEhv6s Say a website is served through a load balancer linked to 2 html servers If you log in you make a session with only one of the servers - stored in server memory Now the "state" is inconsistent - deploy a datastore (redis/memcache etc.) to store sessions This way your web server system is now stateless (nothing is unique in memory) Any data stored in K8 pods is temporary (in memory) - so a db is lost when pod destroyed Usually we mount such data in volumes (persistent data) - stores data on the actual node But this is an issue if running many nodes, a node is wiped or data is very large Can also mount some external storage - but now all pods will share same storage (corruption) Also if pods interact directly they (without a service) network issues will get in the way If you create stateful set pods they are build "delicately" and are given their own DNS name Called "pod-N" and, if destroyed, is remade with the same DNS name These pods also have their own persistent volumes automatically When scaling down, it goes down the list (pod9->pod0) and retains the storage Run Redis on stateful kubernetes: kubectl create namespace example kubectl get storageclass # Get the name from this # statefulset is almost identical to a deployment # statefulset.yaml (almost identical to a deployment) apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-cluster spec: serviceName: redis-cluster replicas: 6 selector: matchLabels: app: redis-cluster template: metadata: labels: app: redis-cluster spec: containers: - name: redis image: redis:5.0.1-alpine ports: - containerPort: 6379 name: client - containerPort: 16379 name: gossip command: ["/conf/update-node.sh", "redis-server", "/conf/redis.conf"] env: - name: POD_ID valueFrom: fieldRef: fieldPath: status.podIP volumeMounts: - name: conf mountPath: /conf readOnly: false - name: data # Redis will writes to a dynamically provisioned volume mountPath /data readOnly: false volumes: - name: conf configMap: name: redis-cluster defaultMode: 0755 VolumeClaimTemplates: # Template for volumes to be made - metadata: name: data spec: accessModels: ["ReadWriteOnce"] storageClassName: "hostpath" # Value from `kubectl get storageclass` resources: requests: storage: 50Mi # Size of storage to request inside volume --- apiVersion: v1 kind: Service # Required service for redis to be exposed metadata: name: redis-cluster spec: type: ClusterIP ports: - port: 6379 targetPort: 6379 name: client - port: 16379 targetPort: 16379 name: gossip selector: app: redis-cluster kubectl -n example apply -f statefulset.yaml kubectl get pods # Can see pods being created one by one kubectl get pv # Lists volumes being created # Now set up the redis clusters (makes 3 master + 3 slave nodes) $IPs = $(kubectl -n example get pods -l app=redis-cluster -o \ jsonpath='{range.items[*]}{.status.podIP}:6379 ') kubectl -n example exec -it redis-cluster -- /bin/sh -c "redis-cli -h 127.0.0.1 -p 6379 \ --cluster create ${IPs}" # Connect to a node and build cluster kubectl -n example apply -f example-app.yaml This example is a website with a counter whenever you refresh Data persists even after killing random nodes - can also scale down kuebctl -n example scale statefulset redis-cluster --replicas 4 # Scale down to 4 # Volumes persist even when you scale down number of pods But if you scale down to 1 pod the redis cluster fails - need to know how to recover If you scale back up, cluster is still down
########################################################### 2024-01-07 18:40 ########################################################### That Devops Guy... Persistent Volumes on Kubernetes https://www.youtube.com/watch?v=ZxC6FwEc9WQ A container/pod is just a process - can be stateful/less (does not store important data) Memory allows fast access to cached data. But container file systems are non-persistent Test this with postgres running in Docker docker run -d --rm -e POSTGRES_DB=postgresdb -e POSTGRES_USER=admin \ -e POSTGRES_PASSWORD=admin123 postgres:10.4 docker exec -it CONTAINER_NAME psql --username=admin postgresdb CREATE TABLE ... # Create table with a given structure \dt # List tables docker rm -f CONTAINER_NAME # Kill container # Rerun and repeat steps - table is not there anymore - data loss Use persistent volumes to keep data after deletion docker volume create postgres docker run -d --rm -v postgres:/var/lib/postgresql/data ... # Mount volume # Now repeat the steps, delete and remake, and database is still there kubectl create namespace postgres kubectl apply -n postgres -f postgres-no-pv.yaml # Create pod without persistent volume # Same exact situation as before - deleting the pod deletes the data # Now create with persistent volume kubectl apply -n postgres -f persistedvolume.yaml # Create the volume kubectl apply -n postgres -f persistedvolumeclaim.yaml # Create "claim" to the volume kubectl apply -n postgres -f postgres-with-pv.yaml # Create pod with persistent volume # persistedvolume.yaml apiVersion: v1 kind: PersistentVolume metadata: name: example-volume labels: type: local spec: storageClassName: hostpath # From `kubectl get storageclass` (gets node storage info) capacity: storage: 1Gi accessModes: - ReadWriteOnce hostPath: # Where to mount on the host operating system path: "/mnt/data" The "storage class" is where things get complex - this is the provisioner of the storage Kubernetes needs to know how to interface with the storage (provisioner can be, say, AWS S3) The "local" storageclass is just a basic storage type Persistent volumes are not bound by namespaces So make a storage class, a persistent volume and then a claim (to a portion of the storage) # persistentvolumeclaim.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-claim spec: storageClassName: hostpath accesModes: - ReadWriteOnce resources: requests: storage: 50Mi kubectl apply -n postgres -f persistentvolumeclaim.yaml # Apply this within a namespace kubectl -n postgres get pvc # List claims # To apply this make a stateful set with a container that mounts some "data" volume # Then add a volume that links to the volume claim (name is arbitrary) # (part of) postgres-with-pv.yaml volumes: - name: data persistentVolumeClaim: claimname: example-claim
That Devops Guy... DaemonSets Explained https://www.youtube.com/watch?v=RGSeeN-o-kQ For apps we use deployments - for databases we use stateful sets DaemonSets make sure pods are running on every node in distinct sets e.g. making sure there is monitoring, logging and a database on each node Can use "kind" to run a kubernetes cluster inside a docker container # kind.yaml apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster nodes: - role: control-plane - role: worker - role: worker - role: worker kind create cluster --name daemonsets --image kindest/node:v1.20.2 --config kind.yaml kubectl get nodes Now apply a daemon set to run pods on each node # daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: example-daemonset namesapce: default labels: app: example-daemonset spec: selector: matchLabels: name: example-daemonset template: metadata: labels: name: example-daemonset spec: tolerations: # Run on master nodes "if you nede to" - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: example-daemonset image: alpine:latest env: - naem: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName # Use k8 to get the nodename and pass as variable command: - "bin/sh" - "-c" - "echo 'Hello! I am running on '$NODE_NAME; while true; do sleep 300s; done;" resources: limits: memroy: 200Mi requests: cpu: 100m memory: 200Mi terminationGracePeriodSeconds: 30 kubectl apply -f daemonsets.yaml # Pod will be running on each node kubectl get pods -o wide # Show nodes and IP addresses kubectl logs POD_NAME # Shows pod logging its nodename kubetl delete daemonset DAEMONSET_NAME DaemonSet pods are scheduled before anything else - tolerate system issues more than other pods Can connect to them with a clusterIP service which selects pods by labels Can also use hostPort (knowing their IP) or with a headless service (gives pods domain names) If you run an nginx http server on a node you can communicate very simply through that Can use "initContainers" to run initial commands within a node (e.g. make an index.html file) # clusterip-service.yaml apiVersion: v1 kind: Service metadata: name: daemonset-svc-clusterip spec: type: ClusterIP selector: # Selects which pods are part of the service name: daemonset-communicate ports: # Expose port 80 - protocol: TCP name: "http" port: 80 targetPort: 80 # Apply this and make sure it has IP endpoints (proof it works) kubectl describe svc/daemnset-svc-clusterip # Can use name of service as dns # Inside a pod, run this to communicate between pods while true; do curl http://daemonset-svc-clusterip; sleep 1s; done # Round robins between pods Can use "type: LoadBalancer" to expose this service publicly To talk to a specific pod, add a "port" definition inside the pod spec # (part of) daemonset-communication.yaml ports: - containerPort: 80 hostPort: 80 name: "http" # Reapply this and all nodes (one at a time) will have ports exposed curl NODEIP:80 # Can ping each individually In a headless service, each pod gets its own domain name # headless-service.yaml apiVersion: v1 kind: Service metadata: name: daemonset-svc-headless spec: clusterIP: None # Starts it headless selector: name: daemonset-communicate ports: - protocol: TCP name: "http" port: 80 targetPort: 80 kubectl apply -f headless-service.yaml # So how can you get the endpoints of the pods? dig daemonset-svc-headless.default.svc.cluster.local # Inside the pods, dig to query the DNS kubectl describe endpoints daemonset-svc-headless # Can query kubernetes API # Can also curl them by IP or DNS curl http://10.244.0.7 curl http://10-244-0-7.default.pod.cluster.local
########################################################### 2024-01-15 16:50 ########################################################### That Devops Guy... Detect Kubernetes misconfiguration with Datree https://www.youtube.com/watch?v=aqiOyXPPadk Kubernetes is very open ended so it doesn't have guardrails for any issues Datree essentially tests kubernetes manifests for errors or misconfigurations Shifting left means moving the testing phase to earlier - fixing issues before they occur # Install inside a simple docker container docker run -it -v ${PWD}:/work -v ${HOME}/.kube/:/root/.kube/ -w /work --net host alpine sh apk add curl unzip bash sudo curl https://get.datre.io | /bin/bash First thing it can do is yaml validation datree test deployment.yaml # Outputs yaml validation with specific line checking # Can also do Kuberntes schema validation - checking if upgrade will work datree test --schema-version 1.14.0 example.yaml # Test for k8s 1.14.0 datree test yaml/*.y*ml # Can do wildcard checks Policy checks are predefined rules for formatting or requirements (e.g. have a CPU limit) Output of checks gives a URL, you log in, and explain/manage policies on a dashboard Can also enable policy-as-code (from the website) # policyrule.yaml apiVersion: v1 policies: - name: Default isDefault: true rules: - identifier: CONTAINERS_MISSING_IMAGE_VALUE_VERSION messageOnFailure: Incorrect value for key `image` - identifier: DEPLOYMENT_MISSING_LABEL_ENV_VALUE messageOnFailure: Missing label object `env` Datree can also check Helm and Kustomize helm datree test example-app -- --valus values.yaml datree kustomize test ./kustomize/application/ Can also link this to CICD - there are different examples for each tool # datree.yaml (github actions) on: workflow_dispatch: push: branches: [ datree ] pull_request: branches: [ datree ] env: DATREE_TOKEN: ${{ secrets.DATREE_TOKEN }} # This is the user token tied to datree website jobs: k8sPolicyCheck: runs-on: ubuntu-latest steps: - name: checkout uses: actions/checkout@v2 - name: run datree policy check uses: datreeio/action-datree@main with: # Here you pass whatever you want datree to test path: 'kubernetes/datree/example/deployment.yaml' cliArguments: '--only-k8s-files' - name: docker login env: DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} run: | docker login -u $DOCKER_USER -p $DOCKER_PASSWORD # Run this and the policy check will prevent pipelines continuing when it finds an issue What about when people apply kubernetes changes direct to the cluster? There is an admission webhook you can add inside your cluster to validate/reject operations kind create cluster --name datree --image kindest/node:v1.23.6 export DATREE_TOKEN=... # Export datree key from website/account/settings/token-management # Create kubernetes webhook for datree curl -L https://get.datree.io/admission-webhook -o datree.sh; bash datree.sh kubectl -n datree get pods # With this running it protects the cluster # Any "apply" is rejected unless it passes all policy requirements What about resources already in the cluster? Use a kubectl plugin for datree bash /tmp/kubectl-plugin.sh # Download kubectl-datree from github as kubectl-plugin.sh kubectl datree test -- --namespace examples # Test a running namespace
########################################################### 2024-01-16 08:30 ########################################################### That Devops Guy... Kubernetes Affinity https://www.youtube.com/watch?v=5EkOIKn9EeM Affinity is an "attraction" to something - wants to run with (or against) something else e.g. Pod must run in us-east but prefers to run on an ssd (has a choice) # kind.yaml apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster nodes: - role: control-plane - role: worker # Worker 1 (east-ssd) labels: zone: us-east type: ssd - role: worker # Worker 2 (west-ssd) labels: zone: us-west type: ssd - role: worker # Worker 3 (east) labels: zone: us-east - role: worker # Worker 4 (west) labels: zone: us-west kind create cluster --name demo --image kindest/node:v1.28.0 --config kind.yaml kubectl get nodes # List active nodes # nod-affinity.yaml [Deployment] spec: selector: matchLabels: app: app-disk replicas: 1 template: metadata: labels: app: app-disk spec: containers: - name: app-disk image: nginx:latest affinity: nodeAffinity: # IgnoredDuringExecution = don't unassig if these values change requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: # So require us-east - key: zone operation: In values: - us-east preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: # So prefer ssd - key: type operator: In values: - ssd kubectl apply -f node-affinity.yaml # Will always go to worker 1 So what happens if you scale up? kubectl scale deploy app-disk replicas 10 # All go to worker 1 (no capacity issues) kubectl taint nodes demo-worker type=ssd:NoSchedule # Prevent scheduling to worker 1 kubectl scale deploy app-disk replicas 15 # New nodes are being made on worker 3 # Now if you make BOTH unavailable kubectl taint nodes demo-worker3 type=ssd:NoSchedule kubectl scale deploy app-disk replicas 20 # New pods are now "Pending" kubectl tain nodes demo-worker type=ssd:NoSchedule- # Remove taint (run this on both) # Scale to 0 then to 1 to reset Affinity is useful as you can force a db to run as close as possible to its web server # pod-affinity.yaml [Deployment] spec: selector: matchLabels: app: web-disk replicas: 1 template: metadata: labels: app: web-disk spec: containers: - name: web-disk image: nginx:latest affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution - labelSelector: matchExpressions: # Select node where app=app-disk - key: app operator: In values: - app-disk topologyKey: "kubernetes.io/hostname" # Label on node where this applies # Use "hostname" as you want the same host as the database - could also use region kubectl apply -f pod-affinity.yaml kubectl scale deploy app-disk --replicas 3 # And repeat also for web-app kubectl get pods -owide # All app -> worker1. All web UI -> app -> worker1 Can tell pods not to schedule on nodes already running some db pod - anti-affinity # (extending) pod-affinity.yaml podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - app-disk topologyKey: "kubernetes.io/hostname" # Scaling this spreads out databases - but as "required" can only run 2 at a time # Scale web-disk to 2 and equally it adds to worker1 and worker3
That Devops Guy... Understanding Kubernetes RBAC https://www.youtube.com/watch?v=jvhKOAyD8S8 RBAC = role based access control - roles are bound to users by their needs You have Users, Groups and Roles at a namespace level (limit access to namespaces) Cluster level permissions have ClusterRoles and ClusterUsers (use infrequently) When you make a "kind" cluster it modifies .kube/config to add a cert-authority Users are just certificates approved by the cluster - managed k8 uses OAuth Look in /etc/kubernetes/pki for CA certificates (or ~/.minikube/ for minikube) docker exec -it rbac-control-plane bash # Go inside master node ls -l /etc/kubernetes/pki # Can see ca.crt and ca.key # Signs user access # Copy out these keys to manually create users docker cp rbac-control-plane:/etc/kubernetes/pki/ca.crt ca.crt # Repeat for ca.key docker run -it -v ${PWD}:/work -w /work -v ${HOME}:/root/ --net host alpine sh apk add openssl # Explore inside docker container and install openssl # Make a certificate signing request (CSR) specifying group ownership # e.g. Name = Bob Smith - Group = Shopping openssl genrsa -out bob.key 2048 # Generate a private key for "Bob Smith" openssl req -new -key bob.key -out bob.csr -subj "/CN=Bob Smith/O=Shopping" # Generate CSR # Sign the request using the key/crt pair - approve it for 1 day (always have expiration) openssl x509 -req -in bob.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out bob.crt -days 1 # "Signature ok. subject=CN = Bob Smith, O = Shopping. Getting CA Private Key" Now you have a signed certificate that can be used as a user access key (bob.crt) Need to embed this certificate in a kubeconfig (Users: user+cert, Cluster: CA+K8address) Contexts bind clusters and users - so you can have a prod/dev context # .kube/config (main config file) apiVersion: v1 kind: Config preferences: {} current-context: kind-rbac clusters: - ... - cluster: certificate-authority-data: ... # Likely CA public key server: https://127.0.0.1:52807 name: kind-rbac # Name of the cluster contexts: - ... - context: # Bind a cluster to a user cluster: kind-rbac user: kind-rbac name: kind-rbac users: - ... - name: kind-rbac # Admin account for a cluster user: client-certificate-data: ... # bob.crt (bob.csr after signed) client-key-data: ... # bob.key # Can change this config with KUBECONFIG variable Now try to make your own config file inside a container # Open new container and add kubectl export KUBECONFIG=~/.kube/new-config kubectl get nodes # Will say there is no config defined # Make a new cluster (using IP of old cluster) kubectl config set-cluster dev-cluster --server=https://127.0.0.1:52807 \ --certificate-authority=ca.crt --embed-certs=true cat $KUBECONFIG # Has created a basic config file with just cluster info # Make user "Bob Smith" (username is taken from inside the certificate) kubectl config set-credentials --client-certificate=bob.crt \ --client-key=bob.key --embed-certs=true # Embed keys in config file # Make context DEV kubectl config set-context --cluster=dev-cluster --namespace=shopping --user=bob kubectl get nodes # Still won't work as not pointed to a cluster kubectl config use-context dev # Can have different people use different clusters at once kubectl get pods # "Forbidden" as no permissions are set for this user Now need to define roles that allow the user to actually do something # role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: shopping # Rules defined only in a given namespace name: manage-pods # Give the user the ability to manage pods rules: - apiGroups: [""] resources: ["pods", "pods/exec"] # Work on pods - allow full control verbs: ["get", "watch", "list", "create", "delete"] - apiGroups: ["apps"] resources: ["deployments"] # Work on apps/deployments - allow full control verbs: ["get", "watch", "list", "delete", "create"] kubectl create ns shopping kubectl -n shopping apply -f ./role.yaml # Creates role but does not give access # rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: manage-pods namespace: shopping subjects: - kind: User name: "Bob Smith" # Bind this user apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: manage-pods # Bind this role apiGroup: rbac.authorization.k8s.io kubectl -n shopping apply -f ./rolebinding.yaml # Access now works But what about for applications? Custom tools need service accounts (similarly need roles) Service accounts "give pods identity" - use service account tokens # serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: shopping-api # pod.yaml (example pod using this service account) apiVersion: v1 kind: Pod metadata: name: shopping-api spec: containers: - image: nignx name: shopping-api serviceAccountName: shopping-api # Point to service account being used kubectl -n shopping apply -f serviceaccount.yaml kubectl -n shopping apply -f pod.yaml # This will have access to k8 api (but no permissions) # Bob has the "pod/exec" permission to can access inside pods kubectl -n shopping exec -it shopping-api -- bash # Access pod SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount # Location of token NAMESPACE=$(cat ${SERVICEACCONT}/namespace) # Get value of this pod's namespace TOKEN=$(cat ${SERVICEACCONT}/token) # Extract ServiceAccount token CACERT=${SERVICEACCONT}/ca.crt # Point to internal CA certificate # Use kubernetes API to list the pods - gives "failure forbidden" curl --cacert ${CACERT} --header "Authorization: Bearer $TOKEN" \ -s ${APISERVER}/api/v1/namespaces/shopping/pods/ To give access for this service account it also needs a role and rolebinding # serviceaccount-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: shopping-api namespace: shopping rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"] # Allow it to get info on pods # serviceaccount-rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: shopping-api namespace: shopping subjects: - kind: ServiceAccount name: shopping-api roleRef: kind: Role name: shopping-api apiGroup: rbac.authorization.k8s.io kubectl -n shopping apply -f serviceaccount-role.yaml kubectl -n shopping apply -f serviceaccount-rolebinding.yaml # Now if you rerun that curl command it is successful
That Devops Guy... Python to Kubernetes https://www.youtube.com/watch?v=d1ZMnV4yM1U To run in Docker you need to download docker-desktop or docker # src/requirements.txt Flask == 1.0.3 # src/server.py from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello World" # dockerfile FROM python:3.7.3-alpine3.9 RUN mkdir -p /app WORKDIR /app COPY ./src/requirements.txt /app/requirements.txt RUN pip install -r requirements.txt COPY ./src/ /app/ ENV FLASK_APP=server.py CMD flask run -h 0.0.0.0 -p 5000 # docker-compose.yaml version: "3.4" services: python: build: context: . container_name: python image: aimvector/python:1.0.0 # Tag for wherever this belongs ports: - 5000:5000 docker-compose build python # Builds and tags it docker-compose up python # Running on localhost:5000 docker-compose push python # Push image to dockerhub Now add to kubernetes - so install kubernetes/kubectl - make sure it is in PATH Also in Docker-desktop, go to Kubernetes and enable it kubectl get nodes # Check if running properly # kubernetes/deployments/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: example-deploy labels: app: example-app spec: selector: matchLabels: app: example-app # Link pods to deployments replicas: 2 strategy: # Will always make sure 1 pod is running, when updating pods in cluster type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 template: metadata: labels: app: example-app spec: containers: - name: example-app image: aimvector/python:1.0.0 imagePullPolicy: always # Always get latest version ports: - containerPort: 5000 resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "256Mi" cpu: "500m" kubectl apply -f ./kubernetes/deployments/deployment.yaml kubectl get deploy; kubectl get pods # kubernetes/services/service.yaml apiVersion: v1 kind: Service metadata: name: example-service spec: type: LoadBalancer # Make publicly accessible selector: app: example-app # Apply rules to all items with this tag ports: - protocol: TCP port: 80 targetPort: 5000 kubectl apply ./kubernetes/services/service.yaml kubectl get svc # Shows a load balancer is running So now you can access your website from "localhost:80"