Setup k8s cluster with kind using different k8s releases

#!/bin/bash
#! /bin/bash
# This script sets up k8s cluster using metallb and istio
# Make sure you have the following executable in your path
#     kubectl
#     kind
#     istioctl

# Setup some colors
ColorOff='\033[0m'        # Text Reset
Black='\033[0;30m'        # Black
Red='\033[0;31m'          # Red
Green='\033[0;32m'        # Green

K8S_RELEASE=$1
# Get all the available releases
alltags=$(wget -q https://registry.hub.docker.com/v1/repositories/kindest/node/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}')

rm -rf ~/.kube/*

if [ -z $K8S_RELEASE ]; then
  kind create cluster
else
  if [[ "$alltags" == *"$K8S_RELEASE"* ]]; then
    kind create cluster --image=kindest/node:$K8S_RELEASE
  else
    echo "Available k8s releases are $alltags"
    exit 1
  fi
fi

# The following procedure is to setup load balancer
kubectl cluster-info --context kind-kind

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/master/manifests/namespace.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/master/manifests/metallb.yaml

PREFIX=$(docker network inspect -f '{{range .IPAM.Config }}{{ .Gateway }}{{end}}' kind | cut -d '.' -f1,2)

cat <

k8s service full domain name

 

orderer-sample.default.svc.cluster.local

<service-name>.<namespace>.svc.cluster.local

 

Run playbook to access k8s with service account mounted.

 

quay.io/operator-framework/ansible-operator:v1.10.0

That image will work if service account with some required collections installed

collections:

- name: community.kubernetes

version: "1.2.1"

- name: operator_sdk.util

version: "0.2.0"

- name: community.general

version: "3.4.0"

- name: community.crypto

version: "1.7.1"

 Use this command to run this playbook

- name: Start fabric operations
  hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - name: Search for the matching secret of the CA organization
      community.kubernetes.k8s_info:
        kind: Secret
      register: casecret

    - debug:
        var: casecret

ansible-playbook test.yaml -e "ansible_python_interpreter=/usr/bin/python3.8"

echo parse and display a certificate

 You often gets a certificate in base64 encoded format, but you want to see what is in the certificate, here is the one liner to do this in linux

echo "<<this is the encoded certificate>>" | base64 -d | openssl x509 -noout -text

That is all it takes to see what is inside of the certificate.

How apps running inside k8s uses service account?

 Many articles talked about using service account and how service account secrets get mounted onto a pod (every pod will have a service account secret mounted to it even if you never reference one), but not many really talked about how these mounted tokens or secrets get used.

Here I will talk about this little missed step.

When a pod gets created, k8s will always mount a service account (default service account if not one specified), which will mount the service account secret onto a path like this by default:

  rootCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

tokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token"

 

The magic for applications like K8S operators simply uses client class to do all sort of operations against k8s is because the client class actually uses this method InClusterConfig defined in client-go/rest/config.go file which will read secrets and tokens to return the in cluster configuration, then go on to authenticate with K8S API server for operations such as get, create, list K8S resources. Here is link to the method https://github.com/kubernetes/client-go/blob/v0.21.2/rest/config.go#L483 In this method, it will read the environment variables such as KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT to get K8S API server, read the token file, create tls configuration, then return this kube configuration file.

 

kubernetes python client package does the same thing, in this file

https://github.com/kubernetes-client/python-base/blob/master/config/incluster_config.py

Exactly same logic gets used to deal with service account secret and token.

SERVICE_PORT_ENV_NAME = "KUBERNETES_SERVICE_PORT" SERVICE_TOKEN_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/token" SERVICE_CERT_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" SERVICE_HOST_ENV_NAME = "KUBERNETES_SERVICE_HOST"

 

what is happening when istio gets installed?

After using istioctl install command to install istio, then run the uninstall command to remove istio from the cluster, many resources will be removed, that also means, the install process created these resources during the install

When use this command to remove istio,

istioctl x uninstall --purge

The following things will happen:

  Removed IstioOperator:istio-system:installed-state.
  Removed HorizontalPodAutoscaler:istio-system:istio-ingressgateway.
  Removed HorizontalPodAutoscaler:istio-system:istiod.
  Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
  Removed PodDisruptionBudget:istio-system:istiod.
  Removed Deployment:istio-system:istio-ingressgateway.
  Removed Deployment:istio-system:istiod.
  Removed Service:istio-system:istio-ingressgateway.
  Removed Service:istio-system:istiod.
  Removed ConfigMap:istio-system:istio.
  Removed ConfigMap:istio-system:istio-sidecar-injector.
  Removed Pod:istio-system:istio-ingressgateway-6968d58d88-9dq7k.
  Removed Pod:istio-system:istiod-74d4864d8d-psjs8.
  Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
  Removed ServiceAccount:istio-system:istio-reader-service-account.
  Removed ServiceAccount:istio-system:istiod-service-account.
  Removed RoleBinding:istio-system:istio-ingressgateway-sds.
  Removed RoleBinding:istio-system:istiod-istio-system.
  Removed Role:istio-system:istio-ingressgateway-sds.
  Removed Role:istio-system:istiod-istio-system.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:stats-filter-1.10.
  Removed EnvoyFilter:istio-system:stats-filter-1.9.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.10.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.9.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector.
  Removed ValidatingWebhookConfiguration::istiod-istio-system.
  Removed ClusterRole::istio-reader-istio-system.
  Removed ClusterRole::istiod-istio-system.
  Removed ClusterRoleBinding::istio-reader-istio-system.
  Removed ClusterRoleBinding::istiod-istio-system.
  Removed CustomResourceDefinition::authorizationpolicies.security.istio.io.
  Removed CustomResourceDefinition::destinationrules.networking.istio.io.
  Removed CustomResourceDefinition::envoyfilters.networking.istio.io.
  Removed CustomResourceDefinition::gateways.networking.istio.io.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.
  Removed CustomResourceDefinition::peerauthentications.security.istio.io.
  Removed CustomResourceDefinition::requestauthentications.security.istio.io.
  Removed CustomResourceDefinition::serviceentries.networking.istio.io.
  Removed CustomResourceDefinition::sidecars.networking.istio.io.
  Removed CustomResourceDefinition::telemetries.telemetry.istio.io.
  Removed CustomResourceDefinition::virtualservices.networking.istio.io.
  Removed CustomResourceDefinition::workloadentries.networking.istio.io.
  Removed CustomResourceDefinition::workloadgroups.networking.istio.io.

After istioctl operator init, there are these things created. The list is created when using the uninstall --purge.

  Removed Deployment:istio-operator:istio-operator.
  Removed Service:istio-operator:istio-operator.
  Removed ServiceAccount:istio-operator:istio-operator.
  Removed ClusterRole::istio-operator.
  Removed ClusterRoleBinding::istio-operator.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.

After do the following:

kubectl apply -f - <<EOF

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-operator
  name: example-istiocontrolplane
spec:
  profile: default
EOF

If remove everything, these are the things will be removed.

  Removed IstioOperator:istio-system:example-istiocontrolplane.
  Removed HorizontalPodAutoscaler:istio-system:istio-ingressgateway.
  Removed HorizontalPodAutoscaler:istio-system:istiod.
  Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
  Removed PodDisruptionBudget:istio-system:istiod.
  Removed Deployment:istio-operator:istio-operator.
  Removed Deployment:istio-system:istio-ingressgateway.
  Removed Deployment:istio-system:istiod.
  Removed Service:istio-operator:istio-operator.
  Removed Service:istio-system:istio-ingressgateway.
  Removed Service:istio-system:istiod.
  Removed ConfigMap:istio-system:istio.
  Removed ConfigMap:istio-system:istio-sidecar-injector.
  Removed Pod:istio-system:istio-ingressgateway-6968d58d88-wcmvt.
  Removed Pod:istio-system:istiod-84cb7c8f48-7q6rx.
  Removed ServiceAccount:istio-operator:istio-operator.
  Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
  Removed ServiceAccount:istio-system:istio-reader-service-account.
  Removed ServiceAccount:istio-system:istiod-service-account.
  Removed RoleBinding:istio-system:istio-ingressgateway-sds.
  Removed RoleBinding:istio-system:istiod-istio-system.
  Removed Role:istio-system:istio-ingressgateway-sds.
  Removed Role:istio-system:istiod-istio-system.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:stats-filter-1.10.
  Removed EnvoyFilter:istio-system:stats-filter-1.9.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.10.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.9.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector.
  Removed ValidatingWebhookConfiguration::istiod-istio-system.
  Removed ClusterRole::istio-operator.
  Removed ClusterRole::istio-reader-istio-system.
  Removed ClusterRole::istiod-istio-system.
  Removed ClusterRoleBinding::istio-operator.
  Removed ClusterRoleBinding::istio-reader-istio-system.
  Removed ClusterRoleBinding::istiod-istio-system.
  Removed CustomResourceDefinition::authorizationpolicies.security.istio.io.
  Removed CustomResourceDefinition::destinationrules.networking.istio.io.
  Removed CustomResourceDefinition::envoyfilters.networking.istio.io.
  Removed CustomResourceDefinition::gateways.networking.istio.io.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.
  Removed CustomResourceDefinition::peerauthentications.security.istio.io.
  Removed CustomResourceDefinition::requestauthentications.security.istio.io.
  Removed CustomResourceDefinition::serviceentries.networking.istio.io.
  Removed CustomResourceDefinition::sidecars.networking.istio.io.
  Removed CustomResourceDefinition::telemetries.telemetry.istio.io.
  Removed CustomResourceDefinition::virtualservices.networking.istio.io.
  Removed CustomResourceDefinition::workloadentries.networking.istio.io.
  Removed CustomResourceDefinition::workloadgroups.networking.istio.io. 

In operator case, the deployment of istio-operator uses image docker.io/istio/operator:1.10.2 which deploys into namespace istio-operatorby default. it will only create the operator crd, only when control plan gets created by using the following command:

kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: default
EOF

Then all other crds will be created. The deployment of istiod will use docker.io/istio/pilot:1.10.2, istiod (or pilot) is now only watching the cluster, does the certificate and configuration. It is not doing what istio operator does which is to accept crd then convert them to various k8s resources. If just use istioctl, then there is no operator to interpret these requests, istioctl will convert all the request and create k8s resources, vs in operator case, it is the operator takes the request and create k8s resources.

Regardless using istioctl or istio operator or helm, the istiod will have to be deployed

Use protoc to generate go code using protobuf

This example is based on istio.io/api project to generate all the go code based on provided proto files in various sub directories.

 

protoc --proto_path=$(pwd) --go_out=$(pwd) --go_opt=paths=source_relative networking/**/*.proto security/**/*.proto type/**/*.proto analysis/**/*.proto authentication/**/*.proto meta/**/*.proto telemetry/**/*.proto