How apps running inside k8s uses service account?

 Many articles talked about using service account and how service account secrets get mounted onto a pod (every pod will have a service account secret mounted to it even if you never reference one), but not many really talked about how these mounted tokens or secrets get used.

Here I will talk about this little missed step.

When a pod gets created, k8s will always mount a service account (default service account if not one specified), which will mount the service account secret onto a path like this by default:

  rootCAFile = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

tokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token"

 

The magic for applications like K8S operators simply uses client class to do all sort of operations against k8s is because the client class actually uses this method InClusterConfig defined in client-go/rest/config.go file which will read secrets and tokens to return the in cluster configuration, then go on to authenticate with K8S API server for operations such as get, create, list K8S resources. Here is link to the method https://github.com/kubernetes/client-go/blob/v0.21.2/rest/config.go#L483 In this method, it will read the environment variables such as KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT to get K8S API server, read the token file, create tls configuration, then return this kube configuration file.

 

kubernetes python client package does the same thing, in this file

https://github.com/kubernetes-client/python-base/blob/master/config/incluster_config.py

Exactly same logic gets used to deal with service account secret and token.

SERVICE_PORT_ENV_NAME = "KUBERNETES_SERVICE_PORT" SERVICE_TOKEN_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/token" SERVICE_CERT_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" SERVICE_HOST_ENV_NAME = "KUBERNETES_SERVICE_HOST"

 

what is happening when istio gets installed?

After using istioctl install command to install istio, then run the uninstall command to remove istio from the cluster, many resources will be removed, that also means, the install process created these resources during the install

When use this command to remove istio,

istioctl x uninstall --purge

The following things will happen:

  Removed IstioOperator:istio-system:installed-state.
  Removed HorizontalPodAutoscaler:istio-system:istio-ingressgateway.
  Removed HorizontalPodAutoscaler:istio-system:istiod.
  Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
  Removed PodDisruptionBudget:istio-system:istiod.
  Removed Deployment:istio-system:istio-ingressgateway.
  Removed Deployment:istio-system:istiod.
  Removed Service:istio-system:istio-ingressgateway.
  Removed Service:istio-system:istiod.
  Removed ConfigMap:istio-system:istio.
  Removed ConfigMap:istio-system:istio-sidecar-injector.
  Removed Pod:istio-system:istio-ingressgateway-6968d58d88-9dq7k.
  Removed Pod:istio-system:istiod-74d4864d8d-psjs8.
  Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
  Removed ServiceAccount:istio-system:istio-reader-service-account.
  Removed ServiceAccount:istio-system:istiod-service-account.
  Removed RoleBinding:istio-system:istio-ingressgateway-sds.
  Removed RoleBinding:istio-system:istiod-istio-system.
  Removed Role:istio-system:istio-ingressgateway-sds.
  Removed Role:istio-system:istiod-istio-system.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:stats-filter-1.10.
  Removed EnvoyFilter:istio-system:stats-filter-1.9.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.10.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.9.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector.
  Removed ValidatingWebhookConfiguration::istiod-istio-system.
  Removed ClusterRole::istio-reader-istio-system.
  Removed ClusterRole::istiod-istio-system.
  Removed ClusterRoleBinding::istio-reader-istio-system.
  Removed ClusterRoleBinding::istiod-istio-system.
  Removed CustomResourceDefinition::authorizationpolicies.security.istio.io.
  Removed CustomResourceDefinition::destinationrules.networking.istio.io.
  Removed CustomResourceDefinition::envoyfilters.networking.istio.io.
  Removed CustomResourceDefinition::gateways.networking.istio.io.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.
  Removed CustomResourceDefinition::peerauthentications.security.istio.io.
  Removed CustomResourceDefinition::requestauthentications.security.istio.io.
  Removed CustomResourceDefinition::serviceentries.networking.istio.io.
  Removed CustomResourceDefinition::sidecars.networking.istio.io.
  Removed CustomResourceDefinition::telemetries.telemetry.istio.io.
  Removed CustomResourceDefinition::virtualservices.networking.istio.io.
  Removed CustomResourceDefinition::workloadentries.networking.istio.io.
  Removed CustomResourceDefinition::workloadgroups.networking.istio.io.

After istioctl operator init, there are these things created. The list is created when using the uninstall --purge.

  Removed Deployment:istio-operator:istio-operator.
  Removed Service:istio-operator:istio-operator.
  Removed ServiceAccount:istio-operator:istio-operator.
  Removed ClusterRole::istio-operator.
  Removed ClusterRoleBinding::istio-operator.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.

After do the following:

kubectl apply -f - <<EOF

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-operator
  name: example-istiocontrolplane
spec:
  profile: default
EOF

If remove everything, these are the things will be removed.

  Removed IstioOperator:istio-system:example-istiocontrolplane.
  Removed HorizontalPodAutoscaler:istio-system:istio-ingressgateway.
  Removed HorizontalPodAutoscaler:istio-system:istiod.
  Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
  Removed PodDisruptionBudget:istio-system:istiod.
  Removed Deployment:istio-operator:istio-operator.
  Removed Deployment:istio-system:istio-ingressgateway.
  Removed Deployment:istio-system:istiod.
  Removed Service:istio-operator:istio-operator.
  Removed Service:istio-system:istio-ingressgateway.
  Removed Service:istio-system:istiod.
  Removed ConfigMap:istio-system:istio.
  Removed ConfigMap:istio-system:istio-sidecar-injector.
  Removed Pod:istio-system:istio-ingressgateway-6968d58d88-wcmvt.
  Removed Pod:istio-system:istiod-84cb7c8f48-7q6rx.
  Removed ServiceAccount:istio-operator:istio-operator.
  Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
  Removed ServiceAccount:istio-system:istio-reader-service-account.
  Removed ServiceAccount:istio-system:istiod-service-account.
  Removed RoleBinding:istio-system:istio-ingressgateway-sds.
  Removed RoleBinding:istio-system:istiod-istio-system.
  Removed Role:istio-system:istio-ingressgateway-sds.
  Removed Role:istio-system:istiod-istio-system.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:stats-filter-1.10.
  Removed EnvoyFilter:istio-system:stats-filter-1.9.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.10.
  Removed EnvoyFilter:istio-system:tcp-metadata-exchange-1.9.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.10.
  Removed EnvoyFilter:istio-system:tcp-stats-filter-1.9.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector.
  Removed ValidatingWebhookConfiguration::istiod-istio-system.
  Removed ClusterRole::istio-operator.
  Removed ClusterRole::istio-reader-istio-system.
  Removed ClusterRole::istiod-istio-system.
  Removed ClusterRoleBinding::istio-operator.
  Removed ClusterRoleBinding::istio-reader-istio-system.
  Removed ClusterRoleBinding::istiod-istio-system.
  Removed CustomResourceDefinition::authorizationpolicies.security.istio.io.
  Removed CustomResourceDefinition::destinationrules.networking.istio.io.
  Removed CustomResourceDefinition::envoyfilters.networking.istio.io.
  Removed CustomResourceDefinition::gateways.networking.istio.io.
  Removed CustomResourceDefinition::istiooperators.install.istio.io.
  Removed CustomResourceDefinition::peerauthentications.security.istio.io.
  Removed CustomResourceDefinition::requestauthentications.security.istio.io.
  Removed CustomResourceDefinition::serviceentries.networking.istio.io.
  Removed CustomResourceDefinition::sidecars.networking.istio.io.
  Removed CustomResourceDefinition::telemetries.telemetry.istio.io.
  Removed CustomResourceDefinition::virtualservices.networking.istio.io.
  Removed CustomResourceDefinition::workloadentries.networking.istio.io.
  Removed CustomResourceDefinition::workloadgroups.networking.istio.io. 

In operator case, the deployment of istio-operator uses image docker.io/istio/operator:1.10.2 which deploys into namespace istio-operatorby default. it will only create the operator crd, only when control plan gets created by using the following command:

kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: default
EOF

Then all other crds will be created. The deployment of istiod will use docker.io/istio/pilot:1.10.2, istiod (or pilot) is now only watching the cluster, does the certificate and configuration. It is not doing what istio operator does which is to accept crd then convert them to various k8s resources. If just use istioctl, then there is no operator to interpret these requests, istioctl will convert all the request and create k8s resources, vs in operator case, it is the operator takes the request and create k8s resources.

Regardless using istioctl or istio operator or helm, the istiod will have to be deployed

Use protoc to generate go code using protobuf

This example is based on istio.io/api project to generate all the go code based on provided proto files in various sub directories.

 

protoc --proto_path=$(pwd) --go_out=$(pwd) --go_opt=paths=source_relative networking/**/*.proto security/**/*.proto type/**/*.proto analysis/**/*.proto authentication/**/*.proto meta/**/*.proto telemetry/**/*.proto

Rebase with upstream master branch

1. Make sure that the upstream repo is added to your configuration.

2. Do a git fetch

git fetch upstream

3. Do rebase

git rebase --interactive upstream/master

4. Push to remote

git push -f 

Install tekton and its dashboard on IBM Cloud

1. Install tekton 0.21.0

kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.21.0/release.yaml

2. Install latest tekton dashboard

kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/openshift-tekton-dashboard-release.yaml --validate=false

3. Expose the dashboard UI

kubectl patch svc tekton-dashboard -n tekton-pipelines -p '{"spec": {"type": "LoadBalancer"}}'

4. Access the dashboard by using the external IP (hostname) and port 9097

kubectl get svc tekton-dashboard -n tekton-pipelines

The external IP or hostname may take a bit of time to be available.

 

For tectoncd/pipeline development, to use ko, set the docker repo env to be the full path like below

export KO_DOCKER_REPO=registry.hub.docker.com/email4tong

 The short name no longer works.

Expose TCP traffic examples

After istio is installed, follow these steps:

0. Label the default namespace for istio sidecar injection

kubectl label namespace default istio-injection=enabled --overwrite

1. Patch istio-ingressgateway service so that the new port is supported.

Create a file named patch-service.yaml with the following content:

spec:
  ports:
  - name: tcp-31400
    protocol: TCP
    port: 31400
    targetPort: 31400

Run the following command

kubectl -n istio-system patch service istio-ingressgateway --patch "$(cat patch-service.yaml)"  

2. Create deployment, service, gateway and virtual service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment
spec:
  selector:
    matchLabels:
      greeting: hello
      department: world
  replicas: 1
  template:
    metadata:
      labels:
        greeting: hello
        department: world
    spec:
      containers:
      - name: hello
        image: "email4tong/pathecho:latest"
        imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  selector:
    greeting: hello
    department: world
  ports:
  - protocol: TCP
    port: 7000
    targetPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tcp-echo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 31400
      name: tcp
      protocol: TCP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tcp-echo
spec:
  hosts:
  - "*"
  gateways:
  - tcp-echo-gateway
  tcp:
  - match:
    - port: 31400
    route:
    - destination:
        host: hello-world.default.svc.cluster.local
        port:
          number: 7000

3. Now use the istio-ingressgateway service external endpoint (IP or hostname) and port 31400 to access the service. In above example, it is a simple http echo, so use curl to test is fine. If the actual service is not http but using any other tcp protocols, then you cannot use curl to test

Istio tcp ingress traffic

 To allow tcp traffic on a specific port using Istio, one will have to patch the istio-ingressgateway service to add the new port if the port is not 80 or 443. Then one will have to create a gateway, a virtual service, and also make sure that the deployments have the sidecar injected automatically. So to summarize:

1. Patch istio-ingressgateway to add new tcp port

2. Create gateway resource

3. Create virtual service resource

4. Make sure that the actual pod is injected with sidecar.

It is very important that the tcp port to be added before the gateway and virtual service resource were created, otherwise, it wont work.