K3s part 3: Traefik proxy

Updated August 12, 2023 4 minutes

Traefik is a proxy service that will allow HTTP and TCP services running inside your cluster to be exposed to the public internet. It follows the standard Kubernetes Ingress API. Traefik handles automatic and transparent TLS (HTTPS) with ACME certificates (Let’s Encrypt).

Create Traefik YAML manifests

Configure the git repository directory you created in part 1 along with other config variables:

## Git directory for cluster config
FLUX_INFRA_DIR=${HOME}/git/flux-infra
## Domain name of the cluster
CLUSTER=k3s.example.com
## Traefik version
TRAEFIK_VERSION=v2.10.4

Configure Let’s Encrypt as your ACME provider. This will generate free TLS certificates.

## For staging use https://acme-staging-v02.api.letsencrypt.org/directory
## This is the production ACME server for Let's Encrypt:
ACME_SERVER=https://acme-v02.api.letsencrypt.org/directory
## This is your real email address:
ACME_EMAIL=you@example.com

To create traefik.rbac.yaml, which creates a ServiceAccount and ClusterRole for traefik:

mkdir -p ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system
cat <<EOF > ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system/traefik.rbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    app.kubernetes.io/name: traefik
    app.kubernetes.io/instance: traefik
  annotations:

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system
EOF

To create traefik.pvc.yaml, which creates a PersistentVolumeClaim to request a volume to store the ACME certificates generated by Traefik:

cat <<EOF > ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system/traefik.pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: traefik-data
  namespace: kube-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100M
  storageClassName: local-path
EOF

To create traefik.yaml, which creates the Traefik DaemonSet:

cat <<EOF > ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system/traefik.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: traefik-ingress-lb
  name: traefik
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
      name: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      containers:
      - args:
        - --api
        - --log.level=INFO
        - --api.insecure=false
        - --api.dashboard=false
        - --accesslog
        - --global.checknewversion=true
        - --entryPoints.web.address=:80
        - --entryPoints.websecure.address=:443
        - --entrypoints.web.http.redirections.entryPoint.to=websecure
        - --entrypoints.websecure.http.tls.certResolver=default
        - --ping=true
        - --providers.kubernetescrd=true
        - --providers.kubernetesingress=true
        - --certificatesresolvers.default.acme.storage=/traefik-data/acme.json
        - --certificatesresolvers.default.acme.tlschallenge=true
        - --certificatesresolvers.default.acme.caserver=${ACME_SERVER}
        - --certificatesresolvers.default.acme.email=${ACME_EMAIL}
        - --entrypoints.ssh.address=:2222
        image: traefik:${TRAEFIK_VERSION}
        name: traefik-ingress-lb
        volumeMounts:
        - name: traefik-data
          mountPath: /traefik-data
        ports:
        - containerPort: 80
          hostPort: 80
          name: web
        - containerPort: 443
          hostPort: 443
          name: websecure
        - containerPort: 2222
          hostPort: 2222
          name: ssh
        securityContext:
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      volumes:
      - name: traefik-data
        persistentVolumeClaim:
          claimName: traefik-data
EOF

The Traefik project distributes an http service called whoami, useful for testing that the proxy functions and that certificate generation is working. The IngressRoute connects the internal service to the outside network, through Traefik.

To create whoami.yaml:

cat <<EOF | sed 's/@@@/`/g' > ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system/whoami.yaml
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  ports:
  - name: web
    port: 80
    protocol: TCP
  selector:
    app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: whoami
  namespace: default

spec:
  weighted:
    services:
      - name: whoami
        weight: 1
        port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: whoami
  name: whoami
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - image: containous/whoami
        name: whoami
        ports:
        - containerPort: 80
          name: web
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(@@@whoami.${CLUSTER}@@@)
    services:
    - name: whoami
      port: 80
  tls:
    certResolver: default
EOF

To create kustomization.yaml, which lists all the resource files, including one from the upstream Traefik distribution URL.

cat <<EOF > ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/traefik/traefik/${TRAEFIK_VERSION}/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
- traefik.rbac.yaml
- traefik.pvc.yaml
- traefik.yaml
- whoami.yaml
EOF

Apply configuration

Apply the configuration to your cluster:

kustomize build ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system | kubectl apply -f -

You may see errors like this:

no matches for kind "IngressRoute" in version "traefik.containo.us/v1alpha1"
no matches for kind "TraefikService" in version "traefik.containo.us/v1alpha1"

Just run the command again:

kustomize build ${FLUX_INFRA_DIR}/${CLUSTER}/kube-system | kubectl apply -f -

The second time will resolve the dependency disorder.

Test it

You can examine the whoami service in your web browser, open https://whoami.k3s.example.com

You now know how to expose any pod to the internet, just follow the example of whoami.yaml, for your own services.

You can discuss this blog on Matrix (Element): #blog-rymcg-tech:enigmacurry.com

This blog is copyright EnigmaCurry and dual-licensed CC-BY-SA and MIT. The source is on github: enigmacurry/blog.rymcg.tech and PRs are welcome. ❤️