GitOps makes configuration management seamless by creating a single source of truth for configuration changes, so changes can be transparent, validated, and low-risk. In this article, I will show you how Traefik Proxyt and Flux can work together to help you implement GitOps principles. But before we jump in, what are these tools, and why do they help?
Flux is a tool created by Weaveworks, the inventor of GitOps. It is the operator that makes GitOps happen in your cluster, ensuring cluster configurations match the ones in Git. Flux exposes an interface to this tool that helps you build and apply YAML to your clusters so you can automate your deployments. It also uses Helm.
Traefik Proxy is an edge router that can expose your services. In the Kubernetes ecosystem, this is called as an ingress controller. Traefik Proxy is continuously updated to work with the latest version of Kubernetes. Traefik Proxy provides its own CRDs such as ingress route, TCP/UDP Ingres route, and TraefikService — an abstraction layer running on top of Kubernetes services and middleware. The middleware is applied to routers and can tweak a request before it hits service. These objects help you expose services to create a configuration. Traefik Proxy is also integrated with a feature called certificate resolver that lets you obtain certificates automatically from Lets Encrypt. Traefik Proxy’s middleware constitutes small applications that you can use to tweak requests. It will help you, for example, add headers or circuit breaker rate limits. Traefik proxy also supports canary deployments and we recently added middleware for TCP ingress routes. Traefik Proxy is a leading modern reverse proxy and load balancer that makes deploying microservices easier.
When used together, Traefik Proxy and Flux can help you implement GitOps and automate at scale. Keep reading for step-by-step instructions on how to deploy multiple Traefik Proxy instances on two clusters with Flux using a GitOps approach. As Flux is low on resource utilization, this is a viable strategy for scaling your Kubernetes deployments and maintaining multiple configurations.
How to deploy Traefik Proxy using Flux
Before anything else, there are a few prerequisites you will need:
- Two Kubernetes clusters acting as e.g. staging and production environments
- An empty GitHub repo
- Exported environment variables GITHUB_TOKEN, GITHUB_USER, GITHUB_REPO
- The latest Flux CLI installed on a workstation
Step 1: Create the infrastructure repository
Create the Git repository where all configuration files will be stored. In our example, we will use a GitHub repo. The following command can be used to create a GitHub repo:
gh repo create flux-traefik-demo --public --description "Flux and Traefik - demo" --clone
This command assumes you are in an empty directory with no Git repository and will have the same effect as running Git init. If your home directory is a Git repository, you might want to run Git init in an empty directory first, before running the above command. This will create the remote repository and take care of setting up the Git remote.
Step 2: Create the repository structure
The following command will create the top directory structures needed to keep manifests that describe the entire infrastructure:
- Apps contain custom manifests per cluster as well as Helm releases.
- Infrastructure contains common infrastructure tools, such as Helm repository definitions or common infrastructure components.
- Clusters contain Flux configurations.
mkdir -pv ./apps/{base,staging,production}/traefik ./clusters/{production,staging} ./infrastructure/{sources,crds}
├── apps
│ ├── base
│ ├── production
│ └── staging
├── infrastructure
│ ├── crds
│ └── sources
└── clusters
├── production
└── staging
Step 3: Create the Traefik base configuration
First, create your RBAC resources.
cat > ./apps/base/traefik/rbac.yaml <<EOF
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- middlewaretcps
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: traefik-ingress-controller
namespace: traefik
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: traefik
EOF
Then create your Traefik Proxy deployment resource.
cat > ./apps/base/traefik/traefik.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik
template:
metadata:
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- name: traefik
image: traefik:2.5.4
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
readinessProbe:
httpGet:
path: /ping
port: 9000
failureThreshold: 1
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /ping
port: 9000
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 100m
memory: 50Mi
ports:
- name: web
containerPort: 8000
protocol: TCP
- name: websecure
containerPort: 8443
protocol: TCP
- name: traefik
containerPort: 9000
protocol: TCP
volumeMounts:
- mountPath: /data
name: storage-volume
volumes:
- name: storage-volume
emptyDir: {}
EOF
Next, create a load balancer type service that exposes Traefik Proxy.
cat > ./apps/base/traefik/svc.yaml << EOF
---
apiVersion: v1
kind: Service
metadata:
name: traefik
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
spec:
selector:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: 80
name: web
targetPort: web
protocol: TCP
- port: 443
name: websecure
targetPort: websecure
protocol: TCP
EOF
Create a Kustomization file that lists all the resources that have been created.
cat > ./apps/base/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
- traefik.yaml
- svc.yaml
EOF
Step 4: Customize Traefik Proxy configurations for your production cluster
Create a namespace for your Traefik resources.
cat > ./apps/production/traefik/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: traefik-production
EOF
Create a patch for a Traefik Proxy deployment to happen in your cluster.
cat > ./apps/production/traefik/traefik-patch.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
spec:
template:
spec:
containers:
- name: traefik
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
- "--certificatesresolvers.myresolver.acme.storage=/data/acme.json"
- "--certificatesresolvers.myresolver.acme.tlschallenge=true"
- "--certificatesresolvers.myresolver.acme.email=jakub.hajek+webinar@traefik.io"
EOF
Then, create Kustomization to add resources.
cat > ./apps/production/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik-production
resources:
- namespace.yaml
- ../../base/traefik
patchesStrategicMerge:
- traefik-patch.yaml
EOF
Step 5: Customize Traefik Proxy configurations for your staging cluster
Create a namespace for Traefik Proxy deployment on a staging cluster.
cat > ./apps/staging/traefik/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: traefik-staging
EOF
Create a Traefik Proxy patch for your staging cluster.
cat > ./apps/staging/traefik/traefik-patch.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
spec:
template:
spec:
containers:
- name: traefik
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
- "--certificatesresolvers.myresolver.acme.storage=/data/acme.json"
- "--certificatesresolvers.myresolver.acme.tlschallenge=true"
- "--certificatesresolvers.myresolver.acme.email=jakub.hajek+webinar@traefik.io"
- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
EOF
Create a Kustomization resource for deploying your Traefik Proxy-related resources.
cat > ./apps/staging/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik-staging
resources:
- namespace.yaml
- ../../base/traefik
patchesStrategicMerge:
- traefik-patch.yaml
EOF
Step 6: Create a Traefik Proxy CRD
Traefik Proxy requires you have Custom Resources deployed on each cluster. The following command will create a common resource (including Treafik's CRD) that will be deployed on each cluster.
cat > ./infrastructure/crds/traefik-crds.yaml << EOF
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: traefik-crds
namespace: flux-system
spec:
interval: 30m
url: https://github.com/traefik/traefik-helm-chart.git
ref:
tag: v10.3.0
ignore: |
# exclude all
/*
# path to crds
!/traefik/crds/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: traefik-api-crds
namespace: flux-system
spec:
interval: 15m
prune: false
sourceRef:
kind: GitRepository
name: traefik-crds
namespace: flux-system
healthChecks:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressroutes.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressroutetcps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressrouteudps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: middlewares.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: middlewaretcps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: serverstransports.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: tlsoptions.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: tlsstores.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: traefikservices.traefik.containo.us
EOF
Create Kustomization for your Traefik Proxy CRDS.
cat > ./infrastructure/crds/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
- traefik-crds.yaml
EOF
Create a Kustomization file that deploys the file contained in the CRDS directory.
cat > ./infrastructure/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- crds
EOF
Step 7: Create the initial Flux configuration for your production cluster
cat > ./clusters/production/apps.yaml << EOF
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m0s
dependsOn:
- name: infrastructure
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/production
prune: true
wait: true
EOF
cat > ./clusters/production/infrastructure.yaml << EOF
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure
prune: true
wait: true
EOF
Step 8: Create the initial Flux configuration for your staging cluster
cat > ./clusters/staging/apps.yaml << EOF
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m0s
dependsOn:
- name: infrastructure
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/staging
prune: true
wait: true
EOF
cat > ./clusters/staging/infrastructure.yaml << EOF
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure
prune: true
wait: true
EOF
Step 9: Create the initial commit
Once all the initial configurations have been created, you can commit and push the code to your repository. The next step is to bootstrap clusters with the Flux CLI command.
├── apps
│ ├── base
│ │ └── traefik
│ │ ├── kustomization.yaml
│ │ ├── rbac.yaml
│ │ ├── svc.yaml
│ │ └── traefik.yaml
│ ├── production
│ │ └── traefik
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ └── traefik-patch.yaml
│ └── staging
│ └── traefik
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── traefik-patch.yaml
├── clusters
│ ├── production
│ │ ├── apps.yaml
│ │ └── infrastructure.yaml
│ └── staging
│ ├── apps.yaml
│ └── infrastructure.yaml
└── infrastructure
├── crds
│ ├── kustomization.yaml
│ └── traefik-crds.yaml
├── kustomization.yaml
└── sources
Step 10: Bootstrap your clusters
Once the configuration files are created, you can bootstrap Flux on both clusters. Before running the bootstrap command, ensure that the following environment variables are exported.
export GITHUB_TOKEN
export GITHUB_USER
export GITHUB_REPO
Look at your staging clusters.
flux bootstrap github \
--branch=main \
--context=t1.aws.traefiklabs.tech \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--path=clusters/staging \
--components-extra=image-reflector-controller,image-automation-controller \
--personal
Then look at your production clusters.
flux bootstrap github \
--branch=main \
--context=t2.aws.traefiklabs.tech \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--path=clusters/production \
--components-extra=image-reflector-controller,image-automation-controller \
--personal
Step 11: Create a base deployment for a whoami application
Create a base configuration for whoami application.
mkdir -pv ./apps/{base,staging,production}/whoami
cat > ./apps/base/whoami/deployment.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
labels:
name: whoamiv1
spec:
replicas: 1
selector:
matchLabels:
task: whoamiv1
template:
metadata:
labels:
task: whoamiv1
spec:
containers:
- name: whoamiv1
image: traefik/traefikee-webapp-demo:v2
args:
- -ascii
- -name=FOO
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /ping
port: 80
failureThreshold: 1
initialDelaySeconds: 2
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 2
resources:
requests:
cpu: 10m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
name: whoamiv1
namespace: app
spec:
ports:
- name: http
port: 80
selector:
task: whoamiv1
EOF
Create an IngressRoute object that exposes the whoami application.
cat > ./apps/base/whoami/ingressroute.yaml << EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(\`fix.me\`)
services:
- kind: Service
name: whoamiv1
port: 80
tls:
certResolver: myresolver
EOF
cat > ./apps/base/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- ingressroute.yaml
EOF
Step 12: Create a custom release for your staging environment
Create a namespace for the whoami app to be deployed.
cat > ./apps/staging/whoami/namespace.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: whoami-staging
EOF
Create patch for the whoami application.
cat > ./apps/staging/whoami/whoami-patch.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
spec:
replicas: 4
template:
spec:
containers:
- name: whoamiv1
args:
- -ascii
- -name=STAGING
EOF
Create a patch that updates the Host rule for the whoami application.
cat > ./apps/staging/whoami/ingressroute-patch.yaml <<EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
routes:
- kind: Rule
match: Host(\`whoami.t1.demo.traefiklabs.tech\`)
services:
- kind: Service
name: whoamiv1
port: 80
EOF
Create a Kustomization configuration to deploy the whoami application on your staging cluster.
cat > ./apps/staging/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: whoami-staging
resources:
- namespace.yaml
- ../../base/whoami
patchesStrategicMerge:
- whoami-patch.yaml
- ingressroute-patch.yaml
EOF
Step 13: Create a custom release for your production environment
Create a namespace for the whoami app to be deployed to production.
cat > ./apps/production/whoami/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: whoami-production
EOF
Create a patch for the whoami application.
cat > ./apps/production/whoami/whoami-patch.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
spec:
replicas: 8
template:
spec:
containers:
- name: whoamiv1
args:
- -ascii
- -name=PRODUCTION
EOF
Create a patch that updates the Host rule for the whoami application.
cat > ./apps/production/whoami/ingressroute-patch.yaml << EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
routes:
- kind: Rule
match: Host(\`whoami.t2.demo.traefiklabs.tech\`)
services:
- kind: Service
name: whoamiv1
port: 80
EOF
Create a Kustomization configuration to deploy the whoami application to your staging cluster.
cat > ./apps/production/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: whoami-production
resources:
- namespace.yaml
- ../../base/whoami
patchesStrategicMerge:
- whoami-patch.yaml
- ingressroute-patch.yaml
EOF
And there you have it. If you have followed these steps, you have successfully deployed Traefik Proxy in Flux using a GitOps approach. This makes it easier for you to maintain multiple configurations in Kubernetes. To participate in our community, get involved in our forum where you can search for answers, ask questions, post answers, and engage in discussions. I hope to see you there!
This is a companion discussion topic for the original entry at https://traefik.io/blog/deploy-traefik-proxy-using-flux-and-gitops/