[Kubernetes-K3s]Domain not reachable through Cloudflare tunnnel > Traefik

Traefik Traefik v2
,
1

Hi,

I'm new to Kubernetes, and can't get my public subdomain resolved.
I want to use a Cloudflare tunnel to my home server so I don't need to open any ports.

My Setup:

Certificates

My certificated are created with Cert-Manager in namespace "Default".

  • Local domain certificates

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: home-mydomain-com
  namespace: default
spec:
  secretName: home-mydomain-com-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  commonName: "*.home.mydomain.com"
  dnsNames:
    - "home.mydomain.com"
    - "*.home.mydomain.com"
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "cloudflare,traefik,pihole,nginx,download-clients"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "cloudflare,traefik,nginx"

  • Public domain certificates

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mydomain-com
  namespace: default
spec:
  secretName: mydomain-com-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  commonName: "*.mydomain.com"
  dnsNames:
    - "mydomain.com"
    - "*.mydomain.com"
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "cloudflare,traefik,pihole,nginx,download-clients"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "cloudflare,traefik,nginx"

Traefik - Helm (values.yml)

globalArguments:
  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  # - "--log.level=INFO"
  - "--log.level=DEBUG"

deployment:
  enabled: true
  replicas: 2 
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []

nodeSelector: 
  worker: "true"

ports:
  web:
    redirectTo:
      port: websecure
      priority: 10
  websecure:
    tls:
      enabled: true
      
ingressRoute:
  dashboard:
    enabled: false

providers:
  kubernetesCRD:
    enabled: true
    ingressClass: traefik-external
    allowExternalNameServices: true
  kubernetesIngress:
    enabled: true
    allowExternalNameServices: true
    publishedService:
      enabled: false

rbac:
  enabled: true

ssl:
  insecureSkipVerify: true

service:
  enabled: true
  type: LoadBalancer
  annotations: {}
  labels: {}
  spec:
    loadBalancerIP: 10.20.15.171
  loadBalancerSourceRanges: []
  externalIPs: []

Cloudflare Tunnel

I created 2 CF tunnels to test.

10.20.15.171 = Traefik LB ip-address

  1. Local deployment, ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared-local
  namespace: cloudflare
data:
  config.yaml: |
    tunnel: k3s-cluster-notlsverify
    credentials-file: /etc/cloudflared/creds/credentials.json
    metrics: 0.0.0.0:2000
    no-autoupdate: true
    ingress:
    - hostname: hello.mydomain.com
      service: hello_world
    - service: https://10.20.15.171:443
      originRequest:
      noTLSVerify: true

  1. Tunnel deployment through CF Zero Trust Dashboard

Created nginx.mydomain.com hostname in tunnel settings: and enabled:

"No TLS Verify" enabled

image
image3394×1430 230 KB

Nginx Deployment:

  • Deployment

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 2
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 2
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest

  • Service

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx
  ports:
  - name: http
    targetPort: 80
    port: 80

  • IngressRoute

---
## LOCAL DNS ##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nginx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`www.nginx.home.mydomain.com`)
      kind: Rule
      services:
        - name: nginx
          port: 80
    - match: Host(`nginx.home.mydomain.com`)
      kind: Rule
      services:
        - name: nginx
          port: 80
      middlewares:
        - name: default-headers
  tls:
    secretName: home-mydomain-com-tls

---
## PUBLIC DNS ##
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nginx-public
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`www.nginx.mydomain.com`)
      kind: Rule
      services:
        - name: nginx
          port: 80
    - match: Host(`nginx.mydomain.com`)
      kind: Rule
      services:
        - name: nginx
          port: 80
      middlewares:
        - name: default-headers
  tls:
    secretName: mydomain-com-tls

What works:

  • Internal/local DNS resolving through Traefik (nginx.home.mydomain.com)
  • Cloudflare tunnel to my home server (hello.mydomain.com resolves)

    ---> from CF documentation:
    This rule sends traffic to the built-in hello-world HTTP server. This can help debug connectivity issues. If hello.example.com resolves and tunnel.example.com does not, then the problem is in the connection from cloudflared to your local service, not from the internet to cloudflared.

What doesn't work:

I can't see any related error's in the log of Traefik / Cloudflare.
So I don't know where the issue is related to (not enough knowledge :sweat:)

The same setup in docker works perfectly, so what am I missing...

If anyone could direct me in the right direction?

Thanks a lot!