Hi guys,
I came from v1.7 and trying to convert my v1 to v2 config.toml
following the https://docs.traefik.io/migration/v1-to-v2/ link or the binary tool https://github.com/containous/traefik-migration-tool .
My goal is to use Traefik v2.0 into GKE cluster.
Tested with my v1.7 conf and no problem, but with my v2.0 config, dashboard is inaccessible (404) and http redirect to https don't work.
Here's the version :
$ kubectl exec -it traefik-784d8c545f-5wf7f traefik version
Version: 2.0.4
Codename: montdor
Go version: go1.13.3
Built: 2019-10-28T20:23:57Z
OS/Arch: linux/amd64
And some logs but nothing is logged when i'm tringy to access in https or via traefik endpoints, but if I activate the api.insecure
I have logs and access to the dashboard is succefull.
$ kubectl logs -f traefik-784d8c545f-5wf7f
kubectl logs -f traefik-784d8c545f-5wf7f
time="2019-11-02T19:03:17Z" level=info msg="Configuration loaded from file: /etc/traefik/config/traefik.toml"
time="2019-11-02T19:09:05Z" level=error msg="Error while Peeking first byte: read tcp 10.40.0.16:8080->10.164.0.30:44296: read: connection reset by peer"
10.164.0.31 - - [02/Nov/2019:19:09:05 +0000] "GET / HTTP/1.1" - - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" 1 - - 0ms
10.164.0.31 - - [02/Nov/2019:19:09:07 +0000] "GET / HTTP/1.1" - - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" 2 - - 0ms
10.164.0.30 - - [02/Nov/2019:19:14:34 +0000] "GET /notls HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0" 3 - - 0ms
10.164.0.30 - - [02/Nov/2019:19:14:35 +0000] "GET /favicon.ico HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0" 4 - - 0ms
# ps faux
PID USER TIME COMMAND
1 root 0:02 traefik traefik --configFile=/etc/traefik/config/traefik.toml
24 root 0:00 sh
31 root 0:00 ps faux
And my full manifest for my GKE cluster, based on the official doc :
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutes
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutetcps
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: default
---
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- protocol: TCP
name: web
port: 80
- protocol: TCP
name: admin
port: 8080
- protocol: TCP
name: websecure
port: 443
type: LoadBalancer
selector:
app: traefik
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-config
data:
traefik.toml: |
[global]
checkNewVersion = true
sendAnonymousUsage = false
[ping]
entryPoint = "web"
[api]
insecure = false
dashboard = true
debug = true
[accessLog]
format = "common"
bufferingSize = 0
[accessLog.fields]
defaultMode = "keep"
[accessLog.fields.names]
"%!s(int=0)" = ""
[accessLog.fields.headers]
defaultMode = "keep"
[accessLog.fields.headers.names]
"%!s(int=0)" = ""
# [log]
# level = "DEBUG"
# # filePath = "foobar"
# format = "json"
[metrics]
[metrics.prometheus]
entryPoint = "traefik"
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"
[entryPoints.traefik]
address = ":8080"
[http.routers]
[http.routers.api]
rule = "Host(`traefik.mydom.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
entrypoints = ["traefik"]
service = "api@internal"
middlewares = ["myAuth"]
[http.routers.api.tls]
[http.routers.web]
rule = "HostSNI(`*`)"
entrypoints = ["web"]
middlewares = ["redirect"]
[http.middlewares]
[http.middlewares.myAuth.basicAuth]
users = [
"test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/"
]
[http.middlewares.redirect.redirectScheme]
scheme = "https"
[providers]
providersThrottleDuration = "2s"
[providers.kubernetesCRD]
throttleDuration = "0s"
[certificatesResolvers]
[certificatesResolvers.default]
[certificatesResolvers.default.acme]
email = "test@mydom.com"
storage = "/etc/traefik/acme/acme.json"
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
[certificatesResolvers.default.acme.httpChallenge]
entryPoint = "web"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: demo-acme
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
volumes:
- name: config
configMap:
name: traefik-config
- name: demo-acme
persistentVolumeClaim:
claimName: demo-acme
containers:
- name: traefik
image: traefik:v2.0
args:
- --configFile=/etc/traefik/config/traefik.toml
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080
volumeMounts:
- mountPath: "/etc/traefik/config"
name: config
- mountPath: /etc/traefik/acme
name: demo-acme
What's going wrong ?
Thanks for the help.
(after that I have some problem with x-forward-for
to get the real public client IP).
Thanks a lot.
DR
Hello,
After a lot of re-reading the entire docs, I can saying it's a SUCCESS.
All works like a charm on GKE.
Ticket can be closed.
ty
DR
1 Like
bygui86
November 12, 2019, 9:23am
3
Hi,
sorry @DnR-iData can you post the final solution?
I have the same problem and I can't solve it.
Hi @bygui86 ,
Here's my full working sample :
1_ingressRoute.yml
:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutes
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutetcps
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: default
2_middlewares.yml
:
# Declaring the user list
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-auth
spec:
basicAuth:
secret: authsecret
---
apiVersion: v1
kind: Secret
metadata:
name: authsecret
namespace: default
data:
users: |2
dGVzdDokYXByMSRINnVza2trVyRJZ1hMUDZld1RyU3VCa1RycUU4d2ovCnRlc3QyOiRhcHIxJGQ5
aHI5SEJCJDRIeHdnVWlyM0hQNEVzZ2dQL1FObzAK
---
# Redirect to https
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-redirectscheme
spec:
redirectScheme:
scheme: https
---
# Enable gzip compression
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-compress
spec:
compress: {}
---
# Exclude from `X-Forwarded-For`
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-ipwhitelist
spec:
ipWhiteList:
sourceRange:
- xxx.xxx.xxx.xxx/xx
3_deployments.yml
:
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-config
data:
traefik.toml: |
[global]
checkNewVersion = false
sendAnonymousUsage = false
[api]
[ping]
[accessLog]
format = "common"
bufferingSize = 0
[accessLog.fields]
defaultMode = "keep"
[accessLog.fields.names]
"%!s(int=0)" = ""
[accessLog.fields.headers]
defaultMode = "keep"
[accessLog.fields.headers.names]
"%!s(int=0)" = ""
[metrics]
[metrics.prometheus]
[entryPoints]
[entryPoints.web]
address = ":8000"
[entryPoints.websecure]
address = ":4443"
[providers]
providersThrottleDuration = "2s"
[providers.kubernetesCRD]
throttleDuration = "0s"
[providers.file]
filename = "/etc/traefik/config/dynamic_conf.toml"
[certificatesResolvers]
[certificatesResolvers.default]
[certificatesResolvers.default.acme]
email = "xx@xxxxx.xx"
storage = "/etc/traefik/acme/acme.json"
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
[certificatesResolvers.default.acme.tlschallenge]
dynamic_conf.toml: |
[http.routers.my-api]
rule = "Host(`traefik.xxxxx.xxx`)"
service = "api@internal"
middlewares = [
"loc-auth",
"loc-ipwhitelist"
]
[http.routers.my-api.tls]
[http.middlewares]
[http.middlewares.loc-auth.basicAuth]
users = [
"test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/",
"test2:$apr1$d9hr9HBB$4HxwgUir3HP4EsggP/QNo0",
]
[http.middlewares.loc-ipwhitelist.ipWhiteList]
sourceRange = [
"xxx.xxx.xxx.xxx/xx"
]
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
volumes:
- name: config
configMap:
name: traefik-config
containers:
- name: traefik
image: traefik:v2.0
args:
- --configFile=/etc/traefik/config/traefik.toml
ports:
- name: web
containerPort: 8000
- name: websecure
containerPort: 4443
volumeMounts:
- mountPath: "/etc/traefik/config"
name: config
4_services.yml
:
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- protocol: TCP
name: http
port: 80
targetPort: 8000
- protocol: TCP
name: https
port: 443
targetPort: 4443
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: traefik
5_routers.yml
(HAVE BEEN REMOVED DON'T USE) :
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: apiingressroutetls
namespace: default
spec:
entryPoints:
- websecure
routes:
- match: Host(`xxx.xxx.xx`) # && PathPrefix(`/api`) || Host(`xxx.xxx.xx`) && PathPrefix(`/dashboard`) || Host(`xxx.xxx.xx`) && PathPrefix(`/ping`) || Host(`xxx.xxx.xx`) && PathPrefix(`/api/`) || Host(`xxx.xxx.xx`) && PathPrefix(`/dashboard/`)
kind: Rule
services:
- name: traefik
port: 8080
middlewares:
- name: test-auth
- name: test-compress
- name: test-ipwhitelist
tls:
certResolver: default
SAMPLE 99_test_app.yml
:
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: whoami
labels:
app: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: containous/whoami
ports:
- name: web
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
ports:
- protocol: TCP
name: web
port: 80
selector:
app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: simpleingressroute
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(`xxx.xxx.xx`) #&& PathPrefix(`/notls`)
kind: Rule
services:
- name: whoami
port: 80
middlewares:
- name: test-redirectscheme
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroutetls
namespace: default
spec:
entryPoints:
- websecure
routes:
- match: Host(`xxx.xxx.xx`) #&& PathPrefix(`/tls`)
kind: Rule
services:
- name: whoami
port: 80
middlewares:
# - name: test-auth
- name: test-compress
# - name: test-ipwhitelist
tls:
certResolver: default
I have a problem, I can't secure *:8080
/ xxx.xxx.xx:8080
, seems IngressRoute
don't give a f* when api.insecure=true
, but in GKE we can't set api.insecure=false
and use service api@internal
.
It's in the doc, I can't find the link where it's say (I'm trying to find, I past later if I can get it).
If you have sugestion I'm open for every solutions
Thanks.
DR
Well well well,
@bygui86 ,
After a good night and some brain motivation lol, I've found a solution for secure dashboard, here's my final solution :
3_deployments.yml
:
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-config
data:
traefik.toml: |
[global]
checkNewVersion = false
sendAnonymousUsage = false
[api]
[ping]
[accessLog]
format = "common"
bufferingSize = 0
[accessLog.fields]
defaultMode = "keep"
[accessLog.fields.names]
"%!s(int=0)" = ""
[accessLog.fields.headers]
defaultMode = "keep"
[accessLog.fields.headers.names]
"%!s(int=0)" = ""
[metrics]
[metrics.prometheus]
[entryPoints]
[entryPoints.web]
address = ":8000"
[entryPoints.websecure]
address = ":4443"
[providers]
providersThrottleDuration = "2s"
[providers.kubernetesCRD]
throttleDuration = "0s"
[providers.file]
filename = "/etc/traefik/config/dynamic_conf.toml"
[certificatesResolvers]
[certificatesResolvers.default]
[certificatesResolvers.default.acme]
email = "xx@xxxxx.xx"
storage = "/etc/traefik/acme/acme.json"
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
[certificatesResolvers.default.acme.tlschallenge]
dynamic_conf.toml: |
[http.routers.my-api]
rule = "Host(`traefik.xxxxx.xxx`)"
service = "api@internal"
middlewares = [
"loc-auth",
"loc-ipwhitelist"
]
[http.routers.my-api.tls]
[http.middlewares]
[http.middlewares.loc-auth.basicAuth]
users = [
"test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/",
"test2:$apr1$d9hr9HBB$4HxwgUir3HP4EsggP/QNo0",
]
[http.middlewares.loc-ipwhitelist.ipWhiteList]
sourceRange = [
"xxx.xxx.xxx.xxx/xx"
]
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
volumes:
- name: config
configMap:
name: traefik-config
containers:
- name: traefik
image: traefik:v2.0
args:
- --configFile=/etc/traefik/config/traefik.toml
ports:
- name: web
containerPort: 8000
- name: websecure
containerPort: 4443
volumeMounts:
- mountPath: "/etc/traefik/config"
name: config
4_services.yml
:
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- protocol: TCP
name: http
port: 80
targetPort: 8000
- protocol: TCP
name: https
port: 443
targetPort: 4443
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: traefik
5_routers.yml
:
REMOVED
Hi @DnR-iData , thanks a lot for your solution!!
Do you know if this solution works as well without CRDs?
Currently I'm using the Kubernetes Ingress definitions instead of Traefik IngressRoute and all other CRDs.
Hi @bygui86 ,
I haven't tested without CRDs
If you try, let me know if it was a success, plz.
Thanks.
DR
Hi @DnR-iData ,
may I ask you why you went for CRDs instead of Kubernetes plain ingress?
anyway I'm testing your config with Kubernetes ingress these days.
bygui86
December 4, 2019, 10:30am
9
Hi @DnR-iData ,
I tried and your configurations don't work with plain Kubernetes Ingress.
Honestly I think there is a bug, because if I put a new router and a new middleware in the config file (traefik.toml or traefik.yaml, I tried both) they get ignored.
Here is my traefik.toml
:
[global]
checkNewVersion = false
sendAnonymousUsage = false
[entryPoints]
[entryPoints.web]
address = ":8080"
[entryPoints.websecure]
address = ":8443"
[entryPoints.ping]
address = ":8081"
[entryPoints.traefik]
address = ":8881"
[entryPoints.metrics]
address = ":9090"
[api]
insecure = true
dashboard = true
debug = true
[ping]
entryPoint = "ping"
[accessLog]
format = "common"
bufferingSize = 0
[metrics]
[metrics.prometheus]
entryPoint = "metrics"
addEntryPointsLabels = true
addServicesLabels = true
buckets = [0.1, 0.3, 1.2, 5.0]
[providers]
providersThrottleDuration = "2s"
[providers.kubernetesIngress]
ingressClass = "traefik"
namespaces = ["default", "monitoring", "logging", "broker"]
[http]
[http.routers]
[http.routers.my-api]
entryPoints = ["web", "websecure"]
rule = "Host(`traefik.swissblock.tech`)"
service = "api@internal"
middlewares = ["loc-auth"]
[http.middlewares]
[http.middlewares.loc-auth.basicAuth]
users = ["test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/", "test2:$apr1$d9hr9HBB$4HxwgUir3HP4EsggP/QNo0"]
And here a couple of screenshots:
Kubernetes yaml manifests are almost identical to yours.
As you can see, my-api
router and loc-auth
middleware get completely ignored.
I will try with CRDs, to better understand if I'm doing something wrong or what...
If I don't achieve anything with these configurations today, I will give up and switch to another ingress-controller
Re @bygui86 ,
I've never tested, because CRD provided me a working solution directly for GKE.
I think maybe [http.routers]
, [http.services]
, [http.middlewares]
have to be into a dynamic_conf
:
I've tried many times get them works into static and nothing happens, but un dynamic it worked.
DR
bygui86
December 4, 2019, 11:22am
11
Re @DnR-iData
I'm on the GKE as well.
Right now I'm testing the dynamic-conf and it doesn't get ignored, so fine. I'm still struggling with configs in general but it seems better... Anyway I think I will switch to CRDs as well, I feel I could have more fine grain control without tons of annotations everywhere...