forwardAuth problem

Hello,
I'm trying to configure forwardAuth in traefik 2.0 and encountered weird problem
Here are my configs:

IngressRoute for Auth Service which handles permissions

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  creationTimestamp: null
  name: cerberus-extauth-dev
  namespace: stage
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`innsmouth.dev`) && PathPrefix(`/api/extauth`)
    middlewares: []
    priority: 0
    services:
    - name: cerberus
      port: 80

Auth middleware which handles redirect:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: cerberus-auth-api
spec:
  forwardAuth:
    address: http://innsmouth.dev/api/extauth
    authResponseHeaders:
    - "uber-trace-id"
    - "x-flow-id"
    - "x-request-id"
    trustForwardHeader: true

IngressRoute for service which should authenticate:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  creationTimestamp: null
  name: admin-api-dev
  namespace: development
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`innsmouth.dev`) && PathPrefix(`/development/admin`)
    middlewares:
    - name: cerberus-auth-api
      namespace: stage
    priority: 0
    services:
    - name: admin
      port: 80

Then I send curl to admin-api-dev service and expect full request path to be send to forwardAuth address like
http://innsmouth.dev/api/extauth/my-path-here like it was in traefik@1.7.x
or maybe in header, no matter, however there is no request path at all, neither in forwardAuth address path, nor in header.

How can I get full request path? I kinda need it for permission model.

Thx in advance

I don't know the answer, but I have the following comments:

  • You are saying what you are not seeing, but you are not saying what you are seeing, that would be extremely useful to those who can answer the question
  • You say "I send curl to admin-api-dev service". This is vague at best and an inference at worst :wink: curl is not aware about concept of a service be it kubernetes service or traefik service. What you probably wanted to say is that you are issuing a request with curl to such-and-such url and you are expected it to be routed to certain service. It would be good if you could say what you actually specified with curl
  • Your middle-ware reference in admin-api-dev Ingress route for cerberus-auth-api specify namespace staging, but in the middleware definition itself the namespace is omitted.

From what I understand the middleware is supposed to include the requiest URI with X-Forwarded-Uri header. If you specify trustForwardHeader: true and the request already has X-Forwarded-Uri header when it arrives it's not going to be replaced.

These all are pure speculation though, as you are not showing either your request or response along with the headers.

Good luck!

Sorry, my first question here :sweat_smile:
So, request is
curl --request GET
--url http://10.55.36.6/development/admin/commissions/commissions?type=2
--header 'host:innsmouth.dev'
--header 'Authorization: bearer here_is_jwt'

Traefik correctly routes it to /api/extauth as specified in forwardAuth middleware
with headers

{
    host: "innsmouth.dev",
    user-agent: "curl/7.58.0",
    accept: "*/*",
    accept-encoding: "gzip",
    authorization: "bearer here_is_jwt",
    uber-trace-id: "136759771c11fb87:6398d799a17ef8ed:136759771c11fb87:1",
    x-forwarded-for: "10.233.118.0",
    x-forwarded-host: "innsmouth.dev",
    x-forwarded-port: "80",
    x-forwarded-proto: "http",
    x-forwarded-server: "traefik-7dbd8b596b-tww2b",
    x-real-ip: "10.233.118.0"
}

There is no X-Forwarded-Uri header
As for no namespace in middleware it's because it was specified via kubectl apply, sorry not to mention that

Hello, it looks like you are doing something wrong :wink:

I just checked a setup similar to your own and I'm getting X-Forwarded-Uri header just alright.

Here is a reproducible set up I used.

# Demo to prove X-Forwarded-Uri is passed by traefik

# Using devd to log incoming requests
# This will stand in for auth server. 
# It won't do any auth, but at least we see what's being passed
# https://github.com/cortesi/devd
# https://github.com/thurt/docker-devd
# https://hub.docker.com/r/tahurt/docker-devd
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devd
  labels:
    app: devd
    stamp: "1"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devd
  template:
    metadata:
      labels:
        app: devd
    spec:
      containers:
      - name: devd
        image: tahurt/docker-devd
        args: ["-H","/static"]
        ports:
        - containerPort: 8000
# Expose devd
---
apiVersion: v1
kind: Service
metadata:
  name: devd
spec:
  selector:
    app: devd
  ports:
  - name: web
    port: 80
    targetPort: 8000
# A stand-in service we are protecting with forwad Auth
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  labels:
    app: whoami
    stamp: "1"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: containous/whoami
# Expose whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  selector:
    app: whoami
  ports:
  - name: web
    port: 80
    targetPort: 80
# This is the ingress route for whoami for traefik
# We use fa middleware here
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: whoami
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    # make sure that this resoves to the traefik instance
    match: Host(`fatest.internal`)
    middlewares:
    - name: fa
    services:
    - name: whoami
      port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: fa
spec:
  forwardAuth:
    # Note that we need correct namespace after `devd` here so that traefik container could resolve the name
    address: http://devd.my-default-namespace
    authResponseHeaders:
    - "uber-trace-id"
    - "x-flow-id"
    - "x-request-id"
    trustForwardHeader: true

fatest.internal here needs to resolve to the traefik instance which should be exposed on port 80 externally. The yaml does not specify a namespace so when you apply it with kubectl apply -f fa.yaml the objects are created in your default namespace. I used my-default-namespace for the name but you should change that to your own.

Now when you do curl http://fatest.internal/hello/world you see something similar to:

Hostname: whoami-5c8d94f78-v5zng
IP: 127.0.0.1
IP: 10.244.3.8
RemoteAddr: 10.244.1.9:60256
GET /hello/world HTTP/1.1
Host: fatest.internal
User-Agent: curl/7.55.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.20.1.16
X-Forwarded-Host: fatest.internal
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-ingress-controller-sd49r
X-Real-Ip: 10.20.1.16

And when you run kubectl logs -l app=devd --tail=100, you'll see something like that:

20:50:08: GET /
                X-Forwarded-Server: traefik-ingress-controller-sd49r
                Accept-Encoding:    gzip
                Accept:             */*
                X-Forwarded-For:    10.20.1.16
                X-Forwarded-Method: GET
                X-Forwarded-Port:   80
                X-Forwarded-Uri:    /hello/world
                X-Real-Ip:          10.20.1.16
                User-Agent:         curl/7.55.1
                X-Forwarded-Host:   fatest.internal
                X-Forwarded-Proto:  http
        <- 200 OK 141 B
                Last-Modified:  Wed, 17 Jan 2018 22:15:21 GMT
                Content-Type:   text/html; charset=utf-8
                Content-Length: 141

Note how this includes X-Forwarded-Uri.

I believe that your problem is not with traefik. I suggest trying my configuration, making sure that you could reproduce my result and than trying step-by-step bridging the gap between your example and mine (from either side). Hopefully the case of disappearing header could be solved this way. Good luck!

Thx, I changed our auth service to devd and header was there, will look into our service code.