Error calling http://oauth2-proxy:4180

soubriquet · September 30, 2023, 3:01pm

I can't connect to my oauth2-proxy service. This is the error I'm getting:

2023-09-30T14:39:13Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/forward.go:122 > Error calling http://oauth2-proxy:4180. Cause: Get "http://oauth2-proxy:4180": dial tcp 192.168.128.7:4180: connect: connection refused middlewareName=oauth2-proxy@docker middlewareType=ForwardedAuthType

Here is my docker-compose.yml:

version: '3'
services:
  upload:
    image: mydomain-upload:v3-staging
    build:
      context: .
      dockerfile: src/services/upload/Dockerfile.upload
    restart: always
    ports:
      - "8004:8004"
    depends_on:
      - oauth2-proxy
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.upload.rule=(Host(`mydomain.com`) || Host(`www.mydomain.com`)) && PathPrefix(`/v3/upload`)"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.address=http://oauth2-proxy:4180"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.trustForwardHeader=true"
      - "traefik.http.routers.upload.entrypoints=websecure"
      - "traefik.http.services.upload.loadbalancer.server.port=8004"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowmethods=GET,POST,OPTIONS"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolalloworiginlist=https://mydomain.com,https://www.mydomain.com"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowheaders=*"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowcredentials=true"
      - "traefik.http.middlewares.upload-cors.headers.addvaryheader=true"
      - "traefik.http.routers.upload.middlewares=upload-cors,oauth2-proxy"
    environment:
      - MODE=staging
  oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.4.0
    ports:
      - 4180:4180
    networks:
      - proxy
    volumes:
      - /var/log:/var/log
    command:
      - --provider=oidc
      - --email-domain=*
      - --oidc-issuer-url=https://accounts.google.com
      - --cookie-secure=true
      - --cookie-secret=${OAUTH2_PROXY_COOKIE_SECRET}
      - --client-id=${OAUTH2_PROXY_CLIENT_ID}
      - --client-secret=${OAUTH2_PROXY_CLIENT_SECRET}
      - --upstream=http://traefik:80
      - --pass-access-token=true
      - --pass-authorization-header=true
      - --set-authorization-header=true
      - --redirect-url=https://www.mydomain.com/oauth2/redirect
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.oauth2-proxy.rule=(Host(`mydomain.com`) || Host(`www.mydomain.com`))"
      - "traefik.http.routers.oauth2-proxy.entrypoints=websecure"
      - "traefik.http.routers.oauth2-proxy.tls.certresolver=myresolver"
  nginx:
    image: mydomain-nginx:v3-staging
    build:
      context: .
      dockerfile: src/static/Dockerfile.nginx.staging
    restart: always
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx.rule=(Host(`mydomain.com`) || Host(`www.mydomain.com`)) && PathPrefix(`/`) && !PathPrefix(`/v3`)"
      - "traefik.http.routers.nginx.entrypoints=websecure"
      - "traefik.http.routers.nginx.tls.certresolver=myresolver"
  traefik:
    image: traefik:v3.0
    restart: always
    depends_on:
      - oauth2-proxy
      - nginx
    ports:
      - "80:80"
      - "443:443"
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/letsencrypt:/letsencrypt
      - /var/log:/var/log
    command:
      - --providers.docker.network=proxy
      - --api.dashboard=true
      - --log.level=DEBUG
      - --log.filepath=/var/log/traefik.log
      - --accesslog=true
      - --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=admin@mydomain.com
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.myresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myrouter.tls.domains[0].main=www.mydomain.com"
      - "traefik.http.routers.myrouter.tls.certresolver=myresolver"
      - "traefik.http.routers.mydashboard.rule=Host(`monitor.mydomain.com`)"
      - "traefik.http.routers.mydashboard.entrypoints=websecure"
      - "traefik.http.routers.mydashboard.tls.certresolver=myresolver"
      - "traefik.http.routers.mydashboard.service=api@internal"
      - "traefik.http.routers.mydashboard.middlewares=myauth"
      - "traefik.http.middlewares.myauth.basicauth.users=admin:$$2y$$05$$/hJpV/MlsvaKB9bAWe8nDeq416TAjd9qU4bfs56ibkFU9smKP/O0S"

networks:
  proxy:
    name: proxy

I tried a few things:

Setting entrypoints=web label in oauth2-proxy service and using forwardauth.address=http://oauth2-proxy:4180 in the upload service and also using websecure and https instead, but both scenarios failed.
oauth2-proxy is running and 4180 is exposed according to docker ps.
I checked the client-id and client-secret match with what's in my GCP.

@bluepuma77 was wondering if you could help me again with an issue.

bluepuma77 · September 30, 2023, 7:37pm

The error indicates that Traefik can not connect to oauth-proxy. Both connected to Docker network, oauth-proxy container is running, is the port number correct and oauth-proxy listening to it?

soubriquet · September 30, 2023, 9:10pm

Yes, both are connected to the same Docker network and the oauth2-proxy container is running and exposes port 4180.

Here is docker ps:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

442cb667d791 traefik:v3.0 "/entrypoint.sh --pr…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp backend_traefik_1

e21dc272f933 mydomain-upload:v3-staging "uvicorn server:app …" About a minute ago Up About a minute 0.0.0.0:8004->8004/tcp, :::8004->8004/tcp backend_upload_1

69ec3a1fbb97 quay.io/oauth2-proxy/oauth2-proxy:v7.4.0 "/bin/oauth2-proxy -…" About a minute ago Up About a minute 0.0.0.0:4180->4180/tcp, :::4180->4180/tcp backend_oauth2-proxy_1

fc40a4435f2e mydomain-nginx:v3-staging "/docker-entrypoint.…" About a minute ago Up About a minute 80/tcp, 443/tcp backend_nginx_1

bluepuma77 · October 1, 2023, 7:07am

You probably should not set ports for any other service than Traefik to open ports on the host. All other services should only be reachable via internal Docker network, which works without ports.

Are your sure this is correct? Your Traefik does a redirect on port 80 to 443:

This may not make sense. LE will create certs for the Host() domains automatically. You would use main and sans for wildcards with dnsChallenge:

This is not needed on labels, as set as global default on entrypoint:

I think it really makes sense for every service to add loadbalancer.server.port to the labels, to explicitly tell Traefik which port to use internally. Otherwise the first Dockerfile expose is used. Check simple Traefik example.

But I am not sure any of this will solve your problem.

Maybe check some tutorials like 1, 2.

soubriquet · October 2, 2023, 2:32pm

Thanks for pointing out the ports. I removed those now.

I also removed --upstream. My current understanding is that it tries to redirect the request to traefik instead of continuing to the upload service. But even though I removed it, I'm still getting 500 when I try to access my upload route and still do not see any requests hitting the upload service in my docker logs.

Should I remove this?

- "traefik.http.routers.myrouter.tls.domains[0].main=www.mydomain.com"

I think it really makes sense for every service to add loadbalancer.server.port to the labels, to explicitly tell Traefik which port to use internally. Otherwise the first Dockerfile expose is used.

Is it ok to not use loadbalancer.server.port as long as I'm careful to define different ports in my Dockerfiles?

Here is my updated docker-compose.yml:

version: '3'
services:
  upload:
    image: mydomain-upload:v3-staging
    build:
      context: .
      dockerfile: src/services/upload/Dockerfile.upload
    restart: always
    depends_on:
      - oauth2-proxy
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.upload.rule=(Host(`mydomain.com`) || Host(`www.mydomain.com`)) && PathPrefix(`/v3/upload`)"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.address=https://oauth2-proxy-backend:4180"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.trustForwardHeader=true"
      - "traefik.http.routers.upload.entrypoints=websecure"
      - "traefik.http.services.upload.loadbalancer.server.port=8004"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowmethods=GET,POST,OPTIONS"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolalloworiginlist=https://mydomain.com,https://www.mydomain.com"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowheaders=*"
      - "traefik.http.middlewares.upload-cors.headers.accesscontrolallowcredentials=true"
      - "traefik.http.middlewares.upload-cors.headers.addvaryheader=true"
      - "traefik.http.routers.upload.middlewares=upload-cors,oauth2-proxy"
    environment:
      - MODE=staging
  oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.4.0
    networks:
      - proxy
    volumes:
      - /var/log:/var/log
    command:
      - --provider=oidc
      - --email-domain=*
      - --oidc-issuer-url=https://accounts.google.com
      - --cookie-secure=true
      - --cookie-secret=${OAUTH2_PROXY_COOKIE_SECRET}
      - --client-id=${OAUTH2_PROXY_CLIENT_ID}
      - --client-secret=${OAUTH2_PROXY_CLIENT_SECRET}
      - --pass-access-token=true
      - --pass-authorization-header=true
      - --set-authorization-header=true
      - --redirect-url=https://www.mydomain.com/oauth2/redirect
      - --request-logging=true
      - --auth-logging=true
      - --standard-logging=true
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.address=https://oauth2-proxy-backend:4180"
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.trustForwardHeader=true"
  nginx:
    image: mydomain-nginx:v3-staging
    build:
      context: .
      dockerfile: src/static/Dockerfile.nginx.staging
    restart: always
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx.rule=(Host(`mydomain.com`) || Host(`www.mydomain.com`)) && PathPrefix(`/`) && !PathPrefix(`/v3`)"
      - "traefik.http.routers.nginx.entrypoints=websecure"
      - "traefik.http.routers.nginx.tls.certresolver=myresolver"
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "4318:4318"
    environment:
      - LOG_LEVEL=debug
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.jaeger.rule=Host(`jaeger.mydomain.com`)"
      - "traefik.http.routers.jaeger.entrypoints=websecure"
      - "traefik.http.routers.jaeger.tls.certresolver=myresolver"
      - "traefik.http.services.jaeger.loadbalancer.server.port=16686"
      - "traefik.http.routers.jaeger.middlewares=myauth"
  traefik:
    image: traefik:v3.0
    restart: always
    depends_on:
      - jaeger
      - oauth2-proxy
      - nginx
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/letsencrypt:/letsencrypt
      - /var/log:/var/log
    command:
      - --providers.docker.network=proxy
      - --api.dashboard=true
      - --log.level=DEBUG
      - --log.filepath=/var/log/traefik.log
      - --accesslog=true
      - --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=admin@mydomain.com
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.myresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
      - --metrics.prometheus=true
      - --tracing.jaeger=true
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myrouter.tls.domains[0].main=www.mydomain.com"
      - "traefik.http.routers.myrouter.tls.certresolver=myresolver"
      - "traefik.http.routers.mydashboard.rule=Host(`monitor.mydomain.com`)"
      - "traefik.http.routers.mydashboard.service=api@internal"
      - "traefik.http.routers.mydashboard.middlewares=myauth"
      - "traefik.http.middlewares.myauth.basicauth.users=admin:$$2y$$05$$/hJpV/MlsvaKB9bAWe8nDeq416TAjd9qU4bfs56ibkFU9smKP/O0S"

networks:
  proxy:
    name: proxy

soubriquet · October 4, 2023, 3:05am

@bluepuma77 I figured it out. Posting in case anyone else runs into this issue.

I had two errors causing this connection refused error to my oauth2-proxy service:

upload (my service I want to protect with OAuth2) and oauth2-proxy should communicate over http - not https. Because they're on the same Docker network, it's not necessary to use https. So --cookie-secure label should be set to false and forwardauth.address should use http://<oauth service name>:4180.

upload:
    image: parkourtheory-upload:v3-staging
    build:
      context: .
      dockerfile: src/services/upload/Dockerfile.upload
    restart: always
    depends_on:
      - oauth2-proxy
    networks:
      - proxy
    labels:
      - "traefik.http.middlewares.oauth2-proxy.forwardauth.address=http://oauth2-proxy:4180"
     # ... rest of my labels
oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.4.0
    networks:
      - proxy
    expose:
      - "4180"
    volumes:
      - /var/log:/var/log
    command:
      - --cookie-secure=false
      #... rest of my commands

If you use https, you'll get the following error:

Invalid Domain for ACME Certificate: Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [oauth2-proxy-backend]: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for \"oauth2-proxy-backend\": Domain name needs at least one dot" acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["oauth2-proxy-backend"] providerName=myresolver.acme routerName=oauth2-proxy-backend@docker rule=Host(`oauth2-proxy-backend`)

oauth2-proxy was not listening on port 4180. I went into the container with

docker exec -it backend_oauth2-proxy_1 sh

and looked for open ports with

netstat -tuln

I found the following:

sudo docker exec -it backend_oauth2-proxy_1 sh
/ $ netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.11:44961        0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:4180          0.0.0.0:*               LISTEN      
udp        0      0 127.0.0.11:40876        0.0.0.0:*

oauth2-proxy was listening on port 4180 , but only on the loopback address 127.0.0.1 , not on all interfaces (0.0.0.0 ). This meant that the service was not accessible from outside this container on port 4180.

To enable Traefik to communicate with oauth2-proxy, I needed to add --http-address=0.0.0.0:4180 to my oauth2-proxy commands:

oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.4.0
    networks:
      - proxy
    expose:
      - "4180"
    volumes:
      - /var/log:/var/log
    command:
      - --http-address=0.0.0.0:4180
      # rest of my commands

After making those changes, netstat -tuln should show:

sudo docker exec -it backend_oauth2-proxy_1 sh
/ $ netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.11:39213        0.0.0.0:*               LISTEN      
tcp        0      0 :::4180                 :::*                    LISTEN      
udp        0      0 127.0.0.11:34591        0.0.0.0:*

and now I can see my POST requests from my client hit my oauth2-proxy in my Docker logs.

system · October 7, 2023, 3:06am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Help setting up ForwardAuth using Keycloak and traefik-forward-auth Traefik v2 docker	1	7562	October 9, 2022
Traefik + Oauth2-proxy : forwardauth middleware forgets URL Traefik v2 middleware	2	8252	August 23, 2022
502 bad gateway connection refused Traefik v2 docker , rest-api	3	2545	January 7, 2024
Traefik 2.0 middleware does not apply Traefik v2 docker	10	3742	October 10, 2019
Traefik2 + Nextcloud = tons of errors Traefik v2 docker	1	772	February 11, 2024

Error calling http://oauth2-proxy:4180

Related topics