Traefik returns 500 error after 30s requesting page

Traefik Traefik v2
,
1

Hi all, I need your help with this issue.

I am setting an authentication service in docker swarm environment using Keycloak as a provider and OAuth2-proxy with forwardauth and errors middlewares. Actually, I have set up these services for 3 servers and they are working properly but when I try to set it up on a new server, I got 500 error HTTP .

There is no more log information about the error, when I looked at traefik look I only saw.

time="2022-09-26T04:29:39Z" level=debug msg="Error calling https://oauth.mydomain.org/oauth2/auth. Cause: Get \"https://oauth.mydomain.org/oauth2/auth\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" middlewareName=oauth@docker middlewareType=ForwardedAuthType
14.168.51.149 - - [26/Sep/2022:04:29:09 +0000] "GET / HTTP/2.0" 500 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2412 "auth_testing@docker" "-" 30000ms
time="2022-09-26T04:29:41Z" level=debug msg="Error calling https://oauth.mydomain.org/oauth2/auth. Cause: Get \"https://oauth.mydomain.org/oauth2/auth\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" middlewareType=ForwardedAuthType middlewareName=oauth@docker

Also attach my configuration for my testing service and oauth:

  oauth:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.1.1
    deploy:
      placement:
        constraints: [node.labels.web==web]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.oauth.rule=Host(`oauth.${HOST_ABBR}.domain.org`) || PathPrefix(`/oauth2`)"
        - "traefik.http.routers.oauth.entrypoints=websecure"
        - "traefik.http.routers.oauth.tls=true"
        - "traefik.http.routers.oauth.tls.certresolver=leresolver"
        # Set up service
        - "traefik.http.routers.oauth.service=oauth@docker"
        - "traefik.http.services.oauth.loadbalancer.server.port=4185"
        # Set up middlewares
        - 'traefik.http.middlewares.oauth.forwardauth.address=https://oauth.${HOST_ABBR}.doamin.org/oauth2/auth'
        - 'traefik.http.middlewares.oauth.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.oauth.forwardauth.authResponseHeaders=X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email, X-Auth-Request-Preferred-Username, X-Auth-Request-Access-Token'
        - "traefik.http.middlewares.oauth-signin.errors.service=oauth@docker"
        - "traefik.http.middlewares.oauth-signin.errors.status=401,403"
        - "traefik.http.middlewares.oauth-signin.errors.query=/oauth2/sign_in"
    environment:
      OAUTH2_PROXY_CLIENT_ID: 'coffee_authentication'
      OAUTH2_PROXY_CLIENT_SECRET: 'xxxxxxx99af13b51' 
      
      OAUTH2_PROXY_PROVIDER: 'keycloak'
      OAUTH2_PROXY_SCOPE: 'openid profile email'
      OAUTH2_PROXY_OIDC_ISSUER_URL: '${MY_DOMAIN}/auth/realms/coffee'
      OAUTH2_PROXY_LOGIN_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/auth'
      OAUTH2_PROXY_REDEEM_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/token'
      OAUTH2_PROXY_PROFILE_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/userinfo'
      OAUTH2_PROXY_VALIDATE_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/userinfo'
      
      OAUTH2_PROXY_PASS_ACCESS_TOKEN: 'true'
      OAUTH2_PROXY_PASS_USER_HEADERS: 'true'
      OAUTH2_PROXY_PASS_BASIC_AUTH: 'true'
      OAUTH2_PROXY_SET_XAUTHREQUEST: 'true'
      OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: 'true'
      OAUTH2_PROXY_COOKIE_DOMAINS: '${MY_DOMAIN}'
      OAUTH2_PROXY_HTTP_ADDRESS: '0.0.0.0:4185'
      OAUTH2_PROXY_COOKIE_REFRESH: '12h'
      OAUTH2_PROXY_COOKIE_SECURE: 'false'
      # python -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'
      OAUTH2_PROXY_COOKIE_SECRET: '${SECRET_COOKIE}'
      OAUTH2_PROXY_AUTHENTICATED_EMAILS_FILE: '/oauth-conf/authorized_emails.txt'
      #OAUTH2_PROXY_EMAIL_DOMAINS: '*'
      OAUTH2_PROXY_WHITELIST_DOMAINS: '*'
      OAUTH2_PROXY_REVERSE_PROXY: 'true'

And here is my service with traefik labels:

      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.auth_testing.rule=Host(`test.${HOST_ABBR}.domain.org`)"
        - "traefik.http.routers.auth_testing.entrypoints=websecure"
        - "traefik.http.routers.auth_testing.tls=true"
        - "traefik.http.routers.auth_testing.tls.certresolver=leresolver"
        #- "traefik.http.middlewares.auth_testing_auth.basicauth.users=${HOST_USER}:${HOST_HASHED_PASS}"
        #- "traefik.http.routers.auth_testing.middlewares=auth_testing_auth"
        - "traefik.http.routers.auth_testing.middlewares=oauth-signin,oauth"
        # Set up service
        - "traefik.http.services.auth_testing_svc.loadbalancer.server.port=5000"
        - "traefik.http.routers.auth_testing.service=auth_testing_svc"

Hope you can help, I have encountered this issue for a week and still checking it.

Thank you so much.

2

This could be a Docker Swarm configuration issue. I noticed the constraint, did you set the required node label on your new server? Did you check that the oauth service is successfully running? Is the oauth service reachable from within the Traefik container?

3

thank you for your suggestion :grin: . I'll check my ansible config again

4

Somehow the Traefik in this server could not route requests to OAuth with HTTPS URL https://oauth.${HOST_ABBR}.doamin.org/oauth2/auth, so I tried with the internal request: http://oauth:4185/oauth2/auth and it works for me.

Replace this line

#'traefik.http.middlewares.oauth.forwardauth.address=https://oauth.${HOST_ABBR}.mydomain.org/oauth2/auth'

By

- 'traefik.http.middlewares.oauth.forwardauth.address=http://oauth:4185/oauth2/auth'
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.