Can Traefik handle DevOps Server?

Traefik Traefik v3 (latest)
1

I'm currently using a docker-based nginx proxy server with LE companion. This handles our docker-based services (as well as one additional non-docker internal service).

However... I'm not entirely satisfied with nginx and I'm researching other options. For one thing traefik has support for docker services with label-defined parameters (great!) which can work alongside dynamic file-based configurations (double great!). That'd make my life a little bit easier vs our existing nginx, but it's not a big enough reason to switch on its own.

There's however one service we're currently not using nginx for and that's our local MS DevOps Server. It's being handled by our firewall (Sophos UTM, using Apache under the hood AFAIK) and it's kind of killing our firewall, sadly. I'd like to move it to a docker-based proxy where I have a lot more freedom in assigning resources (CPU/RAM).

To that end - can Traefik handle services using NTLM authentication? I know the free version of nginx can't do that (only the paid version can), so that's a bust.

Also, I'd need to permit things like websockets (I think Traefik handles those, so that shouldn't be a problem) and have control over any limits (ex. body size; DevOps hosts nuget packages, and those CAN be a bit larger in size).

Can Traefik handle this use-case? I'd rather know in advance if I should start learning the ropes and whipping up configurations...

2

It seems like Traefik cannot handle NTLM properly, unless there are some hidden requirements. :frowning:

I've whipped up a quick config (with a bit of hackery to get things going on our internal network) and I'm essentially endlessly getting the NTLM login prompt from DevOps.

The config I'm testing with:

http:
  middlewares:
    limit:
      buffering:
        maxRequestBodyBytes: 20000000
  routers:
    to-devops:
      rule: "Host(`devops.domain.local`)"
      middlewares:
        - limit
      service: devops
      entrypoints: websecure
      tls:
        certResolver: le
  services:
    devops:
      loadBalancer:
        serversTransport: devops-transport
        servers:
        - url: https://<our.public.devops.site>
  serversTransports:
    devops-transport:
      disableHTTP2: true
      insecureSkipVerify: true

Note that the DevOps server has also been configured (IIS) to handle devops.domain.local, and I've set up devops.domain.local in my own HOSTS file for testing...

3

I don’t see any NTLM specifics in the dynamic config. How does it work?

Enable and check Traefik debug log and Traefik access log in JSON format.

4

I'm not sure I understand your question. How does NTLM work in general? From what I understand NTLM is Microsoft's login scheme which violates the HTTP standard as it's stateful.

If you're asking about NTLM-specific settings for Traefik - then I couldn't find anything in the documentation, so I'm not sure how that's supposed to work to be perfectly honest nor what I'd need to put into the config.

I also found this thread, but it's:

  1. for Traefik 2, not 3 (however I'd expect newer versions to have more features, not less, so I don't think it matters)
  2. one person claims NTLM works with Traefik, but then a lot of questions are posted and at the end there's no definitive answer...

Anyway, I've attempted to replicate the suggestions from that thread by adding a TCP router and I'm still coming up empty.

New dynamic configuration:

http:
  middlewares:
    limit:
      buffering:
        maxRequestBodyBytes: 20000000
  routers:
    to-devops:
      rule: "Host(`devops.domain.local`)"
      middlewares:
        - limit
      service: devops
      entrypoints: websecure
      tls:
        certResolver: le
  services:
    devops:
      loadBalancer:
        serversTransport: devops-transport
        servers:
        - url: https://<our current public devops site>
  serversTransports:
    devops-transport:
      disableHTTP2: true
      insecureSkipVerify: true

tcp:
  routers:
    to-devops:
      rule: HostSNI(`devops.domain.local`)
      entrypoints: websecure
      service: devops
      tls:
        passthrough: true
  services:
    devops:
      loadBalancer:
        serversTransport: devops-transport
        servers:
        - address: "<our current public devops site>:443"
  serversTransports:
    devops-transport:
      insecureSkipVerify: true

Logs:

{"level":"info","version":"3.1.2","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/cmd/traefik/traefik.go:101","message":"Traefik version 3.1.2 built on 2024-08-06T13:37:51Z"}
{"level":"debug","staticConfiguration":{"global":{"checkNewVersion":true},"serversTransport":{"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"},"entryPoints":{"traefik":{"address":":8080","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"readTimeout":"1m0s","idleTimeout":"3m0s"}},"forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"udp":{"timeout":"3s"}},"web":{"address":":80","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"readTimeout":"1m0s","idleTimeout":"3m0s"}},"forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"udp":{"timeout":"3s"}},"websecure":{"address":":443","transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"readTimeout":"1m0s","idleTimeout":"3m0s"}},"forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"udp":{"timeout":"3s"}}},"providers":{"providersThrottleDuration":"2s","docker":{"watch":true,"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock"},"file":{"directory":"/services/","watch":true}},"api":{"insecure":true,"dashboard":true},"log":{"level":"DEBUG","format":"json","filePath":"/var/log/traefik.log"},"certificatesResolvers":{"le":{"acme":{"email":"admin@arpideas.pl","caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","storage":"acme.json","keyType":"RSA4096","certificatesDuration":2160,"httpChallenge":{"entryPoint":"web"}}}}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108","message":"Static configuration loaded [json]"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/cmd/traefik/traefik.go:617","message":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73","message":"Starting provider aggregator aggregator.ProviderAggregator"}
{"level":"debug","entryPointName":"traefik","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:228","message":"Starting TCP Server"}
{"level":"debug","entryPointName":"web","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:228","message":"Starting TCP Server"}
{"level":"debug","entryPointName":"websecure","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:228","message":"Starting TCP Server"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202","message":"Starting provider *file.Provider"}
{"level":"debug","config":{"directory":"/services/","watch":true},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203","message":"*file.Provider provider configuration"}
{"level":"debug","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/file/file.go:122","message":"add watcher on: /services/"}
{"level":"debug","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/file/file.go:122","message":"add watcher on: /services/devops.yml"}
{"level":"debug","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/file/file.go:122","message":"add watcher on: /services/sup.yml"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202","message":"Starting provider *traefik.Provider"}
{"level":"debug","config":{},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203","message":"*traefik.Provider provider configuration"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202","message":"Starting provider *docker.Provider"}
{"level":"debug","config":{"watch":true,"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock"},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203","message":"*docker.Provider provider configuration"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202","message":"Starting provider *acme.ChallengeTLSALPN"}
{"level":"info","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202","message":"Starting provider *acme.Provider"}
{"level":"debug","config":{"email":"admin@arpideas.pl","caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","storage":"acme.json","keyType":"RSA4096","certificatesDuration":2160,"httpChallenge":{"entryPoint":"web"},"ResolverName":"le","store":{},"TLSChallengeProvider":{},"HTTPChallengeProvider":{}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203","message":"*acme.Provider provider configuration"}
{"level":"debug","config":{},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203","message":"*acme.ChallengeTLSALPN provider configuration"}
{"level":"debug","providerName":"file","config":{"http":{"routers":{"to-devops":{"entryPoints":["websecure"],"middlewares":["limit"],"service":"devops","rule":"Host(`devops.domain.local`)","tls":{"certResolver":"le"}},"to-sup":{"entryPoints":["web"],"service":"sup","rule":"Host(`sup.arp.local`)"}},"services":{"devops":{"loadBalancer":{"servers":[{"url":"https://<public devops site>"}],"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"serversTransport":"devops-transport"}},"sup":{"loadBalancer":{"servers":[{"url":"https://10.150.1.50:4443/"}],"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"serversTransport":"sup-transport"}}},"middlewares":{"limit":{"buffering":{"maxRequestBodyBytes":20000000}}},"serversTransports":{"devops-transport":{"disableHTTP2":true},"sup-transport":{"insecureSkipVerify":true}}},"tcp":{"routers":{"to-devops":{"entryPoints":["websecure"],"service":"devops","rule":"HostSNI(`devops.domain.local`)","tls":{"passthrough":true}}},"services":{"devops":{"loadBalancer":{"servers":[{"address":"<public devops site>:443","tls":true}]}}}},"udp":{},"tls":{}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227","message":"Configuration received"}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213","message":"Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\""}
{"level":"info","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:795","message":"Testing certificate renew..."}
{"level":"debug","providerName":"internal","config":{"http":{"routers":{"acme-http":{"entryPoints":["web"],"service":"acme-http@internal","rule":"PathPrefix(`/.well-known/acme-challenge/`)","ruleSyntax":"v3","priority":9223372036854775807},"api":{"entryPoints":["traefik"],"service":"api@internal","rule":"PathPrefix(`/api`)","ruleSyntax":"v3","priority":9223372036854775806},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"service":"dashboard@internal","rule":"PathPrefix(`/`)","ruleSyntax":"v3","priority":9223372036854775805}},"services":{"acme-http":{},"api":{},"dashboard":{},"noop":{}},"middlewares":{"dashboard_redirect":{"redirectRegex":{"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/","permanent":true}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"udp":{},"tls":{}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227","message":"Configuration received"}
{"level":"debug","providerName":"le.acme","config":{"http":{},"tcp":{},"udp":{},"tls":{}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227","message":"Configuration received"}
{"level":"debug","providerName":"docker","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:90","message":"Provider connection established with docker 27.1.1 (API 1.46)"}
{"level":"debug","providerName":"docker","container":"traefik-traefik-db2779adade0b82438ac34c775df7da91b1939618731751a62458567dab9c7e5","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184","message":"Filtering disabled container"}
{"level":"debug","providerName":"docker","config":{"http":{"routers":{"whoami":{"entryPoints":["web"],"service":"whoami-traefik","rule":"Host(`whoami.arp.local`)"}},"services":{"whoami-traefik":{"loadBalancer":{"servers":[{"url":"http://172.18.0.2:80"}],"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"}}}}},"tcp":{},"udp":{},"tls":{}},"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227","message":"Configuration received"}
{"level":"debug","tlsStoreName":"default","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321","message":"No default certificate, fallback to the internal generated certificate"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_stripprefix@internal","middlewareType":"StripPrefix","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32","message":"Creating middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_stripprefix@internal","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17","message":"Creating middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18","message":"Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"traefik","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"web","routerName":"to-sup@file","serviceName":"sup@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:267","message":"Creating load-balancer"}
{"level":"debug","entryPointName":"web","routerName":"to-sup@file","serviceName":"sup@file","serverName":"d0a0a1018f9292a9","target":"https://10.150.1.50:4443/","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:309","message":"Creating server"}
{"level":"debug","entryPointName":"web","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:267","message":"Creating load-balancer"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","serverName":"418fe3cf165a5b76","target":"https://<public devops site>","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:309","message":"Creating server"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","middlewareType":"Buffer","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/buffering/buffering.go:27","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","middlewareType":"Buffer","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/buffering/buffering.go:28","message":"Setting up buffering: request limits: 0 (mem), 20000000 (max), response limits: 0 (mem), 0 (max) with retry: ''"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"websecure","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237","message":"Adding route for devops.domain.local with TLS options default"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/tcp/service.go:95","message":"Creating TCP server"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:332","message":"Adding Passthrough route for \"HostSNI(`devops.domain.local`)\""}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:384","message":"Trying to challenge certificate for domain [devops.domain.local] found in HostSNI rule"}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:851","message":"Looking for provided certificate(s) to validate [\"devops.domain.local\"]..."}
{"level":"debug","tlsStoreName":"default","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321","message":"No default certificate, fallback to the internal generated certificate"}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","domains":["devops.domain.local"],"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:897","message":"Domains need ACME certificates generation for domains \"devops.domain.local\"."}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:619","message":"Loading ACME certificates [devops.domain.local]..."}
{"level":"debug","entryPointName":"web","routerName":"whoami@docker","serviceName":"whoami-traefik@docker","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:267","message":"Creating load-balancer"}
{"level":"debug","entryPointName":"web","routerName":"whoami@docker","serviceName":"whoami-traefik@docker","serverName":"b9597fe0f5ca1856","target":"http://172.18.0.2:80","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:309","message":"Creating server"}
{"level":"debug","entryPointName":"web","routerName":"to-sup@file","serviceName":"sup@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:267","message":"Creating load-balancer"}
{"level":"debug","entryPointName":"web","routerName":"to-sup@file","serviceName":"sup@file","serverName":"d0a0a1018f9292a9","target":"https://10.150.1.50:4443/","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:309","message":"Creating server"}
{"level":"debug","entryPointName":"web","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_stripprefix@internal","middlewareType":"StripPrefix","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32","message":"Creating middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_stripprefix@internal","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17","message":"Creating middleware"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","middlewareType":"RedirectRegex","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18","message":"Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/"}
{"level":"debug","entryPointName":"traefik","routerName":"dashboard@internal","middlewareName":"dashboard_redirect@internal","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"traefik","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:267","message":"Creating load-balancer"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","serverName":"418fe3cf165a5b76","target":"https://<public devops site>","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/service.go:309","message":"Creating server"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","middlewareType":"Buffer","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/buffering/buffering.go:27","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","middlewareType":"Buffer","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/buffering/buffering.go:28","message":"Setting up buffering: request limits: 0 (mem), 20000000 (max), response limits: 0 (mem), 0 (max) with retry: ''"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","middlewareName":"limit@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33","message":"Adding tracing to middleware"}
{"level":"debug","entryPointName":"websecure","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22","message":"Creating middleware"}
{"level":"debug","entryPointName":"websecure","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237","message":"Adding route for devops.domain.local with TLS options default"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","serviceName":"devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/tcp/service.go:95","message":"Creating TCP server"}
{"level":"debug","entryPointName":"websecure","routerName":"to-devops@file","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:332","message":"Adding Passthrough route for \"HostSNI(`devops.domain.local`)\""}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:384","message":"Trying to challenge certificate for domain [devops.domain.local] found in HostSNI rule"}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:851","message":"Looking for provided certificate(s) to validate [\"devops.domain.local\"]..."}
{"level":"debug","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","domains":["devops.domain.local"],"time":"2024-08-21T07:34:44Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:895","message":"No ACME certificate generation required for domains"}
{"level":"debug","providerName":"le.acme","time":"2024-08-21T07:34:45Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251","message":"Building ACME client..."}
{"level":"debug","providerName":"le.acme","time":"2024-08-21T07:34:45Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257","message":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"info","providerName":"le.acme","time":"2024-08-21T07:34:46Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:371","message":"Register..."}
{"level":"debug","lib":"lego","time":"2024-08-21T07:34:46Z","caller":"github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48","message":"[INFO] acme: Registering account for admin@arpideas.pl"}
{"level":"debug","providerName":"le.acme","time":"2024-08-21T07:34:46Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:322","message":"Using HTTP Challenge provider."}
{"level":"debug","lib":"lego","time":"2024-08-21T07:34:46Z","caller":"github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48","message":"[INFO] [devops.domain.local] acme: Obtaining bundled SAN certificate"}
{"level":"error","providerName":"le.acme","acmeCA":"https://acme-staging-v02.api.letsencrypt.org/directory","providerName":"le.acme","ACME CA":"https://acme-staging-v02.api.letsencrypt.org/directory","routerName":"to-devops@file","rule":"Host(`devops.domain.local`)","error":"unable to generate a certificate for the domains [devops.domain.local]: acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Invalid identifiers requested :: Cannot issue for \"devops.domain.local\": Domain name does not end with a valid public suffix (TLD)","domains":["devops.domain.local"],"time":"2024-08-21T07:34:47Z","caller":"github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:396","message":"Unable to obtain ACME certificate for domains"}
{"level":"debug","time":"2024-08-21T07:34:48Z","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"devops.domain.local\""}
{"time":"2024-08-21T07:34:48Z","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.18.0.1:57718: remote error: tls: unknown certificate"}
{"level":"debug","time":"2024-08-21T07:34:51Z","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"devops.domain.local\""}
{"time":"2024-08-21T07:34:51Z","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.18.0.1:57734: remote error: tls: unknown certificate"}
{"level":"debug","time":"2024-08-21T07:34:51Z","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"devops.domain.local\""}
{"level":"debug","time":"2024-08-21T07:34:51Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 418fe3cf165a5b76"}
{"level":"debug","time":"2024-08-21T07:34:53Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 418fe3cf165a5b76"}
{"level":"debug","time":"2024-08-21T07:34:55Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 418fe3cf165a5b76"}
{"level":"debug","time":"2024-08-21T07:34:55Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 418fe3cf165a5b76"}
{"level":"debug","time":"2024-08-21T07:34:55Z","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 418fe3cf165a5b76"}

Right now if I attempt to open https://devops.domain.local I get a 404 response...

5

I'm not sure I can post the logs, as my last post got removed by some anti-spam mechanism on this forum when I did that.

But what specifics would I need to set anyway? I couldn't find anything NTLM-related in the documentation...

6

How does NTLM work? Shouldn’t the target app use it internally for auth. with a backend? Has it anything to do with a proxy between client/browser and app?

7

The DevOps server itself does exactly that; it'll handle the authentication with the required backend all on its own.

However NTLM is... well, it's a bit of Microsoft authentication hackery that doesn't follow proper HTML standards and I suspect that's the main problem, tripping proxy servers like nginx or Traefik (at least without dedicated modules / confiugrations). In very short - NTLM breaks the statelessness of HTML.

8

Traefik Proxy can forward request compatible with NTLM. You'll need to disable HTTP2, see Traefik tries to use HTTP/2 with NTLM · Issue #6608 · traefik/traefik · GitHub and Add ability to disable HTTP/2 in dynamic config by jcuzzi · Pull Request #7645 · traefik/traefik · GitHub

9

I already did that; see my configuration.

10

I've managed to get some kind of initial setup going, but there's an annoying issue with it.

First the setup:

http:
 routers:
   to-devops:
     rule: "Host(`devops-test.public.com`)"
     service: devops
     entrypoints: websecure
 services:
   devops:
     loadBalancer:
       serversTransport: devops-transport
       sticky: true
       servers:
       - url: https://devops.domain.local
 serversTransports:
   devops-transport:
     disableHTTP2: true
     insecureSkipVerify: true

tcp:
 routers:
   to-devops:
     rule: HostSNI(`devops-test.public.com`)
     entrypoints: websecure
     service: devops
     tls:
       passthrough: true
 services:
   devops:
     loadBalancer:
       servers:
       - address: "devops.domain.local:443"
         tls: true

The above does appear to work. I need to do extensive testing if the user sessions do not bleed from one user to another (which was an issue when I tried something similar back in the day with haproxy).

However there's an annoying issue with initial user experience - the user will get the NTLM sign-in twice. The first one is always ignored (even if correct credentials are provided). I've no clue why that is... What's even crazier is that I've tried inspecting what's going on with Fiddler... and when using Fiddler I did NOT get a dual-login prompt!

I'm also not sure if the TCP setup is required for this; I suppose if there are any websocket-type elements in DevOps then it is?

11

WebSockets usually work fine with http.

Note that sticky: true seems useless, as you only have one target address.

12

You're correct - the sticky flag was pointless. I was using tcp because of this thread, where someone wrote the following:

I can confirm that Windows Integrated authentication works successfully with Traefik 2.0 using TCP routers with successful logins proven on Windows/Mac using Safari/Chrome/IE.

... but perhaps it was a Traefik v2 thing.

However, I'm still left with the weird two-time logon issue, and that's going to be an issue with... well, everything (in this proxy case at least) - since MS DevOps is also used for GIT as well as Nuget operations - if the first request always ends up failing this is bound to cause a lot of problems.