Using ExternalNames results in many Requests all answered with 499

Traefik Traefik v2 (latest)
,
1

Hi,

I want to delegate an incoming request to another service with a different domain.
Traefik-Problem-01

As I am developing this scenario in one cluster, I use one Traefik to serve both domains.
Traefik-Problem-02

When sending a request to "https://publicdomain/myapp" the request never ends and I have the following log entries in the Traefik log.

About 160 entries of the follwing two lines:

time="2022-04-05T07:52:50Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/myapp/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Authorization\":[\"Bearer XXX\"],\"Cookie\":[\"vsid=XXX\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"99\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36\"],\"X-Forwarded-Host\":[\"publicdomain\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"ingress-external-traefik-567d99474f-2l8ph\"],\"X-Real-Ip\":[\"10.212.88.5\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"publicdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.212.88.5:26370\",\"RequestURI\":\"/myapp/\",\"TLS\":null}"
time="2022-04-05T07:52:50Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="https://internaldomain:443" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/myapp/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Authorization\":[\"Bearer XXX\"],\"Cookie\":[\"vsid=XXX\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"99\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36\"],\"X-Forwarded-Host\":[\"publicdomain\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"ingress-external-traefik-567d99474f-2l8ph\"],\"X-Real-Ip\":[\"10.212.88.5\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"publicdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.212.88.5:26370\",\"RequestURI\":\"/myapp/\",\"TLS\":null}"

And then about 2000 entries of the following 4 lines

time="2022-04-05T07:52:55Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
time="2022-04-05T07:52:55Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/myapp/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Authorization\":[\"Bearer XXX\"],\"Cookie\":[\"vsid=XXX\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"99\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36\"],\"X-Forwarded-Host\":[\"publicdomain\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"ingress-external-traefik-567d99474f-2l8ph\"],\"X-Real-Ip\":[\"10.212.88.5\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"publicdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.212.88.5:7330\",\"RequestURI\":\"/myapp/\",\"TLS\":null}"
time="2022-04-05T07:52:55Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
10.212.88.5 - - [05/Apr/2022:07:52:12 +0000] "GET /myapp/ HTTP/2.0" 499 21 "-" "-" 14009 "default-myapp-service-eu-ingress-58e1f2f9f4d33d141d61@kubernetescrd" "https://internaldomain:443" 42830ms

I have no idea why so many requests are triggerd internally and why they all result in 499.

I use the following configuration:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik-external
  labels:
    name: myapp-ingress
  name: myapp-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: >-
        Host(`publicdomain`) &&
        PathPrefix(`/myapp`)
      middlewares:
        - name: header-middleware-default
          namespace: default
        - name: error-middleware-default
          namespace: default
        - name: auth-middleware-default
          namespace: default
      services:
        - kind: Service
          name: myapp-proxy-service
          namespace: default
          port: 443
          serversTransport: myapp-eu-transport
  tls: {}
  
---

apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: myapp-transport
  namespace: default
spec:
  insecureSkipVerify: true
  rootCAsSecrets:
    - myapp-internal-ca-certificate
  serverName: internaldomain

---

kind: Service
apiVersion: v1
metadata:
  name: myapp-proxy-service
  namespace: default
spec:
  type: ExternalName
  externalName: internaldomain
  
---

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  namespace: default
  labels:
    name: myapp-service
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
  selector:
    app: myapp-service
    name: myapp-pod
  type: ClusterIP

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik-external
  labels:
    name: myapp-ingress
  name: myapp-ingress
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: >-
        Host(`internaldomain`) &&
        PathPrefix(`/myapp`)
      middlewares:
        - name: header-middleware-default
          namespace: default
        - name: error-middleware-default
          namespace: default
        - name: myapp-service-rewrite
          namespace: default
      services:
        - kind: Service
          name: myapp-service
          namespace: default
          port: 443
          serversTransport: myapp-service-transport
  tls: {}

Thanks for your help and kindest regards,
Andi

2

Problem solved.

The ExternalName changes the domain into an IP-address.
So the rule for internaldomain does not match.
Adding a Host Header with a middleware solved the problem.

No idea why this results in the Http Status Code 499 and so much requests.

2 Likes
3

What host header did you add exactly? How does your solution look like in config? Thank you.