HTTP entrypoint 404s with 2.2.0-rc2

Traefik Traefik v2
,
1

Hi,

Using traefik 2.2.0-rc2 with kubernetes native ingress, and both with and without middleware annotations, HTTP requests result in 404 where they did not when using traefik 2.0. There is no error in the logs, only the 404s. Traefik's dashboard indicates the ingress was configured correctly on both HTTP and HTTPS entrypoints, and when middleware annotations are added those show up as correctly configured in the dashboard as well.

Relevant entrypoints configuration:

    entryPoints:
      http:
        address: ":80"
      https:
        address: ":443"

Middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirect-to-ssl
  namespace: <namespace>
spec:
  redirectScheme:
    permanent: true
    scheme: https

Ingress annotations:

traefik.ingress.kubernetes.io/router.entrypoints: http,https
traefik.ingress.kubernetes.io/router.middlewares: <namespace>-redirect-to-ssl@kubernetescrd

Is this a case of misconfiguration or a possible bug?

2

Hello,

could you provide a reproducible example (full static configuration, full ingress files, ...)

3

Sure thing. Traefik is only started with -c /path/to/traefik.yaml, no other command line parameters. I've installed the CRDs and RBAC from here and here respectively, though with slightly different names. kubectl auth indicates they were installed correctly:

$ kubectl auth can-i get middlewares.traefik.containo.us --namespace=<namespace> --as=system:serviceaccount:<namespace>:traefik-ingress-controller
yes

traefik.yaml:

    global:
      checkNewVersion: false

    serversTransport:
      insecureSkipVerify: true

    entryPoints:
      http:
        address: ":80"
      https:
        address: ":443"
      metrics:
        address: ":8080"
      health:
        address: ":8081"
      metrics:
        address: ":8082"

    providers:
      kubernetesIngress:
        namespaces:
          - <namespace>
        ingressEndpoint:
          publishedService: <namespace>/traefik-lb
      kubernetesCRD:
        namespaces:
          - <namespace>

    api:
      insecure: true
      dashboard: true

    metrics:
      prometheus:
        entryPoint: metrics

    ping:
      entryPoint: health

    log:
      level: INFO
      format: json

    accessLog:
      format: json
      fields:
        names:
          ClientAddr: drop
          ClientPort: drop
          RequestPath: drop
          RequestProtocol: drop
          StartLocal: drop
        headers:
          names:
            Authorization: drop
            Etag: drop

    tls:
      options:
        defaultOptions:
          minVersion: VersionTLS12
          cipherSuites:
          - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
          - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
          - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
          - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
          - "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
          - "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"

ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: http,https
    traefik.ingress.kubernetes.io/router.middlewares: <namespace>-redirect-to-ssl@kubernetescrd
  labels:
    app.kubernetes.io/instance: nextcloud
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nextcloud
    helm.sh/chart: nextcloud-1.9.12
  name: nextcloud
  namespace: <namespace>
spec:
  rules:
  - host: nextcloud.<redacted>
    http:
      paths:
      - backend:
          serviceName: nextcloud
          servicePort: 8080
  tls:
  - hosts:
    - nextcloud.<redacted>
    secretName: nextcloud-tls

bit of relevant log:

{"ClientHost":"<redacted>","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":311072,"Overhead":311072,"RequestAddr":"nextcloud.<redacted>","RequestContentSize":0,"RequestCount":615,"RequestHost":"nextcloud.<redacted>","RequestMethod":"HEAD","RequestPort":"-","RequestScheme":"http","RetryAttempts":0,"StartUTC":"2020-03-12T10:15:18.551293313Z","level":"info","msg":"","time":"2020-03-12T10:15:18Z"}

Please let me know if you need anything else.

4

ok there is several mistakes in your configuration.

First you have mixed dynamic and static configuration.
The TLS section cannot be define in the static configuration.

There are 2 solutions:

I recommend the TLSOption CRD.

traefik.ingress.kubernetes.io/router.entrypoints: http,https
traefik.ingress.kubernetes.io/router.middlewares: <namespace>-redirect-to-ssl@kubernetescrd

This will not work because the TLS is on the created routers. But you cannot have TLS on HTTP.

There are 2 solutions:

I recommend the 2nd solution.

5

Thanks for pointing out my mistakes, I will rewrite the configuration to use a TLSOption resource at least. However I don't quite understand how I'm supposed to configure traefik if I don't want the http entrypoint to always redirect to https, and it's at the very least inefficient to create two ingress resources for the same target just because I want to enable both . Can you expand on that?

Additionally, without the annotations on the ingress resource requests on HTTP also result in a 404. Considering that this does work on both traefik 2.0 and 2.1 using the same configuration it seems like a regression.

6

Hi @ncgee,

I tested a simple ingress without annotation like :

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: whoami
  labels:
    app: whoami
#  annotations:
#    kubernetes.io/ingress.class: 'traefik'
spec:
  rules:
    - host: whoami
      http:
        paths:
          - backend:
              serviceName: whoami
              servicePort: 80

And it works for both the traefik v2.0 and v2.1

Could you provide some logs and the configuration you used ?

7

Hi @jbd,

The problem occurs when using traefik 2.2.0-rc2, not with 2.0 or 2.1. I wanted to try 2.2 to test the newly supported annotations. I used the same configuration as in my previous post, both with and without the annotations on the Ingress resource the result was a 404 on the http entrypoint. The log line I provided in my previous post is associated with the problem:

{"ClientHost":"<redacted>","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":311072,"Overhead":311072,"RequestAddr":"nextcloud.<redacted>","RequestContentSize":0,"RequestCount":615,"RequestHost":"nextcloud.<redacted>","RequestMethod":"HEAD","RequestPort":"-","RequestScheme":"http","RetryAttempts":0,"StartUTC":"2020-03-12T10:15:18.551293313Z","level":"info","msg":"","time":"2020-03-12T10:15:18Z"}

Just tried it again, with fixed TLS configuration as previously pointed out by @ldez, with the same result (again no annotations on the ingress):

{"ClientHost":"<redacted>","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":123317,"Overhead":123317,"RequestAddr":"nextcloud.<redacted>","RequestContentSize":0,"RequestCount":8,"RequestHost":"nextcloud.<redacted>","RequestMethod":"HEAD","RequestPort":"-","RequestScheme":"http","RetryAttempts":0,"StartUTC":"2020-03-12T18:05:06.753721713Z","level":"info","msg":"","time":"2020-03-12T18:05:06Z"}

Merely changing the image yields different results. Before, running 2.0.1:

$ curl -I nextcloud.<redacted>
HTTP/1.1 302 Found

After the update to 2.2.0-rc2:

$ curl -I nextcloud.<redacted>
HTTP/1.1 404 Not Found

There are no other errors in the traefik logs besides warnings about the deprecated apiVersion of the ingress objects (incidentally also for one that was created with networking.k8s.io/v1beta1).

8

Ok, I reproduced the issue with the v2.2.0-rc2 on a simple ingress with a TLS configuration and without any annotation.
I'm investigating more deeply to find out how it happens.
Thanks for testing the RC :slight_smile:

9

After investigating, with the same ingress configuration:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: whoami
  labels:
    app: whoami
#  annotations:
#    kubernetes.io/ingress.class: 'traefik'
spec:
  rules:
    - host: whoami
      http:
        paths:
          - backend:
              serviceName: whoami
              servicePort: 80
  tls:
    - secretName: traefik-ui-tls-cert

Before the v2.2.0-rc1, two routers were created one HTTP and one HTTPS and it was a bug.
With the RC, only one HTTPS router is created, as there is only one TLS ingress, so you can't access the service with an HTTP request.
So, the v2.2.0-rc2 has the right behavior.

10

@jbd I'm also currently testing the 2.2.0-rc2. Can you elaborate how you would handle http to https redirect without using IngressRoute? For me It's really a hard nut to crack.

11

Thanks for looking into it, but to be honest I don't understand why the behavior in 2.2.0-rc2 would be correct. As stated in the Kubernetes API reference, it assumes both port 80 and 443 for ingress objects:

Currently the port of an Ingress is implicitly :80 for http and :443 for https

To disable the router on port 80 if TLS is enabled more or less breaks compatibility with native Kubernetes ingress. It's also a pretty big deviation from how every previous version of Traefik handled ingress objects.

I'm genuinely sorry to say that if this going to be how Traefik functions going forward I will have to look at alternatives. Up to the release of 2.x I've had a very pleasant experience with it, and the (re-)addition of annotation support on native ingress objects made it seem like I could finally move ahead with replacing 1.7 (being unable to completely buy into the CRD method), but being forced to double up on ingress objects just to support basic functionality like a (non-global) redirect to HTTPS makes it hard to keep using Traefik.

12

You don't need to say that, we listen the feedback of the community.
The prove: we added the annotations in the v2.2 because a lot of users requested that.

You explained your problem and we will analyze that, the goal of a RC phase is to get these kind of feedback.

13

@ldez I apologize if my response came across as some sort of vague threat, that was not my intention. Your efforts towards the community is one of the reasons I've had such a pleasant experience with Traefik so far (I should have added that to my previous response).

However, @jbd saying v2.2.0-rc2 has the right behavior does not indicate to me that it is still under investigation. If it is, great, if it isn't, I'd like to understand why it's considered desirable behavior, and why a simple HTTP-to-HTTPS redirect has to go from a single annotation (ingress.kubernetes.io/redirect-to-ssl: true) on an ingress object to such strange "workarounds" as duplicating ingresses.

14

It's a bit more subtle:

Redirection, defined on an entry point, creates a simple router with the "HostRegexp({host:.+})" rule on the entry point web and redirection middleware.

This router will catch all request on entry point web but you can override this behavior.

As it is only a simple router, you can play with the priorities: if you want to expose an application on the entry point web, you just have to define a higher priority traefik.ingress.kubernetes.io/router.priority: "42" on this entry, and/or you can also restrict the point entry applied on the router.

So the redirect will be applied on all the ingress except for the ingress that you don't want a redirect.

If you prefer it's a kind the opposite of the non-standard ingress.kubernetes.io/redirect-to-ssl annotation.

From my point of view, it is safer to redirect by default, and to allow only when you want a non-TLS connection.

15

Thanks a bunch for elaborating.

This explains a lot, and I would agree that it's safer to redirect to HTTPS by default but unfortunately we don't always have that option with every application in the Kubernetes ecosystem, especially when those applications generate and create ingress objects for example.

I will look into leveraging priorities to achieve what we want. If I remember correctly Traefik will sensibly merge routers for overlapping host + path combinations, is that the case? If so, would creating a high-priority router with a widely matching host (like your example HostRegexp({host:.+})) and a specific path (e.g. PathPrefix('/foo') make sure we don't get redirects to HTTPS for ingress objects with more specific hosts and a matching path (e.g. Host('example.com') && Path('/foo/bar'), but without a priority annotation?

16

To explain why I said that the previous version exhibits a wrong behavior :
In the previous versions, for an ingress with TLS defined, Traefik creates two routers, one with TLS and one without TLS.
If no entrypoint is defined, both the routers will listen on all of them.
So we can request both in HTTP and HTTPS on all entrypoints ports that is pretty confusing and not secured.

17

@ncgee stay tune, we are thinking about a way to improve the behavior.

18

Ah, I understand now, that makes sense.

Thanks again!

19

You can try the rc3 with the following configuration:

    entryPoints:
      http:
        address: ":80"
      https:
        address: ":443"
        http:
          tls: {}
20

I've just implemented your suggested configuration with the expected result, requests to port 80 no longer result in a 404. Thanks for your efforts on this!

I'm not sure if this is an unintended side effect, but using the redirect middleware now via the router.middlewares annotation also has the expected result, even though the middleware is applied to the https entrypoint/router as well:

$ curl -v http://<redacted>
* Rebuilt URL to: http://<redacted>
*   Trying <redacted>...
* Connected to <redacted> (<redacted>) port 80 (#0)
> GET / HTTP/1.1
> Host: <redacted>
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Location: https://<redacted>
< Date: Fri, 27 Mar 2020 10:27:46 GMT
< Content-Length: 17
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host <redacted> left intact
Moved Permanently

Curiously, I get a 308 with a HEAD request. There is no global redirect in the configuration.