Issue report instances unavailable. Tokens don’t verify

Traefik Pilot
#1

Report a problem

Steps to reproduce

  1. On page pilot.traefik.io
  2. When I click on instances it says error fetching instances.
  3. This also has blocked all incoming traffic to our k8s as it can’t verify the token.

Expected behavior
Show instances and verify tokens
Current behavior
No instances and k8s doesn’t verify tokens.

Version: Apple Computer, Inc. 5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1

2 Likes
#2 
"Configuration loaded from flags."                                                                                ││ time="2021-01-28T21:43:02Z" level=info msg="Traefik version 2.3.1 built on 2020-09-29T15:49:06Z"                                                             ││ time="2021-01-28T21:43:02Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTranspo ││ time="2021-01-28T21:43:02Z" level=info msg="Stats collection is enabled."                                                                                    ││ time="2021-01-28T21:43:02Z" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your  ││ time="2021-01-28T21:43:02Z" level=info msg="Help us improve Traefik by leaving this feature on :)"                                                           ││ time="2021-01-28T21:43:02Z" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"                                   ││ time="2021-01-28T21:43:02Z" level=debug msg="loading of plugin: pathauth: github.com/aarlint/pathauth@v0.2.3"                                                ││ 2021/01/28 21:43:03 traefik.go:76: command traefik error: failed to download plugin github.com/aarlint/pathauth: error: 400: {"error":"invalid token"}       ││ stream closed
#3

image
image1356×546 19.1 KB

1 Like
#4

I am also having this issue, exact same error and webpage. I've confirmed it's not the firewall, and as far as I can tell you can't remove and re-add the instance in Pilot when it's in this error state.

#5

I even tried to create a new account and generate new instances and you cannot at this time. I have 2 k8s clusters using pilot and a middleware I wrote. They are both in failed state after traefik container restarted.

#6

Here's what I get in the console when I visit https://pilot.traefik.io/instances, it does seem that this is a Traefik issue and not a user config issue.

PilotErrors
PilotErrors1376×314 85.7 KB

#7

it actually looks like the token that is failing is traefik pilots token to git:
failed to download plugin github.com/aarlint/pathauth: error: 400: {"error":"invalid token"}

It might also make sense if they store instance definitions in private git repos. Hopefully they find the fix. They said they are on it :slight_smile:

#8

We are on it, stay tune!

2 Likes
#9

@Aarlint So indeed we do have an issue on Pilot currently that prevent verifying tokens. Sorry for that.
What surprises me though is the fact that all your incoming traffic seems blocked. We don't get this part. Could you elaborate? What's blocked exactly? Every request? What are the associated logs and access logs?

#10

I posted them above. It starts just fine but then it says:
time="2021-01-28T22:05:29Z" level=debug msg="loading of plugin: pathauth: github.com/aarlint/pathauth@v0.2.3" ││ 2021/01/28 22:05:30 traefik.go:76: command traefik error: failed to download plugin github.com/aarlint/pathauth: error: 400: {"error":"invalid token"}

then kills the container/pod.

#11

And now its back. So whatever you did fixed it both on the k8s side and on the website

#12

To be clear, the ingress traffic wasnt working because the pod never became ready. The startup inside traefik container would fail when it tried to pull down the middleware from git.

#13

The problem is, that your instance won't start if the token can't be verified or your cluster can't reach the authentication service.
I had a similar problem due to the fact, that my cluster couldn't reach the service because it was behind a firewall.

So if you use traefik with plugins and the authentication service is down, you are out of luck.

I think it would be nice, if the traefik instance could start without verification.

#14

We may have to rethink our strategy after this too. Needing to "phone home" to start the ingress with plugins just cost a lot of downtime. Would be nice to have a grace period or some other mechanism.

#15

It should be up again folks. Sorry for the inconvenience. The healthcheck from a database we use was broken.
But still, the plugin issue that prevented to start your pod should be addressed. We will work on that.

3 Likes
#16

Thank you for the quick resolve!

But the ability to use plugins without the need to get authenticated by pilot would be really great. I always had a bad feeling about this. Besides that, I really love traefik.

1 Like
#17

So, if you use Traefik with plugins and the authentication service is down, you do not have a chance.
I think it would be fine, if the Traefik instance could start without verification.

#18

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.