Not sure if this is something I need to fix on my end, but I have an agent running in my Docker standalone, and I had a service published and it's been working fine until recently, and when I went to the service it now returns a self-signed cert.
The certificate is valid from today until next year, so something must have changed today in Traefik Hub's load balancers/proxies...
Please advise.
Hello @tbjers and thank you for your interest in Traefik Hub.
We haven't been able to reproduce the issue.
Could you provide your Traefik and Traefik Hub agent logs together with your domain name to allow us to dig deeper?
Thanks in advance.
I cannot access my Traefik logs since I am out of town and I do not have any way of exposing an ingress from here, especially since Traefik Hub has stopped working for me.
Try this URL:
https://accessible-puma-wh1td5.nhkkihsr.traefikhub.io
Chrome shows the following:
When I proceed anyway, I get a text-only response of "404 page not found."
Furthermore, I can see no statistics in my Traefik Hub performance for the service, even going out 30 days. It worked over the weekend.
This is the certificate I see:
Is this coming from Hub, or is it somehow returned from my Traefik instance in Docker? Why would it return a default cert if I have certs in the config file?
Never seen this before.
Hello @tbjers,
Thank you for the additional information.
However, to help you in a relevant way, we really need you to share more information such as the logs, the static configuration, and your docker-compose file.
Thanks in advance.
Totally understandable. I will see if I can figure out a way to get an SSH ingress into my home lab, I donät have anything set up currently. That was actually what I was trying to do when I ran into this issue because I was going to open a Cloudflared tunnel into my security camera network.
@nicomengin I sent you some more info via private message, can you inspect logs for those resources from your end? The agent is online, is there any way for you to diagnose under the hood?
Thanks for the follow-up.
We cannot inspect the agent logs from the platform, we need you to provide them to move forward.
@nicomengin Heard. I'll set a reminder for when I get home and make sure to send y'all the logs. Where's the best place to share the logs that isn't a public forum post?
I'm facing the same issue.
Even with a custom domain the self-signed default Traefik Certificate is returned.
It worked perfectly for a few weeks now.
On Sep 27 the error occured for the first time.
Since yesterday (06.10.2022) it has not recovered.
Environment:
microk8s Cluster running on a local Proxmox Host.
I tried restarting all Pods & Deployments within the traefik-hub namespace.
Even tried updating the helm Chart to it's newest version.
Upon restarting the hub-agent-tunnel the following error is recorded in the logs:
{"level":"debug","component":"tunnel-client","method":"GET","url":"https://platform.hub.traefik.io/agent/tunnel-endpoints","time":1665140194,"message":"Performing request"}
{"level":"error","error":"dial: dial tcp 10.152.183.204:9901: connect: connection refused","time":1665140209,"message":"Unable to proxy the tunnel traffic to the cluster endpoint"}
{"level":"error","error":"dial: dial tcp 10.152.183.204:9901: connect: connection refused","time":1665140210,"message":"Unable to proxy the tunnel traffic to the cluster endpoint"}
{"level":"debug","component":"tunnel-client","method":"GET","url":"https://platform.hub.traefik.io/agent/tunnel-endpoints","time":1665140254,"message":"Performing request"}
{"level":"debug","component":"tunnel-client","method":"GET","url":"https://platform.hub.traefik.io/agent/tunnel-endpoints","time":1665140314,"message":"Performing request"}
Skipping the Certificate-Error-Message returns 404.
Best regards
@tbjers, you can send me a private message.
Hello @cronixx,
The logs mean that your Traefik Hub agent cannot reach your Traefik instance to create a tunnel.
Are you using the default Hub Helm Chart?
Or are you using a custom instance of Traefik?
Could you check if the port 9901 is open and reachable on your Traefik pod?
@nicomengin Has this port changed at any point? I have made no changes to the agent or the Docker install, so not sure what happened. Has something changed upstream and we need to update the agents?
Thanks for the reply.
I'm using the default helm chart.
I just uninstalled and reinstalled the complete agent, hub components including all pods, secrets etc.
Same result.
Will check the connection within the kubernetes instance and will provide feedback tomorrow.
Without having done any additional actions, the service seems to have recovered by itself.
The correct custom-domain is provided with SSL Certificate.
@nicomengin this is a bit strange, my certs are all in order locally, I've verified all my hosts.
Here's what my traefik logs say:
# docker compose up
[+] Running 2/2
⠿ Container traefik Created 0.1s
⠿ Container hub-agent Created 0.1s
Attaching to hub-agent, traefik
traefik | time="2022-10-10T11:24:56Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
hub-agent | {"level":"info","version":"v0.7.2","module":"github.com/traefik/hub-agent-traefik","commit":"e5cd16b","built":"2022-07-01_02:50:11PM","go_version":"go1.17.11","os":"linux","arch":"amd64","time":1665401096}
hub-agent | {"level":"info","addr":"http://hub-agent","time":1665401097,"message":"Using Agent reachable address"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":1000,"time":1665401097,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":1500,"time":1665401098,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":2250,"time":1665401100,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":3375,"time":1665401102,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":5062.5,"time":1665401105,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":7593.75,"time":1665401110,"message":"Unable to clone topology repository"}
hub-agent | {"level":"warn","error":"create local repository: exit status 128 fatal: No directory name could be guessed.\nPlease specify a directory on the command line\n","retry_in":11390.625,"time":1665401118,"message":"Unable to clone topology repository"}
^CGracefully stopping... (press Ctrl+C again to force)
[+] Running 2/2
⠿ Container hub-agent Stopped 10.2s
⠿ Container traefik Stopped 2.2s
canceled
This was after tearing the stack down and pulling fresh images. So, same error as before. I am assuming something's changed in how traefik hub is to be configured?
@nicomengin Here's my docker-compose.yml file:
version: '3'
services:
hub-agent:
image: ghcr.io/traefik/hub-agent-traefik:v0.7.2
container_name: hub-agent
restart: always
command:
- run
- --hub.token=<TOKEN>
- --auth-server.advertise-url=http://hub-agent
- --traefik.host=traefik
- --traefik.tls.insecure=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- traefik
networks:
- proxy
traefik:
container_name: traefik
image: traefik:v2.8
restart: always
ports:
- "443:443"
- "80:80"
- 8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/tbjers/traefik/etc/:/etc/traefik/
- /home/tbjers/traefik/log/:/var/log/traefik/
labels:
- traefik.enable=true
- traefik.http.routers.api.rule=Headers(`X-Forwarded-Host`, `traefik.example.org`)
- traefik.http.routers.api.entrypoints=web
- traefik.http.routers.api.service=api@internal
- traefik.port=8080
networks:
- proxy
networks:
proxy:
driver: bridge
name: proxy
@nicomengin Alright, so I updated the docker-compose.yml file and changed to v0.8.0 of hub-agent, created a new agent, and now things are working again. I guess a breaking change was introduced, and since it's < 1.0 that's fine... But, still pretty frustrating.
Hey @tbjers,
Glad to know you fixed your issue!
Indeed you used a pretty old version, and the new one brings some changes that can explain that now your stack is working again as expected.
Even if we are still in Beta, we do our best to bring no breaking change, but it's a good practice to always have the agent up-to-date.
Thank you for your feedback.