Wrong status code while sending configuration: 500: Internal Server Error

Traefik Pilot is (apparently?) working fine for me, but these kinds of messages are appearing in my traefik logs every day:

time="2022-04-30T23:04:59Z" level=error msg="retry in 17.817328828s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-05-01T11:04:57Z" level=error msg="retry in 11.365517975s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-04-30T23:04:53Z" level=info msg="Anonymous stats sent to https://collect.traefik.io/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"healthcheck\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{}},\"web\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"webtls\",\"scheme\":\"https\",\"permanent\":true,\"priority\":1}}}},\"webtls\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"letsencrypt\"}}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"xxxx\",\"defaultRule\":\"xxxx\",\"network\":\"edge-router\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"directory\":\"/etc/traefik/conf/dynamic\",\"watch\":true}},\"api\":{\"dashboard\":true},\"ping\":{\"entryPoint\":\"healthcheck\",\"manualRouting\":true,\"terminatingStatusCode\":503},\"log\":{\"level\":\"INFO\",\"format\":\"common\"},\"certificatesResolvers\":{\"letsencrypt\":{\"acme\":{\"email\":\"xxxx\",\"caServer\":\"xxxx\",\"storage\":\"/etc/traefik/acme/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"tlsChallenge\":{}}}},\"pilot\":{\"token\":\"xxxx\"},\"experimental\":{\"plugins\":{\"checkheadersplugin\":{\"moduleName\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"}}}}"
time="2022-04-29T23:08:21Z" level=error msg="wrong status code while sending configuration: 500: Internal Server Error"
time="2022-04-30T11:06:05Z" level=error msg="retry in 1m22.197287142s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-04-29T23:04:57Z" level=error msg="retry in 18.966819668s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-05-01T11:05:08Z" level=error msg="retry in 57.025471497s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-04-29T23:04:54Z" level=info msg="Testing certificate renew..." providerName=letsencrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2022-04-29T23:04:53Z" level=error msg="retry in 3.990696596s due to: wrong status code while sending configuration: 500: {\"error\":\"internal server error\"}\n "
time="2022-04-30T23:05:17Z" level=error msg="retry in 44.405695228s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-05-01T11:07:37Z" level=error msg="wrong status code while sending configuration: 500: Internal Server Error"
time="2022-04-30T11:05:13Z" level=error msg="retry in 51.311459468s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-04-30T11:08:04Z" level=error msg="wrong status code while sending configuration: 500: Internal Server Error"
time="2022-04-30T11:07:27Z" level=error msg="retry in 37.088397479s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-04-29T23:07:22Z" level=error msg="retry in 59.307618448s due to: wrong status code while sending configuration: 500: Internal Server Error "
time="2022-05-01T11:04:53Z" level=error msg="retry in 3.613183145s due to: wrong status code while sending configuration: 500: {\"error\":\"internal server error\"}\n "

Are these indicative of upstream issues? I know my key is correct because metrics are being uploaded and I'm using a plugin.

Hello @NReilingh,

Thanks for reporting this issue. As you can see in the code here, your dynamic anonymized configuration is sent to Pilot every 12 hours and it seems that we are not able to read some content of it. We have identified the issue and we are working on a fix. It should be fixed in a couple of days.

Sorry for that and the noisy logs

We will keep you updated once the issue is fixed

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.