Can't Connect to Docker, - level=error msg="Provider connection error panic in operation: %!s(<nil>),

Traefik Traefik v2 (latest)
1

I have Traefik v2.1.1 setup to connect to a socket-proxy as per the instructions, with proper SELinux mounts (mounting with Z). I followed this guide; https://medium.com/@containeroo/traefik-2-0-paranoid-about-mounting-var-run-docker-sock-22da9cb3e78c

Here is my Traefik toml config:

 [providers]
  providersThrottleDuration = 42
  [providers.docker]
    endpoint = "tcp://socket-proxy:2375"
    watch = true
    exposedByDefault = false
    useBindPortIP = false
    swarmMode = false
    network = "bridge"
    [providers.docker.tls]
      caOptional = true
      insecureSkipVerify = true

When I run traefik, it doesn't see any of my other containers (which are set to traefik.enabled=true) this is the output I see from the traefik container;

time="2019-12-14T08:19:26Z" level=error msg="Provider connection error error during connect: Get https://socket-proxy:2375/v1.24/version: http: server gave HTTP response to HTTPS client, retrying in 2.011541491s" providerName=docker
time="2019-12-14T08:19:28Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!: -1" module=api status_code=-1
time="2019-12-14T08:19:28Z" level=error msg="Failed to retrieve information of the docker client and server host: error during connect: Get https://socket-proxy:2375/v1.24/version: http: server gave HTTP response to HTTPS client" providerName=docker
time="2019-12-14T08:19:28Z" level=error msg="Provider connection error error during connect: Get https://socket-proxy:2375/v1.24/version: http: server gave HTTP response to HTTPS client, retrying in 2.635284008s" providerName=docker

It seems to be asking the proxy for https when it should just be straight tcp. I don't know what more to do, the tcp is specified in the endpoint.

This is what the socket proxy says,

Proxy dockerbackend started.
Proxy dockerfrontend started.
[NOTICE] 347/070115 (1) : New worker #1 (11) forked
X.X.X.X:60264 [14/Dec/2019:07:27:55.857] dockerfrontend dockerfrontend/<NOSRV> -1/-1/-1/-1/0 400 188 - - PR-- 1/1/0/0/0 0/0 "<BADREQ>"

Any help would be greatly appreciated, thanks.

2

OK, I fixed part of the problem.

I disabled

[providers.docker.tls]

and now I don't have the https to http issue anymore.

Instead I have another error message from traefik:

time="2019-12-14T23:32:08Z" level=error msg="Error in Go routine: runtime error: index out of range [0] with length 0"
time="2019-12-14T23:32:08Z" level=error msg="Stack: goroutine 43 [running]:\nruntime/debug.Stack(0xc000100000, 0x2550df9, 0x17)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/containous/traefik/v2/pkg/safe.defaultRecoverGoroutine(0x235a380, 0xc000871c20)\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:160 +0xb1\ngithub.com/containous/traefik/v2/pkg/safe.OperationWithRecover.func1.1(0xc00061bd78)\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:168 +0x56\npanic(0x235a380, 0xc000871c20)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/containous/traefik/v2/pkg/config/parser.DecodeToNode(0xc00028c6f0, 0x252f8bb, 0x7, 0xc00061b3a0, 0x2, 0x2, 0x10, 0x21fd9e0, 0x29f5b01)\n\t/go/src/github.com/containous/traefik/pkg/config/parser/labels_decode.go:24 +0x6ec\ngithub.com/containous/traefik/v2/pkg/config/parser.Decode(0xc00028c6f0, 0x21158e0, 0xc00024e260, 0x252f8bb, 0x7, 0xc00061b3a0, 0x2, 0x2, 0x75ffba, 0x241de40)\n\t/go/src/github.com/containous/traefik/pkg/config/parser/parser.go:11 +0x6d\ngithub.com/containous/traefik/v2/pkg/config/label.DecodeConfiguration(0xc00028c6f0, 0xc000409d10, 0x29f5b60, 0xc0003b6d20)\n\t/go/src/github.com/containous/traefik/pkg/config/label/label.go:16 +0xfb\ngithub.com/containous/traefik/v2/pkg/provider/docker.(*Provider).buildConfiguration(0xc00018c2a0, 0x29ac080, 0xc000231740, 0xc0005de600, 0x21, 0x21, 0x21)\n\t/go/src/github.com/containous/traefik/pkg/provider/docker/config.go:31 +0x326\ngithub.com/containous/traefik/v2/pkg/provider/docker.(*Provider).Provide.func1.1(0x0, 0x0)\n\t/go/src/github.com/containous/traefik/pkg/provider/docker/docker.go:193 +0x657\ngithub.com/containous/traefik/v2/pkg/safe.OperationWithRecover.func1(0x0, 0x0)\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:172 +0x6c\ngithub.com/cenkalti/backoff/v3.RetryNotify(0xc000438b70, 0x296a180, 0xc000684200, 0xc00061bef0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/cenkalti/backoff/v3@v3.0.0/retry.go:37 +0xb8\ngithub.com/containous/traefik/v2/pkg/provider/docker.(*Provider).Provide.func1(0x29abfc0, 0xc000263d40)\n\t/go/src/github.com/containous/traefik/pkg/provider/docker/docker.go:293 +0x2d6\ngithub.com/containous/traefik/v2/pkg/safe.(*Pool).GoCtx.func1()\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:63 +0x84\ngithub.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x262c6e0, 0xc0006841e0)\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57\ncreated by github.com/containous/traefik/v2/pkg/safe.GoWithRecover\n\t/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49\n"

and from socket-proxy;

172.16.0.26:58020 [14/Dec/2019:23:34:57.337] dockerfrontend dockerbackend/dockersocket 0/0/0/1/1 200 6066 - - ---- 3/3/0/0/0 0/0 "GET /v1.24/containers/892a993919fdbf6cb67c5f42f802830e2842fb021d8a3236595defe77474bd17/json HTTP/1.1"
3

Hello,

This error comes when you have an invalid label.

This bug (panic) is already reported #6024 and already fixed #6028

4

Is there anyway for me to figure our which label is invalid based on the error? I've gone through my docker-compose yml files, I have dozens of labels, but I don't see any of them ending with a period.

5

Nevermind. I was able to find The container causing the issue.