Letsencrypt not working from podman container

Traefik Traefik v2 (latest)
1

Hi,
I am using Traefik inside a podman pod with the docker provider. The docker-provider is connected to the podman socket (this works fine).

If a new container is started, podman recognizes it, and tries to call letsencrypt. After about 30 seconds I get the following error message:

cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.46.172:443: i/o timeout

So I try wget from inside the traefik container and it works fine:

podman exec -it traefik wget -O - -S https://acme-staging-v02.api.letsencrypt.org/directory
Connecting to acme-v02.api.letsencrypt.org (172.65.46.172:443)
  HTTP/1.1 200 OK
  Server: nginx
  Date: Wed, 14 Jul 2021 16:36:52 GMT
  Content-Type: application/json
  Content-Length: 658
  Connection: close
  Cache-Control: public, max-age=0, no-cache
  X-Frame-Options: DENY
  Strict-Transport-Security: max-age=604800

 ...body removed...

So I think it isn't a networking (routing or firewall) problem. What are my thinking errors?

traefik.yml

log:
    level: DEBUG
    
api:
    insecure: true
    dashboard: true

providers:
    docker: {}

certificatesResolvers:
    letsencrypt:
        acme:
            email: my-mail@example.com # REPLACED
            storage: acme.json
            tlsChallenge: {}
            caserver: https://acme-staging-v02.api.letsencrypt.org/directory

podman:

podman network create traefik_backbone -ip-range 10.0.1.0/24
podman pod create --name traefik_pod --network traefik_backbone
podman container run --rm --name traefik --pod traefik_pod --expose 8080 --expose 8000 \
    --volume /srv/podman/traefik/config/traefik.yml:/etc/traefik/traefik.yml \podman container run --rm --name traefik --pod traefik_pod --expose 8080 --expose 8000 \
    --volume /srv/podman/traefik/config/traefik.yml:/etc/traefik/traefik.yml \
    --volume /srv/podman/podman.sock:/var/run/docker.sock \
    traefik:2.4
2

I seem to be hitting the same issue. Traefik + podman + lets encrypt are working okay for the dashboard. I see a perfectly fine certificate from lets encrypt. However, I am not sure about traefik picking up the labels for dynamically configuring new services. Might be podman, might traefik. I cannot tell. When I bring up a container like so:

podman run --rm -detach \
  --name whoami \
  --label 'traefik.enable=true' \
  --label 'traefik.http.routers.whoami.entrypoints=websecure' \
  --label 'traefik.http.routers.whoami.rule=Host(`who.example.com`)' \   # my domain replaced by example.com
  --label 'traefik.http.routers.whoami.tls.certresolver=letsencrypt' \
  containous/whoami

I cannot find it back in the dashboard. I guess there should be an additional route and service for this container.

Regards,
Gerben

3

Right. I got it working. Two things: 1) I did not enable the docker provider in the static configuration. However, that did not solve the issue initially. I was running everything, i.e. traefik and a test container, in rootless mode. 2) Run everything in rootfull mode. Now I do get new certificates from lets encrypt.

Running the working setup again in rootless mode shows the following in the traefik logs:

level=error msg="service \"whoami\" error: unable to find the IP address for the container \"/whoami\": the server is ignored" providerName=docker container=whoami-226bbc68d9001a0b2f5b5c0b838b9089e9293cc213963d72e2a60a392009fe22

The above error is due to running in rootless mode and there is documentation on this: https://podman.io/getting-started/network.

For now it looks like you to need to be running rootfull containers in case you want traefik and the nice service discovery

4

Rootless Podman might work if you put both the Traefik container and the proxied container in the same CNI network.