Letsencrypt not working from podman container

Hi,
I am using Traefik inside a podman pod with the docker provider. The docker-provider is connected to the podman socket (this works fine).

If a new container is started, podman recognizes it, and tries to call letsencrypt. After about 30 seconds I get the following error message:

cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp 172.65.46.172:443: i/o timeout

So I try wget from inside the traefik container and it works fine:

podman exec -it traefik wget -O - -S https://acme-staging-v02.api.letsencrypt.org/directory
Connecting to acme-v02.api.letsencrypt.org (172.65.46.172:443)
  HTTP/1.1 200 OK
  Server: nginx
  Date: Wed, 14 Jul 2021 16:36:52 GMT
  Content-Type: application/json
  Content-Length: 658
  Connection: close
  Cache-Control: public, max-age=0, no-cache
  X-Frame-Options: DENY
  Strict-Transport-Security: max-age=604800

 ...body removed...

So I think it isn't a networking (routing or firewall) problem. What are my thinking errors?

traefik.yml

log:
    level: DEBUG
    
api:
    insecure: true
    dashboard: true

providers:
    docker: {}

certificatesResolvers:
    letsencrypt:
        acme:
            email: my-mail@example.com # REPLACED
            storage: acme.json
            tlsChallenge: {}
            caserver: https://acme-staging-v02.api.letsencrypt.org/directory

podman:

podman network create traefik_backbone -ip-range 10.0.1.0/24
podman pod create --name traefik_pod --network traefik_backbone
podman container run --rm --name traefik --pod traefik_pod --expose 8080 --expose 8000 \
    --volume /srv/podman/traefik/config/traefik.yml:/etc/traefik/traefik.yml \podman container run --rm --name traefik --pod traefik_pod --expose 8080 --expose 8000 \
    --volume /srv/podman/traefik/config/traefik.yml:/etc/traefik/traefik.yml \
    --volume /srv/podman/podman.sock:/var/run/docker.sock \
    traefik:2.4

I seem to be hitting the same issue. Traefik + podman + lets encrypt are working okay for the dashboard. I see a perfectly fine certificate from lets encrypt. However, I am not sure about traefik picking up the labels for dynamically configuring new services. Might be podman, might traefik. I cannot tell. When I bring up a container like so:

podman run --rm -detach \
  --name whoami \
  --label 'traefik.enable=true' \
  --label 'traefik.http.routers.whoami.entrypoints=websecure' \
  --label 'traefik.http.routers.whoami.rule=Host(`who.example.com`)' \   # my domain replaced by example.com
  --label 'traefik.http.routers.whoami.tls.certresolver=letsencrypt' \
  containous/whoami

I cannot find it back in the dashboard. I guess there should be an additional route and service for this container.

Regards,
Gerben

Right. I got it working. Two things: 1) I did not enable the docker provider in the static configuration. However, that did not solve the issue initially. I was running everything, i.e. traefik and a test container, in rootless mode. 2) Run everything in rootfull mode. Now I do get new certificates from lets encrypt.

Running the working setup again in rootless mode shows the following in the traefik logs:

level=error msg="service \"whoami\" error: unable to find the IP address for the container \"/whoami\": the server is ignored" providerName=docker container=whoami-226bbc68d9001a0b2f5b5c0b838b9089e9293cc213963d72e2a60a392009fe22

The above error is due to running in rootless mode and there is documentation on this: https://podman.io/getting-started/network.

For now it looks like you to need to be running rootfull containers in case you want traefik and the nice service discovery

Rootless Podman might work if you put both the Traefik container and the proxied container in the same CNI network.