Issue requesting SSL using ACME

I have been at this for a while now and haven't identified what I am doing wrong.

I am trying to configure my server with an SSL certificate and I don't know what I am doing wrong as no certificate is being generated for me. I am currently using the staging caServer to be on the safer side as requesting from the production server doesn't also generate the certificate.

Below are my commands and configurations

traefik.yml

providers:
  file:
    directory: /etc/traefik
    watch: true
  docker:
    exposedByDefault: false
    watch: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          scheme: https
  web_secure:
    address: ":443"
    http:
      tls: {}

middleware:
  gzip:
    compress: true

tls:
  certificates:
    - certFile: '/etc/traefik/cert.crt'
      keyFile: '/etc/traefik/cert.key'

log:
  level: DEBUG

api:
  insecure: true
  dashboard: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: loladipupo@plap.xyz
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      storage: /etc/traefik/acme.json
      httpChallenge:
        entryPoint: web

docker-compose.yml

  backend:
    build:
      target: production
    labels:
      - traefik.enable=true
      - traefik.http.routers.backend.rule=Host(`${APP_HOST}`)
      - traefik.http.routers.backend.middlewares=gzip
      - "traefik.http.routers.backend.entrypoints=web,web_secure"
      - traefik.http.routers.backend.tls=true
      - traefik.http.routers.backend.tls.certresolver=letsencrypt
      - traefik.http.routers.backend.tls.domains[0].main=$APP_HOST
      - "traefik.http.services.backend_service.loadbalancer.server.port=8080"
      - "traefik.docker.network=plap-staging_default"

traefik container (not inside docker compose)

 docker run --name proxy --rm -v /home/lawal/.traefik:/etc/traefik/ -p "8080:8080" -p "80:80" -p "443:443" -v /var/run/docker.sock:/var/run/docker.sock traefik:v3.1
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:101 > Traefik version 3.1.3 built on 2024-09-16T15:08:54Z version=3.1.3                                                                                                                                                                         
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"insecure":true},"certificatesResolvers":{"letsencrypt":{"acme":{"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration
":2160,"email":"loladipupo@plap.xyz","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/etc/traefik/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"id
leTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeou
ts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web_secure":{"address":":443","forwardedHeaders":{},"http":{"tls":{}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}
},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true},"file":{"directory":"/etc/traefik","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConns
PerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}                                                                                                                                                                                                                                              
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:617 >                                                                                                                                                                                                                                           
Stats collection is disabled.                                                                                                                                                                                                                                                                                                 
Help us improve Traefik by turning this feature on :)                                                                                                                                                                                                                                                                         
More details on: https://doc.traefik.io/traefik/contributing/data-collection/                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                              
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator aggregator.ProviderAggregator                                                                                                                                                                     
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web_secure                                                                                                                                                                                
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web                        
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik                    
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider                            
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"directory":"/etc/traefik","watch":true}
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik                                            
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/acme.json                                  
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/acme.json.coolify                          
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/cert.crt                                   
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/cert.key                                   
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/traefik.yml                                
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider                         
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}          
2024-09-18T23:36:10Z ERR github.com/traefik/traefik/v3/pkg/provider/traefik/internal.go:153 > Unable to create redirection: the entry point or the port is missing entryPointName=web providerName=internal
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN                    
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider                            
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"letsencrypt","TLSChallengeProvider":{},"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"loladipupo@plap.xyz","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/etc/traefik/acme.json","store":{}}                     
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:795 > Testing certificate renew... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2024-09-18T23:36:10Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.Provider                          
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true}
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"models":{"web_secure":{"tls":{}}},"routers":{"acme-http":{"entryPoints":["web"],"priority":9223372036854775807,"rule":"PathPrefix(`/.well-known/acme-challenge/`)","ruleSyntax":"v3","service":"acme-http@internal"},"api":{"entryPoints":["traefik"],"priority":9223372036854775806,"rule":"PathPrefix(`/api`)","ruleSyntax":"v3","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":9223372036854775805,"rule":"PathPrefix(`/`)","ruleSyntax":"v3","service":"dashboard@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"acme-http":{},"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=file
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=letsencrypt.acme
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}     
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:90 > Provider connection established with docker 26.0.2 (API 1.45) providerName=docker
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=proxy-8ee3e13d3808153c7ec660c16848be2f049581638a4d71c79159381861a20d94 providerName=docker
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=redis-dbb47308f53aecc5c5f6988961ed635aa3138713d7f7f2c4f40518f07eee446d providerName=docker
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=database-b87a53d61084305f50824840b8f95891c9cc9cc3566ddfc1c4b97cd11697ad56 providerName=docker
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"backend":{"entryPoints":["web","web_secure"],"rule":"Host(`test.api.plap.xyz`)","service":"backend"}},"services":{"backend":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.18.0.2:8080"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=docker                        
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal                                                                             
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:97 > No store is defined to add the certificate MIIDazCCAlOgAwIBAgIUUaVHt/4UdySmqpzAhiJOOaUYbe4wDQ, it will be added to the default store
2024-09-18T23:36:10Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:131 > Adding certificate for domain(s)                                           
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal                                                                             
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:268 > Creating load-balancer entryPointName=web routerName=backend@docker serviceName=backend@docker
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:310 > Creating server entryPointName=web routerName=backend@docker serverName=da81a48dc1e3586e serviceName=backend@docker target=http://172.18.0.2:8080
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web_secure middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-09-18T23:36:11Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for test.api.plap.xyz with TLS options default entryPointName=web_secure
2024-09-18T23:37:49Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:37:49Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:37:49Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:6401: remote error: tls: unknown certificate                            
2024-09-18T23:37:49Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:6402: remote error: tls: unknown certificate                            
2024-09-18T23:39:21Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:21Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:31490: remote error: tls: unknown certificate                           
2024-09-18T23:39:21Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:21Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:31491: remote error: tls: unknown certificate                           
2024-09-18T23:39:27Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:27Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:27Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:32258: remote error: tls: unknown certificate                           
2024-09-18T23:39:27Z DBG log/log.go:245 > http: TLS handshake error from 41.217.89.160:32259: remote error: tls: unknown certificate                           
2024-09-18T23:39:28Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:28Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:39:28Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: da81a48dc1e3586e              
2024-09-18T23:40:29Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "test.api.plap.xyz"                
2024-09-18T23:40:29Z DBG log/log.go:245 > http: TLS handshake error from 54.176.87.200:37352: EOF                                                              
2024-09-18T23:45:47Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: da81a48dc1e3586e
2024-09-18T23:45:51Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: da81a48dc1e3586e
^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A2024-09-18T23:48:49Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=file
2024-09-18T23:48:49Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file             
2024-09-18T23:48:51Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=file
2024-09-18T23:48:51Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file             
2024-09-18T23:50:27Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=file
2024-09-18T23:50:27Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=file             

I can confirm HTTP is working as I could connect to it but using https uses my default certificates (self signed).

Thank you.