Traefik not issuing a LE certificate instead issuing fake LE due to "unknown certifcate authority"

Hi all,

Start off fresh here. I'm trying to serve services through traefik through my private domain with a CA. I've been able to serve the traefik dashboard successfully and flawlessly. Now I'm trying to serve a publicly accessible website. Traefik looks like it successfully requests a certificate but continues to serve a fake LE cert. Is this an issue with domain definition?

Traefik.toml


[global]
  checkNewVersion = true
  sendAnonymousUsage = true

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.web.http]
    [entryPoints.web.http.redirections]
      [entryPoints.web.http.redirections.entryPoint]
        to = "websecure"
        scheme = "https"

  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.http.tls]
      [[entryPoints.websecure.http.tls.domains]]
        main = "omit.tech"
        sans = ["*.omit.tech", "*.omit.com", "omit.com"]


[log]
level = "DEBUG"

[api]
  dashboard = true
  insecure = false

[ping]


  [http.routers.my-api]
    rule = "Host(`traefik.ca1.omit.tech`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
    service = "api@internal"
    entryPoints = ["websecure"]
  



[providers]

  [providers.file]
  directory = "/etc/traefik/"
  watch=true

  [providers.docker]
    endpoint = "tcp://ca1.omit.omit:2376"
    [providers.docker.tls]
    caOptional = false
    insecureSkipVerify = false
    cert = "/certs/ca1.crt"
    key = "/certs/ca1.key"
    ca = "/certs/ca.pem"


[tls.stores]
 [tls.stores.default]
   [tls.stores.default.defaultCertificate]
     certFile = "/certs/ca1.crt"
     keyFile  = "/certs/ca1.key"


[certificatesResolvers.mydnschallenge.acme]

  # File or key used for certificates storage.
  #
  # Required
  #
  email = "omit@omit.com"
  storage = "acme.json"

  # CA server to use.
  # Uncomment the line to use Let's Encrypt's staging server,
  # leave commented to go to prod.
  #
  # Optional
  # Default: "https://acme-v02.api.letsencrypt.org/directory"
  # "https://acme-staging-v02.api.letsencrypt.org/directory"
  #
  caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
  [certificatesResolvers.mydnschallenge.acme.dnschallenge]
    provider = "cloudflare"
    resolvers = ["1.1.1.1:53", "8.8.8.8:53"]

Juicy parts of the traefik logs

time="2020-06-03T14:15:18Z" level=debug msg="Using DNS Challenge provider: cloudflare" providerName=mydnschallenge.acme
time="2020-06-03T14:15:18Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Obtaining bundled SAN certificate"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/61622013"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Could not find solver for: tls-alpn-01"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Could not find solver for: http-01"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: use dns-01 solver"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Preparing to solve DNS-01"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] cloudflare: new record for test.anthonyrabbito.com, ID 14ff5c1f5dfb2e4a358de16e1b292e84"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Trying to solve DNS-01"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Checking DNS record propagation using [1.1.1.1:53 8.8.8.8:53]"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]"
time="2020-06-03T14:15:19Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Waiting for DNS record propagation."
time="2020-06-03T14:15:20Z" level=debug msg="Serving default certificate for request: \"traefik.ca1.rabbito.tech\""
time="2020-06-03T14:15:21Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Waiting for DNS record propagation."
time="2020-06-03T14:15:26Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] The server validated our request"
time="2020-06-03T14:15:26Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Cleaning DNS-01 challenge"
time="2020-06-03T14:15:26Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] acme: Validations succeeded; requesting certificates"
time="2020-06-03T14:15:29Z" level=debug msg="legolog: [INFO] [test.anthonyrabbito.com] Server responded with a certificate."
time="2020-06-03T14:15:29Z" level=debug msg="Certificates obtained for domains [test.anthonyrabbito.com]" providerName=mydnschallenge.acme routerName=whoami@docker rule="Host(`test.anthonyrabbito.com`)"
time="2020-06-03T14:15:29Z" level=debug msg="Configuration received from provider mydnschallenge.acme: {\"http\":{},\"tls\":{}}" providerName=mydnschallenge.acme
time="2020-06-03T14:15:29Z" level=debug msg="Adding certificate for domain(s) test.anthonyrabbito.com"

time="2020-06-03T14:15:29Z" level=debug msg="Try to challenge certificate for domain [test.anthonyrabbito.com] found in HostSNI rule" routerName=whoami@docker rule="Host(`test.anthonyrabbito.com`)" providerName=mydnschallenge.acme
time="2020-06-03T14:15:29Z" level=debug msg="Looking for provided certificate(s) to validate [\"test.anthonyrabbito.com\"]..." routerName=whoami@docker rule="Host(`test.anthonyrabbito.com`)" providerName=mydnschallenge.acme
time="2020-06-03T14:15:29Z" level=debug msg="No ACME certificate generation required for domains [\"test.anthonyrabbito.com\"]." providerName=mydnschallenge.acme routerName=whoami@docker rule="Host(`test.anthonyrabbito.com`)"

docker-compose.yml

version: "3.7"
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    extra_hosts: 
      - host.docker.internal:172.17.0.1
    command:
      - --providers.file.directory=/etc/traefik
      - --providers.file.filename=traefik.toml
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    environment:
      - CF_API_EMAIL=${CF_EMAIL}
      - CF_API_KEY=${CF_API_KEY}
    volumes:
      - /docker/traefik/traefik.toml:/etc/traefik/traefik.toml
      - /certs/:/certs
  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.rule=Host(`test.anthonyrabbito.com`)
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.tls.certresolver=mydnschallenge
      - traefik.http.services.whoami.loadbalancer.server.port=80
curl -XGET -H "Content-type: application/json" 'https://traefik.ca1.omt.tech/api/http/services'

[{
	"status": "enabled",
	"usedBy": ["my-api@file"],
	"name": "api@internal",
	"provider": "internal"
}, {
	"status": "enabled",
	"name": "dashboard@internal",
	"provider": "internal"
}, {
	"status": "enabled",
	"usedBy": ["web-to-websecure@internal"],
	"name": "noop@internal",
	"provider": "internal"
}, {
	"status": "enabled",
	"usedBy": ["ping@internal"],
	"name": "ping@internal",
	"provider": "internal"
}, {
	"loadBalancer": {
		"servers": [{
			"url": "http://172.24.0.2:80"
		}],
		"passHostHeader": true
	},
	"status": "enabled",
	"usedBy": ["traefik-docker@docker", "websecure-traefik-docker@docker"],
	"serverStatus": {
		"http://172.24.0.2:80": "UP"
	},
	"name": "traefik-docker@docker",
	"provider": "docker",
	"type": "loadbalancer"
}, {
	"loadBalancer": {
		"servers": [{
			"url": "http://172.24.0.3:80"
		}],
		"passHostHeader": true
	},
	"status": "enabled",
	"usedBy": ["whoami@docker"],
	"serverStatus": {
		"http://172.24.0.3:80": "UP"
	},
	"name": "whoami@docker",
	"provider": "docker",
	"type": "loadbalancer"
}]

Update: this seems like expected behavior as your browsers don't install the staging certificate. Wooot!