Hi all, I've had traefik with LE w/ acme azure dns-01 working a few days ago, but as of 7/30/22 I ran into some bizarre issues when I try to create a new cert. I've tried to sandbox this to just a basic setup (see docker compose below). After googling for the past 24h w/o much success, I'm hoping someone here can help point me in the right direction 
Also of note, I can confirm the same azure environments (app id, client secret, tenant, zones) used with certbot + azure extension (from certbot-dns -azure works successfully; I can confirm a live cert is generated and see audits of created and delete TXT records in the Azure DNS. This leads me to think something is different in the setup/configuration/implementation with traefik and/or the LEGO provider. Any assistance to move forward is greatly appreciated; if anyone has a bare working docker-compose for dns challenge in azure, that would also be helpful.
My high level roadblocks observed from logs:
The traefik log error of interest seems to be this line (see section Initial Logs with Authorization Failed below for full logs):
level=error msg="Unable to obtain ACME certificate for domains \"[redacted1].org,*.[redacted1].org\" : unable to generate a certificate for the domains [[redacted1].org *.[redacted1].org]: error: one or more domains had a problem:\n[*.[redacted1].org] [*.[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: **StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid**. If access was recently granted, please refresh your credentials.\"\n[[redacted1].org] [[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n" providerName=myresolver.acme
The first thought is the service principal wasn't granted proper access. However, I've double checked the service principal was part of the DNZ Zone Contributor, and even tried granting it ownership access all the way to the subscription level. When I do that, I get a different error about resource not found (see section below Secondary Logs with Resource Not Found for full logs):
Unable to obtain ACME certificate for domains \"[redacted1].org,*.[redacted1].org\" : unable to generate a certificate for the domains [[redacted1].org *.[redacted1].org]: error: one or more domains had a problem:\n[*.[redacted1].org] [*.[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"**The Resource 'Microsoft.Network/dnszones/org' under resource group 'rg-wus2-dns' was not found**. For more details please go to https://aka.ms/ARMResourceNotFoundFix\"\n[[redacted1].org] [[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Network/dnszones/org' under resource group 'rg-wus2-dns' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix\"\n" providerName=myresolver.acme
Thanks in advance for reading and any help!
Additional References
Host System
OS: Ubuntu 18.04 LTS
Docker CE Version: 19.03
docker-compose.yaml:
version: '3.3'
services:
traefik:
image: traefik:v2.8.1
container_name: traefik
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.dnschallenge=true"
- "--certificatesresolvers.myresolver.acme.dnschallenge.provider=azure"
- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.myresolver.acme.email=[redacted4]@outlook.com"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
- "8080:8080"
environment:
- AZURE_CLIENT_ID=[redacted6]
- AZURE_CLIENT_SECRET=[redacted7]
- AZURE_SUBSCRIPTION_ID=[redacted3]
- AZURE_TENANT_ID=[redacted8]
- AZURE_RESOURCE_GROUP=rg-wus2-dns
volumes:
- "./letsencrypt:/letsencrypt"
- /var/run/docker.sock:/var/run/docker.sock
#- ./config:/etc/traefik
whoami-a:
image: "containous/whoami"
container_name: "whoami-a"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami-a.rule=Host(`test.[redacted1].org`) && Path(`/whoami-a`)"
- "traefik.http.routers.whoami-a.entrypoints=websecure"
- "traefik.http.routers.whoami-a.tls.certresolver=myresolver"
- "traefik.http.routers.whoami-a.tls.domains[0].main=[redacted1].org"
- "traefik.http.routers.whoami-a.tls.domains[0].sans=*.[redacted1].org"
Initial Logs with Authorization Failed:
whoami-a | Starting up on port 80
traefik | time="2022-07-31T23:27:04Z" level=info msg="Configuration loaded from flags."
traefik | time="2022-07-31T23:27:04Z" level=info msg="Traefik version 2.8.1 built on 2022-07-11T14:06:57Z"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"myresolver\":{\"acme\":{\"email\":\"[redacted4]@outlook.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"azure\"}}}},\"pilot\":{\"dashboard\":true}}"
traefik | time="2022-07-31T23:27:04Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik | time="2022-07-31T23:27:04Z" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
traefik | time="2022-07-31T23:27:04Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Starting TCP Server" entryPointName=websecure
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Starting TCP Server" entryPointName=traefik
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Starting TCP Server" entryPointName=web
traefik | time="2022-07-31T23:27:04Z" level=info msg="Starting provider *traefik.Provider"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="*traefik.Provider provider configuration: {}"
traefik | time="2022-07-31T23:27:04Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
traefik | time="2022-07-31T23:27:04Z" level=info msg="Starting provider *docker.Provider"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
traefik | time="2022-07-31T23:27:04Z" level=info msg="Starting provider *acme.Provider"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"[redacted4]@outlook.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"azure\"},\"ResolverName\":\"myresolver\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik | time="2022-07-31T23:27:04Z" level=info msg="Testing certificate renew..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=myresolver.acme
traefik | time="2022-07-31T23:27:04Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Provider connection established with docker 19.03.12 (API 1.40)" providerName=docker
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" container=traefik-traefiktmp-32e8f8f718fd1c5bc7cb779e66970469f4dbadf99ed8d28daf57600c113837ba providerName=docker
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=sabnzbd-sabnzbd-dba5ab545a3471b1afca0844eda618e77c7de0f583544e45c270609d029ef856
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=influxdb-monitor-ee338c086eb41b8ed730584f82ca489b270d157ee1b9a0360fc914f5147b8a9d
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=grafana-monitor-f096ca1dcf9784b09283e472aee2ddab7f4ff0b5a21b9ea6b644ce66f09dd828
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=promtail-monitor-eca90b51d5b0e11ec428462e315ed056961804f4740c305702ab5355ea8b7885
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=loki-monitor-386d176f33d71e1615bb0e0ff624791013d54fe0250b3d584716eefb40604c06
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=gluetun-gluetunvpn-73ec9a860a2335dd19c16e33645a6f920bfdbeb356e774df076eb9f027b3fa2e
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=redis-insight-redis-b754bcf78fdc5f633fe28fddc510b6afed433cbcb34e5618de6f719ca9bdd7e0
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Filtering disabled container" providerName=docker container=redis-commander-redis-6437c9160176bf7c0acfeb23541045d0e7fb38b89db6e1b44db7e3e3fa7cc835
traefik | time="2022-07-31T23:27:04Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"whoami-a\":{\"entryPoints\":[\"websecure\"],\"service\":\"whoami-a-traefiktmp\",\"rule\":\"Host(`test.[redacted1].org`) \\u0026\\u0026 Path(`/whoami-a`)\",\"tls\":{\"certResolver\":\"myresolver\",\"domains\":[{\"main\":\"[redacted1].org\",\"sans\":[\"*.[redacted1].org\"]}]}}},\"services\":{\"whoami-a-traefiktmp\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.16.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@internal entryPointName=traefik
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
traefik | time="2022-07-31T23:27:05Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
traefik | time="2022-07-31T23:27:05Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal entryPointName=traefik
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=whoami-a@docker serviceName=whoami-a-traefiktmp middlewareName=pipelining middlewareType=Pipelining
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=whoami-a@docker serviceName=whoami-a-traefiktmp
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating server 0 http://192.168.16.3:80" serverName=0 routerName=whoami-a@docker serviceName=whoami-a-traefiktmp entryPointName=websecure
traefik | time="2022-07-31T23:27:06Z" level=debug msg="child http://192.168.16.3:80 now UP"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Propagating new UP status"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Added outgoing tracing middleware whoami-a-traefiktmp" routerName=whoami-a@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Adding route for test.[redacted1].org with TLS options default" entryPointName=websecure
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Looking for provided certificate(s) to validate [\"[redacted1].org\" \"*.[redacted1].org\"]..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Domains [\"[redacted1].org\" \"*.[redacted1].org\"] need ACME certificates generation for domains \"[redacted1].org,*.[redacted1].org\"." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Loading ACME certificates [[redacted1].org *.[redacted1].org]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Building ACME client..." providerName=myresolver.acme
traefik | time="2022-07-31T23:27:06Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
traefik | time="2022-07-31T23:27:06Z" level=debug msg="Using DNS Challenge provider: azure" providerName=myresolver.acme
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [[redacted1].org, *.[redacted1].org] acme: Obtaining bundled SAN certificate"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [*.[redacted1].org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3192531834"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [[redacted1].org] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3192531844"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [*.[redacted1].org] acme: use dns-01 solver"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [[redacted1].org] acme: Could not find solver for: tls-alpn-01"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [[redacted1].org] acme: Could not find solver for: http-01"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [[redacted1].org] acme: use dns-01 solver"
traefik | time="2022-07-31T23:27:06Z" level=debug msg="legolog: [INFO] [*.[redacted1].org] acme: Preparing to solve DNS-01"
traefik | time="2022-07-31T23:27:07Z" level=debug msg="legolog: [INFO] [[redacted1].org] acme: Preparing to solve DNS-01"
traefik | time="2022-07-31T23:27:08Z" level=debug msg="legolog: [INFO] [*.[redacted1].org] acme: Cleaning DNS-01 challenge"
traefik | time="2022-07-31T23:27:08Z" level=debug msg="legolog: [WARN] [*.[redacted1].org] acme: cleaning up failed: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid. If access was recently granted, please refresh your credentials.\" "
traefik | time="2022-07-31T23:27:08Z" level=debug msg="legolog: [INFO] [[redacted1].org] acme: Cleaning DNS-01 challenge"
traefik | time="2022-07-31T23:27:09Z" level=debug msg="legolog: [WARN] [[redacted1].org] acme: cleaning up failed: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid. If access was recently granted, please refresh your credentials.\" "
traefik | time="2022-07-31T23:27:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3192531834"
traefik | time="2022-07-31T23:27:09Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3192531844"
traefik | time="2022-07-31T23:27:09Z" level=error msg="Unable to obtain ACME certificate for domains \"[redacted1].org,*.[redacted1].org\" : unable to generate a certificate for the domains [[redacted1].org *.[redacted1].org]: error: one or more domains had a problem:\n[*.[redacted1].org] [*.[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n[[redacted1].org] [[redacted1].org] acme: error presenting token: azure: dns.ZonesClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '[redacted2]' with object id '[redacted2]' does not have authorization to perform action 'Microsoft.Network/dnsZones/read' over scope '/subscriptions/[redacted3]/resourceGroups/rg-wus2-dns/providers/Microsoft.Network/dnsZones/org' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n" providerName=myresolver.acme