Rootless mode, plugins-storage permissions and logs pb

Traefik Traefik v3 (latest)
,
1

Hello,
The problems of the day:

  1. unable to create plugins client: unable to create directory /plugins-storage/sources: mkdir plugins-storage: read-only file system
  2. traefik logs file does not populate has it should be.

My rootless PATH where all my dockers are:
/home/user/.local/share/docker/volumes/
My logs PATH:
/home/user/.local/share/docker/volumes/logs/crowdsec

My traefik.yml file:

api:
  dashboard: true
  insecure: true
  debug: true
entryPoints:
  http:
    address: ":80"
    forwardedHeaders:
      trustedIPs: 
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      middlewares:
        - crowdsec-bouncer@file
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
    forwardedHeaders:
      trustedIPs: 
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      middlewares: 
        - crowdsec-bouncer@file
  http-external: 
    address: ":1181"
    http:
      middlewares:
        - crowdsec-bouncer@file
      redirections:
        entrypoint:
          to: https-external
          scheme: https
  https-external:
    address: ":11444"
    http:
      middlewares:
        - crowdsec-bouncer@file
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    network: mynet
    endpoint: "tcp://t-docker-socket-proxy:2375"
    exposedByDefault: false
  file:
    filename: /dynamic_conf.yml
    watch: true
certificatesResolvers:
  dns-cloudflare:
    acme:
      email: email
      storage: ./letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
      caServer: https://acme-v02.api.letsencrypt.org/directory
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
experimental:
  plugins:
    traefik-get-real-ip:
      moduleName: "github.com/Paxxs/traefik-get-real-ip"
      version: "v1.0.3"

log:
  level: "INFO"
  filePath: "/home/user/.local/share/docker/volumes/logs/crowdsec/traefik.log"
accessLog:
  filePath: "/home/user/.local/share/docker/volumes/logs/crowdsec/access.log"
bufferingSize: 50

My dynamic file:

http:
  middlewares:
    default-whitelist:
      ipWhiteList:
        sourceRange:
          - 173.245.48.0/20
          - 103.21.244.0/22
          - 103.22.200.0/22
          - 103.31.4.0/22
          - 141.101.64.0/18
          - 108.162.192.0/18
          - 190.93.240.0/20
          - 188.114.96.0/20
          - 197.234.240.0/22
          - 198.41.128.0/17
          - 162.158.0.0/15
          - 104.16.0.0/13
          - 104.24.0.0/14
          - 172.64.0.0/13
          - 131.0.72.0/22
          - 2400:cb00::/32
          - 2606:4700::/32
          - 2803:f800::/32
          - 2405:b500::/32
          - 2405:8100::/32
          - 2a06:98c0::/29
          - 2c0f:f248::/32
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true
    secured:
      chain:
        middlewares:
          - default-whitelist
    crowdsec-bouncer:
      forwardauth:
        address: http://bouncer-traefik:8080/api/v1/forwardAuth
        trustForwardHeader: true  
    authentik:
      forwardauth:
        address: http://authentik-server:9000/outpost.goauthentik.io/auth/traefik
        trustForwardHeader: true
        authResponseHeaders:
          - X-authentik-username
          - X-authentik-groups
          - X-authentik-email
          - X-authentik-name
          - X-authentik-uid
          - X-authentik-jwt
          - X-authentik-meta-jwks
          - X-authentik-meta-outpost
          - X-authentik-meta-provider
          - X-authentik-meta-app
          - X-authentik-meta-version 
    my-traefik-get-real-ip:
            plugin:
                traefik-get-real-ip:
                    Proxy:
                        - proxyHeadername: X-From-Cdn
                          proxyHeadervalue: cdn1
                          realIP: X-Forwarded-For
                        - proxyHeadername: X-From-Cdn
                          proxyHeadervalue: cdn2
                          realIP: Client-Ip
                        - overwriteXFF: "true"
                          proxyHeadername: X-From-Cdn
                          proxyHeadervalue: cdn3
                          realIP: Cf-Connecting-Ip
                        - proxyHeadername: '*'
                          realIP: RemoteAddr 
                                                               
 #region routers 
  routers:
    proxmox-rtr:
      rule: "Host(`pve.domain.org`)"
      entryPoints:
        - "http-external"
      middlewares:
        - https-redirectscheme
        - authentik
      tls: {}
      service: proxmox-svc
    secure-webserver:
      entryPoints:
        - "https-external"
      rule: "Host(`www.domain`)"
      middlewares:
        - https-redirectscheme
        #- authentik
      tls: {}
      service: secure-webserver
    homeassistant:
      entryPoints:
        - "https-external"
      rule: "Host(`haoss.domain`)"
      middlewares:
        - https-redirectscheme
        #- authentik
      tls: {}
      service: homeassistant
    authentik:
      entryPoints:
        - "https-external"
      rule: "Host(`authentik.domain`)"
      middlewares:
        - https-redirectscheme
        #- authentik
      priority: 10
      tls: {}
      service: authentik
    gotify:
      entryPoints:
        - "https-external"
      rule: "Host(`gotify.domain`)"
      middlewares:
        - https-redirectscheme
      priority: 9
      tls: {}
      service: gotify
    uptime-kuma:
      entryPoints:
        - "https-external"
      rule: "Host(`uptime-kuma.domain`)"
      middlewares:
        - https-redirectscheme
      priority: 5
      tls: {}
      service: uptime-kuma
      
#region services
  services:
    proxmox-svc:
      loadBalancer:
        servers:
          - url: "https://192.xxxxxxx:8006/"
        passHostHeader: true
    secure-webserver:
      loadBalancer:
        servers:
          - url: "http://192xxxxxxx:80"
        passHostHeader: true
    homeassistant:
      loadBalancer:
        servers:
          - url: "http://192.xxxxxxxx:8123"
        passHostHeader: true
    gotify:
      loadBalancer:
        servers:
          - url: "http://192.xxxxxxx:80"
        passHostHeader: true
    uptime-kuma:
      loadBalancer:
        servers:
          - url: "http://192.xxxxxx:3001"
        passHostHeader: true
    authentik:
      loadBalancer:
        servers:
          - url: "http://authentik-server:9000/outpost.goauthentik.io"
        passHostHeader: true

My compose file:

services:
  traefik:
    image: traefik:v3.0.0
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    read_only: true
    mem_limit: 2G
    cpus: 0.75
    depends_on:
      - dockerproxy
    networks:
      - $MYNET
      - socket-t
  
    ports:
      - 1180:80
      - 11443:443
      - 8087:8080
      - 1181:1181
      - 11444:11444
    environment:
      CF_API_EMAIL: $EMAIL
      CF_DNS_API_TOKEN: $TOKEN
      TZ: Europe/Helsinki
      GID: ${GID-988}
      
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - $BASE/logs/crowdsec/:/$BASE/logs/crowdsec/
      - $BASE/traefik/data/traefik.yml:/traefik.yml:ro
      - $BASE/traefik/data/dynamic_conf.yml:/dynamic_conf.yml:ro
      - $BASE/letsencrypt:/letsencrypt


    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=$MYNET"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.$MYDOMAIN`)"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.$MYDOMAIN`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=dns-cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=$MYDOMAIN"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.$MYDOMAIN"
      - "traefik.http.routers.traefik-secure.service=api@internal"
      # middlewares
      - "traefik.http.middlewares.traefik-auth.basicauth.removeheader=true"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=login:xxxxxxxxxxxx"
      # middlewares security headers
      - "traefik.http.middlewares.security-headers.headers.accesscontrolallowmethods=GET, OPTIONS, PUT"
      - "traefik.http.middlewares.security-headers.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.security-headers.headers.addvaryheader=true"
      - "traefik.http.middlewares.security-headers.headers.hostsproxyheaders=X-Forwarded-Host"
      - "traefik.http.middlewares.security-headers.headers.sslredirect=true"
      - "traefik.http.middlewares.security-headers.headers.sslproxyheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.security-headers.headers.stsseconds=63072000"
      - "traefik.http.middlewares.security-headers.headers.stsincludesubdomains=true"
      - "traefik.http.middlewares.security-headers.headers.stspreload=true"
      - "traefik.http.middlewares.security-headers.headers.forcestsheader=true"
      - "traefik.http.middlewares.security-headers.headers.framedeny=true"
      - "traefik.http.middlewares.security-headers.headers.contenttypenosniff=true"
      - "traefik.http.middlewares.security-headers.headers.browserxssfilter=true"
      - "traefik.http.middlewares.security-headers.headers.referrerpolicy=same-origin"
      - "traefik.http.middlewares.security-headers.headers.featurepolicy=camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
      - "traefik.http.middlewares.security-headers.headers.customresponseheaders.X-Robots-Tag=none,noarchive,nosnippet,notranslate,noimageindex"


  dockerproxy:
    image: wollomatic/socket-proxy:1.3.1
    container_name: t-docker-socket-proxy
    command:
      - '-loglevel=debug'
      - '-allowfrom=0.0.0.0/0'
      - '-listenip=0.0.0.0'
      - '-allowGET=/v1\..{1,2}/(version|containers/.*|events.*)'
      - '-watchdoginterval=3600'
      - '-stoponwatchdog'
      - '-shutdowngracetime=10'
    restart: unless-stopped
    read_only: true
    mem_limit: 64M
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
    user: 65534:988 # change gid from 998 to the gid of the docker group on your host
    volumes:
      #- /var/run/docker.sock:/var/run/docker.sock:ro
      - /run/user/1000/docker.sock:/var/run/docker.sock:ro
    networks:
      - socket-t

networks:
  mynet:
    name: $MYNET
    external: true
  socket-t:
    driver: bridge
    internal: true
    attachable: false
2

What I have in my traefik log file .... Even when I am testing my web server, I only have this logs ...
Where should be the folder /plugins-storage ?

2024-05-15T09:15:03+03:00 ERR Plugins are disabled because an error has occurred. error="unable to create plugins client: unable to create directory /plugins-storage/sources: mkdir plugins-storage: read-only file system"
2024-05-15T09:15:03+03:00 INF Starting provider aggregator aggregator.ProviderAggregator
2024-05-15T09:15:03+03:00 INF Starting provider *file.Provider
2024-05-15T09:15:03+03:00 INF Starting provider *acme.ChallengeTLSALPN
2024-05-15T09:15:03+03:00 INF Starting provider *docker.Provider
2024-05-15T09:15:03+03:00 INF Starting provider *acme.Provider
2024-05-15T09:15:03+03:00 INF Starting provider *traefik.Provider
2024-05-15T09:15:03+03:00 INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme
2024-05-15T09:19:21+03:00 ERR Error while Peeking first byte error="read tcp 172.18.0.6:1181->172.18.0.1:52642: i/o timeout"
3

As explained before, this is more a Docker issue than a Traefik one. Check Docker Community Forum.

You can go into your Traefik container and test write access. If it does not work as expected, then something on host or in compose is not configured correctly.

docker exec -it <container-id> sh

Note that this looks wrong, it seems a : is missing (doc):

4

Hi come back to this forum ...

I have modified the volumes in my traefik compose file and add :

$BASE/traefik/plugins-storage/sources:/plugins/sources

I still don't know where the folder should be ???

After doing a

"rootlesskit bash"

root@proxmox-ve-node1-serverdockers:~/.local/share/docker/volumes/traefik# ll
total 0
drwxrwxr-x  4 theboss docker  41 mai   17 03:25 ./
drwx-w---x 14 theboss docker 229 mai   17 03:09 ../
drwxrwxr-x  2 theboss docker  49 mai   10 09:36 data/
drwxr-xr-x  3 root    root    21 mai   17 03:25 plugins-storage/

Even if I change the permission to theboss/docker ... I still do have the same pb.

In the traefik compose file, I have modified GID and UID ... same pb

UID: ${UID:-1000}
GID: ${GID:-998}

I might need to go back to non rootless mode ....

5

Just to late informed.
I have move back to a normal use of Docker. :frowning:
By accident, I saw something strange. If I am using Traefik v3.0.0, I still have the error concerning the folder "plugins-storage" ... If I downgrade to 2.11.2, this error desappear and traefik is working as usual.
Thx again @bluepuma77 and @wollomatic for your precious help.

1 Like