Can't use plugins - error="mkdir plugins-storage: permission denied"

Traefik Traefik v2 (latest)
1

Welcome!

  • Yes, I've searched similar issues on GitHub and didn't find any.
  • Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Tried to configure: Plugin

# Static configuration

experimental:
  plugins:
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin

# Dynamic configuration

http:
  routers:
    my-router:
      rule: host(`woami.localhost`)
      service: service-foo
      entryPoints:
        - web
      middlewares:
        - crowdsec

  services:
    service-foo:
      loadBalancer:
        servers:
          - url: http://127.0.0.1:5000
  
  middlewares:
    crowdsec:
      plugin:
        bouncer:
          enabled: false
          updateIntervalSeconds: 60
          defaultDecisionSeconds: 60
          crowdsecMode: live
          crowdsecLapiKey: privateKey
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiScheme: http
          forwardedHeadersTrustedIPs: 
            - 10.0.10.23/32
            - 10.0.20.0/24
          forwardedHeadersCustomName: X-Custom-Header

What did you see instead?

time="2022-11-03T16:09:42+01:00" level=error msg="Plugins are disabled because an error has occurred." error="mkdir plugins-storage: permission denied"

What version of Traefik are you using?

level=info msg="Traefik version 2.8.4 built on 2022-09-02T14:42:59Z"

What is your environment & configuration?

[Unit]
Description=traefik proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal
EnvironmentFile=/etc/traefik/.env

; User and group the process will run as.
User=traefik
Group=traefik

; Always set "-root" to something safe in case it gets forgotten in the traefikfile.
ExecStart=/opt/traefik/traefik --configfile=/etc/traefik/traefik.yml

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use private /tmp and /var/tmp, which are discarded after traefik stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/traefik, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/traefik
#ReadWriteDirectories=/opt/traefik # doesn't work

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by traefik. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

If applicable, please paste the log output in DEBUG level

No response

2

This seems like a bug because when ran as root from the terminal, traefik creates the directories properly, and when I just transfer ownership for the entire /opt/traefik/plugins-storage directory to the traefik user, I still get the same error.

3

Hello @zoomba,

Thank you for using our Plugin, i'm one of the co autors of GitHub - maxlerebourg/crowdsec-bouncer-traefik-plugin: Traefik plugin to apply crowdsec decisions from local API

I've found this thread randomly when I encountered the same error you did.
I'm writing at the moment an exemple on how to use the plugin using Traefik and crowdsec as binaries in a virtual machine.

In the log file traefik.log (or output to journalctl), you can see that Traefik cannot create the directory plugins-storage.

This is because your service file is a bit too restrictive on what the process launched can do.

Blockquote
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full

This will prevent Traefik to write files in etc

Blockquote
ReadWriteDirectories=/etc/traefik

This should let Traefik write in the /etc/traefik directory

But does he try to write there ?

I believe it does'nt after looking at Traefik code which loads the plugins

const outputDir = "./plugins-storage/"

Here it is using a relative path for a folder named plugins-storage

currentPath, err := os.Getwd()
	if err != nil {
		return err
	}

	if strings.HasPrefix(currentPath, dirPath) {
		return fmt.Errorf("cannot be deleted: the directory path %s is the parent of the current path %s", dirPath, currentPath)
	}

	err = os.RemoveAll(dir)
	if err != nil {
		return err
	}

	return os.MkdirAll(dir, 0o755)

Here is it using the current directory Traefik is launched with.

After looking in the container what is the default, I found out it's /

So Traefik is trying to write /plugins-storage

To do it properly and bypass this error, I had to set this current directory in the service file:

WorkingDirectory=/etc/traefik

And I added also because i'm in the etc directory with ProtectSystem=full

ReadWriteDirectories=/etc/traefik/plugins-storage

Please feel free to open issues in our repository if you have any question about the plugin and the connection between Traefik and Crowdsec

Best
Mathieu