Proxmox behind Traefik

Traefik Traefik v2
1

Hello together!

Maybe someone can help me with the following case: I would like to include my proxmox behind my Traefik reverse proxy.

To do this, I tried to follow the instructions in the Proxmox wiki, which describes instructions for a Proxmox behind an nginx reverse proxy.

This is my current configuration at the moment:

http:
routers:
proxmox-https-router:
rule: "Host(proxmox.home.xxxxx.yyy)"
service: my-proxmox
entryPoints:
- https
tls:
certResolver: http
middlewares:
- basicAuth

proxmox-http-router:
  rule: "Host(`proxmox.home.xxxxx.yyy`)"
  service: my-proxmox
  entryPoints:
    - http
  middlewares:
    - redirect

services:
my-proxmox:
loadBalancer:
servers:
- url: https://192.168.1.97:8006

middlewares:
https_redirect:
redirectScheme:
scheme: https
permanent: true

I've gotten as far as being able to access the web interface, but VNC or SPICE don't work at all.
And I also see many settings in the sample nginx configuration that are certainly missing here.
Unfortunately I am not that experienced in the field of reverse proxies. Can someone help me here maybe?

TIA

2

Hello @candoom,

Thank you for your interest in Traefik!

Can you format your configuration file to make it more readable?

Regarding SPICE, the proxmox documentation says:

This daemon listens on TCP port 3128, and implements an HTTP proxy to forward CONNECT request from the SPICE client to the correct Proxmox VE VM. It runs as user www-data and has very limited permissions.

Regarding VNC it seems to be listening on TCP port 5900.

Your current configuration is routing all requests matching the Host matcher "proxmox.home.xxxxx.yyy" to "https://192.168.1.97:8006/", so you only set up a route to reach the web interface.

For SPICE, as I understand it expects CONNECT request, you'll need to add a new router with a rule with the previous host matcher plus a method matcher: Method(`CONNECT`). You'll also have to create the appropriate service to reach the correct port on proxmox.

For VNC, you'll probably have to set up a TCP router and service.

3

Hello @rtribotte,

Thank you very much for your answer. I have adjusted my configuration according to your suggestions. It makes sense what you said. I think it should be right now?

http:
  routers:
    proxmox-https-router:
      rule: "Host(`proxmox.home.xxxxx.yyy`)"
      service: my-proxmox
      entryPoints:
        - https
      tls:
        certResolver: http
#      middlewares:
#        - basicAuth

    proxmox-http-router:
      rule: "Host(`proxmox.home.xxxxx.yyy`)"
      service: my-proxmox
      entryPoints:
        - http
      middlewares:
        - redirect
		
	proxmox-spice-router:
      rule: "Host(`proxmox.home.xxxxx.yyy`) &&  Method(`CONNECT`)"
      service: spice-proxmox
      entryPoints:
        - http
      middlewares:
        - redirect

  services:
    my-proxmox:
      loadBalancer:
        servers:
          - url: https://192.168.1.97:8006
		  
  middlewares:
    https_redirect:
      redirectScheme:
        scheme: https
        permanent: true
tcp:
  services:
	spice-proxmox:
	  loadbalancer:
	    servers:
		  - address: 192.168.1.97:3128
4

Hello @candoom,

Sorry for the delay, the proxmox-spice-router cannot reference a TCP service, it should be an HTTP service.

5

Hello @candoom

I have traefik running on my server and its routing traffic to my bitwarden, I want to use traefik to route traffic to proxmox. What you have above is if the config file, can you please share your traefik docker-compose file to see how the config file is called.

Thanks

6

I was able to get this working.
However, this will only work with 1 proxmox host/cluster.
I have not yet found a way to redirect via SNI.

In my traefik.yml

entryPoints:
# SPICE Proxmox Proxy
  spice:
    address: ":3128"

In my Spice file config

tcp:
  routers:
    spice:
      rule: HostSNI(`*`)
      entrypoints: spice
      tls: false
      service: spice
  services:
    spice:
      loadbalancer:
        servers:
          - address: "proxmox.host.tld:3128"
1 Like
7

Hi @djarbz

I tried this same solution but cannot get it working.

The error I get from remote-viewer is: Cannot determine the connection type from URI

The generated pve-spice.vv file is:

[virt-viewer]
host-subject=OU=PVE Cluster Node,O=Proxmox Virtual Environment,CN=pve.home.papasmurf.nl
title=VM 101 - ubuntu-test
delete-this-file=1
tls-port=61000
type=spice
proxy=http://proxmox.home.papasmurf.nl:3128
ca=-----BEGIN CERTIFICATE-----xxxxx-the certificate is left out-xxxxx\n-----END CERTIFICATE-----\n
password=xxxxxx-the-password-xxxxxx
release-cursor=Ctrl+Alt+R
secure-attention=Ctrl+Alt+Ins
toggle-fullscreen=Shift+F11
host=pvespiceproxy:626525b4:101:pve::dc88a7f15a905e5d8fb50bd34f15a50b582e9988

In my traefik.yml:

entryPoints:                                                                                        
  web:                                                                                              
    address:·":80"                                                                                  
    http:                                                                                           
      redirections:                                                                                 
        entryPoint:                                                                                 
          to:·web-secure                                                                            
          scheme:·https                                                                             
          permanent:·true                                                                           
  web-secure:                                                                                       
address:·":443"                                                                                 
  spice:                                                                                            
    address:·":3128"

config.yml:

tcp:                                                                                                           
  routers:                                                                                          
    spice:                                                                                          
      rule: HostSNI(`*`)                                                                            
      entrypoints: spice                                                                            
      tls: false                                                                                    
      service: spice                                                                                
  services:                                                                                         
    spice:                                                                                          
      loadBalancer:                                                                                 
        servers:                                                                                    
          - address: "192.168.144.10:3128"                                                          
                                                                                                    
http:                                                                                               
  routers:  
(here follows my http router config)

Part of the http config is a router for the Proxmox web interface

...
proxmox:
  rule: "Host(`proxmox.home.papasmurf.nl`) "
  service: proxmox
...
services:
  proxmox:
    loadBalander:
      servers:
        - url: "https://192.168.144.10:8006"

By the way: the hostname of the proxmox server is `pve'.
When I do nothing in the Traefik config, and I quickly change the url in the pve-spice file to 'pve.home.papasmurf.nl', then I can connect successfully.

8

It looks like you are from the Netherlands, does your pve-spice.vv file contain any non-ASCII characters?
HERE is an issue where a Cyrillic file name was causing this issue.

Other than that, your configuration looks correct.

9

I know its been a while. Did this config work? I'm having the same problem and I tried this config however, I still can't connect. Do you have a working config now you can share?

10

it's my config, work well

[http.routers]
  [http.routers.pve-rtr]
      entryPoints = ["https"]
      rule = "Host(`pve.domain.com`)"
      service = "pve-svc"
      # middlewares = ["chain-basic-auth"]
      tls = {} 

[http.services]
  [http.services.pve-svc]
    [http.services.pve-svc.loadBalancer]
      passHostHeader = true
      serversTransport = "pve"
      [[http.services.pve-svc.loadBalancer.servers]]
        url = "https://10.0.0.222:8006"  # or whatever your external host's IP:port is 

[http.serversTransports.pve]
  insecureSkipVerify = true
1 Like
12

Thanks for sharing. This config looks like its for the Proxmox web interface? I have that part working.

My specific issue; now that I'm going through Traefik it breaks SPICE console access. I get "connection refused" when I try to connect through proxy.

When I bypass proxy and go through IP direct it works just fine. Is there a configuration work around in Traefik to make SPICE work?

13

I assume you mean "Simple Protocol for Independent Computing Environments"? That’s a TCP protocol, you need to setup an additional TCP router (and service) for that. If it uses TLS you can use the same entrypoint with HostSNI() in rule.

14

Under config.yml, I created the following TCP settings and Traefik dashboard shows "Success" for both TCP Routers/Services for this config. However, I still get "Connection refused" when I use SPICE console. Is there something I need to modify within the TCP router/service to make it accept the connection?

tcp:
  routers:
    spice:
      entryPoints:
        - "https"
      rule: "HostSNI(`proxmox.example.com`)"
      tls: {}
      service: spice
  services:
    spice:
      loadbalancer:
        servers:
          - address: "1.2.3.4:3128"
15

So I've tried the HTTP and TCP methods in this post and neither work. I experimented with exposing port 3128 on Traefik container I get "404 error" instead of "connection refused" like I typically get. But , I don't think I should have exposed that?

If there is a way to proxy SPICE console I couldn't find a way to do it.

16

The question is how Proxmox is handling the port. Is the app using the port in the browser to connect to and the server to listen? Or can you configure it differently?

If both need the special port, then you need to open the port and create an entrypoint. If you get 404 on the special port, you can try HostSNI(`*`) in rule.

17

This configuration is working for me:

Static file: :stop_sign:PLEASE READ YOUR "pve_spice.vv" AND MAKE SURE OF YOUR tls-port. :stop_sign:

entryPoints:
  spice:
    address: ":3128"
  spice-tls:
    address: ":61001"

Dynamic file:

tcp:
  routers:
    spice:
      rule: HostSNI(`*`)
      entrypoints: spice
      tls: false
      service: spice
    spice-tls:
      entryPoints:
        - spice-tls
      service: spice-tls
      rule: HostSNI(`proxmox.domain.tld`)
      tls:
        passthrough: true

  services:
    spice:
      loadbalancer:
        servers:
          - address: "10.10.10.100:3128"
    spice-tls:
      loadBalancer:
        servers:
          - address: "10.10.10.100:61001"

Hope it can help someone else with the same problem.

18

Interesting, I would have expected that this

will enable TLS in Traefik and it therefore will create a custom Traefik TLS cert which your client doesn’t know.

TCP connections are usually not decrypted, but always passed plain (encrypted), so I would expect you don’t need the setting.

Update: never mind, you do use HostSNI with a domain, so there must be a know cert.

19

what would be the file setup if I was just using the proxmox web interface at 192.168.0.121

20

Something like this. Pretty standard router and service as indicated in the documentation. Link

http:
  routers:
    pve:
      rule: Host(`proxmox.domain.tld`)
      service: pve-service
      entryPoints:
        - "https"
      tls:
        certResolver: myresolver

  services:
    pve-service:
      loadBalancer:
        servers:
          - url: "https://192.168.0.121:8006"
        healthCheck:
          path: "/#v1:0:18:4:::::::"
          interval: "10s"
          timeout: "3s"
1 Like
21

Ok, Finally made it work. I'll share my entire setup:

My entire setup is based on TechnoTim's tutorial. Follow his tutorial for the proxy network, Cloudflare API token, dashboard credentials and .env and acme.json files.

Here goes:

docker-compose.yml

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
      - 3128:3128 ## SPICE PORTS
      - 61000:61000
      # - 443:443/tcp # Uncomment if you want HTTP3
      # - 443:443/udp # Uncomment if you want HTTP3
    environment:
      CF_DNS_API_TOKEN: ${CF_DNS_API_TOKEN}
      TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.${TOP_LEVEL_DOMAIN}`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.${TOP_LEVEL_DOMAIN}`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=${TOP_LEVEL_DOMAIN}"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.${TOP_LEVEL_DOMAIN}"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

data/traefik.yml

api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

  # for proxmox's SPICE protocol
  spice:
    address: ":3128"
  spice-tls:
    address: ":61000" # check if your spice.vv uses 61000 or 61001

serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml

certificatesResolvers:
  cloudflare:
    acme:
      email: your@email.com
      storage: acme.json
      caServer: https://acme-v02.api.letsencrypt.org/directory # prod (default)
      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        #delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is ready before verification is attempted
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

data/config.yml

# HTTP
http:
# Routers
  routers:
    proxmox:
      entryPoints:
        - "https"
      rule: "Host(`pve.domain.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: proxmox

# HTTP Services
  services:
    proxmox:
      loadBalancer:
        servers:
          - url: "https://0.0.0.0:8006" # your proxmox IP

# HTTP Middlewares
  middlewares:
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true
    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipAllowList:
        sourceRange:
        - "0.0.0.0/24" #your subnet

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

## TCP
tcp:
  routers:
    proxmox-spice:
      rule: "HostSNI(`*`)"
      entrypoints: "spice"
      tls: false
      service: proxmox-spice
    proxmox-spice-tls:
      rule: "HostSNI(`pve.domain.com`)"
      entrypoints: "spice-tls"
      tls:
        passthrough: true
      service: proxmox-spice-tls

  services:
    proxmox-spice:
      loadbalancer:
         servers:
          - address: "0.0.0.0:3128" # your proxmox IP
    proxmox-spice-tls:
      loadbalancer:
         servers:
          - address: "0.0.0.0:61000" # proxmox IP.  Also check if your instance uses 61000 or 61001