Problem Description
I'm running gRPC services behind a Traefik reverse proxy in Docker, but I'm getting 404 errors when making gRPC calls. REST calls work fine, and simple paths work, but complex gRPC method paths fail to match any router.
Environment
- Traefik: v3.x (latest)
- gRPC Services: Go-based with dual setup (gRPC + REST Gateway)
- Docker Compose: Services running in Docker containers
- Protocol: HTTP/2 with h2c for gRPC backend communication
Configuration
Docker Compose Labels
template:
image: template-local:latest
networks:
- backend-network
labels:
traefik.enable: "true"
traefik.http.routers.template.rule: "Host(`api.backend.local`) && PathPrefix(`/template`)"
traefik.http.routers.template.entrypoints: "websecure"
traefik.http.services.template.loadbalancer.server.port: "50001"
traefik.http.middlewares.template-strip.stripprefix.prefixes: "/template"
traefik.http.routers.template.middlewares: "template-strip"
traefik.http.routers.template.tls: "true"
traefik.http.services.template.loadbalancer.server.scheme: "h2c"
gRPC Client Code
conn, err := grpc.NewClient(
"api.backend.local:443",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithUnaryInterceptor(prefixUnaryClientInterceptor("/template")),
)
func prefixUnaryClientInterceptor(prefix string) grpc.UnaryClientInterceptor {
return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
modifiedMethod := prefix + method
return invoker(ctx, modifiedMethod, req, reply, cc, opts...)
}
}
What Works vs What Doesn't
Working Requests
// REST call
{"RequestPath": "/template/", "RouterName": "template@docker", "DownstreamStatus": 415}
// Simple paths
{"RequestPath": "/template/test", "RouterName": "template@docker", "DownstreamStatus": 415}
{"RequestPath": "/template/commonpb.HealthService", "RouterName": "template@docker", "DownstreamStatus": 415}
Failing Request
// gRPC method call
{"RequestPath": "/template/commonpb.HealthService/CheckHealth", "RouterName": "", "DownstreamStatus": 404}
Verification Steps Completed
- gRPC Server is Running:
netstatconfirms service listening on port 50001 - gRPC Service Available:
grpcurl -plaintext localhost:50001 listshowscommonpb.HealthService - Router Exists: Simple paths successfully match
template@dockerrouter - Backend Connectivity: 415 errors confirm requests reach the backend
Error Pattern
The issue is specifically with complex gRPC method paths containing multiple forward slashes. The pattern shows:
/template/ Works/template/commonpb.HealthService Works/template/commonpb.HealthService/CheckHealth No router found (404)
Client Error
could not check health: code=Unimplemented, message=unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
Additional Context
- The same router configuration works perfectly for REST endpoints
- The gRPC server responds correctly when accessed directly (not through Traefik)
- Using
h2cscheme for HTTP/2 cleartext communication with gRPC backend - Both services (template and user) have identical label configurations, but only complex gRPC method paths fail
Any insights on why Traefik's router matching behaves differently for complex gRPC method paths would be greatly appreciated!
Static Traefik Configuration (traefik.yml)
global:
checkNewVersion: false
sendAnonymousUsage: false
entryPoints:
web:
address: :80
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: :443
http:
tls: {}
certificatesResolvers:
staging:
acme:
email: company+traefik@backend.com
storage: /tmp/acme.json
caServer: https://acme-staging-v02.api.letsencrypt.org/directory
httpChallenge:
entryPoint: web
production:
acme:
email: company+traefik@backend.com
storage: /tmp/acme.json
caServer: https://acme-v02.api.letsencrypt.org/directory
httpChallenge:
entryPoint: web
log:
level: DEBUG
filePath: /var/log/traefik/traefik.log
accessLog:
format: json
filePath: /var/log/traefik/access.log
api:
insecure: true
dashboard: true
ping:
entryPoint: traefik
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
network: backend_backend-network
constraints: "!Label(`traefik.disable`, `true`)"
watch: true
file:
directory: /etc/traefik/dynamic
watch: true
Dynamic Traefik Configuration (config.yml)
http:
middlewares:
stripServicePrefix:
stripPrefixRegex:
regex:
- "^/[^/]+"
tls:
certificates:
- certFile: "/etc/traefik/dynamic/certs/api.backend.local.pem"
keyFile: "/etc/traefik/dynamic/certs/api.backend.local-key.pem"
Traefik Docker Compose Service Definition
traefik:
image: traefik:latest
command: --configFile=/etc/traefik/traefik.yml
security_opt:
- no-new-privileges:true
ports:
- 80:80 # this will be forwarded to https
- 443:443 # for https
- 8080:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- $PWD/proxy/traefik.yml:/etc/traefik/traefik.yml:ro
- $PWD/proxy/config.yml:/etc/traefik/dynamic/config.yml:ro
- $PWD/proxy/certs:/etc/traefik/dynamic/certs:ro
- $PWD/proxy/acme.json:/tmp/acme.json
- $PWD/proxy/logs:/var/log/traefik
restart: unless-stopped
networks:
- backend-network
environment:
TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}