Invalid memory address failure

kechf · January 28, 2020, 10:27am

Hi everyone,
our traefik containers on different swarm nodes failed randomly with following error...

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x578f3f]




goroutine 3866 [running]:


bufio.(*Writer).Available(...)


	/usr/local/go/src/bufio/bufio.go:607


bufio.(*Writer).WriteString(0x0, 0x2d06c01, 0x19, 0x8, 0xc0008f4e00, 0x42c7bf)


	/usr/local/go/src/bufio/bufio.go:688 +0x7f


net/http.(*expectContinueReader).Read(0xc0008d15e0, 0xc000786000, 0x4000, 0x4000, 0x7f88895ac2b0, 0xc0008d15e0, 0x40bcd9)


	/usr/local/go/src/net/http/server.go:890 +0x13b


github.com/containous/traefik/vendor/golang.org/x/net/http2.(*clientStream).writeRequestBody(0xc0005f8000, 0x7f88895ac290, 0xc0008d15e0, 0x7f88895ac2b0, 0xc0008d15e0, 0x0, 0x0)


	/go/src/github.com/containous/traefik/vendor/golang.org/x/net/http2/transport.go:1054 +0x526


github.com/containous/traefik/vendor/golang.org/x/net/http2.(*Transport).getBodyWriterState.func1()


	/go/src/github.com/containous/traefik/vendor/golang.org/x/net/http2/transport.go:2250 +0xc2


sync.(*Once).Do(0xc000fce600, 0xc000e17830)


	/usr/local/go/src/sync/once.go:44 +0xb3


github.com/containous/traefik/vendor/golang.org/x/net/http2.(*Transport).getBodyWriterState.func2()


	/go/src/github.com/containous/traefik/vendor/golang.org/x/net/http2/transport.go:2268 +0x37


created by time.goFunc


	/usr/local/go/src/time/sleep.go:169 +0x44

Can everybody help us?

We use traefik version 1.7.20!

daniel.tomcej · January 28, 2020, 10:35am

Hello @kechf,

That error can be caused by an "out of disk space" situation.

Can you confirm that you have available space?

The buffering middleware writes to the /tmp dir on the container, which is configured as a volume in the dockerfile, so it may use more disk space on the host.

kechf · January 28, 2020, 11:11am

It seems, that some very large requests could raise these errors. Can this be possible?

daniel.tomcej · January 28, 2020, 11:19am

@kechf How large are you talking?

The buffering middleware creates a file to store the buffered request.

It is also possible that there may be an issue with the body being closed prematurely.

Can you provide your Traefik configuration as well?

kechf · January 28, 2020, 11:35am

Do you want the complete toml configuration of my traefik instances?

daniel.tomcej · January 28, 2020, 1:23pm

Just static config, and the labels on one of your containers that its crashing with.

Just to figure out the configuration of the middlewares etc.

daniel.tomcej · January 28, 2020, 3:58pm

Hmmmm ok, it doesn't appear that the buffering middleware is enabled on those backends, so its very possible that the large request headers may be causing issues.

Do you happen to know the size of the headers (in KB)?

May help pin down where the problem is.

kechf · January 28, 2020, 4:48pm

Here, for example, the beginning of one of the crazy requests (from the log in traefik, before the complete container failed)...

time="2020-01-28T08:00:16Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/identity.service/connect/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/json, application/

This is only the beginning of that requests...

kechf · January 28, 2020, 4:50pm

I don't know the concrete size of these request headers. But in this case min. 17 Kb.

kechf · January 29, 2020, 10:43am

Hi Daniel,

wie found out, that too large requests (with large headers) cause that issue!
We can reproduce that issue with specific requests.

Up to approx 20.0000 characters in headers causes this issue in traefik!!

kechf · February 1, 2020, 2:28pm

Hi Daniel,

here our static config...

################################################################
# Global configuration
################################################################

# Enable debug mode
#
# Optional
# Default: false
#
# debug = true

# Log level
#
# Optional
# Default: "ERROR"
#
logLevel = "DEBUG"

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
#
# Optional
# Default: ["http"]
#
defaultEntryPoints = ["http", "https"]

insecureSkipVerify = true

################################################################
# Entrypoints configuration
################################################################

# Entrypoints definition
#
# Optional
# Default:
[entryPoints]
    [entryPoints.http]
    address = ":80"
    [entryPoints.https]
    address = ":443"
        [entryPoints.https.tls]
            [[entryPoints.https.tls.certificates]]
            certFile = "/certs/prd.de.mycompany.com.pem"
            keyFile = "/certs/prd.de.mycompany.com.key"

################################################################
# Traefik logs configuration
################################################################

# Traefik logs
# Enabled by default and log to stdout
#
# Optional
#
[traefikLog]

# Sets the filepath for the traefik log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
#
# Optional
# Default: os.Stdout
#
# filePath = "log/traefik.log"

# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format = "common"

################################################################
# Access logs configuration
################################################################

# Enable access logs
# By default it will write to stdout and produce logs in the textual
# Common Log Format (CLF), extended with additional fields.
#
# Optional
#
[accessLog]

# Sets the file path for the access log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
#
# Optional
# Default: os.Stdout
#
# filePath = "/path/to/log/log.txt"

# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format = "common"

################################################################
# API and dashboard configuration
################################################################

# Enable API and dashboard
[api]

  # Name of the related entry point
  #
  # Optional
  # Default: "traefik"
  #
  # entryPoint = "traefik"

  # Enabled Dashboard
  #
  # Optional
  # Default: true
  #
  # dashboard = false

################################################################
# Ping configuration
################################################################

# Enable ping
[ping]

  # Name of the related entry point
  #
  # Optional
  # Default: "traefik"
  #
  # entryPoint = "traefik"

################################################################
# Docker configuration backend
################################################################

# Enable Docker configuration backend
[docker]

# Docker server endpoint. Can be a tcp or a unix socket endpoint.
#
# Required
# Default: "unix:///var/run/docker.sock"
#
# endpoint = "tcp://xx.xx.xx.xx:2375"

# Default domain used.
# Can be overridden by setting the "traefik.domain" label on a container.
#
# Optional
# Default: ""
#
domain = "prd.de.mycompany.com"

# Expose containers by default in traefik
#
# Optional
# Default: true
#
exposedByDefault = false

# Enable watch docker changes
#
# Optional
#
watch = true

# Use Docker Swarm Mode as data provider
#
# Optional
#
swarmmode = true

# Polling interval (in seconds) for Swarm Mode.
#
# Optional
# Default: 15
#
swarmModeRefreshSeconds = 15

# Define a default docker network to use for connections to all containers.
# Can be overridden by the traefik.docker.network label.
#
# Optional
#
network = "gateway-prd_traefik-net"

constraints = ["tag==prd"]

# Metrics definition
[metrics]
  #...

  # To enable Traefik to export internal metrics to Prometheus
  [metrics.prometheus]

    # Name of the related entry point
    #
    # Optional
    # Default: "traefik"
    #
    entryPoint = "traefik"

    # Buckets for latency metrics
    #
    # Optional
    # Default: [0.1, 0.3, 1.2, 5.0]
    #
    buckets = [0.1,0.3,1.2,5.0]

And here our dynamic file configuration with the concrete backend, which occurs that issue...
When we disable that front/-backend traefik works well...

[backends.identity-service]
    [backends.identity-service.servers]
      [backends.identity-service.servers.server0]
        url = "https://xxx.mycompany.corp/Identity.Service:443"
        weight = 1
      [backends.identity-service.servers.server1]
        url = "https://xxx.mycompany.corp/Identity.Service:443"
        weight = 1
    [backends.identity-service.healthCheck]
      path = "Identity.Service/api/status/check"
      interval = "2s"
      port = 443
    [backends.identity-service.loadBalancer]
      method = "wrr"
      [backends.identity-service.loadBalancer.stickiness]


[frontends.identity-service]
    entryPoints = ["http", "https"]
    backend = "identity-service"
    passHostHeader = true
    [frontends.identity-service.routes]
      [frontends.identity-service.routes.route1]
        rule = "Host:identity-service.prd.de.mycompany.com;ReplacePathRegex: ^(/identity.service)?(.*) /identity.service$2"
    [frontends.identity-service.redirect]
      entryPoint = "https"
      permanent = true

We found out, that some requests, which calls the identity-service frontend have very large headers and bodies (e.g. the header is hundreds of lines long). We assume, that these requests are the reasons for our issues in the moment.

daniel.tomcej · February 3, 2020, 3:45pm

Hello @kechf,

That's awesome to hear that you found the cause of this issue.

Can you please open a ticket on github, and provide this information?

Large requests should not cause a panic.

Thanks!

Topic		Replies	Views
Help Requested! Error during Docker Run Traefik v2 docker	1	526	August 13, 2022
Invalid memory address or nil pointer dereference after updating to 2.4.11 Traefik v2 docker	2	745	July 16, 2021
Error 504 with Traefik on Docker Swarm - again Traefik v2 docker , docker-swarm	10	520	January 4, 2024
Traefik docker container keeps crashing Traefik v2 docker	5	676	December 2, 2023
Custom SSL certificate results in runtime error Traefik v2 docker , file	2	1423	February 6, 2021

Invalid memory address failure

Related topics