Routing the SSH port for GitLab container

Traefik Traefik v2
1

I set up Traefik and GitLab in docker containers. They work, except I can't clone from GitLab through SSH. I saw several discussions about this but I can't get mine to work.

Notes:

  • The server's sshd uses port 22
  • port 2232 is forwarded, it's also allowed by my firewall
  • cloning through HTTPS works

I followed these setups (among others):
https://github.com/realtarget/traefik2-docker-stack
(see the issue which suggests HostSNI(`*`))
And this: https://github.com/harveyconnor/traefikv2-docker-gitlab/blob/master/gitlab/docker-compose.yml

My config files:

docker-compose.yml for Traefik:

version: '3.7'

networks:
  default:
    driver: bridge
  traefik_proxy:
    name: traefik_proxy
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.12.0/24

services:
  reverse-proxy:
    container_name: reverse-proxy
    image: traefik:2.2.7
    restart: unless-stopped
    environment:
      - CF_API_EMAIL=$CLOUDFLARE_EMAIL
      - CF_API_KEY=$CLOUDFLARE_API_KEY
    security_opt:
      - no-new-privileges:true
    networks:
      traefik_proxy:
        ipv4_address: 192.168.12.34
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 22
        published: 2232
        protocol: tcp
        mode: host
    volumes:
      - ./:/etc/traefik/
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.toml:/traefik.toml
      - ./traefik.log:/traefik.log
      - /srv/certs:/etc/certs
      - ./shared:/shared
    labels:
      - 'traefik.enable=true'
      - 'traefik.docker.network=traefik_proxy'

docker-compose.yml for GitLab:

version: '3.7'

networks:
  #default:
    #driver: bridge
  traefik_proxy:
    #driver: bridge
    external:
      name: traefik_proxy

secrets:
  gitlab_root_password:
    file: ./secrets/root_password

services:
  gitlab:
    container_name: gitlab
    image: gitlab/gitlab-ce:latest
    restart: always
    hostname: 'gitlab.mydomain.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.mydomain.com'
        gitlab_rails['gitlab_shell_ssh_port'] = 2232
        gitlab_rails['initial_root_password'] = File.read('/run/secrets/gitlab_root_password')
        gitlab_rails['lfs_enabled'] = true
        nginx['listen_https'] = false
        nginx['listen_port'] = 80
        nginx['proxy_set_headers'] = {
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }
    secrets:
      - gitlab_root_password
    networks:
      - traefik_proxy
      - default
    #ports:
    #  - '2232:22'
    volumes:
      - $GITLAB_HOME/config:/etc/gitlab:Z
      - $GITLAB_HOME/logs:/var/log/gitlab:Z
      - $GITLAB_HOME/data:/var/opt/gitlab:Z
    labels:
      - 'traefik.enable=true'
      - 'traefik.docker.network=traefik_proxy'
      - 'traefik.port=80'
      - 'traefik.http.routers.gitlab.entrypoints=websecure'
      - 'traefik.http.routers.gitlab.rule=Host(`gitlab.mydomain.com`)'
      - 'traefik.http.routers.gitlab.tls=true'
      - 'traefik.http.routers.gitlab.service=gitlab-svc' 
      - 'traefik.http.services.gitlab-svc.loadbalancer.server.port=80'
      # Headers
      - 'traefik.http.routers.gitlab.middlewares=gitlab-headers'
      - 'traefik.http.middlewares.gitlab-headers.headers.customrequestheaders.X_FORWARDED_PROTO=https'
      - 'traefik.http.middlewares.gitlab-headers.headers.customrequestheaders.X_Forwarded-Ssl=on'
      - 'traefik.http.middlewares.gitlab-headers.headers.customresponseheaders.X_FORWARDED_PROTO=https'
      - 'traefik.http.middlewares.gitlab-headers.headers.customresponseheaders.X_Forwarded-Ssl=on'
      # Registry
      - 'traefik.http.routers.gitlab-registry.rule=Host(`gitlab-registry.mydomain.com`)'
      - 'traefik.http.routers.gitlab-registry.entrypoints=websecure'
      #- "traefik.http.routers.gitlab-registry.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.gitlab-registry.service=gitlab-registry'
      - 'traefik.http.services.gitlab-registry.loadbalancer.server.port=8500'
      # SSH
      - 'traefik.tcp.routers.gitlab-ssh.entrypoints=ssh-gitlab'
      - 'traefik.tcp.routers.gitlab-ssh.rule=HostSNI(`*`)'
      - 'traefik.tcp.routers.gitlab-ssh.service=gitlab-ssh-svc'
      - 'traefik.tcp.services.gitlab-ssh-svc.loadbalancer.server.port=22'

(see the labels for SSH at the bottom).

traefik.toml:

[log]
  level = "WARN"

[providers]
  [providers.docker]
    exposedByDefault = false
    watch = true
    network = "traefik-proxy"
    endpoint = "unix:///var/run/docker.sock"
  [providers.file]
    directory = "/etc/traefik/dynamic"
    watch = true

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.forwardedHeaders]
      trustedIPs = ["... [omitted] ..."]
  [entryPoints.ssh-gitlab]
    address = ":2232"
  [entryPoints.traefik]
    address = ":9000"

[accessLog]
  filePath = "/traefik.log"
  bufferingSize = 100
  [accessLog.filters]
    statusCodes = "400-499"

[api]
  dashboard = true

Dynamic config file traefik_dynamic.toml:

[http.routers.http-catchall]
  entryPoints = ["web"]
  middlewares = ["https-redirect"]
  rule = "HostRegexp(`{any:.+}`)"
  service = "noop"

[http.services]
  [http.services.noop.loadBalancer]

[[tls.certificates]]
  certFile = "/etc/certs/mydomain.com/mydomain.com.pem"
  keyFile = "/etc/certs/mydomain.com/mydomain.com.key"

[http.routers.traefik-rtr]
  entryPoints = ["websecure"]
  rule = "Host(`traefik.mydomain.com`)"
  service = "api@internal"
  middlewares = ["chain-authelia@file"]
  [http.routers.traefik-rtr.tls]
    #certResolver = "dns-cloudflare" # Comment out after first run, to force wildcard certs
    [[http.routers.traefik-rtr.tls.domains]]
      main = "mydomain.com"
      sans = ["*.mydomain.com"]

When I try to clone over SSH, it hangs for a good minute or two, then says:

ssh: connect to host gitlab.mydomain.com port 2232: Network is unreachable
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

My SSH keys are set up correctly.

What can I do?

Thanks for taking a look!

2

It looks like you have a mistake in the Docker Compose definition for Traefik:

      - target: 22
        published: 2232
        protocol: tcp
        mode: host

This forwards the Host's port 2232 to the container's port 22.

However, in the Traefik configuration, you have this:

  [entryPoints.ssh-gitlab]
    address = ":2232"

You need to choose only one place in the configuration to proxy port 2232 to the GitLab container's port 22. Either do it in Docker Compose (in which case Traefik should be listening on port 22), or do it in Traefik (in which case Docker Compose should have the target port set to 2232).

However, I would not do either of those things. Instead, I would move the server's sshd to an alternate port and let GitLab use port 22. There are two reasons for this:

  1. Configuring the git command to use an alternate SSH port is annoying, and all of your users will have to do it.
  2. There are always a lot more automated attacks against the standard port number. If an attack would somehow succeed, the damage would be confined within the GitLab container (which already locks down SSH to prevent shell access).
3 Likes
3

Thanks for the alternative suggestion.

Unfortunately I still can't get it to work.
As you suggested I let GitLab use port 22, and changed the sshd port to something else.

I set GitLab's environment var for SSH just in case:

  environment:
    GITLAB_OMNIBUS_CONFIG: |
    ...
    gitlab_rails['gitlab_shell_ssh_port'] = 22

and I removed SSH routing from Traefik's configs, and the SSH labels for GitLab's container.

But I still couldn't clone over SSH, "network unreachable", and HTTPS clone is working fine.

Am I doing something wrong again?

4

Here is the config that I am using:

Entrypoints in traefik.toml 

[entryPoints]
  [entryPoints.ssh]
    address = ":22"

  [entryPoints.http]
    address = ":80"
    [entryPoints.http.http.redirections.entryPoint]
      to = "https"
      scheme = "https"

  [entryPoints.https]
    address = ":443"

docker-compose.yml 

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    ports:
    - "22:22"
    - "80:80"
    - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./traefik/traefik.toml:/traefik.toml"
      - "./traefik/acme.json:/acme.json"

  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab
    restart: always
    hostname: 'git.example.com'
    volumes:
      - './gitlab/config:/etc/gitlab'
      - './gitlab/logs:/var/log/gitlab'
      - './gitlab/data:/var/opt/gitlab'
    labels:
      - traefik.enable=true

      - traefik.http.routers.gitlab.entrypoints=https
      - "traefik.http.routers.gitlab.rule= Host(`git.example.com`) || Host(`registry.example.com`)"
      - traefik.http.routers.gitlab.tls=true
      - traefik.http.routers.gitlab.tls.certresolver=letsencrypt
      - traefik.http.services.gitlab.loadbalancer.server.port=80

      - traefik.tcp.routers.gitlab-ssh.rule=HostSNI(`*`)
      - traefik.tcp.routers.gitlab-ssh.entrypoints=ssh
      - traefik.tcp.routers.gitlab-ssh.service=gitlab-ssh
      - traefik.tcp.services.gitlab-ssh.loadbalancer.server.port=22

I have nothing in the GitLab configuration (which I have in the volume mounted from ./gitlab/config to /etc/gitlab) about SSH at all.

3 Likes
5

Hmm.. I have separate docker-compose files and use docker networks, and some of my labels are different. But the web app and cloning with HTTPS work, so the network setup shouldn't be a problem.

traefik.toml:

[entryPoints]
    [entryPoints.ssh]
        address = ":22"
    [entryPoints.web]
    	address = ":80"
  	[entryPoints.websecure]
    	address = ":443"

Traefik docker compose file:

networks:
  default:
    driver: bridge
  traefik_proxy:
    name: traefik_proxy
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.12.0/24
  socket_proxy:
    name: socket_proxy
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.13.0/24

services:
  reverse-proxy:
    container_name: reverse-proxy
    image: traefik:2.2.7
    restart: unless-stopped
    environment:
      - CF_API_EMAIL=#...
      - CF_API_KEY=#...
    depends_on:
      - socket-proxy
    security_opt:
      - no-new-privileges:true
    networks:
      traefik_proxy:
        ipv4_address: 192.168.12.200
      socket_proxy:
    ports:
	  - target: 22
	    published: 22
	    protocol: tcp
	    mode: host
	  - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - ./:/etc/traefik/
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.toml:/traefik.toml
      - ./traefik.log:/traefik.log
      - /etc/certs:/etc/certs
      - ./shared:/shared
    labels:
      - 'traefik.enable=true'
      - 'traefik.docker.network=traefik_proxy'

  socket-proxy:
    container_name: socket-proxy
    image: tecnativa/docker-socket-proxy
    restart: always
    networks:
      socket_proxy:
        ipv4_address: 192.168.13.200
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

GitLab docker compose file:

version: '3.7'

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  private:
    name: private
    external: false
  gitlab_network:
    name: gitlab_network
    external: false

services:
  gitlab:
    container_name: gitlab
    image: gitlab/gitlab-ce:latest
    restart: always
    hostname: 'gitlab.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        gitlab_rails['gitlab_shell_ssh_port'] = 22
        gitlab_rails['lfs_enabled'] = true
        nginx['listen_https'] = false
        nginx['listen_port'] = 80
        nginx['proxy_set_headers'] = {
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }
    networks:
      - traefik_proxy
      - default
      - gitlab_network
      - private
    volumes:
      - $GITLAB_HOME/config:/etc/gitlab:Z
      - $GITLAB_HOME/logs:/var/log/gitlab:Z
      - $GITLAB_HOME/data:/var/opt/gitlab:Z
    labels:
      - 'traefik.enable=true'
      - 'traefik.docker.network=traefik_proxy'
      - 'traefik.port=80'
      - 'traefik.http.routers.gitlab.entrypoints=websecure'
      - 'traefik.http.routers.gitlab.rule=Host(`gitlab.example.com`)'
      - 'traefik.http.routers.gitlab.tls=true'
      - 'traefik.http.routers.gitlab.service=gitlab-svc'
      - 'traefik.http.services.gitlab-svc.loadbalancer.server.port=80'
      # Headers
      - 'traefik.http.routers.gitlab.middlewares=gitlab-headers'
      - 'traefik.http.middlewares.gitlab-headers.headers.customrequestheaders.X_FORWARDED_PROTO=https'
      - 'traefik.http.middlewares.gitlab-headers.headers.customrequestheaders.X_Forwarded-Ssl=on'
      - 'traefik.http.middlewares.gitlab-headers.headers.customresponseheaders.X_FORWARDED_PROTO=https'
      - 'traefik.http.middlewares.gitlab-headers.headers.customresponseheaders.X_Forwarded-Ssl=on'
      # Registry
      - 'traefik.http.routers.gitlab-registry.rule=Host(`gitlab-registry.example.com`)'
      - 'traefik.http.routers.gitlab-registry.entrypoints=websecure'
      - 'traefik.http.routers.gitlab-registry.service=gitlab-registry-svc'
      - 'traefik.http.services.gitlab-registry-svc.loadbalancer.server.port=80'
      # SSH
      - 'traefik.tcp.routers.gitlab-ssh.entrypoints=ssh'
      - 'traefik.tcp.routers.gitlab-ssh.rule=HostSNI(`*`)'
      - 'traefik.tcp.routers.gitlab-ssh.service=gitlab-ssh-svc'
      - 'traefik.tcp.services.gitlab-ssh-svc.loadbalancer.server.port=22'
7

I also had the problem that it did not work. My configuration was similar to yours. In the end removing

gitlab_rails['gitlab_shell_ssh_port'] = 22

solved my problem