Traefik 2.2 & Keycloak - Service port not found

Hi!

I'm running a k8s cluster (v1.17.4) with Traefik v2.2, Elastic ECK (1.1.1) and Keycloak 7.0.0. I've successfully deployed the ECK stack and is able to reach, say, Kibana through a Traefik serviced Ingress, however, when I add the Keycloak sidecar I'm unable to reach Kibana (through the "sidecar proxy"). The Keycloak proxy launches successfully and contacts the Keycloak server.

I added the Keycloack proxy to Kibana:

containers:
- name: <name>
  image: <image url>
  args:
    - --config=<config>
  ports:
    - containerPort: 3000

Kibana launches a Service which I refer to in the Ingress. The Ingress I'm using is:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: <...>
  namespace: <...>
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: http
    traefik.ingress.kubernetes.io/router.middlewares: <...>
spec:
  rules:
  - http:
      paths:
      - path: <...>
        backend:
          serviceName: <Kibana service name>
          servicePort: 3000

In the ConfigMap for the Keycloak sidecar I have (among other settings):

listen: 0.0.0.0:3000 (tried :3000 as well)
redirection-url: <URL to Ingress path>
upstream: http://127.0.0.1:5601 (Kibana port)

The error I get is:

time="2020-09-30T11:59:33Z" level=error msg="Cannot create service: service port not found" providerName=kubernetes namespace=<elastic namespace> ingress=<Ingress name here> serviceName=<service name here> servicePort=3000

time="2020-09-30T11:59:34Z" level=error msg="Cannot create service: service port not found" servicePort=3000 serviceName=<service name here> providerName=kubernetes ingress=<Ingress name here> namespace=<elastic namespace>

I've added a NetworkPolicy in the Traefik and Elastic namespaces looking like (in order to allow traffic between them. No other NP exist so I know defining them is not needed but...):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: <name>
  namespace:<namespace name>
spec:
  podSelector: {}
  ingress:
  - from
      - namespaceSelector: {}

I must be missing something here. No certificates are involved yet - I wanted to get this up and running before adding them.

Many thanks in advance!

The first thing I'd try is to eliminate (or prove) Traefik as a suspect. Try reaching your Kibana (through the "sidecar proxy") via the service, the ingress is pointing to. If that works, then we'll focus on traefik, if not, then something else is wrong with your configuration.

Oh, and remove the NetworkPolicy, before you do anything else.

Hi!

Many thanks for you reply and suggestions! I've now "exposed" the service through a NodePort to 32767 and when I access the service using http://host:32767 I'm presented with the Keycloak login screen. I can see that the redirectionUrl is present in the URL pointing to my Kibana URL as configured in my Gatekeeper ConfigMap. After successful login in I can see that I get redirected to the correct URL however with "/oauth/callback?state..." added which will not be picked up by my Traefik Ingress which assume the Kibana ServerBasePath is simply "/service/logging/kibana". I "think" this could be the problem I'm facing. Do you have any thoughts about how to fix this?

Again many thanks :slight_smile:

What's in your path?

You may set the Kibana ServerBasePath while deploying it and I've set it to /service/logging/kibana. The same goes for the Ingress path.

Well, this is what the matcher is using then, it won't match anything else.

Here is the routers documentation: https://doc.traefik.io/traefik/routing/routers/#rule

And I fully understand that :slight_smile: I guess I have to read up on the Keycloak documentation... Thank you for your efforts :slight_smile:

Yeah, sorry, so many different technologies and product are used with traefik, I don't think I can be expert in all of them, and the experts in other products mostly do not come to traefik forum to talk about it :wink: If you can translate what you want to achieve in traefik terms, we can take it from there!

I fully understand, there are many parts involved in this and add to that my lack of experience integrating these two. Now I've prepared a rather lengthy explanation in order to try to explain the problem I'm facing.

I want to use Keycloak as a standard way of authenticating users to applications running in our Kubernetes clusters. One of the clusters is running the Elastic ECK component (v1.1.1) and we use the operator to deploy Elastic clusters and Kibana as a frontend. In order to keep things as simple as possible I’ve done the following. Please note that I've removed all the "http":s as I was not allowed to post this otherwise.

Deployed Kibana

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: {{ .Values.kibana.name }}
  namespace: {{ .Release.Namespace }}
  annotations:
    traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
spec:
  version: {{ .Values.kibana.version }}
  count: {{ .Values.kibana.instances }}
  elasticsearchRef:
    name: {{ .Values.kibana.elasticCluster }}
    namespace: {{ .Release.Namespace }}
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
        - name: SERVER_BASEPATH
          value: {{ .Values.kibana.serverBasePath }}
        resources:
          requests:
            {{- if not .Values.kibana.cpu.enableBurstableQoS }}
            cpu: {{ .Values.kibana.cpu.requests }}
            {{- end }}
            memory: {{ .Values.kibana.memory.requests }}Gi
          limits:
            {{- if not .Values.kibana.cpu.enableBurstableQoS }}
            cpu: {{ .Values.kibana.cpu.limits }}
            {{- end }}
            memory: {{ .Values.kibana.memory.limits }}Gi
  http:
    tls:
      selfSignedCertificate:
        disabled: true

Created Ingress

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: kibana-{{ .Values.kibana.name }}-stripprefix
  namespace: {{ .Release.Namespace }}
spec:
  stripPrefix:
    prefixes: 
      - {{ .Values.kibana.serverBasePath }}
    forceSlash: true

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: {{ .Values.kibana.name }}-ingress
  namespace: {{ .Release.Namespace }}
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: http
    traefik.ingress.kubernetes.io/router.middlewares: {{ .Release.Namespace }}-kibana-{{ .Values.kibana.name }}-stripprefix@kubernetescrd
spec:
  rules:
  - http:
      paths:
      - path: {{ .Values.kibana.serverBasePath }}
        backend:
            servicePort: {{ .Values.kibana.port }}
            serviceName: {{ .Values.kibana.name }}-kb-http

Result
Deploying the above works perfectly fine. I’m able to reach the Kibana UI using the external IP exposed by our MetalLB component and being correctly routed through Traefik. I simply enter "external IP/service/logging/kibana" and I’m presented to the Kibana log in screen and I can log on using the “built in” authentication process.

Adding the Keycloak Gatekeeper
Now, if I add the following to the Kibana manifest, effectively adding the Keycloak Gatekeeper sidecar to the Kibana Pod:

  - name: {{ .Values.kibana.name }}-gatekeeper
    image: "{{ .Values.kibana.keycloak.gatekeeper.repository }}/docker-r/keycloak/keycloak-gatekeeper:{{ .Values.kibana.keycloak.gatekeeper.version }}"
    args:
      - --config=/etc/keycloak-gatekeeper.conf
    ports:
      - containerPort: 3000
        name: proxyport
    volumeMounts:
    - name: gatekeeper-config
      mountPath: /etc/keycloak-gatekeeper.conf
      subPath: keycloak-gatekeeper.conf
  volumes:
    - name: gatekeeper-config
      configMap:
        name: {{ .Release.Name }}-gatekeeper-config

with the following ConfigMap which is "mounted":

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-gatekeeper-config 
  namespace: {{ .Release.Namespace }}
data: 
  keycloak-gatekeeper.conf: |+
    redirection-url: {{ .Values.kibana.keycloak.gatekeeper.redirectionUrl }}
    discovery-url: https://.../auth/realms/{{ .Values.kibana.keycloak.gatekeeper.realm }}
    skip-openid-provider-tls-verify: true
    client-id: kibana
    client-secret: {{ .Values.kibana.keycloak.gatekeeper.clientSecret }}
    enable-refresh-tokens: true
    encryption-key: ...
    listen: :3000
    tls-cert:
    tls-private-key:
    secure-cookie: false
    upstream-url: {{ .Values.kibana.keycloak.gatekeeper.upstreamUrl }}
    resources:
    - uri: /*
    groups:
    - kibana

The upstream-url points to "127.0.0.1:5601"

and add an intermediary service:
In order to explicitly address the Gatekeeper proxy I added another service, “keycloak-proxy” as such:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.kibana.name }}-keycloak-proxy
  namespace: {{ .Release.Namespace }}
spec:
  type: ClusterIP
  selector:
    common.k8s.elastic.co/type: kibana
    kibana.k8s.elastic.co/name: cap-logging
  ports:
    - name: http
      protocol: TCP
      port: 8888
      targetPort: proxyport

and change the backend definition in the Kibana definition to:

servicePort: 8888
serviceName: {{ .Values.kibana.name }}-keycloak-proxy

and then issue the same URL as above, "external IP/service/logging/kibana", I’m redirected to "external IP/oauth/authorize?state=0db97b79-b980-4cdc-adbe-707a5e37df1b" and get an “404 Page not found” error.

If I reconfigure the “keycloak-proxy” service and convert it into a NodePort and expose it on, say, port 32767 and issue an "host IP:32767" I’m presented to the Keycloak login screen on the Keycloak server!

If I look into the Gatekeeper startup log I find the following:

1.6018108005048046e+09 info starting the service {"prog": "keycloak-gatekeeper", "author": "Keycloak", "version": "7.0.0 (git+sha: f66e137, built: 03-09-2019)"}
1.6018108005051787e+09 info attempting to retrieve configuration discovery url {"url": "https://.../auth/realms/...", "timeout": "30s"}
1.601810800537417e+09 info successfully retrieved openid configuration from the discovery
1.6018108005392597e+09 info enabled reverse proxy mode, upstream url {"url": "http://127.0.0.1:5601"}
1.6018108005393562e+09 info using session cookies only for access and refresh tokens
1.6018108005393682e+09 info protecting resource {"resource": "uri: /*, methods: DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT,TRACE, required: authentication only"}
1.6018108005398147e+09 info keycloak proxy service starting {"interface": ":3000"}

This is what I get when I try to access Kibana through the Gatekeeper proxy:
host/service/logging/kibana (gets redirected to) host/oauth/authorize?state=4dbde9e7-674c-4593-83f2-a8e5ba7cf6b5

and the Gatekeeper log:
1.601810963344485e+09 error no session found in request, redirecting for authorization {"error": "authentication session not found"}