The Ingress Object itself already has a long history with K8s. It is still considered beta, which is kinda surprising for something that has been so long present in K8s. But why is that? And when will that change?
With the release of Kubernetes 1.18, some improvements have been made to Ingress, which have been overdue for a long time. However, the changes introduced are minor, and some of the issues we’ll be covering in this blog post have gone untackled. In addition to covering the issues mentioned above, we’ll be exploring the new Service API aimed at solving these issues.
In this blog post we’ll cover some of the long-standing issues with the current state of Ingress in Kubernetes, including these topics:
- Inflexible HTTP routing definitions
- Schema differences across vendors
- Extensibility of Ingress
Inflexible HTTP routing definitions
A simple Ingress object example is the following:
--- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: test-ingress spec: rules: - http: paths: - Host: myhost.com path: /testpath pathType: Prefix backend: serviceName: test servicePort: 80
The above Ingress object will route HTTP requests with the URI GET http://myhost.com/testpath and forward the request internally to the service called test on port 80. So far, so good.
The primary focus of an ingress resource is on solving simple HTTP routing cases, similar to the concept of Virtual Hosts with a Path Based routing extension. This leads us to the first issue: How would you configure cases like a redirect?
Schema differences across vendors
Further configuration is usually done through annotations on the Ingress resource. Annotations are like key-value pairs stored in the metadata of an object.
--- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: test-ingress nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - http: paths: - Host: myhost.com path: /testpath pathType: Prefix backend: serviceName: test servicePort: 80
The above example shows that the Ingress Controller (nginx) would rewrite all requests to / (slash) before forwarding to the backend.
Consider that the same configuration in Traefik would look like this:
--- apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: rewrite-slash spec: replacePath: path: / --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: whoami-ingress annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/router.middlewares: [email protected] spec: rules: - host: whoami.localhost http: paths: - path: /foobar backend: serviceName: whoami servicePort: web
As you can see, it’s totally different! Which brings us to another issue: extensibility.
Extensibility of Ingress
Extending Ingress Objects is a requirement and the de-facto standard for that is using annotations. However, annotations often differ between the many different implementations of Ingress Controllers and it's therefore hard to manage for an end-user.
This unmanageability also translates back to the providers who must maintain their ingress controller implementation, who are often constrained by the simplicity of the key / value pair approach.
Service API aka Ingress V2
Announced at Kubecon NA in 2019 by Google there has been substantial effort in creating an “Ingress V2” which is now known as the Service API.
This specification aims to solve a few problems:
- Provide clean separation and role-based control
- Uplevel the Ingress specification
- Specify standard methods of extending the Ingress specification
To do so, the specification currently consists of 4 different CRD Ressources:
The GatewayClass is meant to be a Cluster Scoped resource, which is meant to represent a “category” of gateways. It’s similar to the former `ingress.class` annotation or the now included IngressClass resource.
The expected use-case is to have more than one GatewayClass per Ingress Controller provider. These classes may have a variety of default settings, which are inherited by the cluster-level gateway. Also, this can be used to pass additional configuration down to that gateway. Since it is a cluster scoped resource, it's expected to be managed by the Infrastructure provider.
--- kind: GatewayClass metadata: name: cluster-gateway spec: controller: "acme.io/gateway-controller" parametersRef: apiVersion: core/v1 kind: ConfigMap namespace: acme-system name: internet-gateway
The gateway has a life-cycle which is tied with the infrastructure. For instance, one Gateway could be running an instance of Traefik or one AWS ELB. As already mentioned, it’s linked to a Gateway class for inferring configuration.
The gateway sets listener bindings (Address, Ports, TLS…) and the routes served by the gateway.
--- kind: Gateway name: my-app-gw namespace: my-app spec: class: from-internet listeners: - address: ip: 184.108.40.206 protocols: ["http"] # implies port 80. routes: ... - address: ip: 220.127.116.11 protocols: ["https"] # implies port 443 certificates: - name: my-secret - apiGroup: certmanager.k8s.io kind: Certificate name: lets-encrypt-cert routes: - route: name: http-app-1 namespace: app-1 kind: HTTPRoute
It also has some sane default protocols like: http, https, TCP… which map to predefined ports.
Last but not least, a route is used to describe a way to handle traffic given protocol level descriptions.
A route can be of a different ressource (HTTPRoute, TLSRoute, TCPRoute…) and can therefore have different protocol level descriptions taken into consideration for routing. Each of these Protocols have different attributes which could be used for route matching. It can also be used to delegate to other Route Resources in a multi-tenant scenario, for example, where one team offers a global authentication service where you would want to forward from within your current scope.
--- kind: HTTPRoute name: delegate-1 namespace: other-team rules: - match: http: host: bar.com path: prefix: /store action: backend: name: delegate-1 namespace: other-team kind: HTTPRoute apiGroup: networking.k8s.io
In the above HTTPRoute example it’s listening for traffic with a host of bar.com and looking for a Pathprefix of /store, which is close to some of the examples we’ve explored on traditional Ingress but allows for some additional specifics.
Taking what we just learned to the next level, the question that arrives now is: “How can we extend that, for example, to implement Rewrites?”
For that purpose, the specification currently offers three different levels of support:
Core functionality is guaranteed between all solutions respecting a specific set of API’s. Extended is a standardised API, but functionality is not guaranteed between all solutions.
For special use-cases, which might be vendor specific, there’s a custom layer where the implementation can be custom built to meet specific requirements. Usually, these will end up in CRD’s or custom annotations.
Service API is a new set of forward-looking API’s attempting to solve some issues that have become apparent over the evolving usage of Ingress. However, as it’s a bit more complex and might not solve all the simple use cases Ingress is capable of solving, it’s not meant to replace Ingress but rather provide an alternative for complex use cases.
This is a companion discussion topic for the original entry at https://containo.us/blog/kubernetes-ingress-service-api-demystified/