I face a security issue with Traefik 2.2.8. From an IngressRoute in one namespace, I can access a service in another namespace.
All details are written in issue #7151.
Is this a real issue and the bot did not classify it correctly? Or did I make a mistake in my configuration, as suggested by the bot?
Thanks.
Seems this is a SECURITY ISSUE. Anyone here to confirm or infirm this?
Hello, @jaycci, I've read through your github issue, and I don't see anything wrong with what you described. You configured your IngressRoute to pick up a service from a different namespace, and that's exactly what it does. The whole point of the namespace property of the services stanza is to allow that.
Could you please explain, why do you feel there is any kind of security issue here?
Hello @zespri,
There are 2 main reasons for me to think so.
The namespace next to me may not belong to me. If I have means to gain access to it, or (worse!) to make other people gain access to it (through an Ingress, open to the world), this is a security incident. For a simple example, if my neighbour offers his service through authentication (with some Middleware rules in the Ingress), I could offer his service on the whole Web on another URL and without authentication.
There is an old issue on Traefik, concerning TCP entrypoints. This one tells that TCP entrypoints accross 2 namespaces is a security issue: https://github.com/containous/traefik/issues/5748#issuecomment-547323939. I think it's exactly the same in the case I describe.
What do you think about this?
Good points. Very good reading on the kubernetes issue linked from the issue you linked above.
I can't speak for traefik team, but my guess would be that they in the end decided that this is a reasonable trade off. As the kubernetes thread points out, other products, like istio, are already doing this. There is also a trivial way to do it on vanilla Kubernetes with ExternalName (which does not work on GKE, but that's another issue).
@dduportal Any thoughts?
@dduportal any thoughts?
@zespri, @dduportal, how could we mitigate this security issue? If you don't want to take it into account just because your competitors do the same security issue or because a vanilla k8s conf allows it (well, nearly every vanilla k8s conf breaks security rules, this is off topic I think), how can I do to prevent this situation? Is there any way I could use to prevent an IngressRoute from another namespace to pick up my service?
Thx.
Thanks for the ping @zespri
@jaycci While it's possible to constrain the RBAC for Traefik to a specific namespace with regards to IngressClass, if Kubernetes lets you define a route to a service in another namespace then it would seem that's something that needs to be addressed external to Traefik (and might be solvable by a more advanced network plane that supports ACLs, but I don't know -- I'd reach out to the folks at Weave to find out). As of right now, I don't know of any Ingress controllers which handle whether services may be accessed by a particular User/Role -- that's typically relegated to a network policy provider.
I did some more digging, and apparently, this shouldn't be possible based on this thread. I'm not entirely convinced this is a security issue with Traefik though, because this behavior historically was blocked by the Kubernetes API. It's possible there was a change to the Gateway API which allows this behavior, so it may be worth raising an issue on their repository.
@notsureifkevin as far as I can see nginx ingress controller does not have this issue, it looks like it's traefik specific. The thread you linked that is linked from the traefik thread @jaycci linked discusses exactly that: that someone wants cross-namespace access but kubernetes people argue that it is not reasonable request, because it may be insecure. Let me know if you have read that differently.
On the other hand Traefik does allow that.
because this behavior historically was blocked by the Kubernetes API. It's possible there was a change to the Gateway API which allows this behavior, so it may be worth raising an issue on their repository.
Sorry, can you please expand on that? What is "Gateway API"? As far as I understand traefik is given certain RBAC permissions that allow it to access services in all configured namespaces, and thus, traefik ingress controller will happily do cross-namespace for you if configured so, there is no kubernetes machinery in-between that would prevent such access. What specifically in your view should prevent it?
that's typically relegated to a network policy provider.
This is something that occurred to me too, but I do not feel that this is necessarily the right answer. Network policy provider works at a lower level than either ingresses or services, it works at a pod level. Traefik Labs does not publish what kind of minimal network policies should be in place for your typical default-deny cluster. Setting those up from scratch is an interesting exercise. When you go through it you will realise, that since you do not know what applications and pods will be there in the future (and traefik will need access to them to do its job) you either have to create a specific allow rules for traefik for each future application, when you first deploy it or to have a blanket rule to whitelist traefik globally. In both cases the cross-namespace problem is sidestepped, that is if you give traefik enough permissions to do it's job it will also be able to do that cross-namespace.
Let me give you a more concrete example of the above.
Let's assume you have traefik ingress controller and two applications A and B. Both applications are exposed via a traefik ingress / ingress route. Both app live in their own specific namespace. In order for traefik to do it's work Network Policies has to be in place to access both A and B pods.
Now let's assume that a malicious person with access to namespace A but not B creates another ingress in namespace A that points to a service and then pod in namespace B.
Can you give an example of Network Policy/Policies that will allow traefik to continue exposing App B via its own ingress but prohibit ingress from namespace A from exposing that same App B?
For what I can see this configuration is impossible, and thus network policy provider won't help here. It only can allow or disallow access between traefik itself and the target pod, and that has to be allowed for traefik to function. This means that traefik itself has to have some security controls for preventing this scenario, if it is undesirable.
I will be glad to be wrong here, could you please let me know what you think.
Based on the most recent post in the issue in Kubernetes, it appears to now be possible in 1.18 with nginx.
With regards to NetworkPolicy, I can only speculate as to whether or not that will solve the issue as I’ve not had a use-case for that.
I’d be interested in seeing if cross-namespace service routing is possible in 1.17, and if the issue manifests itself in only the most recent releases of Traefik (2.1->2.2->2.3). I’ve not looked at the commit history for the section of the code which handles talking to the Kubernetes API, but I don’t think it’s changed in any significant way recently (but I could be wrong).
Based on the most recent post in the issue in Kubernetes, it appears to now be possible in 1.18 with nginx.
I'm not sure what's going on in this example, but it appears anecdotal at best. The author itself accepts that he does not know what he is doing. I think, that this example does not have cross-access per-se, it's just and example of URL redirection, and then it's all the same namespace. Again, if you read it differently please let me know.
With regards to
NetworkPolicy, I can only speculate as to whether or not that will solve the issue as I’ve not had a use-case for that.
I did. And based on what I've seen I do not think it does.
I’d be interested in seeing if cross-namespace service routing is possible in
1.17, and if the issue manifests itself in only the most recent releases of Traefik (2.1->2.2->2.3)
It think we discovered that it was not possible in traefik v1 with the issue linked above. This ability was added in v2.
I’ve not looked at the commit history for the section of the code which handles talking to the Kubernetes API, but I don’t think it’s changed in any significant way recently (but I could be wrong).
I did and it has. This is the commit where it became possible.
To sum up:
I can understand how community can be confused about that.
To re-iterate, I personally do not feel that this is a security issue despite the kubernetes thread reasoning because:
But it is not my place to approve or reject this, so that's why I asked for your insight @notsureifkevin. Do you think it may be worth your time discussing this with the dev team?
What’s peculiar is that the issue we’re referencing indicates that IngressRoute at one point wouldn’t allow this to happen (CRDs don’t exist in 1.x). So what changed? https://github.com/traefik/traefik/issues/5748
I will bring this up with the developers, but I’m still curious if this was a regression on our side, or the Kubernetes side.
It is a change in traefik. I think it's a good change. But the concern in OP was raised, and I thought you might want to address it. Thank you for your time 
This is the PR, as you see it's quite meaty. It was associated with an often requested feature to have the same level of control for Kubernetes CRD provider as with File Provider and Docker Provider. Before the PR you could not configure all the aspects of the Service with Kubernetes CRD.
It was part of 2.1.0 release.
I did some digging around in the PR @zespri and I believe I found something that may be useful to the OP.
Is what clued me into what might be a way to configure which namespaces the CRD provider is supposed to watch.
@jaycci by configuring this in the CRD provider, eg:
[providers.kubernetesCRD]
namespaces = ["default", "production"]
# ...
You can specify which namespaces particular instance(s) of Traefik are permitted to watch for. By setting that configuration parameter, Traefik no longer will discover/route to services in unauthorized namespaces. Does this solve your issue?
Yep, I noticed that as well. The problem with that solution is the same as I explained above with regards to the Network Policies. In many scenarios you want traefik to work with both namespaces A and B, so you have to specify them both. What you do not want to do is to be able to access service in namespace B from ingress in namespace A, and this is not something that can be configured this way.
Unless I'm missing something?
I don't think you're missing anything, but I don't see a non-trivial way to handle that use case aside from running separate ingress controllers and managing access permissions for each one separately.
Agreed. So the question is if traefik team considers this to be a security issue? Since the OP's GitHub issue was closed, I guess they do not. If not, it would be interesting to learn what made them change their mind since they did say it is a security issue in the linked GitHub traefik thread. Could be Damien was mistaken, and it is not an issue at all?
Given that I just confirmed this works (by setting namespaces=ns2) and got this in the logs:
time="2020-09-24T23:35:24Z" level=info msg="Configuration loaded from flags."
I0924 23:35:25.672891 1 request.go:621] Throttling request took 1.1977555s, request: GET:https://10.96.0.1:443/apis/traefik.containo.us/v1alpha1/namespaces/production/tlsoptions?limit=500&resourceVersion=0
time="2020-09-24T23:35:26Z" level=error msg="failed to get service ns1/whoami: namespace is not within watched namespaces" providerName=kubernetescrd namespace=ns2 ingress=whoami
time="2020-09-24T23:35:36Z" level=error msg="failed to get service ns1/whoami: namespace is not within watched namespaces" providerName=kubernetescrd ingress=whoami namespace=ns2
I suspect that the security issue is rendered nil since the previous behavior can be maintained via configuration. I'll raise this thread with them though and see if they want to provide any additional context/details around the decision.
I suspect that the security issue is rendered nil since the previous behavior can be maintained via configuration.
You mean by having two traefik instances each for its own set of namespaces?