Unable to migrate to traefik v2 from v1: 404 not found error

Hey we are trying to migarte to traefik v2 from v1.
But for any set of configuration we have tried we are getting 404 errrors.
Apparently we are also not getting any useful logs, when tried to debug.
I have attached the yaml files for both ingress-controller deployment and ingress yaml.
Can anyone let me know, what i might have missed?

This is the only log message that we are gettign from ingress controller

sbhardwaj@MacBook-Pro-4 liftie % kubectl logs ingress-controller-64458b8fd8-gj964 -n mlx
time="2022-05-17T09:15:46Z" level=info msg="Configuration loaded from file: /config/traefik.toml"

ingress-controller.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "12"
    meta.helm.sh/release-name: mlx-mlx
    meta.helm.sh/release-namespace: mlx
  creationTimestamp: "2022-05-16T10:19:52Z"
  generation: 12
  labels:
    app.kubernetes.io/managed-by: Helm
    role: ingress-controller
    version: v0.8.3
  name: ingress-controller
  namespace: mlx
  resourceVersion: "339489"
  uid: d1d88cf3-782e-479d-9912-62e1f30a55e2
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      role: ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2022-05-17T12:19:55+05:30"
      creationTimestamp: null
      labels:
        app: ingress-controller
        role: ingress-controller
        version: v0.8.3
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: role
                operator: In
                values:
                - ingress-controller
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - --log.level=DEBUG
        - --providers.kubernetesingress=true
        - --configfile=/config/traefik.toml
        image: traefik:v2.6
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 8881
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: traefik-ingress-lb
        ports:
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 8881
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /ssl
          name: ssl
        - mountPath: /config
          name: config
      dnsConfig:
        options:
        - name: ndots
          value: "1"
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: jfrog-dev
      nodeSelector:
        role.node.kubernetes.io/infra: "true"
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      serviceAccount: sa-ingress-controller
      serviceAccountName: sa-ingress-controller
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: role.node.kubernetes.io/infra
        operator: Equal
        value: "true"
      volumes:
      - name: ssl
        secret:
          defaultMode: 420
          secretName: traefik-cert
      - configMap:
          defaultMode: 420
          name: traefik-config
        name: config
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2022-05-17T06:16:48Z"
    lastUpdateTime: "2022-05-17T06:16:48Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-05-16T10:19:52Z"
    lastUpdateTime: "2022-05-17T09:15:55Z"
    message: ReplicaSet "ingress-controller-64458b8fd8" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 12
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

And here is the yaml file for ingress:

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/protocol: http
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"ingress.kubernetes.io/protocol":"http","kubernetes.io/ingress.class":"traefik","meta.helm.sh/release-name":"mlx-mlx","meta.helm.sh/release-namespace":"mlx","traefik.ingress.kubernetes.io/router.entrypoints":"web, websecure","traefik.ingress.kubernetes.io/router.tls":"true","traefik.ingress.kubernetes.io/rule-type":"PathPrefixStrip"},"creationTimestamp":"2022-05-16T10:19:52Z","generation":2,"labels":{"app.kubernetes.io/managed-by":"Helm"},"name":"client","namespace":"mlx","resourceVersion":"302275","uid":"6dbd717f-3fa5-44f1-b2e9-896ad69e31b4"},"spec":{"rules":[{"host":"livelog.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"livelog","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"api.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"rest.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"api","port":{"number":8889}}},"path":"/api/v2","pathType":"ImplementationSpecific"}]}},{"host":"api.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"api","port":{"number":8889}}},"path":"/api/v2","pathType":"ImplementationSpecific"}]}},{"host":"ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"api","port":{"number":8889}}},"path":"/api/v2","pathType":"ImplementationSpecific"}]}},{"host":"models.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"consoles.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"www.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"Exact"}]}},{"host":"assets.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"livereload.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":35729}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"grpc.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":20050}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"tcp-ingress.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"tcp-ingress-controller","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"}]}},{"host":"model-metrics.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"model-metrics","port":{"number":80}}},"pathType":"ImplementationSpecific"}]}},{"http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":80}}},"path":"/internal/load-balancer/","pathType":"ImplementationSpecific"}]}}]},"status":{"loadBalancer":{}}}
      kubernetes.io/ingress.class: traefik
      meta.helm.sh/release-name: mlx-mlx
      meta.helm.sh/release-namespace: mlx
      traefik.ingress.kubernetes.io/router.entrypoints: web, websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
      traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
    creationTimestamp: "2022-05-17T08:19:50Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: client
    namespace: mlx
    resourceVersion: "325044"
    uid: 6d4f7024-662d-4662-b873-d3d0b3806a84
  spec:
    rules:
    - host: livelog.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: livelog
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: api.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: rest.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 8889
          path: /api/v2
          pathType: ImplementationSpecific
    - host: api.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 8889
          path: /api/v2
          pathType: ImplementationSpecific
    - host: ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 8889
          path: /api/v2
          pathType: ImplementationSpecific
    - host: models.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: consoles.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: www.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: Exact
    - host: assets.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: livereload.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 35729
          path: /
          pathType: ImplementationSpecific
    - host: grpc.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 20050
          path: /
          pathType: ImplementationSpecific
    - host: tcp-ingress.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: tcp-ingress-controller
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    - host: model-metrics.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: model-metrics
              port:
                number: 80
          pathType: ImplementationSpecific
    - http:
        paths:
        - backend:
            service:
              name: web
              port:
                number: 80
          path: /internal/load-balancer/
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/protocol: h2c
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"ingress.kubernetes.io/protocol":"h2c","meta.helm.sh/release-name":"mlx-mlx","meta.helm.sh/release-namespace":"mlx"},"creationTimestamp":"2022-05-16T10:19:52Z","generation":1,"labels":{"app.kubernetes.io/managed-by":"Helm"},"name":"grpc","namespace":"mlx","resourceVersion":"4236","uid":"98ca5dba-4913-48b4-b543-41a612973a0f"},"spec":{"rules":[{"host":"grpcapi.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"api","port":{"number":8888}}},"path":"/","pathType":"ImplementationSpecific"}]}}]},"status":{"loadBalancer":{}}}
      meta.helm.sh/release-name: mlx-mlx
      meta.helm.sh/release-namespace: mlx
    creationTimestamp: "2022-05-17T08:19:52Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: grpc
    namespace: mlx
    resourceVersion: "325046"
    uid: 9db5a45e-2443-4be9-ae1b-ddc6aaee7612
  spec:
    rules:
    - host: grpcapi.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: api
              port:
                number: 8888
          path: /
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"meta.helm.sh/release-name":"mlx-mlx","meta.helm.sh/release-namespace":"mlx"},"creationTimestamp":"2022-05-16T10:19:52Z","generation":1,"labels":{"app.kubernetes.io/managed-by":"Helm"},"name":"modelproxy-ingress","namespace":"mlx","resourceVersion":"4234","uid":"8a7b946f-c2fe-4b08-9753-ee2e4fc8b2b9"},"spec":{"rules":[{"host":"modelservice.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work","http":{"paths":[{"backend":{"service":{"name":"model-proxy","port":{"number":80}}},"path":"/model","pathType":"ImplementationSpecific"}]}}]},"status":{"loadBalancer":{}}}
      meta.helm.sh/release-name: mlx-mlx
      meta.helm.sh/release-namespace: mlx
    creationTimestamp: "2022-05-17T08:19:53Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: modelproxy-ingress
    namespace: mlx
    resourceVersion: "325049"
    uid: 44734b34-413e-4144-9471-8a79470dfdcd
  spec:
    rules:
    - host: modelservice.ml-160c4e93-d9e.eng-ml-d.xcu2-8y8x.dev.cldr.work
      http:
        paths:
        - backend:
            service:
              name: model-proxy
              port:
                number: 80
          path: /model
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Also the config-map for tarefik ingress controller can be seen here:

apiVersion: v1
data:
  dynamic.toml: |
    # dynamic.toml
    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile = "/ssl/server.pem"
          keyFile  = "/ssl/server-key.pem"

    [tls.options]
      [tls.options.default]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
        ]
  traefik.toml: |
    # traefik.toml
    [global]
      checkNewVersion = false
      sendAnonymousUsage = false

    [entryPoints.web]
      address = ":80"
      [entryPoints.web.http.redirections.entryPoint]
        to = "websecure"
        scheme = "https"

    [entryPoints.websecure]
      address = ":443"

    [entryPoints.ping]
        address = ":8881"

    [ping]
      entryPoint = "ping"

    [providers]
      [providers.file]
        filename = "/config/dynamic.toml"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"dynamic.toml":"# dynamic.toml\n[tls.stores]\n  [tls.stores.default]\n    [tls.stores.default.defaultCertificate]\n      certFile = \"/ssl/server.pem\"\n      keyFile  = \"/ssl/server-key.pem\"\n\n[tls.options]\n  [tls.options.default]\n    minVersion = \"VersionTLS12\"\n    cipherSuites = [\n      \"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\",\n      \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\",\n      \"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305\",\n      \"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305\",\n      \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\",\n      \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\",\n      \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\",\n      \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\",\n    ]\n","traefik.toml":"# traefik.toml\n[global]\n  checkNewVersion = false\n  sendAnonymousUsage = false\n\n[entryPoints.web]\n  address = \":80\"\n  [entryPoints.web.http.redirections.entryPoint]\n    to = \"websecure\"\n    scheme = \"https\"\n\n[entryPoints.websecure]\n  address = \":443\"\n\n[entryPoints.ping]\n    address = \":8881\"\n\n[ping]\n  entryPoint = \"ping\"\n\n[providers]\n  [providers.file]\n    filename = \"/config/dynamic.toml\"\n"},"kind":"ConfigMap","metadata":{"annotations":{"meta.helm.sh/release-name":"mlx-mlx","meta.helm.sh/release-namespace":"mlx"},"creationTimestamp":"2022-05-16T04:44:44Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"name":"traefik-config","namespace":"mlx","resourceVersion":"3710","uid":"0974f6f8-294f-4b3b-b8bb-65f0e6eaa552"}}
    meta.helm.sh/release-name: mlx-mlx
    meta.helm.sh/release-namespace: mlx
  creationTimestamp: "2022-05-17T04:11:34Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: traefik-config
  namespace: mlx
  resourceVersion: "262148"
  uid: b03ec2ea-21d5-442f-8c53-399d27aa102d

Can you please try to remove that annotation? This is only available on v1.7 and no longer exists on v2.