Can I control TLS options for traefic client - ldaps

I am trying to proxy ldaps connections to kanidm.

I have kanidm hosted with a self signed cert. For HTTP I use a serverTransport configured to skip TLS verification. It works fine.

http:
  serversTransports:
    skip-tls-verify:
      insecureSkipVerify: true

For LDAPS a similar setup does not work.
I have a LDAPS entrypoint and I use a TCP router.

entryPoints:
  # REDACTED
  ldaps:
    address: ":3636"

# docker swarm labels config

        traefik.tcp.routers.kanidm-prod-ldaps.entrypoints: ldaps
        traefik.tcp.routers.kanidm-prod-ldaps.rule: HostSNI(`*`)
        traefik.tcp.routers.kanidm-prod-ldaps.tls: "true"
        # traefik.tcp.routers.kanidm-prod-ldaps.tls.passthrough: "true"
        traefik.tcp.routers.kanidm-prod-ldaps.service: kanidm-prod-ldaps@swarm
        traefik.tcp.services.kanidm-prod-ldaps.loadbalancer.server.port: 3636
        traefik.tcp.services.kanidm-prod-ldaps.loadbalancer.serversTransport: ldapsTransport@file

Using TLS passthrough and ignoring certificate on ldap side works, but I can't make it to work if I let Traefik terminate TLS with LetsEncrypt certificate and use a TLS connection to kanidm with the self signed cert.

# this works with TLS passthrough but not without
LDAPTLS_REQCERT=ALLOW ldapsearch -H ldaps://idm.example.com:3636 -b 'dc=idm,dc=example,dc=com' -x '(name=admin)' 

This is what I get on kanidm side:

idm-prod_kanidm.1.5h4xsb8y283k@obi01    | 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: LDAP TLS accept error, continuing -> Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772427, library: "SSL routines", function: "ssl3_get_record", reason: "wrong version number", file: "ssl/record/ssl3_record.c", line: 354 }]))) }
idm-prod_kanidm.1.5h4xsb8y283k@obi01    | 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: LDAP TLS accept error, continuing -> Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772427, library: "SSL routines", function: "ssl3_get_record", reason: "wrong version number", file: "ssl/record/ssl3_record.c", line: 354 }]))) }