I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.

My setup:

• ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: my-etcd-cluster.my-etcd-namespace.svc.cluster.local)
• I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
• I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD

Here's my affected Go code:

cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
    DialTimeout: 5 * time.Second,
    Username:    username,
    Password:    password,
})

if err != nil {
    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}

And here's an error I'm getting when clientv3.New(...) gets executed:

{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227

The same error happens when I'm passing some dummy, invalid credentials.

However, when I tried to access the database in a HTTP API manner:

postBody, _ := json.Marshal(map[string]string{
    "name":     username,
    "password": password,
})
responseBody := bytes.NewBuffer(postBody)

resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()

...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.

When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.

EDIT: I've also added a simple pod to try accessing my etcd db via etcdctl:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: my-controller-namespace
spec:
  containers:
  - name: etcdctl
    image: bitnami/etcd
    command:
    - sleep
    - infinity

When logged into the container via kubectl exec, I was able to access my db:

$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK

I guess the problem is somewhere in the SDK?

Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0. As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):