SSL handshake failed on MacOS (Hopsworks Serverless)

Hopsworks
#1

Hi Hopsworks community,

I’m using a local python environment to access Hopsworks Serverless from MacOS 12.5. Either hopsworks API and FeatureStore API allowed me to create a Feature Group but failed when i tried to insert a DataFrame as following.

ssl://c.app.hopsworks.ai:9092/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 124ms in state SSL_HANDSHAKE)

I have librdkafka packaged installed, OpenSSL 1.1.1, and the C_INCLUDE_PATH and LIBRARY_PATH environment variables set. I also checked that Hopsworks PEM files (downloaded by the API) were in the default folder (/tmp).

Why does the inserting of a DataFrame to a Feature Group is not working in my local Python environment?

It works on Google Colab btw.

Kind regards
Didiosky

#2

Hi @didiosky-zigatta,

I’ve seen a similar issue when a user tries to use the Hopsworks client behind a VPN/Corporate Network and connecting to https://app.hopsworks.ai . The network might be configured to terminate the TLS connection and establish a new one. If this is the case, the certificate the client receives is not the same as the Kafka producer certificate. Resulting in the issue above.

You can run this command to inspect the certificate chain returned when the client tries to establish a connection to the Kafka broker:

openssl s_client -showcerts -connect c.app.hopsworks.ai:9092

You should see something like this:

 0 s:C = SE, ST = Sweden, L = kafka, OU = 1, CN = ip-172-16-4-138.us-east-2.compute.internal
   i:C = SE, ST = Sweden, O = SICS, CN = HopsIntermediateCA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 16 13:48:29 2022 GMT; NotAfter: Jun 13 13:53:29 2032 GMT

If you see your company name in the certificate chain, then this is most likely the issue.

#3

Hi fabio,

Thank for your feedback. I’m testing the Feature store from my Local Service Provider (not from my corporate/network). Actually SSL handshake failed if you don’t specified openssl CAfile, key and cert.

image

However with the right path to the Hopsworks CAfile, key and certificate the verification is OK.

Checking the kafka_api.get_default_config() from my default project i could see that all hopsworks certifications paths are correct, but for some reason the Producer failed using that configuration.

import hopsworks
project = hopsworks.login()
kafka_api = project.get_kafka_api()
producer_config = kafka_api.get_default_config()

from confluent_kafka import Producer
producer = Producer(producer_config)

A temporal solution would be to disable the SSL verification, but i would like to address the underlying issue causing the SSL verification failure in my local python environment (macos).

Any other idea?

#4

Hi @didiosky-zigatta,

We spent some time looking into it. The error is caused by a compatibility issue with the version of librdkafka that gets installed using brew.

The Hopsworks serverless platform runs a slightly older version of the brokers which are not compatible with librdkafka version 2.x.

For the Hopsworks client to work you need to install an older version, for example version 1.9.2. You can run the following commands:

curl -O https://raw.githubusercontent.com/Homebrew/homebrew-core/f7d0f40bbc4075177ecf16812fd95951a723a996/Formula/librdkafka.rb
brew install --build-from-source librdkafka.rb

Let me know if that works for you

–
Fabio

#5

Hi Fabio,

It works for me. After installing librdkafka 1.9.2 it works.

Thank you for your support Fabio.
Regards, Didiosky