Databricks connector for SB2Bi

Today, Databricks is a major player in the data industry. The strategic directions of the Group have led to the construction of an Azure Data Lake, and the data management solution implemented to ingest, process, and expose data in this DL is Databricks.

About a year ago, we had extensive discussions with IBM support to interconnect SB2Bi with the Databricks layer (parquet files), but despite our numerous attempts, we were unable to establish a JDBC connection and thus exploit the data layer from SB2Bi.

At that time, we identified and implemented a workaround solution consisting of exposing data in serverless_db views via Synapse and connecting SB2Bi via JDBC SQL Server. This approach has been working more or less satisfactorily until now but presents serious drawbacks:

• Performance is average or even poor.
• It is necessary to manually configure this serverless layer systematically and solely for SB2Bi whenever we wish to process data with SB2Bi.
• Synapse is a deprecated solution whose development has been abandoned by Microsoft (replaced by Fabric). Consequently, some recent features of Databricks (deletion vectors) are not compatible with Synapse. Therefore, we have to "limit" the use of Databricks (disable this feature) to ensure compatibility with our workaround solution, leading us into a cycle of obsolescence for solutions that is not acceptable in the long term.
• Furthermore, we are gradually replacing Synapse with a full Databricks solution. Eventually, we will no longer be able to expose serverless views for SB2Bi.

We would like IBM to implement a native Databricks connector (service) and enable the exploitation of parquet files.

In your knowledge base, there are currently 43 ideas related to the implementation of a Databricks connector, but none for SB2Bi...