we are trying to use Delta Lake primitive saveAsTable from a Jupyter Notebook with pyspark kernel. When we call the method, we receive the following exception:
An error occurred while calling o223.saveAsTable.
: java.io.IOException: The error typically occurs when the default LogStore implementation, that
is, HDFSLogStore, is used to write into a Delta table on a non-HDFS storage system.
In order to get the transactional ACID guarantees on table updates, you have to use the
correct implementation of LogStore that is appropriate for your storage system.
Is there any option to set or is it an incompatibility with hopsfs / hive?
We noticed also this limitation trying to use Hive ACID table:
metastore.RetryingHMSHandler: MetaException(message:Unable to update transaction database java.sql.SQLSyntaxErrorException: The storage engine for the table doesn’t support SAVEPOINT
We are using Hopsworks 2.3 on prem