Error "org.apache.hadoop.hive.ql.exec.vector.TimestampColumnVector.changeCalendar(ZZ)V" while using Timestamp filed in SQL query

Hopsworks
#1

Hi Team,
We are using Hopsworks 2.4 community edition and trying to execute a sql query having timestamp attribute to create a feature and it gives below error.

An error was encountered:
An error occurred while calling o180.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 765) (h02dyn.maersk.com executor 1): java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.vector.TimestampColumnVector.changeCalendar(ZZ)V

SQL
df=spark.sql(""“select order_id,
order_year,
order_week,
Source_txn_commit,
source_txn_csn,
source_txn_rsn,
order_amount,
row_number()over(partition by order_id,order_year,order_week,order_amount order by Source_txn_commit desc,source_txn_csn desc,source_txn_rsn desc) as Rid
from order
where order_year >= ‘2018’
and COALESCE(Comments,‘Empty’) NOT LIKE (’%sample request%’)
group by 1,2,3,4,5,6,7
order by Source_txn_commit, source_txn_csn, source_txn_rsn desc”"")

#2

HI @Rajendra_Tamboli,

In which format is the data stored and could you send the schema of the table you are trying to query?

–
Fabio

#3

Hi Fabio,

The data is stored in orc in adls and fetched via storage connector. Please find below schema of the order table used in SQL -

root
|-- order_id: integer (nullable = true)
|-- order_year: integer (nullable = true)
|-- order_week: integer (nullable = true)
|-- order_amount: decimal(18,4) (nullable = true)
|-- last_updated: timestamp (nullable = true)
|-- updated_by: string (nullable = true)
|-- status: string (nullable = true)
|-- data_source: string (nullable = true)
|-- comments: string (nullable = true)
|-- source_txn_commit: timestamp (nullable = true)
|-- source_txn_type: string (nullable = true)
|-- source_file_name: string (nullable = true)
|-- source_txn_csn: string (nullable = true)
|-- source_txn_rsn: long (nullable = true)
|-- load_txn_tm: string (nullable = true)
|-- dl_load_flag: string (nullable = true)
|-- etl_load_date: timestamp (nullable = true)
|-- end_date: timestamp (nullable = true)
|-- batchid: long (nullable = true)
|-- errormapcol: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- source_file_name_dl: string (nullable = true)
|-- date_part: date (nullable = true)