Python lib conflict between petastorm and pyarrow

jie8357IOII · May 13, 2020, 12:00pm

Hi, hopsworks team.

I am using Hopswork 1.3.0-SNAPSHOT
When I tried to run PetastormHelloWorld.ipynb, I found python lib conflict issue cause the materialize_dataset won’t run successfully.
Error message below.

An error was encountered:
get_metadata() takes 1 positional argument but 2 were given
Traceback (most recent call last):
File “/srv/hops/anaconda/anaconda/envs/python36/lib/python3.6/contextlib.py”, line 88, in exit
next(self.gen)
File “/srv/hops/anaconda/anaconda/envs/python36/lib/python3.6/site-packages/petastorm/etl/dataset_metadata.py”, line 111, in materialize_dataset
_generate_unischema_metadata(dataset, schema)
File “/srv/hops/anaconda/anaconda/envs/python36/lib/python3.6/site-packages/petastorm/etl/dataset_metadata.py”, line 191, in _generate_unischema_metadata
utils.add_to_dataset_metadata(dataset, UNISCHEMA_KEY, serialized_schema)
File “/srv/hops/anaconda/anaconda/envs/python36/lib/python3.6/site-packages/petastorm/utils.py”, line 98, in add_to_dataset_metadata
arrow_metadata = dataset.pieces[0].get_metadata(dataset.fs.open)
TypeError: get_metadata() takes 1 positional argument but 2 were given

And I found the reason is python lib conflict between pyarrow 0.15.0 and hops-petastorm 0.7.6

github.com

logicalclocks/petastorm/blob/master/petastorm/etl/dataset_metadata.py#L325


"""Split the row groups by reading the footers of the parquet pieces"""

logger.info('Recovering rowgroup information for the entire dataset. This can take a long time for datasets with '
            'large number of files. If this dataset was generated by Petastorm '
            '(i.e. by using "with materialize_dataset(...)") and you still see this message, '
            'this indicates that the materialization did not finish successfully.')

thread_pool = futures.ThreadPoolExecutor()

def split_piece(piece):
    metadata = piece.get_metadata(dataset.fs.open)
    return [pq.ParquetDatasetPiece(piece.path,
                                   row_group=row_group,
                                   partition_keys=piece.partition_keys)
            for row_group in range(metadata.num_row_groups)]

futures_list = [thread_pool.submit(split_piece, piece) for piece in dataset.pieces]
result = [item for f in futures_list for item in f.result()]
thread_pool.shutdown()
return result

I think hops-petastorm 0.7.6 suit for pyarrow 0.13.0.

So would you fix the pyarrow version to 0.13.0 in package? or upgrade hops-petastorm to fit a higher pyarrow version?

Thank you.