I have installed hadoop jar binary, hsfs[hive] and pydoop python lib successfully in an external python environment, while using the TrainginDataset.read() I got an error of permission dened. Can anyone help?
Feature Store: 2.4.0
My ApiKey has the Scope:
["DATASET_CREATE","DATASET_DELETE","FEATURESTORE","PROJECT","DATASET_VIEW","JOB"]
My Error output:
File "/app/app.py", line 80, in fetch_dataset_csv
df: DataFrame = td.read(split=config.get("dataset_split", "train"))
File "/usr/local/lib/python3.8/site-packages/hsfs/training_dataset.py", line 257, in read
return self._training_dataset_engine.read(self, split, read_options)
File "/usr/local/lib/python3.8/site-packages/hsfs/core/training_dataset_engine.py", line 107, in read
return training_dataset.storage_connector.read(
File "/usr/local/lib/python3.8/site-packages/hsfs/storage_connector.py", line 106, in read
return engine.get_instance().read(self, data_format, options, path)
File "/usr/local/lib/python3.8/site-packages/hsfs/engine/hive.py", line 73, in read
df_list = self._read_hopsfs(location, data_format)
File "/usr/local/lib/python3.8/site-packages/hsfs/engine/hive.py", line 108, in _read_hopsfs
path_list = hdfs.ls(location, recursive=True)
File "/usr/local/lib/python3.8/site-packages/pydoop/hdfs/__init__.py", line 307, in ls
dir_list = lsl(hdfs_path, user, recursive)
File "/usr/local/lib/python3.8/site-packages/pydoop/hdfs/__init__.py", line 291, in lsl
top = next(treewalk)
File "/usr/local/lib/python3.8/site-packages/pydoop/hdfs/fs.py", line 631, in walk
top = self.get_path_info(top)
File "/usr/local/lib/python3.8/site-packages/pydoop/hdfs/fs.py", line 406, in get_path_info
return self.fs.get_path_info(path)
PermissionError: [Errno 13] Permission denied