I’m trying to follow the tutorial to run the fraud credit card transactions. As a part of the exercise I’m trying to run some commands in jupyter notebook but failing to start the pyspark application. I have tried it multiple times by creating a different jupyter notebook but its failing every time.
Code:
import json
from pyspark.sql.types import StructField, StructType, StringType, DoubleType, TimestampType, LongType, IntegerType
Error:
2021-10-26 19:39:37,133 INFO YarnClientImpl: Application submission is not finished, submitted application application_1634125593199_0034 is still in NEW_SAVING
Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
c) Restart the kernel.
Hello,
we have the same issue however the error is different here is attached:
from hops import featurestore
The code failed because of a fatal error:
Session 6 unexpectedly reached final status ‘killed’. See logs:
stdout:
2024-04-29 16:45:53,832 INFO EsServiceCredentialProvider: Loaded EsServiceCredentialProvider
stderr:
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/srv/hops/hadoop-3.2.0.12-EE-RC0/share/hadoop/common/lib/log4j-slf4j-impl-2.19.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/srv/hops/hadoop-3.2.0.12-EE-RC0/share/hadoop/hdfs/lib/log4j-slf4j-impl-2.19.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See SLF4J Error Codes for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2024-04-29 16:45:49,630 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
2024-04-29 16:45:50,168 INFO RMProxy: Connecting to ResourceManager at /10.0.0.5:8032
2024-04-29 16:45:50,977 INFO Client: Requesting a new application from cluster with 0 NodeManagers
2024-04-29 16:45:51,072 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (64000 MB per container)
2024-04-29 16:45:51,073 INFO Client: Will allocate AM container, with 2432 MB memory including 384 MB overhead
2024-04-29 16:45:51,074 INFO Client: Setting up container launch context for our AM
2024-04-29 16:45:51,076 INFO Client: Setting up the launch environment for our AM container
2024-04-29 16:45:51,093 INFO Client: Preparing resources for our AM container
2024-04-29 16:45:52,782 INFO Client: Source and destination file systems are the same. Not copying hdfs:/user/spark/log4j2.properties
2024-04-29 16:45:52,907 INFO Client: Source and destination file systems are the same. Not copying hdfs:/Projects/project_genericScore/Resources/mssql-jdbc-12.6.1.jre11.jar
2024-04-29 16:45:53,171 INFO Client: Uploading resource file:/tmp/spark-54bac7d3-29d2-434b-ae87-88ae71762090/__spark_conf__8886233589824668741.zip → hdfs:/Projects/project_genericScore/Resources/.sparkStaging/project_genericScore__fernando/.sparkStaging/application_1714396161248_0007/spark_conf.zip
2024-04-29 16:45:53,757 INFO SecurityManager: Changing view acls to: livy,project_genericScore__fernando
2024-04-29 16:45:53,758 INFO SecurityManager: Changing modify acls to: livy,project_genericScore__fernando
2024-04-29 16:45:53,759 INFO SecurityManager: Changing view acls groups to:
2024-04-29 16:45:53,760 INFO SecurityManager: Changing modify acls groups to:
2024-04-29 16:45:53,760 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy, project_genericScore__fernando); groups with view permissions: Set(); users with modify permissions: Set(livy, project_genericScore__fernando); groups with modify permissions: Set()
2024-04-29 16:45:53,855 INFO Client: Submitting application application_1714396161248_0007 to ResourceManager
2024-04-29 16:45:53,936 INFO YarnClientImpl: Submitted application application_1714396161248_0007
2024-04-29 16:45:53,941 INFO Client: Application report for application_1714396161248_0007 (state: GENERATING_SECURITY_MATERIAL)
2024-04-29 16:45:53,947 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1714409153881
final status: UNDEFINED
tracking URL: https://hopsworks-master.internal.cloudapp.net:8089/proxy/application_1714396161248_0007/
user: project_genericScore__fernando
2024-04-29 16:45:53,952 INFO ShutdownHookManager: Shutdown hook called
2024-04-29 16:45:53,955 INFO ShutdownHookManager: Deleting directory /tmp/spark-f64a37d6-3ff4-4667-bf19-27029ab1f746
2024-04-29 16:45:53,962 INFO ShutdownHookManager: Deleting directory /tmp/spark-54bac7d3-29d2-434b-ae87-88ae71762090
YARN Diagnostics:
Application application_1714396161248_0007 was killed by user livy at 10.0.0.5.
Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
c) Restart the kernel.