Hi,
On a newly deployed single-node Hopswork (Hopswork 1.3.0-SNAPSHOT), I created a new project with all the functionalities (Jupyter, Kafka, Feature Store etc.). However, when I get into Jupyter tab and tried to create a new notebook (either JupyterLab or Classic Jupyter), there is a quick circle bar says: “Connecting to Jupyter …”
. A moment later, a red pop-up appears that says: A generic error occurred, then nothing happens.
Do I have to do any further set up for Jupyter service to work?
Please help me to resolve this.
Thank you!
Hi,
Can you look into application server’s log if there is any error? The log file is /srv/hops/domains/domain1/logs/server.log Try to tail the file while opening JupyterLab.
Generally snapshot versions are under development so there might be a change the nightly build you installed was broken.
Kind regards
Thank you for your promt reply. Here is the log when I clicked the JupyterLab button:
[#|2020-05-07T17:59:50.372+0000|SEVERE|Payara 4.1|io.hops.hopsworks.common.dao.jupyter.config .JupyterConfigFilesGenerator|_ThreadID=301;ThreadName=http-thread-pool::http-listener-2(6); TimeMillis=1588874390372;_LevelValue=1000;|Error in initializing JupyterConfig for project: f lax. com.logicalclocks.servicediscoverclient.exceptions.ServiceNotFoundException: Error: host not found Could not find service ServiceQuery(name=rpc.namenode.service.consul, tags=[])|#]
[#|2020-05-07T17:59:50.373+0000|SEVERE|Payara 4.1|io.hops.hopsworks.api.exception.mapper.REST ApiThrowableMapper|_ThreadID=301;_ThreadName=http-thread-pool::http-listener-2(6);_TimeMillis =1588874390373;LevelValue=1000;|errorCode=120000
io.hops.hopsworks.exceptions.GenericException: java.lang.NullPointerException
at io.hops.hopsworks.api.exception.mapper.RESTApiThrowableMapper.handleUnknownExcepti on(RESTApiThrowableMapper.java:108)
at io.hops.hopsworks.restutils.ThrowableMapper.toResponse(ThrowableMapper.java:94)
at org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.jav a:601)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:476 )
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:31 7)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154 )
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1692)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilter Chain.java:339)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.ja va:209)
at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:30 5)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilter Chain.java:251)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.ja va:209)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256 )
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160 )
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:368)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(Conta inerMapper.java:483)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:1 80)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.jav a:235)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java :119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterCh ain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilte rChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.ja va:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.ja va:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.ja va:539)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy .java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrateg y.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOS trategy.java:56)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(W orkerThreadIOStrategy.java:137)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPo ol.java:593)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool. java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at io.hops.hopsworks.api.jupyter.JupyterService.startNotebookServer(JupyterService.ja va:343)
at io.hops.hopsworks.api.jupyter.JupyterService$Proxy$$$_WeldClientProxy.startNotebo okServer(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1357.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$ 1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1. run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.in voke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$Re sponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.di spatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvok er.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoke r.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoke r.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
… 48 more
|#]
[#|2020-05-07T18:00:00.000+0000|INFO|Payara 4.1|io.hops.hopsworks.common.experiments.tensorbo ard.TensorBoardKillTimer|_ThreadID=77;_ThreadName=__ejb-thread-pool8;_TimeMillis=158887440000 0;_LevelValue=800;|Running TensorBoardKillTimer.|#]
2020-05-07T18:00:00.008+0000|INFO|Payara 4.1|io.hops.hopsworks.common.kafka.ZookeeperTopic CleanerTimer|_ThreadID=55;_ThreadName=__ejb-thread-pool1;_TimeMillis=1588874400008;_LevelValu e=800;|Running ZookeeperTopicCleanerTimer.|#]
It looks like the NameNode is not running ServiceNotFoundException: Error: host not found Could not find service ServiceQuery(name=rpc.namenode.service.consul, tags=[])
I suggest before restarting the service to check the logs and see why it died. NameNode logs are in /srv/hops/hadoop/logs/hadoop-hdfs-namenode-*.log.
You can restart it with sudo systemctl restart namenode It might take up to 20 seconds until the service is discovered. You can try ping namenode.service.consul to see if it’s available. Then you can try again starting Jupyter.
Make sure that all services are running by clicking on the dropdown menu from the top right corner, Admin and Services.
1 Like
I’ve done the following:
- Copy the log folder before restarting namenode service as you guided full namenode log
- I also checked the Services section in Admin before restarting the namenode, and all the services are green and status is Good (including hadoop datanode and namenode)
Then I restart the namenode with sudo systemctl restart namenode and tried Jupyter again but still the same error popped up.
Here is the new error log in server.log when I click the JupyterLab button:
[#|2020-05-07T19:34:12.936+0000|SEVERE|Payara 4.1|io.hops.hopsworks.common.dao.jupyter.config.JupyterConfigFilesGenerator|_ThreadID=29;_ThreadName=http-thread-pool::http-listener-2(3);_TimeMillis=1588880052936;_LevelValue=1000;|Error in initializing JupyterConfig for project: flax. com.logicalclocks.servicediscoverclient.exceptions.ServiceNotFoundException: Error: host not found Could not find service ServiceQuery(name=rpc.namenode.service.consul, tags=[])|#]
[#|2020-05-07T19:34:12.936+0000|SEVERE|Payara 4.1|io.hops.hopsworks.api.exception.mapper.RESTApiThrowableMapper|_ThreadID=29;_ThreadName=http-thread-pool::http-listener-2(3);_TimeMillis=1588880052936;LevelValue=1000;|errorCode=120000
io.hops.hopsworks.exceptions.GenericException: java.lang.NullPointerException
at io.hops.hopsworks.api.exception.mapper.RESTApiThrowableMapper.handleUnknownException(RESTApiThrowableMapper.java:108)
at io.hops.hopsworks.restutils.ThrowableMapper.toResponse(ThrowableMapper.java:94)
at org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.java:601)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:476)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1692)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:339)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:305)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:251)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:368)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:483)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:180)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at io.hops.hopsworks.api.jupyter.JupyterService.startNotebookServer(JupyterService.java:343)
at io.hops.hopsworks.api.jupyter.JupyterService$Proxy$$$_WeldClientProxy.startNotebookServer(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1357.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
… 48 more
|#]
And can you give a bit more details about how can check namenode is normal.
Depending on when you made the installation you might have hit a temporary bug in the recipes where NameNode wasn’t registering with our service discovery system.
Do you have the file /srv/hops/consul/consul.d/nn-consul.hcl ?
I just check, I don’t have that file. Here is the content of consul.d folder:
total 36
-r-x------ 1 consul consul 499 May 7 09:21 airflow-consul.hcl
-rwxr-x— 1 consul consul 515 May 7 09:31 consul.hcl
-r-x------ 1 consul consul 310 May 7 09:26 glassfish-consul.hcl
-r-x------ 1 consul consul 752 May 7 09:36 hiveserver2-consul.hcl
-r-x------ 1 consul consul 316 May 7 09:50 livy-consul.hcl
-r-x------ 1 consul consul 2603 May 7 09:33 logstash-consul.hcl
-r-x------ 1 consul consul 426 May 7 09:35 metastore-consul.hcl
-rwxr-x— 1 consul consul 203 May 7 09:31 systemd_env_vars
-r-x------ 1 consul consul 450 May 7 09:30 zk-consul.hcl
Yeah you hit the bug I mentioned. It was fixed in almost 30 minutes so you were very unlucky 
At this point it is easier if you start the installation from scratch.
You should:
- Stop all the services with
sudo /srv/hops/kagent/kagent/bin/shutdown-all-local-services.sh
- Delete installation directory
sudo rm -rf /srv/hops
- Delete Karamel cache
sudo rm -rf /home/INSTALLATION_USER/.karamel
- Delete Chef cache
sudo rm -rf /tmp/chef-solo
Then you can start the installation process again.
1 Like
I will give it a try. Thank you @antonios for your quick response.
I installed again and everything works fine now. Thank you for your help!
1 Like