Unable to create a new project

Hopsworks
#1

I have installed Hopsworks enterprise on a single-node AWS VM, using a community image of 2.0 snapshot.

Started all the services. Opened the GUI by connecting to the public it over https.
Trying to create a new Project, with the “Serving” and “Feature Store” services enabled, and getting the following error

An argument was not provided or it was malformed.
java.net.UnknownHostException: rpc.namenode.service.consul

#2

Hi,

Did you follow the instructions from https://hopsworks.readthedocs.io/en/stable/getting_started/installation_guide/platforms/aws-image.html to install the image?

If so, Hopsworks version should be 1.4.1, maybe the version was skipped somehow when running the commands on AWS?

Can you try to create the VM again with the 1.4.1 version?

#3

when I search for 1.4.1 I see two images
LC-hopsworks-1.4.1_master__1 - ami-079ec9f61cfd6032f
LC-hopsworks-1.4.1_master__0 - ami-0e339036273422447

does it make a difference which I pick for a single node VM?

#4

Hi @DrGigabit

Pick LC-hopsworks-1.4.1_master__1 which is the latest.

BR,
antonios

#5

OK, so I have the VM running.

I SSH’ed into the machine, and started all the services
sudo /srv/hops/kagent/kagent/bin/start-all-local-services.sh

after that, I was able to log in into the Web UI by navigating https://publicIP/hopsworks

Hovewer, when I click e.g. on the feature store tour, I get the same pop up
×1

An argument was not provided or it was malformed.

java.net.UnknownHostException: rpc.namenode.service.consul

Here are the details of the network viewer:

Summary
URL: https://publicIP/hopsworks-api/api/project/starterProject/featurestore
Status: 422 UNPROCESSABLE_ENTITY
Source: Network
Address: publicIP:443
Initiator: 
vendor.72aa0020.js:6:19727


Request
POST /hopsworks-api/api/project/starterProject/featurestore HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json, text/plain, */*
Authorization: Bearer eyJraWQiOiIzNCIsInR5cCI6IkpXVCIsImFsZyI6IkhTNTEyIn0.eyJhdWQiOiJhcGkiLCJzdWIiOiJtZWIxMDAwMCIsIm5iZiI6MTYwNDQ5MTg5NCwiZXhwTGVld2F5Ijo5MDAsInJlbmV3YWJsZSI6dHJ1ZSwicm9sZXMiOlsiSE9QU19BRE1JTiJdLCJpc3MiOiJob3Bzd29ya3NAbG9naWNhbGNsb2Nrcy5jb20iLCJleHAiOjE2MDQ0OTM2OTQsImlhdCI6MTYwNDQ5MTg5NCwianRpIjoiYTAyNWI0NDUtM2ViMC00Yzk5LWJhNWQtODhlZTY5ODFiYzIyIn0.tIWPFmm60rwblwsxHJS78_q3S2F8_eglBYQj1NGMTbtofBfq2ya8l6W3a0N-WyAyByxC0rgyAnDVyaXkqvLMBg
Accept-Language: en-us
Accept-Encoding: gzip, deflate, br
Host: 3.16.13.180
Origin: https://3.16.13.180
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
Connection: keep-alive
Referer: https://publicIP/hopsworks/
Content-Length: 23
Cookie: JSESSIONIDSSO=231AEC29A56C30AFB84F3A9110782CFA; SESSION=32b5ebf0ea6adb3f512298ea37ee

Response
HTTP/1.1 422 UNPROCESSABLE_ENTITY
Date: Wed, 04 Nov 2020 12:14:05 GMT
Content-Type: application/json
X-Frame-Options: SAMEORIGIN
Content-Length: 182
X-Powered-By: Servlet/3.1 JSP/2.3 (Payara Server 4.1.2.181 #badassfish Java/Private Build/1.8)

Request Data
MIME Type: application/x-www-form-urlencoded
{"type":"featurestore"}

Same happens if I create a new project

Summary
URL: https://publicIP/hopsworks-api/api/project?projectName=test
Status: 422 UNPROCESSABLE_ENTITY
Source: Network
Address: publicIP:443
Initiator:
vendor.72aa0020.js:6:19727

#6

I looked further, and noticed that the datanode service is DEAD.
here is the log

systemctl status datanode.service
● datanode.service - DataNode, blocks server for HDFS.
   Loaded: loaded (/lib/systemd/system/datanode.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/datanode.service.d
           └─limits.conf
   Active: activating (start-pre) since Thu 2020-11-05 07:10:42 UTC; 1min 18s ago
  Process: 29236 ExecStart=/srv/hops/hadoop/sbin/start-dn.sh (code=exited, status=0/SUCCESS)
 Main PID: 29332 (code=exited, status=1/FAILURE); Control PID: 10460 (bash)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/datanode.service
           ├─10460 /bin/bash -c while ! /usr/bin/host rpc.namenode.service.consul ; do if systemctl list-units --full -all | grep -Fq namenode; then if systemctl is-active --quiet n
           └─12812 /bin/sleep 5

Nov 05 07:11:12 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:17 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:22 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:27 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:32 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:37 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:42 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:47 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:52 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
Nov 05 07:11:57 ip-172-31-26-151 bash[10460]: Host rpc.namenode.service.consul not found: 3(NXDOMAIN)
#7

Hi,

I took the same AMI and spun up a Hopsworks instance with no problem. Are you sure you have enabled Private DNS hostnames for your VPC?

Also, could you check /root/ec2init/ec2init.log if there is any error message?