Namenode format hang on "Using ClusterJ DTO Cache: false" and failed

HopsFS
#1

Hi, I manually installed hops3.5 by the document of the GitHub, and I have seen io.hops.exception.StorageInitializtionException: Error getting connection to cluster , it’s a different issue.

when I try

./bin/hdfs namenode -format

it shows

************************************************************/
2022-01-13 12:15:15,350 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-01-13 12:15:15,434 INFO namenode.NameNode: createNameNode [-format]
2022-01-13 12:15:15,647 INFO resolvingcache.Cache: starting Resolving Cache [InMemoryCache]
2022-01-13 12:15:15,690 INFO ndb.ClusterjConnector: Database connect string: 10.0.0.7:1186
2022-01-13 12:15:15,690 INFO ndb.ClusterjConnector: Database name: hops_db
2022-01-13 12:15:15,690 INFO ndb.ClusterjConnector: Max Transactions: 1024
2022-01-13 12:15:15,692 INFO ndb.DBSessionProvider: Database connect string: 10.0.0.7:1186
2022-01-13 12:15:15,692 INFO ndb.DBSessionProvider: Database name: hops_db
2022-01-13 12:15:15,692 INFO ndb.DBSessionProvider: Max Transactions: 1024
2022-01-13 12:15:15,692 INFO ndb.DBSessionProvider: Using ClusterJ Session Cache: false
2022-01-13 12:15:15,692 INFO ndb.DBSessionProvider: Using ClusterJ DTO Cache: false

after a few seconds, I get

2022-01-13 12:15:36,213 ERROR namenode.NameNode: Failed to start namenode.
io.hops.exception.StorageInitializtionException: Error getting connection to cluster
	at io.hops.metadata.ndb.NdbStorageFactory.setConfiguration(NdbStorageFactory.java:72)
	at io.hops.metadata.HdfsStorageFactory.setConfiguration(HdfsStorageFactory.java:123)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.formatHdfs(NameNode.java:951)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1179)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1245)
2022-01-13 12:15:36,223 INFO util.ExitUtil: Exiting with status 1: io.hops.exception.StorageInitializtionException: Error getting connection to cluster
2022-01-13 12:15:36,228 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node6/10.0.0.6
************************************************************/

However, the NDB works well

mysql -uroot -p123 -h10.0.0.7
...
mysql>

and that’s my ndb-config.properties:

com.mysql.clusterj.connectstring=10.0.0.7:1186
com.mysql.clusterj.database=hops_db
com.mysql.clusterj.connection.pool.size=1
com.mysql.clusterj.max.transactions=1024
com.mysql.clusterj.connection.reconnect.timeout=5

io.hops.enable.clusterj.dto.cache=false
io.hops.enable.clusterj.session.cache=false
com.mysql.clusterj.max.cached.instances=256
com.mysql.clusterj.max.cached.sessions=0
com.mysql.clusterj.warmup.cached.sessions=0

io.hops.metadata.ndb.mysqlserver.data_source_class_name = com.mysql.cj.jdbc.MysqlDataSource
io.hops.metadata.ndb.mysqlserver.host=10.0.0.7
io.hops.metadata.ndb.mysqlserver.port=3306
io.hops.metadata.ndb.mysqlserver.username=root
io.hops.metadata.ndb.mysqlserver.password=123
io.hops.metadata.ndb.mysqlserver.connection_pool_size=1
io.hops.session.pool.size=1000
io.hops.session.reuse.count=2147483647

Any help would be appreciated!

#2

Does this file /srv/hops/hadoop/lib/native/libndbclient.so exist? In order to connect to RonDB, HopsFS needs libndbclient.so library. If it does not exist then you can copy it from /srv/hops/mysql/lib/libndbclient.so

#3

Hi, @salman ! Thanks for your reply!
I’m sure that I had libndbclient.so in my $HADOOP_HOME/lib/native/.
If not, it will show another error.

#4

Make sure that all the ports used by RonDB are not blocked by a firewall on the HopsFS node. You will find the ports in the /srv/hops/mysql-cluster/config.ini file. By default, it uses ports 1186 and 10000.