Hi,
We are facing a problem with mysql ndb. The cluster starts correctly but when the mysqld process try to join with ndb we received errors in the log (see below the extraction). We have a 5 nodes installation 1 mysqld, 1 management, 3 data nodes and the sw version is: mysql-ndb-7.6.9, hopsworks 1.0
Thank you for the support
Here is the log extraction:
2021-03-05T12:19:36.063208Z 1 [ERROR] NDB Binlog: Unable to create event in database. Event: REPL$mysql/ndb_schema Error Code: 1 Message: Unknown error code
2021-03-05T12:19:36.063252Z 1 [ERROR] NDB Binlog: FAILED CREATE (DISCOVER) TABLE Event: REPL$mysql/ndb_schema
2021-03-05T12:19:36.063561Z 15 [Note] table ‘./hops/hdfs_users’ opened read only
2021-03-05T12:19:36.073919Z 15 [Note] table ‘./hops/hdfs_users_groups’ opened read only
2021-03-05T12:19:36.083460Z 15 [Note] table ‘./hops/hdfs_variables’ opened read only
2021-03-05T12:19:36.098879Z 15 [Note] table ‘./hops/hdfs_xattrs’ opened read only
2021-03-05T12:19:36.109746Z 15 [Note] table ‘./hops/yarn_applicationattemptstate’ opened read only
2021-03-05T12:19:36.120866Z 15 [Note] table ‘./hops/yarn_applicationstate’ opened read only
2021-03-05T12:19:36.134213Z 15 [Note] table ‘./hops/yarn_container_to_decrease’ opened read only
2021-03-05T12:19:36.144689Z 15 [Note] table ‘./hops/yarn_container_to_signal’ opened read only
2021-03-05T12:19:36.157928Z 15 [Note] table ‘./hops/yarn_containerid_toclean’ opened read only
2021-03-05T12:19:36.167726Z 15 [Note] table ‘./hops/yarn_containers_checkpoint’ opened read only
2021-03-05T12:19:36.179337Z 15 [Note] table ‘./hops/yarn_containers_logs’ opened read only
2021-03-05T12:19:36.190517Z 15 [Note] table ‘./hops/yarn_containerstatus’ opened read only
2021-03-05T12:19:36.203033Z 15 [Note] table ‘./hops/yarn_delegation_key’ opened read only
2021-03-05T12:19:36.213028Z 15 [Note] table ‘./hops/yarn_delegation_token’ opened read only
2021-03-05T12:19:36.223308Z 15 [Note] table ‘./hops/yarn_le_descriptors’ opened read only
2021-03-05T12:19:36.234489Z 15 [Note] table ‘./hops/yarn_nextheartbeat’ opened read only
By the by, could you tell me how I can use a client software to connect to the MySQL Cluster? Such as Workbench, Squirrel-sql, etc.
I’ve already known that there is a database script (mysql-client.sh), which can be used to connect to MySQL clusters. But I haven’t found the username and password yet.
If you run the script as root, it uses the local socket to connect (root has read privileges on the socket).
You can connect mysql workbench. When you installed Hopsworks (which version is it?), how did you install it? A username/password has been set - if you used hopsworks-installer.sh, it is in the ~/cluster-definitions/hopsworks-installation.yml file where you ran the installer. Otherwise, if you used Karamel, and did not set a password, the default is
username: kthfs
password: kthfs
from here:
Thank you for your reply. The version is 2.0. I used hopsworks-installer.sh to install.
I’ve found the MySQL password in hopsworks-installation.yml, but I haven’t found the username. I’ve tried “root” and “kthfs”, but none of them is right.
Hi Jim,
Thank’s a lot for your reply. This is the output of the report memory usage command:
– NDB Cluster – Management Client –
ndb_mgm> ALL REPORT MEMORYUSAGE
Connected to Management Server at: localhost:1186
Node 1: Data usage is 27%(21568 32K pages of total 78064)
Node 1: Index usage is 3%(1936 32K pages of total 58432)
Node 2: Data usage is 27%(21580 32K pages of total 78064)
Node 2: Index usage is 3%(1936 32K pages of total 58420)
Node 3: Data usage is 27%(21561 32K pages of total 78062)
Node 3: Index usage is 3%(1938 32K pages of total 58439)
df -kh
on the command line - is there free space on the server?
Can you post the logs of your mysql server and of the ndb_mgmd service?
tail -444 /srv/hops/mysq-cluster/logs/cluster.log
and
tail -444 /srv/hops/mysq-cluster/logs/mysql_52_out.log
As you say, “mysql -u kthfs -pXXXXX -h 127.0.0.1 hopsworks” works well.
But when I replaced 127.0.0.1 with the host IP address(or host name), an error message was reported:
ERROR 1045 (28000): Access denied for user ‘kthfs’@‘dwfaihead’ (using password: YES)
‘dwfaihead’ is the hostname of the master node, could you please tell me how to solve this issue?
The mysql user/password is for localhost for security reasons.
You can create a new username/password using mysql-client.sh:
grant all privileges on hopsworks.* to ‘kthfs’@‘dwfaihead’ identified by ‘YOUR_PASSWORD’
Thank you very much for your help. By creating a new user, I have solved this issue.
BTW, for version 8 or later of MySQL, it must take two steps, the script is as follows:
use mysql;
select host,user from user;
create user ‘new_user’@’%’ identified by ‘new_user_pw’;
GRANT ALL PRIVILEGES ON . TO ‘new_user’@’%’ WITH GRANT OPTION;
select host,user from user;
FLUSH PRIVILEGES;
Hi @Jim_Dowling,
I’m sorry for the long time waiting for answer. At the end, we solved the problem by setting NDB configuration ndb-join-pushdown=1 because the DB failed when executed particular join (eg. python deps).