Deploying and Starting a Cluster in Linux

[Former Member]

Hi all,

I have installed ESP5.1 SP08 on RHEL 6.0 x64. I completed setup without any error.

Then, I set environment variables and start esp_cluster.db as described in the following link.

Deploying and Starting a Cluster - Installation Guide (Linux) - SAP Library

But, when I start node, I am getting following error. I checked my db is up and running. cluster.cfg exists in the directory. What could the reason of this error?

Aug 14 2014 10:14:12.576 INFO - SAP Event Stream Processor Cluster Node 5.1.08.00/20140618.1/SP08 PL00/linux/x86_64/64-bit/OPT/Wed Jun 18 10:33:37 PDT 2014

Aug 14 2014 10:14:12.969 FATAL - CODE_700418 | Could not load config from database

com.sybase.esp.cluster.config.ConfigException: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor._load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor.load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseManager.load(Unknown Source)

        at com.sybase.esp.cluster.impl.Node.initialize(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.factory(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.main2(Unknown Source)

Caused by: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.config.Config.readData(Unknown Source)

        ... 6 more

Caused by: java.io.EOFException

        at java.io.DataInputStream.readFully(DataInputStream.java:197)

        at java.io.DataInputStream.readLong(DataInputStream.java:416)

        ... 7 more

Aug 14 2014 10:14:12.971 FATAL - CODE_700412 | Factory of new node failed

Thanks and regards,

Bulut

[Former Member · Accepted solution]

Hi Bulut,

I think this is being caused by the use of "localhost.localdomain".  I could not get the cluster manager to communicate with the cluster database when the cluster database was started with either "localhost.localdomain" or just "localhost".

If I kill the "dbsrv" process associated with "esp_cluster_esp1_localhost.localdomain", and then make the following changes it works:

1) Edit the "start_db.sh" file and change the following line to have an actual host name:

ESP_CLUSTER_DB_NAME=esp_cluster_esp1_archer

2) Edit the "cluster.cfg" file and uncomment the "ciper-file" line as the "esp_cluster_node" command will need that line for some commands.

3) Edit the "cluster.xml" file and change the ESP_HOSTNAME to be a fully qualified host name.  If you don't use a fully qualified host name, remote clients won't be able to connect to ESP:

    <Macro name="ESP_HOSTNAME" type="value">archer.acme.com</Macro>

4) Execute "start_db.sh" and use dblocate to confirm it is no longer using the localhost:

source $ESP_HOME/sqla/bin64/*.csh (or .sh if using a different shell)

dblocate

5) Try deploying the cluster configuration to the cluster database:

$ESP_HOME/bin/esp_cluster_node --config cluster.cfg --deploy --config-type file --file cluster.xml

Want to deploy config-version 5

Failed to deploy version 5

NOTE: If it says "Failed to deploy", the configuration changes may have still deployed but the cluster database versioning is out of sync.  You can check the changes were saved (especially the ESP_HOSTNAME) by executing this command:

  $ESP_HOME/esp_cluster_node --config cluster.cfg --show

6) Try starting the cluster manager:

start_node.sh node1

If this doesn't resolve it, you may want to reinstall ESP and when it prompts for the host name fill in a fully qualified host name.

Thanks,

Neal

[Former Member · Accepted solution]

Hi Neal,

Yes, it works. I can start the node finally. Thank you very much.

I have a last question My Linux is RHEL 6 and running as a VM on my machine. I am trying to connect from my Windows machine. Actually, I tried to connect but I could not be successful because of user credentials. What should be the user name? I used my unix user and password but it didn't work. I chose Native OS during the installation and it is seen in the cluster xml as below:

	<Security>
		<Authenticators>
			<xi:include href="auth_native_unix.xml" parse="xml"/>
		</Authenticators>
		<Authorizer enabled="true"/>
		<KeyStore>
			<Type expand="true">JKS</Type>
			<File expand="true">${ESP_CLUSTER_KEYSTORE}</File>
			<Password expand="true" encrypted="true">

u7In3rk3v+bSnOEth8cOlCP6a691nh+TndcVnZkpmiUmH/iK

</Password>

			<Algorithm expand="true">RSA</Algorithm>
		</KeyStore>

[Former Member · Accepted solution]

Hello,

There are some post-installation tasks to make Native OS authentication work:

    Configuration and Administration Guide - SAP Library

I forgot to mention that when I was talking about doing the "grant" permissions above.

So:

• Copy the correct "sybase-csi" to the /etc/pam.d directory
• If the Linux user is local to the VM, make sure that the user starting the cluster manager has the proper permissions to the /etc/shadow file as this is where local user credentials are stored.

Thanks,

Neal

[Former Member · Accepted solution]

I copied correct "sybase-csi" to the /etc/pam.d and gave read permission to /etc/shadow file for my local linux user.

Then to give permission as you said above I tried to connect cluster But, I could not login to cluster with espsysuser. Does it have a default password? I tried with with cluster password but I could not login?

$ESP_HOME/bin/esp_cluster_admin --uri=esp://localhost:19011 --username=espsysusr --password=Password1

[Former Member · Accepted solution]

The espsysusr password should be the same as the cluster password:

     http://help.sap.com/saphelp_esp51sp08lnx/helpdata/en/e7/a0b88e6f0f1014b13294f6f9e312ba/frameset.htm

espsysusr - Configuration and Administration Guide - SAP Library

Did you try "sybase"?

[Former Member · Accepted solution]

Yes, but it didn't work.

I am trying example cluster which includes predefined user. Hope it will work.

Thanks you very much again.

Regards,

Bulut

[Former Member · Accepted solution]

I can run example cluster with predefided user and can connect it from my Windows machine.

I don't understand why I could not connect when NativeOS is the security type. I will investigate it when I have more idle time.

Thanks again Neal.

Bulut

[Former Member · Accepted solution]

Hi Bulut,

The password for the "espsysusr" is the same as the cluster password.  The cluster password is defined in one of two ways:

1) Directly on the command line:
D:\ESP51_SP8\ESP-5_1\cluster\config\esp1>esp_cluster_node --cluster-name=esp1 --cluster-password=Password1 --cluster-password-is-encrypted=false --file cluster.xml --node-name=node1 --jdbc-url=jdbc:sybase:Tds:localhost:19111 --jdbc-username=espdbadm --jdbc-password=Password1 --cipher-file=D:\\ESP51_SP8\\ESP-5_1\\cluster\\keys\\esp1\\cluster.key --node-name=node1

2) In the "cluster.cfg" bootstrap file (esp_cluster_node --config cluster.cfg --config-type=file --file cluster.xml --node-name=node1):
cluster-name=esp1
cluster-password=Password1
cluster-password-is-encrypted=false
#cipher-file=/sybase/ESP-5_1/cluster/keys/esp1/cluster.key

So in these two examples, the password for the "espsysusr" is the same as the cluster password which is "Password1".

If you are not able to connect with "espsysusr", double check:
1) There are no orphaned "esp_cluster_node.exe" processes running with a different password.  If there are, kill them and restart.
2) Make sure your esp_cluster_admin is using the proper "--uri" setting for your node.  If your node is SSL enabled, use "esps://hostname:port".  If your node is not SSL enabled, use "esp://hostname:port".
3) Make sure to uncomment the "cipher-file" parameter out of your "cluster.cfg" file as it is required.

Thanks,

  Neal

[Former Member]

Hi Bulut,

That generally means that the username/password in the "cluster.cfg" file used by the "start_node.sh node1" command does not match the password defined in the cluster database.

Follow these steps to confirm that the password in the config file does not match:

1) Set you Linux PATH environment variable to include the SQL Anywhere executables:

      setenv PATH $ESP_HOME/sqla/bin64:${PATH}

2) Set your Linux LD_LIBRARY_PATH environment variable:

     setenv LD_LIBRARY_PATH $ESP_HOME/sqla/lib64:${LD_LIBRARY_PATH}

3) Issue the "dblocate" command to find the running instance of your cluster database:

% dblocate

SQL Anywhere Server Enumeration Utility Version 16.0.0.1691

Server Name                     Address

-----------                     -------

esp_cluster_esp1_denn00530262a  DENN00530262A.acme.com:19111

4) Test authentication to this database with the credentials you *think* are valid:

dbping -d -c "UID=espdbadm;PWD=Password1;ENG=esp_cluster_esp1_denn00530262a;DBN=esp_cluster"

SQL Anywhere Server Ping Utility Version 16.0.0.1691

Connected to SQL Anywhere 16.0.0.1691 server "esp_cluster_esp1_DENN00530262A" and database "esp_cluster".

Ping database successful.

5) Use those credentials in the "cluster.cfg" bootstrap file.  I would only worry about encrypting them if this is a cluster deployment:

jdbc-url=jdbc:sybase:Tds:localhost:19111

jdbc-username=espdbadm

jdbc-password=Password1

jdbc-password-is-encrypted=false

cluster-name=esp1

cluster-password=Password1

cluster-password-is-encrypted=false

Thanks,

  Neal

[Former Member]

Hi Neal,

While I can connect to databae, I could not start cluster again. I changed cluster.cfg based on my credentials, but I am getting same error. Any idea?

The steps I did:

[sybase@localhost esp1]$ . start_db.sh

SQL Anywhere Start Server In Background Utility Version 16.0.0.1691

[sybase@localhost esp1]$ dbping -d -c "UID=sybase;PWD=sybase;ENG=esp_cluster_esp1_localhost.localdomain;DBN=esp_cluster"

SQL Anywhere Server Ping Utility Version 16.0.0.1691

Connected to SQL Anywhere 16.0.0.1691 server "esp_cluster_esp1_localhost.localdomain" and database "esp_cluster".

Ping database successful.

[sybase@localhost esp1]$ . start_node.sh mynode

[sybase@localhost esp1]$ more mynode.log

Aug 14 2014 13:40:48.033 INFO - SAP Event Stream Processor Cluster Node 5.1.08.00/20140618.1/SP08 PL00/linux/x86_64/64-bit/OPT/Wed Jun 18 10:33:37 PDT 2014

Aug 14 2014 13:40:48.736 FATAL - CODE_700418 | Could not load config from database

com.sybase.esp.cluster.config.ConfigException: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor._load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor.load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseManager.load(Unknown Source)

        at com.sybase.esp.cluster.impl.Node.initialize(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.factory(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.main2(Unknown Source)

Caused by: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.config.Config.readData(Unknown Source)

        ... 6 more

Caused by: java.io.EOFException

        at java.io.DataInputStream.readFully(DataInputStream.java:197)

        at java.io.DataInputStream.readLong(DataInputStream.java:416)

        ... 7 more

Aug 14 2014 13:40:48.743 FATAL - CODE_700412 | Factory of new node failed

[sybase@localhost esp1]$

[sybase@localhost esp1]$

[sybase@localhost esp1]$

[sybase@localhost esp1]$ more cluster.cfg

#config-type=file

#file=cluster.xml

#The following url points to a default location.

#Set as appropriate for your environment.

jdbc-url=jdbc:sybase:Tds:localhost:19111

jdbc-username=sybase

jdbc-password=sybase

jdbc-password-is-encrypted=false

jdbc-type=sqla

jdbc-sqla-ha=false

jdbc-sqla-ha-mirror-primary=

jdbc-sqla-ha-mirror-partner-1=

jdbc-sqla-ha-mirror-partner-2=

cluster-name=esp1

cluster-password=sybase

cluster-password-is-encrypted=false

#cipher-file=/sybase/ESP-5_1/cluster/keys/esp1/cluster.key

[sybase@localhost esp1]$

Thanks.

Bulut

[Former Member]

Hi Bulut,

I just had someone else report something similar that you might check.  Look in $SYBASE/log/init_cluster_db.log.  Are there any errors in there?

If not, try starting the node with the node name "node1" as that is the default the installer normally uses:

    start_node.sh node1

If there were errors, the installer is supposed to create and initialize with the SQL Anywhere cluster database with the admin user and password and then installer is supposed to populate the database with some tables.

If you can ping it, the database admin and user were created and you might have to manually create the tables:

cd $ESP_HOME/cluster/sql

dbisql -c "UID=sybase;PWD=sybase;ENG=esp_cluster_esp1_localhost.localdomain;DBN=esp_cluster" -nogui sqla.sql

After the tables have been created, deploy the cluster configuration to the database:

$ESP_HOME/bin/esp_cluster_node --config cluster.cfg --deploy --config-type file --file cluster.xml

Then try starting the cluster manager again:

start_node.sh node1

After the cluster manager node has been started, you will have to use the special "espsysusr" security user to grant permission to a valid user (valid according to the authentication type you selected at installation):

$ESP_HOME/bin/esp_cluster_admin --uri=esp://localhost:19011 --username=espsysusr --password=Password1

> grant perm all to user my_linux_user

> quit

Then you can test authenticating and doing something with this user:

$ESP_HOME/bin/esp_cluster_admin --uri=esp://localhost:19011 --username=my_linux_user --password=Password1

> add workspace default

[done]

[Former Member]

Hi Neal,

When I tried to create the tables manually, I get following errors which show tables are already created.

----------------------

[sybase@localhost sql]$ dbisql -c "UID=sybase;PWD=sybase;ENG=esp_cluster_esp1_localhost.localdomain;DBN=esp_cluster" -nogui sqla.sql

Could not execute statement.

  Item 'ESP_CLUSTER_CONFIG_ENTRY' already exists

  SQLCODE=-110, ODBC 3 State="42S01"

  File: "sqla.sql" on line 1, column 1

  CREATE TABLE ESP_CLUSTER_CONFIG_ENTRY

  (

          VERSION                          BIGINT NOT NULL,

          SEQUENCE_NUMBER                  INTEGER NOT NULL,

          SEGMENT                          VARBINARY(2048) NOT NULL,

  ...

  You can continue executing or stop.

1. Stop

2. Continue

Select an option: 2

(Continuing after error)

Could not execute statement.

  Item 'ESP_CLUSTER_CONFIG_HISTORY' already exists

  SQLCODE=-110, ODBC 3 State="42S01"

  File: "sqla.sql" on line 11, column 1

  CREATE TABLE ESP_CLUSTER_CONFIG_HISTORY

  (

          VERSION                          BIGINT NOT NULL,

          TIME_CREATED                     TIMESTAMP NOT NULL,

          USER_NAME                        VARCHAR(128) NOT NULL,

  ...

  You can continue executing or stop.

1. Stop

2. Continue

Select an option: 2

(Continuing after error)

Could not execute statement.

  Item 'ESP_CLUSTER_CONFIG_VERSION' already exists

  SQLCODE=-110, ODBC 3 State="42S01"

  File: "sqla.sql" on line 22, column 1

  CREATE TABLE ESP_CLUSTER_CONFIG_VERSION

  (

          CURRENT_VERSION                  BIGINT NULL,

          PREVIOUS_VERSION                 BIGINT NULL

  )

  ...

  You can continue executing or stop.

1. Stop

2. Continue

-----------------------

And again, I could not start node1. Same message lines were writen into node1.log file.

Aug 15 2014 08:14:20.301 INFO - SAP Event Stream Processor Cluster Node 5.1.08.00/20140618.1/SP08 PL00/linux/x86_64/64-bit/OPT/Wed Jun 18 10:33:37 PDT 2014

Aug 15 2014 08:14:20.540 FATAL - CODE_700418 | Could not load config from database

com.sybase.esp.cluster.config.ConfigException: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor._load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseAccessor.load(Unknown Source)

        at com.sybase.esp.cluster.impl.ConfigDatabaseManager.load(Unknown Source)

        at com.sybase.esp.cluster.impl.Node.initialize(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.factory(Unknown Source)

        at com.sybase.esp.cluster.FactoryNode.main2(Unknown Source)

Caused by: com.sybase.esp.cluster.config.ConfigException: java.io.EOFException

        at com.sybase.esp.cluster.config.Config.readData(Unknown Source)

        ... 6 more

Caused by: java.io.EOFException

        at java.io.DataInputStream.readFully(DataInputStream.java:197)

        at java.io.DataInputStream.readLong(DataInputStream.java:416)

        ... 7 more

Aug 15 2014 08:14:20.544 FATAL - CODE_700412 | Factory of new node failed

[Former Member]

Hi Bulut,

I think I will need to see all of the config, log and XML files from your %ESP_HOME%\cluster\config\esp1 directory.  Can you zip them up and attach them here?

Or email them to me (neal dot stack at sap dot com).

Thanks,

Neal