Error after migration of the statistics server on SAP HANA

[THorton]

Hi,

In preparation for converting an existing single containter HANA DB from single container mode to multitenant I migrated the statistics server, per SAP Note 1917938.  All checks detailed in the note and associated documentation were successful during the migration of the statistics server.  Therefore, I proceeded with the multitenant database conversion.  After the multitenant database conversion, I receive the following error in HANA Studio:

SAP DBTech JDBC: [259] (at 42): invalid table name:  Could not find table/view STATISTICS_CURRENT_ALERTS in schema _SYS_STATISTICS: line 1 col 43 (at pos 42)

com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [259] (at 42): invalid table name:  Could not find table/view STATISTICS_CURRENT_ALERTS in schema _SYS_STATISTICS: line 1 col 43 (at pos 42)

at com.sap.db.jdbc.exceptions.SQLExceptionSapDB.createException(SQLExceptionSapDB.java:345)

at com.sap.db.jdbc.exceptions.SQLExceptionSapDB.generateDatabaseException(SQLExceptionSapDB.java:176)

at com.sap.db.jdbc.packet.ReplyPacket.buildExceptionChain(ReplyPacket.java:102)

at com.sap.db.jdbc.ConnectionSapDB.execute(ConnectionSapDB.java:1033)

at com.sap.db.jdbc.ConnectionSapDB.execute(ConnectionSapDB.java:823)

at com.sap.db.jdbc.StatementSapDB.sendCommand(StatementSapDB.java:896)

at com.sap.db.jdbc.StatementSapDB.sendSQL(StatementSapDB.java:945)

at com.sap.db.jdbc.StatementSapDB.execute(StatementSapDB.java:256)

at com.sap.db.jdbc.StatementSapDB.execute(StatementSapDB.java:228)

at com.sap.db.jdbc.trace.Statement.execute(Statement.java:132)

at com.sap.ndb.studio.jdbc.util.SQLExecutor$1.run(SQLExecutor.java:118)

The traces files are also logging the following message:

exception throw location:

1: 0x00007f499ac0f7c0 in StatisticsService::CallInterfaceDirect::install()+0x2b0 at CallInterfaceDirect.cpp:243 (libstatisticsserver2.so)

2: 0x00007f499aa071ba in StatisticsService::NameServerControllerThread::run(void*)+0xd6 at NameServerControllerThread.cpp:306 (libhdbns.so)

3: 0x00007f4990860594 in TrexThreads::PoolThread::run()+0x810 at PoolThread.cpp:256 (libhdbbasement.so)

4: 0x00007f4990862070 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:124 (libhdbbasement.so)

5: 0x00007f49822e5759 in Execution::Thread::staticMainImp(void**)+0x875 at Thread.cpp:492 (libhdbbasis.so)

6: 0x00007f49822e63bd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)

[46268]{-1}[-1/-1] 2015-04-30 14:49:21.124089 e STATS_WORKER     CallInterfaceDirect.cpp(00216) : error updating installation state with statement "update _SYS_STATISTICS.STATISTICS_PROPERTIES set value='Done (error) since '||current_utctimestamp where key='internal.installation.state'", exception 9002003: Unknown catalog object.

[46268]{-1}[-1/-1] 2015-04-30 14:49:21.124157 e STATS_CTRL       NameServerControllerThread.cpp(00327) : error installing: exception  1: no.9002004  (StatisticsServerAsSQLScript/global/CallInterfaceDirect.cpp:243)

    Error during installation: exception 9002002: Duplicate catalog object.

I have not been able to find any documentation on this error.  What do I need to do to resolve it?

Thanks in advance for your help!

Best Regards,

Tommye Horton

[Former Member · Accepted solution]

Hi Tommye,

We have done the ESS migration on our HANA systems earlier this month and I confirm STATISTICS_CURRENT_ALERTS is a view that is available post the ESS migration

Have you done similar ESS migration and Multitenant Database on any other system? Did you find similar errors on them as well?

Are these the only set of errors you see or do you have more of these errors?

Can you do a select * from STATISTICS_PROPERTIES and share the result there must be KEY internal.installation.state and VALUE=Done(okay) since YYYY-MM-DD...

For the first error regarding the View we may overcome it by creating the VIEW but if we have more errrors on the Statisticsserver objects it is good to have a case open with SAP Support right away

[THorton · Accepted solution]

Hi Sunil,

Thank you very much for responding.

No, this is the first ESS migration and multitenant database conversion that I have completed.

The errors that I posted above are the only ones that I have found, and they are being logged repeatedly.

The result of the select * from STATISTICS_PROPERTIES statement is as follows:

Could not execute 'select * from STATISTICS_PROPERTIES'

SAP DBTech JDBC: [259]: invalid table name:  Could not find table/view STATISTICS_PROPERTIES in schema SYSTEM: line 1 col 15 (at pos 14)

I do have a case open with SAP.  However, our experience is that it often takes weeks or even months to receive a response.  I guess they just do not have enough people.

I would appreciate any other input that you have.

Thanks again,

Tommye

[Former Member · Accepted solution]

Hi Tommye,

We have 2 SAP notes for the ESS migration 2092033 and 1917938

You will notice that checking the value of internal.installation.state from the table STATISTICS.PROPERTIES is the first step to validate the successful migration. Did you notice that the table existed right after migration? That would confirm the Multitenant conversion may have caused some issue leading to these errors

[THorton · Accepted solution]

Hi Sunil,

Yes, I ran the check as detailed in the instructions, and the table existed right after the migration.

Thanks!

Tommye

[Former Member · Accepted solution]

That confirms the issue started occurring after the Multitenant setup, it is not straight forward to understand why would Multitenant setup impact the STATISTICS tables

I would expect SAP Support to have a clear understanding of the issue and suggest not to make any changes unless we are asked by SAP Support

This will help you identify any missing links and prepare a plan for doing this smoothly in the next set of systems you will be working on

[Former Member]

Has there been any reply to this by any chance? We seem to have a similar situation on our HANA system.

Thanks!

[Former Member]

Hi Tommye,

Could you please share the solution? We are having the same problem.

Thanks in advance,

[THorton]

Victor and Rodrigo,

We still have an open message with SAP.  The solution is still TBD.

Thanks,

Tommye

[candy_zhan]

Hi Tommye,

What is the error before "exception throw location:
1: 0x00007f499ac0f7c0 in StatisticsService::CallInterfaceDirect::install()+0x2b0 at CallInterfaceDirect.cpp:243 (libstatisticsserver2.so)
..."
in indexserver? Also what is your HANA Revision?

Can you try the following workaround?

1. Clean up the previously incomplete migration:

drop table "_SYS_STATISTICS"."STATISTICS_ALERTS_BASE";
drop table "_SYS_STATISTICS"."STATISTICS_ALERT_INFORMATION";
drop table "_SYS_STATISTICS"."STATISTICS_ALERT_THRESHOLDS";
drop table "_SYS_STATISTICS"."STATISTICS_EMAILRECIPIENTS";
drop table "_SYS_STATISTICS"."STATISTICS_EMAIL_PROCESSING";
drop table "_SYS_STATISTICS"."STATISTICS_PROPERTIES";
drop table "_SYS_STATISTICS"."STATISTICS_SCHEDULE";
drop type "_SYS_STATISTICS"."TT_STATISTICS_ALERT_THRESHOLDS";
drop type "_SYS_STATISTICS"."TT_STATISTICS_USED_VALUES";
drop view "_SYS_STATISTICS"."STATISTICS_ALERTS";
drop view "_SYS_STATISTICS"."STATISTICS_CURRENT_ALERTS";
drop view "_SYS_STATISTICS"."STATISTICS_LAST_CHECKS";
drop procedure "_SYS_STATISTICS"."STATISTICS_SCHEDULABLEWRAPPER";
drop procedure "_SYS_STATISTICS"."STATISTICS_PREPARE_CALL_TIMER";
drop procedure "_SYS_STATISTICS"."STATISTICS_PREPARE_CALL_MANUAL";

2. Reset all threshold with _INFORMATION suffix in statisticsserver.ini back to default.
3. Restart the ESS migration (active=true)
4. Migration will hang in this stage because of another bug(Bug 68720):

[67361]{-1}[-1/-1] 2015-01-30 02:18:12.930518 i STATS_CTRL       NameServerControllerThread.cpp(00409) : waiting for start of old StatisticsServer dellhana9:31005, volume: 4...

Force kill the old statistic server process so the migration will move forward

Let me know the result.

Best regards,
Candy Zhan

[THorton]

Hi Candy,

Thank you for your reply.

I pasted the entire error message in the original post - it is repeatedly getting logged in the trace file.

The HANA Revision is 1.00.091.00.1418659308.

The objects listed above do not exist in the SYSTEM Database.  All of the objects with an _INFORMATION suffix already have the default values.

Best Regards,

Tommye

[Former Member]

I have the same problem, have you solved it?

Regards,