HANA AWS Instance not listening on 30015

Hello,

I have spent nearly all day loading data into our HANA instance on AWS for a demo next week.

I had been connected via JDBC, everything was working as it should.

After several hours of everything working I got this error half way through a SQL script.:

21:20:23 [INSERT - 7962624 row(s), 442.472 secs] Command processed

21:32:26 [INSERT - 0 row(s), 0.000 secs] [Error Code: -708, SQL State: 08006] [-708] Data receive failed [Connection reset].

21:32:26 [INSERT - 0 row(s), 0.000 secs] [Error Code: -708, SQL State: 08006] [-708] Connection T0 could not send data [Socket closed].

Trying to reconnect I get:

An error occurred while establishing the connection:

Type: com.sap.db.jdbc.exceptions.JDBCDriverException Error Code: -813 SQL State: HY000

Message:

SAP DBTech JDBC: Cannot connect to jdbc:sap://<ip-address>:30015 [Cannot connect to host <ip-address>:30015 [Connection refused], -813.].

I check to see if port 30015 is listening on the server:

imdbhdb:/usr/sap/HDB/HDB00> netstat -ant | grep "LISTEN"

tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30005 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30007 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:30008 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:50013 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30000 0.0.0.0:* LISTEN

It is not. I try restarting HANA...

imdbhdb:~ # su - hdbadm

imdbhdb:/usr/sap/HDB/HDB00> netstat -ant | grep "LISTEN"

tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30005 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30007 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:30008 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:50013 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30000 0.0.0.0:* LISTEN

imdbhdb:/usr/sap/HDB/HDB00> stopsap

Stopping the SAP instance HDB00

----------------------------------

Shutdown-Log is written to /usr/sap/HDB/home/stopsap_HDB00.log

Instance on host imdbhdb stopped

Waiting for cleanup of resources..........

imdbhdb:/usr/sap/HDB/HDB00> netstat -ant | grep "LISTEN"

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:50013 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

imdbhdb:/usr/sap/HDB/HDB00> startsap

Starting SAP Instance HDB00

------------------------------

Startup-Log is written to /usr/sap/HDB/home/startsap_HDB00.log

Instance Service on host imdbhdb started

Instance on host imdbhdb started

imdbhdb:/usr/sap/HDB/HDB00> netstat -ant | grep "LISTEN"

tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30003 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30005 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30007 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:30008 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:50013 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:30000 0.0.0.0:* LISTEN

imdbhdb:/usr/sap/HDB/HDB00>

It is still not listening on 30015.

I have rebooted the instance.. same deal. no 30015.

What's wrong?

I'd rather fix this than have to create a new HANA instance and have to reinsert all my data again..

I haven't changed anything on the server..

Thanks,

Peter

SAP Managed Tags:
SQL,
SAP HANA

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Accepted Solutions (0)

Answers (1)

Hi Peter,

What's the content of the startup log file /usr/sap/HDB/home/startsap_HDB00.log?

Also, can you run a "ps -A | grep hdb" and post the output?

--juergen

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi Juergen,

Here is the output that you have requested:

Log output:

imdbhdb:/usr/sap/HDB/HDB00> cat /usr/sap/HDB/home/startsap_HDB00.log

Trace of system startup/check of SAP System HDB on Sat Jul 7 23:20:31 UTC 2012

Called command: /usr/sap/HDB/HDB00/exe/startsap HDB00

Starting SAP Instance HDB00

------------------------------

Instance Service on host imdbhdb started

SAP-R/3-Startup Program Rel 802 V2.0 (2008/11/15)

-------------------------------------------------

Starting at 2012/07/07 23:20:31

Startup Profile: "/usr/sap/HDB/SYS/profile/HDB_HDB00_imdbhdb"

Setup Environment Variables

---------------------------

(5971) SETENV PATH=/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/HDB00/imdbhdb:/usr/sap/HDB/HDB00:/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/HDB00/exe/Python/bin:/usr/sap/HDB/HDB00/exe/dat_bin_dir:.:/usr/sap/HDB/home:/usr/sap/HDB/home/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin

(5971) SETENV SAP_RETRIEVAL_PATH=/usr/sap/HDB/HDB00/imdbhdb

(5971) SETENV PYTHONPATH=/usr/sap/HDB/HDB00/imdbhdb:/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/SYS/global/hdb/custom/python_support:/usr/sap/HDB/HDB00/exe/python_support

(5971) SETENV DIR_LIBRARY=/usr/sap/HDB/HDB00/exe

(5971) SETENV LD_LIBRARY_PATH=/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/HDB00/exe:/usr/sap/HDB/HDB00/exe/Python/lib:/usr/sap/HDB/HDB00/exe/filter:/usr/sap/HDB/HDB00/exe/dat_bin_dir:/usr/sap/HDB/HDB00/exe/AFL

(5971) SETENV SHLIB_PATH=/usr/sap/HDB/HDB00/exe:

(5971) SETENV LIBPATH=/usr/sap/HDB/HDB00/exe:

Execute Pre-Startup Commands

----------------------------

(5972) Local: /usr/sap/HDB/SYS/global/hdb/install/bin/hdbupdconf

SAP HANA Database Installation Manager - HdbupdConf 1.00.28.361821

******************************************************************

No todo file "/usr/sap/HDB/global/hdb/install/config/imdbhdb/hdbupdConf.todo" present: nothing to be done, exiting.

hdbupdConf done.

(5980) Local: rm -f /usr/sap/HDB/HDB00/imdbhdb/trace/py.sapHDB_HDB00

(5981) Local: ln -s -f /usr/sap/HDB/HDB00/exe/Python/bin/python /usr/sap/HDB/HDB00/imdbhdb/trace/py.sapHDB_HDB00

(5982) Local: rm -f /usr/sap/HDB/HDB00/imdbhdb/trace/hdb.sapHDB_HDB00

(5983) Local: ln -s -f /usr/sap/HDB/HDB00/exe/hdbdaemon /usr/sap/HDB/HDB00/imdbhdb/trace/hdb.sapHDB_HDB00

Starting Programs

-----------------

07.07.2012 23:20:31

ShmDetach

(5986) Starting: local /usr/sap/HDB/HDB00/imdbhdb/trace/hdb.sapHDB_HDB00 -d -nw -f /usr/sap/HDB/HDB00/imdbhdb/daemon.ini pf=/usr/sap/HDB/SYS/profile/HDB_HDB00_imdbhdb

(5971) Waiting for Child Processes to terminate.

Instance on host imdbhdb started

PS output:

imdbhdb:/usr/sap/HDB/HDB00> ps -A | grep hdb

5986 ? 00:00:00 hdb.sapHDB_HDB0

6005 ? 00:00:04 hdbnameserver

6053 ? 00:00:01 hdbpreprocessor

6078 ? 00:00:02 hdbstatisticsse

6105 ? 00:00:02 hdbxsengine

Let me know if there's anything else you need.

Thanks,

Peter

hm. Looks like your hdbindexserver process crashed and is not coming up again.

In /usr/sap/HDB/HDB00/imdbhdb/trace you should find a recent crashdump file for the indexserver and a trace file called indexserver_alert_imdbhdb.trc. Have a look into these files to see if there's any hint. Also, please check how much disk space you have left on your instance.

--Juergen

I gave up and have begun recreating the stack again.

So the instance in now gone.

I had looked through some of the trace files earlier, but probably not that one..

Had

I had looked at the available disk space.. From memory, I hadn't used very much at all...

Thanks for your help... I'll repost to this thread if I encounter the issue again.

Peter.

Hey，Juergen, I looked into the trace log and got below information(I followed the steps Peter did):

NOTE: full crash dump will be written to /usr/sap/HDB/HDB00/hanadb/trace/indexserver_hanadb.30003.crashdump.04408.trc

Call stack of crashing context:

1: 0x00007eff7e0ec65b in raise+0x27 (libpthread.so.0)

2: 0x00007eff8aad5c3a in Basis::crashImpl(char const*, int, char const*, ltt::exception const&)+0xb6 at Crash.cpp:583 (libhdbbasis.so)

3: 0x00007eff8aad5c8e in Basis::crashImpl(char const*, int, ltt::exception const&)+0xa at Crash.cpp:523 (libhdbbasis.so)

4: 0x00007eff8aad5e7e in Basis::crashImpl(char const*, int, char const*)+0x6a at Crash.cpp:489 (libhdbbasis.so)

5: 0x00007eff8ab3ba3a in exit_handler+0x46 at FaultProtectionImpl.cpp:531 (libhdbbasis.so)

6: 0x000000000049949a in _exit+0x16 at LinuxMallocInitializer.cpp:143 (hdbindexserver)

7: 0x00007eff81a29e97 in ptime::CheckpointMgr::restarter(void*)+0xd3 at CheckpointMgr.cc:522 (libhdbrskernel.so)

8: 0x00007eff8103e6f0 in ptime::PtimeThread::run(void*)+0x10 at ptime_thread.h:131 (libhdbrskernel.so)

9: 0x00007eff95f779ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

10: 0x00007eff95f79028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

11: 0x00007eff8abcdf42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

12: 0x00007eff8abce10d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[4636]{0}[0] 2013-01-19 15:22:08.923767 f PersistenceL PersistenceFactory.cpp(00557) : start up of newdb persistence failed: exception 1: no.1000002 (ltt/impl/memory.cpp:115)

Out of memory ; $size$=1073741824; $name$=LogRecovery; $type$=pool; $inuse_count$=2; $allocated_size$=544

exception throw location:

1: 0x00007fbe424d703f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

2: 0x00007fbe424cd18e in DataRecovery::RecoveryHandlerImpl::pushLogSegment(DataAccess::LogSegmentBackupInfo const&, DataRecovery::LogSegment*, ltt::smartptr_handle<Stream::ChannelCallback> const&)+0x3ca at RecoveryHandlerImpl.cpp:993 (libhdbdataaccess.so)

3: 0x00007fbe424cd5d7 in DataRecovery::RecoveryHandlerImpl::pushLocalLogSegment(DataRecovery::LogSegment&, DataRecovery::ReplayProgressCallback&)+0x33 at RecoveryHandlerImpl.cpp:1022 (libhdbdataaccess.so)

4: 0x00007fbe424ceea4 in DataRecovery::RecoveryHandlerImpl::RecoveryHandlerImpl(DataRecovery::LoggerImpl&, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*, unsigned long, DataAccess::OnDiskLogState&)+0x15b0 at RecoveryHandlerImpl.cpp:704 (libhdbdataaccess.so)

5: 0x00007fbe424a4fae in DataRecovery::LoggerImpl::prepareRecovery(unsigned long, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*)+0x27a at LoggerImpl.cpp:638 (libhdbdataaccess.so)

6: 0x00007fbe4221a8c2 in DataAccess::PersistenceManager::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool)+0x320 at PersistenceManagerImpl.cpp:1492 (libhdbdataaccess.so)

7: 0x00007fbe4221a9a1 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool)+0x40 at PersistenceManagerImpl.cpp:2709 (libhdbdataaccess.so)

8: 0x00007fbe4221ae56 in DataAccess::PersistenceManager::replay()+0xa2 at PersistenceManagerImpl.cpp:1356 (libhdbdataaccess.so)

9: 0x00007fbe4e63dfb8 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool)+0x64 at PersistenceFactory.cpp:530 (libhdbpersistence.so)

10: 0x00007fbe4e377cdf in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, TREX_ERROR::TRexError*)+0x133b at PersistenceController.cpp:383 (libhdblogger.so)

11: 0x00000000004a7a64 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0xb0 at TREXIndexServer.cpp:355 (hdbindexserver)

12: 0x00000000005250ed in TRexAPI::AssignThread::run(void*)+0x39 at TREXIndexServer.cpp:260 (hdbindexserver)

13: 0x00007fbe4ee179ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

14: 0x00007fbe4ee19028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

15: 0x00007fbe43a6df42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

16: 0x00007fbe43a6e10d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[4636]{0}[0] 2013-01-19 15:22:09.003637 f PersistenceL PersistenceController.cpp(00385) : newdb persistence startup failed

[4636]{0}[0] 2013-01-19 15:22:09.004101 e assign TREXIndexServer.cpp(00364) : assign failed with persistence startup error. stopping service...

[4766]{0}[0] 2013-01-19 15:22:09.567973 e parallel Parallel.cpp(00163) : release; Allocator[702]:Pool/parallel(reset_time=(none),CATEGORY=Pool/parallel,DEPTH=2,INCLUSIVE_SIZE_IN_USE=7680B,INCLUSIVE_COUNT_IN_USE=20,INCLUSIVE_ALLOCATED_SIZE=7680B,INCLUSIVE_DEALLOCATED_SIZE=0B,INCLUSIVE_ALLOCATED_COUNT=20,INCLUSIVE_DEALLOCATED_COUNT=0,INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,INCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_SIZE_IN_USE=7680B,EXCLUSIVE_COUNT_IN_USE=20,EXCLUSIVE_ALLOCATED_SIZE=7680B,EXCLUSIVE_DEALLOCATED_SIZE=0B,EXCLUSIVE_ALLOCATED_COUNT=20,EXCLUSIVE_DEALLOCATED_COUNT=0,EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,EXCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_ALLOC_ERRORS=0,MALLOC_PROXY_CACHE_MISSES=0,FLAGS=(none))

[4527]{0}[0] 2013-01-22 01:32:54.528817 f PersistenceL PersistenceFactory.cpp(00557) : start up of newdb persistence failed: exception 1: no.1000002 (ltt/impl/memory.cpp:115)

Out of memory ; $size$=1073741824; $name$=LogRecovery; $type$=pool; $inuse_count$=2; $allocated_size$=544

exception throw location:

1: 0x00007f16fac0803f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

2: 0x00007f16fabfe18e in DataRecovery::RecoveryHandlerImpl::pushLogSegment(DataAccess::LogSegmentBackupInfo const&, DataRecovery::LogSegment*, ltt::smartptr_handle<Stream::ChannelCallback> const&)+0x3ca at RecoveryHandlerImpl.cpp:993 (libhdbdataaccess.so)

3: 0x00007f16fabfe5d7 in DataRecovery::RecoveryHandlerImpl::pushLocalLogSegment(DataRecovery::LogSegment&, DataRecovery::ReplayProgressCallback&)+0x33 at RecoveryHandlerImpl.cpp:1022 (libhdbdataaccess.so)

4: 0x00007f16fabffea4 in DataRecovery::RecoveryHandlerImpl::RecoveryHandlerImpl(DataRecovery::LoggerImpl&, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*, unsigned long, DataAccess::OnDiskLogState&)+0x15b0 at RecoveryHandlerImpl.cpp:704 (libhdbdataaccess.so)

5: 0x00007f16fabd5fae in DataRecovery::LoggerImpl::prepareRecovery(unsigned long, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*)+0x27a at LoggerImpl.cpp:638 (libhdbdataaccess.so)

6: 0x00007f16fa94b8c2 in DataAccess::PersistenceManager::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool)+0x320 at PersistenceManagerImpl.cpp:1492 (libhdbdataaccess.so)

7: 0x00007f16fa94b9a1 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool)+0x40 at PersistenceManagerImpl.cpp:2709 (libhdbdataaccess.so)

8: 0x00007f16fa94be56 in DataAccess::PersistenceManager::replay()+0xa2 at PersistenceManagerImpl.cpp:1356 (libhdbdataaccess.so)

9: 0x00007f1706d6efb8 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool)+0x64 at PersistenceFactory.cpp:530 (libhdbpersistence.so)

10: 0x00007f1706aa8cdf in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, TREX_ERROR::TRexError*)+0x133b at PersistenceController.cpp:383 (libhdblogger.so)

11: 0x00000000004a7a64 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0xb0 at TREXIndexServer.cpp:355 (hdbindexserver)

12: 0x00000000005250ed in TRexAPI::AssignThread::run(void*)+0x39 at TREXIndexServer.cpp:260 (hdbindexserver)

13: 0x00007f17075489ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

14: 0x00007f170754a028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

15: 0x00007f16fc19ef42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

16: 0x00007f16fc19f10d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[4527]{0}[0] 2013-01-22 01:32:54.590691 f PersistenceL PersistenceController.cpp(00385) : newdb persistence startup failed

[4527]{0}[0] 2013-01-22 01:32:54.590833 e assign TREXIndexServer.cpp(00364) : assign failed with persistence startup error. stopping service...

[4572]{0}[0] 2013-01-22 01:32:54.621219 e parallel Parallel.cpp(00163) : release; Allocator[702]:Pool/parallel(reset_time=(none),CATEGORY=Pool/parallel,DEPTH=2,INCLUSIVE_SIZE_IN_USE=7680B,INCLUSIVE_COUNT_IN_USE=20,INCLUSIVE_ALLOCATED_SIZE=7680B,INCLUSIVE_DEALLOCATED_SIZE=0B,INCLUSIVE_ALLOCATED_COUNT=20,INCLUSIVE_DEALLOCATED_COUNT=0,INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,INCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_SIZE_IN_USE=7680B,EXCLUSIVE_COUNT_IN_USE=20,EXCLUSIVE_ALLOCATED_SIZE=7680B,EXCLUSIVE_DEALLOCATED_SIZE=0B,EXCLUSIVE_ALLOCATED_COUNT=20,EXCLUSIVE_DEALLOCATED_COUNT=0,EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,EXCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_ALLOC_ERRORS=0,MALLOC_PROXY_CACHE_MISSES=0,FLAGS=(none))

[5234]{0}[0] 2013-01-22 15:30:33.470806 f PersistenceL PersistenceFactory.cpp(00557) : start up of newdb persistence failed: exception 1: no.1000002 (ltt/impl/memory.cpp:115)

Out of memory ; $size$=1073741824; $name$=LogRecovery; $type$=pool; $inuse_count$=2; $allocated_size$=544

exception throw location:

1: 0x00007f5939e1a03f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

2: 0x00007f5939e1018e in DataRecovery::RecoveryHandlerImpl::pushLogSegment(DataAccess::LogSegmentBackupInfo const&, DataRecovery::LogSegment*, ltt::smartptr_handle<Stream::ChannelCallback> const&)+0x3ca at RecoveryHandlerImpl.cpp:993 (libhdbdataaccess.so)

3: 0x00007f5939e105d7 in DataRecovery::RecoveryHandlerImpl::pushLocalLogSegment(DataRecovery::LogSegment&, DataRecovery::ReplayProgressCallback&)+0x33 at RecoveryHandlerImpl.cpp:1022 (libhdbdataaccess.so)

4: 0x00007f5939e11ea4 in DataRecovery::RecoveryHandlerImpl::RecoveryHandlerImpl(DataRecovery::LoggerImpl&, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*, unsigned long, DataAccess::OnDiskLogState&)+0x15b0 at RecoveryHandlerImpl.cpp:704 (libhdbdataaccess.so)

5: 0x00007f5939de7fae in DataRecovery::LoggerImpl::prepareRecovery(unsigned long, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*)+0x27a at LoggerImpl.cpp:638 (libhdbdataaccess.so)

6: 0x00007f5939b5d8c2 in DataAccess::PersistenceManager::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool)+0x320 at PersistenceManagerImpl.cpp:1492 (libhdbdataaccess.so)

7: 0x00007f5939b5d9a1 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool)+0x40 at PersistenceManagerImpl.cpp:2709 (libhdbdataaccess.so)

8: 0x00007f5939b5de56 in DataAccess::PersistenceManager::replay()+0xa2 at PersistenceManagerImpl.cpp:1356 (libhdbdataaccess.so)

9: 0x00007f5945f80fb8 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool)+0x64 at PersistenceFactory.cpp:530 (libhdbpersistence.so)

10: 0x00007f5945cbacdf in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, TREX_ERROR::TRexError*)+0x133b at PersistenceController.cpp:383 (libhdblogger.so)

11: 0x00000000004a7a64 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0xb0 at TREXIndexServer.cpp:355 (hdbindexserver)

12: 0x00000000005250ed in TRexAPI::AssignThread::run(void*)+0x39 at TREXIndexServer.cpp:260 (hdbindexserver)

13: 0x00007f594675a9ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

14: 0x00007f594675c028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

15: 0x00007f593b3b0f42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

16: 0x00007f593b3b110d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[5234]{0}[0] 2013-01-22 15:30:33.617058 f PersistenceL PersistenceController.cpp(00385) : newdb persistence startup failed

[5234]{0}[0] 2013-01-22 15:30:33.987463 e assign TREXIndexServer.cpp(00364) : assign failed with persistence startup error. stopping service...

[5281]{0}[0] 2013-01-22 15:30:34.714118 e parallel Parallel.cpp(00163) : release; Allocator[702]:Pool/parallel(reset_time=(none),CATEGORY=Pool/parallel,DEPTH=2,INCLUSIVE_SIZE_IN_USE=7680B,INCLUSIVE_COUNT_IN_USE=20,INCLUSIVE_ALLOCATED_SIZE=7680B,INCLUSIVE_DEALLOCATED_SIZE=0B,INCLUSIVE_ALLOCATED_COUNT=20,INCLUSIVE_DEALLOCATED_COUNT=0,INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,INCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_SIZE_IN_USE=7680B,EXCLUSIVE_COUNT_IN_USE=20,EXCLUSIVE_ALLOCATED_SIZE=7680B,EXCLUSIVE_DEALLOCATED_SIZE=0B,EXCLUSIVE_ALLOCATED_COUNT=20,EXCLUSIVE_DEALLOCATED_COUNT=0,EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,EXCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_ALLOC_ERRORS=0,MALLOC_PROXY_CACHE_MISSES=0,FLAGS=(none))

[4478]{0}[0] 2013-01-23 14:33:45.422033 e Metadata ptl_shm.cc(00520) : ShmSystem::create (size=536870912) - Cannot allocate memory

[4478]{0}[0] 2013-01-23 14:33:45.422426 e Metadata ptl_shm.cc(00661) : ShmSystem: retry attach 0/16

[4478]{0}[0] 2013-01-23 14:33:45.435732 e Metadata ptl_shm.cc(00520) : ShmSystem::create (size=536870912) - Cannot allocate memory

[4478]{0}[0] 2013-01-23 14:33:45.435810 e Metadata ptl_shm.cc(00661) : ShmSystem: retry attach 1/16

[4478]{0}[0] 2013-01-23 14:33:47.604893 e Memory PoolAllocator.cpp(00510) : Destroying allocator 'LogRecovery' with 2 blocks still allocated

[4478]{0}[0] 2013-01-23 14:33:47.604910 e Memory PoolAllocator.cpp(00515) : Block list (including suballocators):

Block 0x00007fa580000000, size 1073741824B, flags 0x80/BigBlock, head 0x00007fa580000000, type <unknown>, alloc Pool/PersistenceManager/LogRecovery, allocated at:

1: 0x00007fa778bb503f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

Block 0x00007fa668bbf700, size 256B, flags 0x1/SmallBlock, head 0x00007fa668bbf700, type DataRecovery::RecoverySegment, alloc Pool/PersistenceManager/LogRecovery, allocated at:

1: 0x00007fa778bab0b0 in DataRecovery::RecoveryHandlerImpl::pushLogSegment(DataAccess::LogSegmentBackupInfo const&, DataRecovery::LogSegment*, ltt::smartptr_handle<Stream::ChannelCallback> const&)+0x2f0 at handle_ref.hpp:430 (libhdbdataaccess.so)

[allocator contains 2 blocks, in total 1073742080B]

[4478]{0}[0] 2013-01-23 14:33:47.647428 f PersistenceL PersistenceFactory.cpp(00557) : start up of newdb persistence failed: exception 1: no.1000002 (ltt/impl/memory.cpp:115)

Out of memory ; $size$=1073741824; $name$=LogRecovery; $type$=pool; $inuse_count$=3; $allocated_size$=1073742368

exception throw location:

1: 0x00007fa778bb503f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

2: 0x00007fa778b94638 in DataRecovery::LogSegment::doSegmentRecovery()+0x84 at LogSegment.cpp:1447 (libhdbdataaccess.so)

3: 0x00007fa778b948fb in DataRecovery::LogSegment::load(ltt::refcounted_handle<Stream::Channel>, DataRecovery::LogSegment::LoadCallback&)+0xd7 at LogSegment.cpp:1427 (libhdbdataaccess.so)

4: 0x00007fa778bab649 in DataRecovery::RecoveryHandlerImpl::pushLocalLogSegment(DataRecovery::LogSegment&, DataRecovery::ReplayProgressCallback&)+0xa5 at RecoveryHandlerImpl.cpp:1030 (libhdbdataaccess.so)

5: 0x00007fa778bacea4 in DataRecovery::RecoveryHandlerImpl::RecoveryHandlerImpl(DataRecovery::LoggerImpl&, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*, unsigned long, DataAccess::OnDiskLogState&)+0x15b0 at RecoveryHandlerImpl.cpp:704 (libhdbdataaccess.so)

6: 0x00007fa778b82fae in DataRecovery::LoggerImpl::prepareRecovery(unsigned long, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*)+0x27a at LoggerImpl.cpp:638 (libhdbdataaccess.so)

7: 0x00007fa7788f88c2 in DataAccess::PersistenceManager::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool)+0x320 at PersistenceManagerImpl.cpp:1492 (libhdbdataaccess.so)

8: 0x00007fa7788f89a1 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool)+0x40 at PersistenceManagerImpl.cpp:2709 (libhdbdataaccess.so)

9: 0x00007fa7788f8e56 in DataAccess::PersistenceManager::replay()+0xa2 at PersistenceManagerImpl.cpp:1356 (libhdbdataaccess.so)

10: 0x00007fa784d1bfb8 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool)+0x64 at PersistenceFactory.cpp:530 (libhdbpersistence.so)

11: 0x00007fa784a55cdf in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, TREX_ERROR::TRexError*)+0x133b at PersistenceController.cpp:383 (libhdblogger.so)

12: 0x00000000004a7a64 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0xb0 at TREXIndexServer.cpp:355 (hdbindexserver)

13: 0x00000000005250ed in TRexAPI::AssignThread::run(void*)+0x39 at TREXIndexServer.cpp:260 (hdbindexserver)

14: 0x00007fa7854f59ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

15: 0x00007fa7854f7028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

16: 0x00007fa77a14bf42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

17: 0x00007fa77a14c10d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[4478]{0}[0] 2013-01-23 14:33:47.744132 f PersistenceL PersistenceController.cpp(00385) : newdb persistence startup failed

[4478]{0}[0] 2013-01-23 14:33:47.744336 e assign TREXIndexServer.cpp(00364) : assign failed with persistence startup error. stopping service...

[4519]{0}[0] 2013-01-23 14:33:47.758861 e parallel Parallel.cpp(00163) : release; Allocator[702]:Pool/parallel(reset_time=(none),CATEGORY=Pool/parallel,DEPTH=2,INCLUSIVE_SIZE_IN_USE=7680B,INCLUSIVE_COUNT_IN_USE=20,INCLUSIVE_ALLOCATED_SIZE=7680B,INCLUSIVE_DEALLOCATED_SIZE=0B,INCLUSIVE_ALLOCATED_COUNT=20,INCLUSIVE_DEALLOCATED_COUNT=0,INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,INCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_SIZE_IN_USE=7680B,EXCLUSIVE_COUNT_IN_USE=20,EXCLUSIVE_ALLOCATED_SIZE=7680B,EXCLUSIVE_DEALLOCATED_SIZE=0B,EXCLUSIVE_ALLOCATED_COUNT=20,EXCLUSIVE_DEALLOCATED_COUNT=0,EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,EXCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_ALLOC_ERRORS=0,MALLOC_PROXY_CACHE_MISSES=0,FLAGS=(none))

[5657]{0}[0] 2013-01-23 14:49:05.799130 f PersistenceL PersistenceFactory.cpp(00557) : start up of newdb persistence failed: exception 1: no.1000002 (ltt/impl/memory.cpp:115)

Out of memory ; $size$=1073741824; $name$=LogRecovery; $type$=pool; $inuse_count$=2; $allocated_size$=544

exception throw location:

1: 0x00007f491789803f in DataRecovery::RecoverySegment::prepareBuffer()+0x2b at RecoveryHandlerImpl.cpp:152 (libhdbdataaccess.so)

2: 0x00007f491788e18e in DataRecovery::RecoveryHandlerImpl::pushLogSegment(DataAccess::LogSegmentBackupInfo const&, DataRecovery::LogSegment*, ltt::smartptr_handle<Stream::ChannelCallback> const&)+0x3ca at RecoveryHandlerImpl.cpp:993 (libhdbdataaccess.so)

3: 0x00007f491788e5d7 in DataRecovery::RecoveryHandlerImpl::pushLocalLogSegment(DataRecovery::LogSegment&, DataRecovery::ReplayProgressCallback&)+0x33 at RecoveryHandlerImpl.cpp:1022 (libhdbdataaccess.so)

4: 0x00007f491788fea4 in DataRecovery::RecoveryHandlerImpl::RecoveryHandlerImpl(DataRecovery::LoggerImpl&, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*, unsigned long, DataAccess::OnDiskLogState&)+0x15b0 at RecoveryHandlerImpl.cpp:704 (libhdbdataaccess.so)

5: 0x00007f4917865fae in DataRecovery::LoggerImpl::prepareRecovery(unsigned long, DataAccess::LogReplayCallback&, DataAccess::LogRecoveryCallback*)+0x27a at LoggerImpl.cpp:638 (libhdbdataaccess.so)

6: 0x00007f49175db8c2 in DataAccess::PersistenceManager::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool)+0x320 at PersistenceManagerImpl.cpp:1492 (libhdbdataaccess.so)

7: 0x00007f49175db9a1 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool)+0x40 at PersistenceManagerImpl.cpp:2709 (libhdbdataaccess.so)

8: 0x00007f49175dbe56 in DataAccess::PersistenceManager::replay()+0xa2 at PersistenceManagerImpl.cpp:1356 (libhdbdataaccess.so)

9: 0x00007f49239fefb8 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool)+0x64 at PersistenceFactory.cpp:530 (libhdbpersistence.so)

10: 0x00007f4923738cdf in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, TREX_ERROR::TRexError*)+0x133b at PersistenceController.cpp:383 (libhdblogger.so)

11: 0x00000000004a7a64 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0xb0 at TREXIndexServer.cpp:355 (hdbindexserver)

12: 0x00000000005250ed in TRexAPI::AssignThread::run(void*)+0x39 at TREXIndexServer.cpp:260 (hdbindexserver)

13: 0x00007f49241d89ee in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

14: 0x00007f49241da028 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

15: 0x00007f4918e2ef42 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:446 (libhdbbasis.so)

16: 0x00007f4918e2f10d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:505 (libhdbbasis.so)

[5657]{0}[0] 2013-01-23 14:49:05.831651 f PersistenceL PersistenceController.cpp(00385) : newdb persistence startup failed

[5657]{0}[0] 2013-01-23 14:49:05.831863 e assign TREXIndexServer.cpp(00364) : assign failed with persistence startup error. stopping service...

[5677]{0}[0] 2013-01-23 14:49:05.860028 e parallel Parallel.cpp(00163) : release; Allocator[702]:Pool/parallel(reset_time=(none),CATEGORY=Pool/parallel,DEPTH=2,INCLUSIVE_SIZE_IN_USE=7680B,INCLUSIVE_COUNT_IN_USE=20,INCLUSIVE_ALLOCATED_SIZE=7680B,INCLUSIVE_DEALLOCATED_SIZE=0B,INCLUSIVE_ALLOCATED_COUNT=20,INCLUSIVE_DEALLOCATED_COUNT=0,INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,INCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_SIZE_IN_USE=7680B,EXCLUSIVE_COUNT_IN_USE=20,EXCLUSIVE_ALLOCATED_SIZE=7680B,EXCLUSIVE_DEALLOCATED_SIZE=0B,EXCLUSIVE_ALLOCATED_COUNT=20,EXCLUSIVE_DEALLOCATED_COUNT=0,EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE=384B,EXCLUSIVE_PEAK_ALLOCATION_SIZE=7680B,EXCLUSIVE_ALLOC_ERRORS=0,MALLOC_PROXY_CACHE_MISSES=0,FLAGS=(none))hanadb:/usr/sap/HDB/HDB00/hanadb/trace>

also, I couldn't see indexserver from HDB info..... I got connection refuse from my HANA server.thanks a lot in advance.

Hi Juergen Schmerder,

I am also facing the same issue on AWS HANA revision 70.

Below is the trace file (indexserver_imdbhdb.30003.001.trc).

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.462328 i Service_Startup IndexManager_rebuild.cc(00416) : Number of indexes: 734

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.462614 i Service_Startup IndexManager_rebuild.cc(00518) : Number of JobEx indexes: 637

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.577509 i Service_Startup IndexManager_rebuild.cc(00598) : Number of large indexes: 0

Number of small indexes: 87

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.583894 i Service_Startup IndexManager_rebuild.cc(00638) : large index rebuilding finished

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.594588 i Service_Startup ptime_master_start.cc(00745) : Rebuilding indexes done.

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.594601 i Service_Startup ptime_master_start.cc(00747) : Index rebuilding time: 0.13 seconds

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862520 e TrexNet BufferedIO.cpp(04284) : active channel 73 from 45313 to 127.0.0.1:30007: protocol mismatch - trying to read type Int but EOF reached

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862555 e TrexNetIO BufferedIO.cpp(01395) : dump of TrexNet buffer: size 21, current position 0x00007f3974c05729 (21 bytes), mode incoming, channel 73, local port 45313, peer 127.0.0.1:30007, content:

BLOB START (Addr: 0x00007f3974c05714 Len: 21)0x00007f3974c05714 18 45 ff c1 81 c1 cb 00 cb 00 cb 00 .E..........

0x00007f3974c05720 cb 00 cb 00 c6 81 81 c5 17 .........

BLOB END

2269[thr=27622]: JobWrk0004 at

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863748 e TrexNet BufferedIO.cpp(04284) : active channel 3 from 46356 to 127.0.0.1:30005: protocol mismatch - trying to read type Int but EOF reached

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863760 e TrexNetIO BufferedIO.cpp(01395) : dump of TrexNet buffer: size 21, current position 0x00007f3974c034d1 (21 bytes), mode incoming, channel 3, local port 46356, peer 127.0.0.1:30005, content:

BLOB START (Addr: 0x00007f3974c034bc Len: 21)0x00007f3974c034bc 18 45 ff c1 .E..

0x00007f3974c034c0 81 c1 cb 00 cb 00 cb 00 cb 00 cb 00 c6 81 81 c5 ................

0x00007f3974c034d0 17 .

BLOB END

2265[thr=27620]: JobWrk0002 at

1: 0x00007f399f34af30 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x300 at ContextFunctions.cpp:622 (libhdbbasis.so)

2: 0x00007f39aac2f2d2 in TrexNet::BufferedNetInput::checkType(unsigned char)+0x5e0 at BufferedIO.cpp:4286 (libhdbbasement.so)

3: 0x00007f39aac30b40 in TrexNet::BufferedNetInput::operator>>(int&)+0x20 at BufferedIO.cpp:3557 (libhdbbasement.so)

4: 0x00007f39b0810c22 in Executor::X2::runCommThread(Executor::X2::SendTaskInfo&, Execution::TraceContextScope*, Executor::X2Job*)+0x730 at X2.cpp:1965 (libhdbexecutor.so)

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862712 e TrexNet BufferedIO.cpp(04286) : 1: 0x00007f399f34af30 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x300 at ContextFunctions.cpp:622 (libhdbbasis.so)

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.872098 i Service_Startup ptime_master_start.cc(01199) : Session service is ready.

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863819 e TrexNet BufferedIO.cpp(04286) : 5: 0x00007f39b08202c9 in Executor::X2Job::run(Execution::JobObject&)+0xe5 at X2.cpp:4382 (libhdbexecutor.so)

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862712 e TrexNet BufferedIO.cpp(04286) : 2: 0x00007f39aac2f2d2 in TrexNet::BufferedNetInput::checkType(unsigned char)+0x5e0 at BufferedIO.cpp:4286 (libhdbbasement.so)

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863819 e TrexNet BufferedIO.cpp(04286) : 6: 0x00007f399f35c202 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x410 at JobExecutorImpl.cpp:436 (libhdbbasis.so)

7: 0x00007f399f36e765 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2b1 at JobExecutorThreads.cpp:299 (libhdbbasis.so)

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.875935 i Service_Startup translog.cc(01697) : Activating private log buffering mode

[27607]{-1}[-1/-1] 2014-03-21 18:50:14.875967 i assign TREXIndexServer.cpp(00725) : persistence started with volume 3

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863819 e TrexNet BufferedIO.cpp(04286) : 8: 0x00007f399f372797 in Execution::JobWorker::run(void*&)+0x173 at JobExecutorThreads.cpp:462 (libhdbbasis.so)

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862712 e TrexNet BufferedIO.cpp(04286) : 3: 0x00007f39aac30b40 in TrexNet::BufferedNetInput::operator>>(int&)+0x20 at BufferedIO.cpp:3557 (libhdbbasement.so)

[27620]{-1}[11/-1] 2014-03-21 18:50:14.863819 e TrexNet BufferedIO.cpp(04286) : 9: 0x00007f399f38ddcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

10: 0x00007f399f38df6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

[27620]{-1}[11/-1] 2014-03-21 18:50:14.878430 e Executor X2.cpp(01987) : X2::runCommThread()() caught TrexNet exception "a protocol mismatch occured" during serialize returns, plan plan1@imdbhdb:30003, server imdbhdb:30005

[27622]{-1}[11/-1] 2014-03-21 18:50:14.862712 e TrexNet BufferedIO.cpp(04286) : 4: 0x00007f39b0810c22 in Executor::X2::runCommThread(Executor::X2::SendTaskInfo&, Execution::TraceContextScope*, Executor::X2Job*)+0x730 at X2.cpp:1965 (libhdbexecutor.so)

5: 0x00007f39b08202c9 in Executor::X2Job::run(Execution::JobObject&)+0xe5 at X2.cpp:4382 (libhdbexecutor.so)

6: 0x00007f399f35c202 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x410 at JobExecutorImpl.cpp:436 (libhdbbasis.so)

7: 0x00007f399f36e765 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2b1 at JobExecutorThreads.cpp:299 (libhdbbasis.so)

8: 0x00007f399f372797 in Execution::JobWorker::run(void*&)+0x173 at JobExecutorThreads.cpp:462 (libhdbbasis.so)

9: 0x00007f399f38ddcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

10: 0x00007f399f38df6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

[27622]{-1}[11/-1] 2014-03-21 18:50:14.885981 e Executor X2.cpp(01987) : X2::runCommThread()() caught TrexNet exception "a protocol mismatch occured" during serialize returns, plan plan1@imdbhdb:30003, server imdbhdb:30007

[27648]{-1}[11/-1] 2014-03-21 18:50:14.886065 e Executor PlanExecutor.cpp(00674) : plan plan1@imdbhdb:30003 failed with rc 2613; executor: communication problem

[27648]{-1}[11/-1] 2014-03-21 18:50:14.886093 e Executor PlanExecutor.cpp(00674) : -- returns for plan1@imdbhdb:30003:

[27648]{-1}[11/-1] 2014-03-21 18:50:14.886099 e Executor PlanExecutor.cpp(00674) : pop=<all>, duration=18:50:14.860 to 18:50:14.886 +0.026

[27648]{-1}[11/-1] 2014-03-21 18:50:14.886116 e Executor PlanExecutor.cpp(00674) : con=<all>, duration=+0.000

[27648]{-1}[11/-1] 2014-03-21 18:50:14.886119 e Executor PlanExecutor.cpp(00674) : -- end executor returns

[27607]{-1}[-1/-1] 2014-03-21 18:50:15.668284 i Service_Startup catalog.cc(08523) : Setting up statistics server security done

[27607]{-1}[14/1637748] 2014-03-21 18:50:15.734902 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27607]{-1}[14/1637748] 2014-03-21 18:50:15.735030 i HistoryManager HistoryManager.cpp(00460) : Maximum garbage collection parallelity set to 4

[27664]{-1}[-1/-1] 2014-03-21 18:50:21.494431 i PersistenceManag PersistenceManagerImpl.cpp(04822) : DTX slave 4 reports rollback at restart completion; pre-restart RTT still awaits restart of volumes [2]

[27665]{-1}[-1/-1] 2014-03-21 18:50:21.518525 i PersistenceManag PersistenceManagerImpl.cpp(04822) : DTX slave 2 reports rollback at restart completion; all known DTX slaves reported rollback at restart completion, pre-restart RTT empty now

[27607]{-1}[-1/-1] 2014-03-21 18:50:21.557226 i assign TREXIndexServer.cpp(01001) : assign to volume 3 finished

[27664]{300048}[27/-1] 2014-03-21 18:51:27.665093 i TraceContext TraceContext.cpp(00699) : UserName=DAPUSER, ApplicationUserName=RajKumar, ApplicationSource=csns.modeler.datapreview.providers.ResultSetDelegationDataProvider.<init>(ResultSetDelegationDataProvider.java:109);csns.modeler.actions.DataPreviewDelegationAction.getDataProvider(DataPreviewDelegationAction.java:256);csns.modeler.actions.DataPreviewDelegationAction.run(DataPreviewDelegationAction.java:230);csns.modeler.actions.DataPreviewDelegationAction.run(DataPreviewDelegationAction.java:126);csns.modeler.command.handlers.DataPreviewHandler.execute(DataPreviewHandler.java:52);org.eclipse.core.commands

[27664]{300048}[27/-1] 2014-03-21 18:51:27.665081 e Basis Crash.cpp(00728) : Crash at Authorization/impl/ProcedureInfoProvider.cpp:178

Reason:

exception 1: no.1000000 (Authorization/impl/ProcedureInfoProvider.cpp:178)

Assertion failed: ti

exception throw location:

1: 0x00007f39920d9bb8 in Authorization::ProcedureInfoProvider::getProcedureInfo(ptime::Transaction&, ltt::basic_string<char, ltt::char_traits<char> > const&, ltt::basic_string<char, ltt::char_traits<char> > const&) const+0x724 at ProcedureInfoProvider.cpp:178 (libhdbrskernel.so)

2: 0x00007f39920d5df7 in Authorization::ProcedureFilter::getOperands(ptime::Transaction&) const+0x83 at ProcedureFilter.cpp:96 (libhdbrskernel.so)

3: 0x00007f39920d310a in Authorization::Filter::getOperandsAsStrings(ptime::Transaction&) const+0x36 at Filter.cpp:36 (libhdbrskernel.so)

4: 0x00007f3992121f70 in AnalyticalAuthorization::XmlAnalyticalPrivilegeFacade::exportAnalyticalPrivilege(ptime::Transaction&, Authorization::StructuredPrivilege const&, bool) const+0xa80 at smart_ptr.hpp:255 (libhdbrskernel.so)

5: 0x00007f39921231db in AnalyticalAuthorization::XmlAnalyticalPrivilegeFacade::findAndRetrieveAnalyticalPrivileges(ptime::Transaction&, Authorization::User&, Authorization::ObjectId const&, ltt::basic_string<char, ltt::char_traits<char> > const&) const+0x497 at XmlAnalyticalPrivilegeFacade.cpp:1367 (libhdbrskernel.so)

6: 0x00007f399211718a in AnalyticalAuthorization::XmlAnalyticalPrivilegeFacade::findAndRetrieveAnalyticalPrivileges(ptime::Transaction&, Authorization::User&, ltt::basic_string<char, ltt::char_traits<char> > const&, ltt::basic_string<char, ltt::char_traits<char> > const&) const+0xa6 at XmlAnalyticalPrivilegeFacade.cpp:1400 (libhdbrskernel.so)

7: 0x00007f39ac99cb5c in TRexAPI::TRexApiSearch::analyticalPrivilegesCheck(TRexConfig::TableOrView, bool&, bool&)+0x1038 at TRexApiSearch.cpp:19693 (libhdbcsapi.so)

8: 0x00007f39ac9d77a9 in TRexAPI::TRexApiSearch::doSearch(TRexAPI::TRexApiSearchTableResult&)+0x34d5 at TRexApiSearch.cpp:6024 (libhdbcsapi.so)

9: 0x00007f39ac9da365 in TRexAPI::TRexApiSearch::search(TRexAPI::TRexApiSearchTableResult&)+0x81 at TRexApiSearch.cpp:5351 (libhdbcsapi.so)

10: 0x00007f39ac9f5b19 in TRexAPI::TRexApiSearch::searchNet(TRexAPI::TRexApiSearchTableResult&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&)+0x985 at TRexApiSearch.cpp:11089 (libhdbcsapi.so)

11: 0x00007f39ac9d492f in TRexAPI::TRexApiSearch::doSearch(TRexAPI::TRexApiSearchTableResult&)+0x65b at TRexApiSearch.cpp:5398 (libhdbcsapi.so)

12: 0x00007f39ac9da365 in TRexAPI::TRexApiSearch::search(TRexAPI::TRexApiSearchTableResult&)+0x81 at TRexApiSearch.cpp:5351 (libhdbcsapi.so)

13: 0x00007f398f72cc43 in ptime::TrexMDFactory::estimateColumnViewSize(ptime::CSViewInfo*, ltt_adp::vector<ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<ltt::pair<ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, ltt_adp::vector<ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, ltt::integral_constant<bool, true> > >, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<ltt::pair<ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, ltt_adp::vector<ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, ltt::integral_constant<bool, true> > >, ltt::integral_constant<bool, true> > const&)+0x3c0 at trex_metadata.cc:1318 (libhdbcswrapper.so)

14: 0x00007f39914fe02b in ptime::qo_size_estimation::getColumnViewRowCount(ptime::qo_Context&, ptime::ClassPtr<ptime::ViewInfo>, ptime::qo_Table*)+0x3d7 at qo_size_estimation.cc:2105 (libhdbrskernel.so)

15: 0x00007f39914feafe in ptime::qo_size_estimation::fetch_all_histogram(ptime::qo_Context&, ptime::NextList<ptime::qo_Histogram>*, ptime::ClassPtr<ptime::ViewInfo>, ptime::qo_Table*)+0x8a at qo_size_estimation.cc:2346 (libhdbrskernel.so)

16: 0x00007f399151062e in ptime::qo_size_estimation::table_estimation(ptime::qo_Context&, ptime::qo_Table*)+0x61a at qo_size_estimation.cc:1079 (libhdbrskernel.so)

17: 0x00007f399151211b in ptime::qo_size_estimation::head_estimation(ptime::qo_Context&, ptime::qo_Rel*)+0x277 at qo_size_estimation.cc:193 (libhdbrskernel.so)

18: 0x00007f39915127c1 in ptime::qo_size_estimation::rel_estimation(ptime::qo_Context&, ptime::qo_Rel*)+0x280 at qo_size_estimation.cc:165 (libhdbrskernel.so)

19: 0x00007f39914e5c0e in ptime::qo_PhysicalEnumerator::physical_plan_enum(ptime::qo_Context&, ptime::qo_Rel*)+0x174a at qo_physical_enumerator.cc:458 (libhdbrskernel.so)

20: 0x00007f39913c94e3 in ptime::QueryOptimizer::optimize_plan(ptime::Transaction&, char const*, ptime::qo_Rel*, ptime::ptl::stp_heap*, ptime::ptl::stp_heap*, ptime::qo_Context&, ptime::ClassPtr<ptime::TableInfo>, bool, ptime::qo_TrexParameters*, bool, ptime::TrexKey*, bool, bool, void*, bool, ltt_adp::map<ptime::Query::Plan::OptTraceVal, double, ltt::less<ptime::Query::Plan::OptTraceVal>, ltt::integral_constant<bool, true> >&)+0x1260 at query_optimizer.cc:581 (libhdbrskernel.so)

21: 0x00007f39913cb90d in ptime::QueryOptimizer::optimize_plan(ptime::Transaction&, char const*, ptime::qc_relation const*, ptime::ptl::stp_heap*, ptime::ptl::stp_heap*, ptime::qo_Context&, ptime::qc_hint*, ptime::ClassPtr<ptime::TableInfo>, bool, ptime::qc_trex_parameters*, bool, ptime::TrexKey*, bool, bool, void*, bool, ltt_adp::map<ptime::Query::Plan::OptTraceVal, double, ltt::less<ptime::Query::Plan::OptTraceVal>, ltt::integral_constant<bool, true> >&)+0x319 at query_optimizer.cc:344 (libhdbrskernel.so)

22: 0x00007f3991973b19 in ptime::Query::_compile(ptime::Transaction&, ptime::Query::Plan*, char const*, ptime::qp_dml*, char const*, ptime::qc_params*, ptime::qc_stmt*&, char const*, ptime::ptl::stp_heap*, ptime::ptl::stp_heap*, ptime::qo_Context&, ptime::Query::PlanVizQueryFlag&, ptime::IdTable*, void*, bool, bool, bool)+0x3a5 at query.cc:3938 (libhdbrskernel.so)

[27664]{300048}[27/-1] 2014-03-21 18:51:27.934510 e Basis FaultProtectionImpl.cpp(01247) : SIGNAL 6 (SIGABRT) caught, sender PID: 27589, PID: 27589, thread: 37791[thr=27664]: SqlExecutor, value int: 282, ptr: 0x000000000000011a, time: 2014-03-21 18:51:27 000 Local

Instance HDB/00, OS Linux imdbhdb 3.0.42-0.7-ec2 #1 SMP Tue Oct 9 11:58:45 UTC 2012 (a8dc443) x86_64

----> Register Dump <----

rax: 0x0000000000000000 rbx: 0x00007f37c6828c50

rcx: 0xffffffffffffffff rdx: 0x0000000000000006

rsi: 0x0000000000006c10 rdi: 0x0000000000006bc5

rsp: 0x00007f37c6828798 rbp: 0x00007f37c6828c00

r08: 0x00007f39722a6fb0 r09: 0x00007f39927e78ca

r10: 0x00007f37c6828540 r11: 0x0000000000000206

r12: 0x00007f39927e78ca r13: 0x00000000000000b2

r14: 0x0000000000000000 r15: 0x00007f37c6828bc0

rip: 0x00007f398c3c065b eflg: 0x0000000000000206

cs: 0x0000 gs: 0x0000

fs: 0x0000

omsk: 0x0000000000000000 cr2: 0x0000000000000000

trap: 0x0000000000000000 err: 0x0000000000000000

----> Floating-point register dump <----

foper: 0xffff frip: 0x0000000000000000 fdata: 0x0

control: 0xf4fa status: 0x8004 tag: 0xffff

mxcsr: 0x0000037f mxcr_mask: 0x0000

fp[0]: 0x0.0000 0000 0000 0000 * 2^0x0

fp[1]: 0x0.1fa4 0000 ffff 0000 * 2^0x34

fp[2]: 0x0.ffff 0000 0000 0000 * 2^0xc

fp[3]: 0x0.ffff 0000 0000 0000 * 2^0x0

fp[4]: 0x0.ffff 0000 0000 0000 * 2^0x0

fp[5]: 0x0.ffff 0000 0000 0000 * 2^0x0

fp[6]: 0x0.0000 0000 0000 0000 * 2^0x0

fp[7]: 0x0.0000 0000 0000 0000 * 2^0x0

xmm[00]: 0x00000000.00000000.00000001.00000000

xmm[01]: 0x0000ffff.00000000.00000000.00000000

xmm[02]: 0x00000000.00000000.00000000.ffff0000

xmm[03]: 0x0000ff00.00000000.354e5a5f.6d697470

xmm[04]: 0x75513565.38797265.315f3053.616c5036

xmm[05]: 0x7a69566e.72657551.00000000.3ff00000

xmm[06]: 0x00000000.00000000.000003a9.000001bb

xmm[07]: 0x00000282.0000039c.00000065.0000038c

xmm[08]: 0x000003f1.00000346.0000037d.00000576

xmm[09]: 0x0000006c.00000018.000003fa.0000037b

xmm[10]: 0x00000280.0000057a.000003ed.00000322

xmm[11]: 0x0000006b.0000037a.000001dd.00000579

xmm[12]: 0x00000578.0000039b.00000284.000003a9

xmm[13]: 0x00000281.000001bb.0000005a.000005da

xmm[14]: 0x00000282.000001ea.000001ba.000002dc

xmm[15]: 0x00000490.000002af.00000071.00000578

NOTE: full crash dump will be written to /usr/sap/HDB/HDB00/imdbhdb/trace/indexserver_imdbhdb.30003.crashdump.20140321-185127.027589.trc

Call stack suppressed, since assertion stack trace already written

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796527 i Basis TraceStream.cpp(00395) : ==== Starting hdbindexserver, version 1.00.70.00.386119 (NewDB100_REL), build linuxx86_64 not set 2013-11-29 15:58:05 ld7270.wdf.sap.corp gcc (SAP release 20130125, based on SUSE gcc43-4.3.4_20091019-0.22.17.5096.1.PTF.799300) 4.3.4 [gcc-4_3-branch revision 152973]

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796681 i Basis TraceStream.cpp(00400) : MaxOpenFiles: 1048576

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796694 i Memory MallocProxy.cpp(01228) : Installed malloc hooks

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796697 i Basis ProcessorInfo.cpp(00657) : Using GDT segment limit to determine current CPU ID

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796698 i Basis Timer.cpp(00584) : Using RDTSC for HR timer

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796700 i Memory AllocatorImpl.cpp(00834) : Allocators activated

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796701 i Memory AllocatorImpl.cpp(00850) : Using big block segment size 134217728

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796703 w Environment Environment.cpp(00284) : Changing environment set IMSLERRPATH=/usr/sap/HDB/HDB00/exe//

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796705 w Environment Environment.cpp(00284) : Changing environment set IMSLSERRPATH=/usr/sap/HDB/HDB00/exe//

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796706 w Environment Environment.cpp(00284) : Changing environment set NODNSSAPTRANSHOST=1

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.796707 w Environment Environment.cpp(00263) : Changing environment set SAP_DIR_GLOBAL=/usr/sap/HDB/SYS/global

[27712]{-1}[-1/-1] 2014-03-21 18:51:31.812422 w Environment Environment.cpp(00284) : Changing environment set SSL_WITH_OPENSSL=0

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.639691 i assign TREXIndexServer.cpp(00618) : assign to volume 3 started

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.640239 i FileIO FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HDB/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.642542 i FileIO FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HDB/SYS/global/hdb/backint/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.645007 i FileIO FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HDB/SYS/global/hdb/log/mnt00001/hdb00003/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.647269 i FileIO FileStatistics.cpp(00287) : FileFactoryConfiguration::initial(path="/usr/sap/HDB/SYS/global/hdb/data/mnt00001/hdb00003/", AsyncWriteSubmitActive=auto,AsyncWriteSubmitBlocks=new,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.649798 i HistoryManager HistoryManager.cpp(00460) : Maximum garbage collection parallelity set to 4

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.649868 i PhysicalPageAcce DataVolumeImpl.cpp(00543) : Open volume file(s) in "/usr/sap/HDB/SYS/global/hdb/data/mnt00001/hdb00003/" (readOnly= 0, diagMode= 0)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.649884 i PhysicalPageAcce DataVolumeImpl.cpp(00556) : --confMaxFileSize= 0MB, fsysMaxFilesSize= 2048000MB --> 2048000MB

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.650015 i PhysicalPageAcce DataVolumeImpl.cpp(00601) : --opened volume file "datavolume_0000.dat"

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674655 i PersistenceManag PersistenceManagerImpl.cpp(03997) : Restart page version 1 loaded (96bytes)

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674688 i PersistenceManag PersistenceManagerImpl.cpp(04022) : Initial maximum known TID after restart: 1637748

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674691 i Logger PersistenceManagerImpl.cpp(04030) : Newest known master commit position: 0x8a1d2cc

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674695 i Logger PersistenceManagerImpl.cpp(04040) : Known last prepare commit position on volume 1: 0x135348

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674697 i Logger PersistenceManagerImpl.cpp(04040) : Known last prepare commit position on volume 2: 0x18de742

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.674698 i Logger PersistenceManagerImpl.cpp(04040) : Known last prepare commit position on volume 4: 0x10eb02

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.676213 i PersistenceManag PersistenceManagerImpl.cpp(04094) : Known DTX volume set [1,2,4], 6144 RTT entries

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.722823 i PersistenceManag PersistenceSessionRegistry.cpp(00255) : Start loading open sessions and history cleanup files

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.781903 i Logger PersistenceSessionRegistry.cpp(00812) : Open session count at restart: 0/0, max known TID at restart: 1637748

[27730]{-1}[-1/-1] 2014-03-21 18:51:32.781920 i PersistenceManag PersistenceSessionRegistry.cpp(00262) : Loading 0 open session(s) and 0 history cleanup file(s) finished in 0.059082 seconds;

[27730]{-1}[-1/-1] 2014-03-21 18:51:33.136620 i Service_Startup ContMgr.cc(00076) : Initializing system catalog.

[27730]{-1}[-1/-1] 2014-03-21 18:51:33.518871 i Service_Startup ContMgr.cc(00198) : Initializing system catalog done.

[27746]{-1}[-1/-1] 2014-03-21 18:51:36.503715 i Service_Startup CheckpointMgr.cc(00669) : 3 RS segments loaded from Persistency in 2.7300000004470348 sec (70.329670318153916MB/s)

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.505478 i Service_Startup MemSystem.cc(00150) : total memory size (MB) = 35014

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.505653 i Logger PersistenceManagerImpl.cpp(02268) : Preparing log recovery handler, start position 0x8a1d302

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.505866 i Logger LoggerImpl.cpp(00444) : Setting log segment backup timeout to 900s, original value was 0s

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.565574 w PersistenceManag PersistenceManagerImpl.cpp(03492) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.565603 i Logger PersistenceManagerImpl.cpp(03696) : Deactivating REDO logging for log replay

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.565606 i Logger PersistenceManagerImpl.cpp(03707) : Start termination of rollback(s) open in restart/backup savepoint

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.565890 i Logger PersistenceManagerImpl.cpp(03715) : Termination of rollback(s) open in restart/backup savepoint finished in 8e-06 seconds;

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.565929 i Logger PersistenceManagerImpl.cpp(03720) : Start garbage collection of history files

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.566032 i Logger PersistenceManagerImpl.cpp(03731) : Garbage collection of history files finished: 0 cleanup files in 9.5e-05 seconds;

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.566040 i Logger PersistenceManagerImpl.cpp(03747) : Expecting savepoint 3700 log record as first log record at 0x8a1d302

[27730]{-1}[-1/-1] 2014-03-21 18:51:36.566176 i Logger RecoveryHandlerImpl.cpp(00936) : RecoveryQueueCount used: 4

[27737]{-1}[-1/-1] 2014-03-21 18:51:54.227972 i Logger RecoveryHandlerImpl.cpp(00374) : Stopping parsing of open segment LogSegment[0:0x8a1c9c0/Writing][GUID=F3E7D3FB-2233-20140321-185012-3C70000001/PrevGUID=F3E7D3FB-2233-20140321-184648-0CB0000001/PersGUID=24C912E8-1261-20140114-150036-2780000001/RestoreGUID=00000000-0000-19720306-084436-2E00000000] at file position 1302528, unused rest 1072439296B, termination reason: Buffer at position 0x13e000 doesn't belong to segment 0x8a1c9c0

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.233804 i Logger PersistenceManagerImpl.cpp(03815) : Starting log replay at position 0x8a1d302

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.233814 i Logger RecoveryHandlerImpl.cpp(01472) : Triggering recovery of remaining log

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.233844 i Logger RecoveryHandlerImpl.cpp(01520) : Finishing log recovery, waiting for in-process and in-load segments

[27757]{-1}[-1/-1] 2014-03-21 18:51:54.233986 i Logger PersistenceManagerImpl.cpp(03017) : Found savepoint 3700 log record

[27757]{-1}[-1/-1] 2014-03-21 18:51:54.247394 i Logger RecoveryHandlerImpl.cpp(01866) : Log recovery cannot expect any more data at position 0x8a21940, load states:

- LogPartition[/usr/sap/HDB/SYS/global/hdb/log/mnt00001/hdb00003/:0]: #segments 0/1 processed, 1/1 loaded, 1 total, last loaded LogSegment[0:0x8a1c9c0/Writing][GUID=F3E7D3FB-2233-20140321-185012-3C70000001/PrevGUID=F3E7D3FB-2233-20140321-184648-0CB0000001/PersGUID=24C912E8-1261-20140114-150036-2780000001/RestoreGUID=00000000-0000-19720306-084436-2E00000000]

[27757]{-1}[-1/-1] 2014-03-21 18:51:54.247437 i Logger RecoveryHandlerImpl.cpp(02166) : Recovery finished at log position 0x8a21940

[27753]{-1}[-1/-1] 2014-03-21 18:51:54.323848 i Logger RecoveryHandlerImpl.cpp(03342) : Signaling finish wait barrier

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.324126 i Logger LogSegment.cpp(00593) : Closing open log segments after log recovery completed at position 0x8a21940

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.324151 i Logger LogSegment.cpp(00676) : Closing open on-disk segment LogSegment[0/2:0x8a1c9c0/GUID=F3E7D3FB-2233-20140321-185012-3C70000001/PrevGUID=F3E7D3FB-2233-20140321-184648-0CB0000001,TS=2014-03-21 18:50:12.967755/Writing/0x0]@0x00007f9ef08f8540 after log recovery completed at position 0x8a21940

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.326529 i Logger LoggerImpl.cpp(00933) : Replayed 1150848B (1MB) of log in 17.7605 seconds; 0.0617964MB/s; max known TID=1637998

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.326658 i Logger LoggerImpl.cpp(00723) : Starting logger with 8 log buffers per partition, 1024KB each, segment size 1024MB, log mode normal

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.599645 i Logger RecoveryHandlerImpl.cpp(01767) : Termination of 0 indoubt transactions was executed in 0.001357 seconds;

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.599836 i Logger PersistenceManagerImpl.cpp(03923) : Activating REDO logging after log replay

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.610691 i LogReplay RowStoreTransactionCallback.cc(00220) : starting master-slave DTX consistency check

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.611467 i LogReplay RowStoreTransactionCallback.cc(00295) : finished master-slave DTX consistency check

[27745]{-1}[-1/-1] 2014-03-21 18:51:54.625661 i PersistenceManag VirtualFileStatsProxy.cpp(00191) : Initialize VirtualFile LOB statistics...

[27743]{-1}[-1/-1] 2014-03-21 18:51:54.625661 i PersistenceManag VirtualFileStatsProxy.cpp(00129) : Initialize VirtualFile statistics...

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.626101 i ContainerDirecto ContainerNameDirectoryImpl.cpp(01113) : Load ContainerNameDirectory...

[27743]{-1}[-1/-1] 2014-03-21 18:51:54.638692 i PersistenceManag VirtualFileStatsProxy.cpp(00170) : VirtualFile statistics initialized: 2780/2780 CD entries in 13.01 msec

[27730]{-1}[-1/-1] 2014-03-21 18:51:54.644153 i ContainerDirecto ContainerNameDirectoryImpl.cpp(01172) : ContainerNameDirectory loaded 2780/2780 in 18.03 msec

[27745]{-1}[-1/-1] 2014-03-21 18:51:54.657993 i PersistenceManag VirtualFileStatsProxy.cpp(00231) : VirtualFile LOB statistics initialized: 15417 CD entries in 32.31 msec

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.263569 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.263595 i PersistenceManag PersistenceManagerImpl.cpp(02359) : Activating periodic savepoint, frequency 300

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.271815 i Service_Startup transmgmt.cc(05550) : Starting up global transaction coordinator

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.901477 i Service_Startup mm_recovery.cc(01426) : Rebuilding page lists.

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.961092 i Service_Startup mm_recovery.cc(00975) : RS: metadata & data are separated

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.961567 i Service_Startup ptime_master_start.cc(00723) : Rebuilding system indexes.

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.999146 i Service_Startup ptime_master_start.cc(00737) : Rebuilding system indexes done.

[27730]{-1}[-1/-1] 2014-03-21 18:51:55.999156 i Service_Startup ptime_master_start.cc(00739) : Rebuilding indexes.

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.000476 i Service_Startup IndexManager_rebuild.cc(00416) : Number of indexes: 734

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.000703 i Service_Startup IndexManager_rebuild.cc(00518) : Number of JobEx indexes: 637

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.115999 i Service_Startup IndexManager_rebuild.cc(00598) : Number of large indexes: 0

Number of small indexes: 87

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.124234 i Service_Startup IndexManager_rebuild.cc(00638) : large index rebuilding finished

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.135460 i Service_Startup ptime_master_start.cc(00745) : Rebuilding indexes done.

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.135478 i Service_Startup ptime_master_start.cc(00747) : Index rebuilding time: 0.14 seconds

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.408268 i Service_Startup ptime_master_start.cc(01199) : Session service is ready.

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411051 e TrexNet BufferedIO.cpp(04284) : active channel 73 from 46419 to 127.0.0.1:30005: protocol mismatch - trying to read type Int but EOF reached

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411087 e TrexNetIO BufferedIO.cpp(01395) : dump of TrexNet buffer: size 21, current position 0x00007f9f2dc7a4d1 (21 bytes), mode incoming, channel 73, local port 46419, peer 127.0.0.1:30005, content:

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411087 e TrexNet BufferedIO.cpp(04284) : active channel 81 from 45376 to 127.0.0.1:30007: protocol mismatch - trying to read type Int but EOF reached

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411099 e TrexNetIO BufferedIO.cpp(01395) : dump of TrexNet buffer: size 21, current position 0x00007f9f2dc7c729 (21 bytes), mode incoming, channel 81, local port 45376, peer 127.0.0.1:30007, content:

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411087 e TrexNetIO BufferedIO.cpp(01395) : BLOB START (Addr: 0x00007f9f2dc7a4bc Len: 21)

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411099 e TrexNetIO BufferedIO.cpp(01395) : BLOB START (Addr: 0x00007f9f2dc7c714 Len: 21)0x00007f9f2dc7a4bc 18 45 ff c1 .E..

0x00007f9f2dc7a4c0 81 c1 cb 00 cb 00 cb 00 cb 00 cb 00 c6 81 81 c5 ................

0x00007f9f2dc7a4d0 17 .

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411087 e TrexNetIO BufferedIO.cpp(01395) : BLOB END0x00007f9f2dc7c714 18 45 ff c1 81 c1 cb 00 cb 00 cb 00 .E..........

0x00007f9f2dc7c720 cb 00 cb 00 c6 81 81 c5 17 .........

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411099 e TrexNetIO BufferedIO.cpp(01395) : BLOB END

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411222 e TrexNet BufferedIO.cpp(04286) : 2263[thr=27742]: JobWrk0002 at

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411299 e TrexNet BufferedIO.cpp(04286) : 2267[thr=27744]: JobWrk0004 at

1: 0x00007fa0d88ddf30 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x300 at ContextFunctions.cpp:622 (libhdbbasis.so)

2: 0x00007fa0e41c22d2 in TrexNet::BufferedNetInput::checkType(unsigned char)+0x5e0 at BufferedIO.cpp:4286 (libhdbbasement.so)

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.414856 i Service_Startup translog.cc(01697) : Activating private log buffering mode

[27730]{-1}[-1/-1] 2014-03-21 18:51:56.414888 i assign TREXIndexServer.cpp(00725) : persistence started with volume 3

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411299 e TrexNet BufferedIO.cpp(04286) : 3: 0x00007fa0e41c3b40 in TrexNet::BufferedNetInput::operator>>(int&)+0x20 at BufferedIO.cpp:3557 (libhdbbasement.so)

4: 0x00007fa0e9da3c22 in Executor::X2::runCommThread(Executor::X2::SendTaskInfo&, Execution::TraceContextScope*, Executor::X2Job*)+0x730 at X2.cpp:1965 (libhdbexecutor.so)

5: 0x00007fa0e9db32c9 in Executor::X2Job::run(Execution::JobObject&)+0xe5 at X2.cpp:4382 (libhdbexecutor.so)

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411222 e TrexNet BufferedIO.cpp(04286) : 1: 0x00007fa0d88ddf30 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x300 at ContextFunctions.cpp:622 (libhdbbasis.so)

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411299 e TrexNet BufferedIO.cpp(04286) : 6: 0x00007fa0d88ef202 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x410 at JobExecutorImpl.cpp:436 (libhdbbasis.so)

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411222 e TrexNet BufferedIO.cpp(04286) : 2: 0x00007fa0e41c22d2 in TrexNet::BufferedNetInput::checkType(unsigned char)+0x5e0 at BufferedIO.cpp:4286 (libhdbbasement.so)

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411299 e TrexNet BufferedIO.cpp(04286) : 7: 0x00007fa0d8901765 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2b1 at JobExecutorThreads.cpp:299 (libhdbbasis.so)

8: 0x00007fa0d8905797 in Execution::JobWorker::run(void*&)+0x173 at JobExecutorThreads.cpp:462 (libhdbbasis.so)

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411222 e TrexNet BufferedIO.cpp(04286) : 3: 0x00007fa0e41c3b40 in TrexNet::BufferedNetInput::operator>>(int&)+0x20 at BufferedIO.cpp:3557 (libhdbbasement.so)

[27744]{-1}[12/-1] 2014-03-21 18:51:56.411299 e TrexNet BufferedIO.cpp(04286) : 9: 0x00007fa0d8920dcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

10: 0x00007fa0d8920f6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

[27744]{-1}[12/-1] 2014-03-21 18:51:56.425511 e Executor X2.cpp(01987) : X2::runCommThread()() caught TrexNet exception "a protocol mismatch occured" during serialize returns, plan plan1@imdbhdb:30003, server imdbhdb:30007

[27742]{-1}[12/-1] 2014-03-21 18:51:56.411222 e TrexNet BufferedIO.cpp(04286) : 4: 0x00007fa0e9da3c22 in Executor::X2::runCommThread(Executor::X2::SendTaskInfo&, Execution::TraceContextScope*, Executor::X2Job*)+0x730 at X2.cpp:1965 (libhdbexecutor.so)

5: 0x00007fa0e9db32c9 in Executor::X2Job::run(Execution::JobObject&)+0xe5 at X2.cpp:4382 (libhdbexecutor.so)

6: 0x00007fa0d88ef202 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x410 at JobExecutorImpl.cpp:436 (libhdbbasis.so)

7: 0x00007fa0d8901765 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x2b1 at JobExecutorThreads.cpp:299 (libhdbbasis.so)

8: 0x00007fa0d8905797 in Execution::JobWorker::run(void*&)+0x173 at JobExecutorThreads.cpp:462 (libhdbbasis.so)

9: 0x00007fa0d8920dcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

10: 0x00007fa0d8920f6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

[27742]{-1}[12/-1] 2014-03-21 18:51:56.433662 e Executor X2.cpp(01987) : X2::runCommThread()() caught TrexNet exception "a protocol mismatch occured" during serialize returns, plan plan1@imdbhdb:30003, server imdbhdb:30005

[27770]{-1}[12/-1] 2014-03-21 18:51:56.433778 e Executor PlanExecutor.cpp(00674) : plan plan1@imdbhdb:30003 failed with rc 2613; executor: communication problem

[27770]{-1}[12/-1] 2014-03-21 18:51:56.433805 e Executor PlanExecutor.cpp(00674) : -- returns for plan1@imdbhdb:30003:

[27770]{-1}[12/-1] 2014-03-21 18:51:56.433811 e Executor PlanExecutor.cpp(00674) : pop=<all>, duration=18:51:56.409 to 18:51:56.433 +0.024

[27770]{-1}[12/-1] 2014-03-21 18:51:56.433829 e Executor PlanExecutor.cpp(00674) : con=<all>, duration=+0.000

[27770]{-1}[12/-1] 2014-03-21 18:51:56.433831 e Executor PlanExecutor.cpp(00674) : -- end executor returns

[27730]{-1}[-1/-1] 2014-03-21 18:51:57.231405 i Service_Startup catalog.cc(08523) : Setting up statistics server security done

[27730]{-1}[14/1640059] 2014-03-21 18:51:57.271496 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27730]{-1}[14/1640059] 2014-03-21 18:51:57.271652 i HistoryManager HistoryManager.cpp(00460) : Maximum garbage collection parallelity set to 4

[27786]{-1}[-1/-1] 2014-03-21 18:52:03.310097 i PersistenceManag PersistenceManagerImpl.cpp(04822) : DTX slave 4 reports rollback at restart completion; pre-restart RTT still awaits restart of volumes [2]

[27785]{-1}[-1/-1] 2014-03-21 18:52:03.332131 i PersistenceManag PersistenceManagerImpl.cpp(04822) : DTX slave 2 reports rollback at restart completion; all known DTX slaves reported rollback at restart completion, pre-restart RTT empty now

[27730]{-1}[-1/-1] 2014-03-21 18:52:03.367513 i assign TREXIndexServer.cpp(01001) : assign to volume 3 finished

[27758]{-1}[-1/-1] 2014-03-21 18:56:57.896688 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:01:57.939235 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:06:57.979347 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:11:58.019074 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:16:58.081004 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:21:58.121557 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:26:58.143377 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:31:58.164238 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:36:58.205855 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:41:58.245833 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:46:58.292558 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:51:58.334941 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 19:56:58.379278 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:01:58.421888 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:06:58.449051 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:11:58.473643 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:16:58.518787 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:21:58.566908 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:26:58.613231 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:31:58.652849 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:36:58.695735 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:41:58.737743 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:46:58.778964 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:51:58.799849 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 20:56:58.825272 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:01:58.877992 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:06:58.922642 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:11:58.963498 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:16:59.009218 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:21:59.049075 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:26:59.090163 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:31:59.132360 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:36:59.155403 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:41:59.177086 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:46:59.221033 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:51:59.261900 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 21:56:59.302002 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:01:59.344349 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:06:59.387264 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:11:59.411916 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:16:59.456652 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:21:59.496709 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:26:59.544231 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:31:59.593326 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:36:59.633745 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:41:59.673108 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:46:59.717942 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:51:59.740390 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 22:56:59.761804 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 23:01:59.802867 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 23:06:59.847900 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27758]{-1}[-1/-1] 2014-03-21 23:11:59.887512 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27712]{-1}[-1/-1] 2014-03-21 23:16:51.901362 i Service_Shutdown TrexService.cpp(05683) : Preparing for shutting service down

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.903368 i assign TREXIndexServer.cpp(02602) : unassign from volume 3

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.903406 i Service_Shutdown TREXIndexServer.cpp(02605) : Preparing to shutdown

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.903832 e TNS TNSClient.cpp(00733) : nameserver imdbhdb:30001 not initialized. retry in 5 sec...

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.903897 e TNS TNSClient.cpp(00749) : process is terminating -> aborting nameserver retries

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.903970 i Service_Shutdown TREXIndexServer.cpp(02633) : Stopping LobGarbageCollectorThread

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.904670 i Service_Shutdown TREXIndexServer.cpp(02649) : Stopping EPM

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.906260 i Service_Shutdown TREXIndexServer.cpp(02653) : Stopping PlanningEngine

[27780]{-1}[-1/-1] 2014-03-21 23:16:51.906855 i Service_Shutdown TREXIndexServer.cpp(02658) : Stopping SQL session service

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.032527 i Service_Shutdown TREXIndexServer.cpp(02693) : Stopping TRexApiSystem

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.033116 i Service_Shutdown TREXIndexServer.cpp(02697) : Stopping executor

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.033205 i Service_Shutdown TREXIndexServer.cpp(02701) : Stopping CalculationEngineManager

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.033284 i Service_Shutdown TREXIndexServer.cpp(02705) : Stopping CacheMgr

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.565959 i Service_Shutdown TREXIndexServer.cpp(02712) : Stopping GraphEngineMgr

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.565985 i Service_Shutdown TREXIndexServer.cpp(02716) : Stopping Invalidator

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.566027 i Service_Shutdown TREXIndexServer.cpp(02720) : Stopping RClient

[27780]{-1}[-1/-1] 2014-03-21 23:16:52.566054 i Service_Shutdown TREXIndexServer.cpp(02724) : Stopping DebuggingBackend

[27780]{-1}[-1/-1] 2014-03-21 23:16:53.168432 i Service_Shutdown TREXIndexServer.cpp(02736) : Stopping udivmgr

[27780]{-1}[-1/-1] 2014-03-21 23:16:53.168465 i Service_Shutdown TREXIndexServer.cpp(02742) : Closing trace files

[27780]{-1}[-1/-1] 2014-03-21 23:16:53.168471 i Service_Shutdown TREXIndexServer.cpp(02746) : Stopping persistenceLayer

[27780]{-1}[-1/-1] 2014-03-21 23:16:58.355238 w Logger SavepointImpl.cpp(02343) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[27780]{-1}[-1/-1] 2014-03-21 23:16:58.355260 i Logger BackupHandlerImpl.cpp(00321) : Shutting down log backup, 0 log backup(s) pending

[27780]{-1}[-1/-1] 2014-03-21 23:16:58.356986 w ResMan ResourceContainer.cpp(01300) : Information about shrink at 21.03.2014 18:51:31 000 Fri:

Reason for shrink: PersistenceManager::shutdown 0

ShrinkCaller

1: 0x00007fa0d8a486e7 in ResourceManager::reportShrink(ResourceManager::ShrinkInformation&, char const*, unsigned long)+0x103 at ResourceContainer.cpp:596 (libhdbbasis.so)

2: 0x00007fa0d8a4c55a in ResourceManager::ResourceContainerImpl::shrink(unsigned long, unsigned long, char const*, unsigned long, unsigned long&)+0x216 at ResourceContainer.cpp:3403 (libhdbbasis.so)

3: 0x00007fa0d8a4dba1 in ResourceManager::ResourceContainer::shrink(unsigned long, char const*, unsigned long)+0x20 at ResourceContainer.cpp:4308 (libhdbbasis.so)

4: 0x00007fa0d5fc8f98 in DataAccess::PersistenceManager::shutdownImpl(bool)+0x1f4 at PersistenceManagerImpl.cpp:2479 (libhdbdataaccess.so)

5: 0x00007fa0e395abdf in PersistenceLayer::PersistenceSystem::finalize()+0xab at PersistenceManager.hpp:868 (libhdbpersistence.so)

6: 0x00007fa0e38cfcf9 in PersistenceLayer::Persistence::removeSystem(PersistenceLayer::PersistenceSystem*)+0x35 at Persistence.cpp:335 (libhdbpersistence.so)

7: 0x00007fa0e394cf70 in PersistenceLayer::PersistenceFactory::shutdownPersistence(bool)+0x60 at PersistenceFactory.cpp:465 (libhdbpersistence.so)

8: 0x00007fa0e367f930 in PersistenceController::shutdown()+0x30 at PersistenceController.cpp:653 (libhdblogger.so)

9: 0x00000000004b3a65 in TRexAPI::TREXIndexServer::prepareToShutdown()+0xa91 at TREXIndexServer.cpp:2770 (hdbindexserver)

10: 0x0000000000566c9b in PrepareToShutdownThread::run(void*)+0x27 at TrexService.cpp:5399 (hdbindexserver)

11: 0x00007fa0e42130cf in TrexThreads::PoolThread::run()+0x9fb at PoolThread.cpp:279 (libhdbbasement.so)

12: 0x00007fa0e4214c38 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:125 (libhdbbasement.so)

13: 0x00007fa0d8920dcb in Execution::Thread::staticMainImp(void**)+0x627 at Thread.cpp:475 (libhdbbasis.so)

14: 0x00007fa0d8920f6d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:543 (libhdbbasis.so)

IPMM short info:

GLOBAL_ALLOCATION_LIMIT (GAL) = 32909886669b (30.64gb), SHARED_MEMORY = 1152420992b (1.07gb), CODE_SIZE = 6754078720b (6.29gb)

PID=4335 (hdbnameserver), PAL=33044104396, AB=1724227584, UA=0, U=1274942907, FSL=134217728

PID=27712 (hdbindexserver), PAL=33044104396, AB=2672967680, UA=0, U=1677982839, FSL=0

Total allocated memory= 12303694976b (11.45gb)

Total used memory = 10859425458b (10.11gb)

Sum AB = 4397195264

Sum Used = 2952925746

Heap memory fragmentation: 11

Top allocators (ordered descending by inclusive_size_in_use).

1: / 1671209951b (1.55gb)

2: Pool 1338288664b (1.24gb)

3: Pool/RowEngine 801038608b (763.92mb)

4: Pool/RowEngine/LockTable 536871680b (512mb)

5: Pool/malloc 245083848b (233.72mb)

6: AllocateOnlyAllocator-unlimited 231466088b (220.74mb)

7: Pool/RowEngine/Internal 218672640b (208.54mb)

8: StackAllocator 99729408b (95.10mb)

9: Pool/malloc/libhdbexpression.so 90502768b (86.30mb)

10: Pool/malloc/libhdbbasement.so 90485480b (86.29mb)

11: AllocateOnlyAllocator-unlimited/ReserveForUndoAndCleanupExec 84029440b (80.13mb)

12: AllocateOnlyAllocator-unlimited/ReserveForOnlineCleanup 84029440b (80.13mb)

13: Pool/PersistenceManager 64471272b (61.48mb)

14: Pool/L 52438608b (50mb)

15: Pool/L/llang 52437904b (50mb)

16: Pool/L/llang/Context 52436480b (50mb)

17: Pool/malloc/libhdbrskernel.so 44179264b (42.13mb)

18: Pool/MergeMonitor 38400384b (36.62mb)

19: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 37748736b (36mb)

20: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1> 37748736b (36mb)

Top allocators (ordered descending by exclusive_size_in_use).

1: Pool/RowEngine/LockTable 536871680b (512mb)

2: Pool/RowEngine/Internal 218672640b (208.54mb)

3: StackAllocator 99729408b (95.10mb)

4: Pool/malloc/libhdbexpression.so 90502768b (86.30mb)

5: Pool/malloc/libhdbbasement.so 90485480b (86.29mb)

6: AllocateOnlyAllocator-unlimited/ReserveForUndoAndCleanupExec 84029440b (80.13mb)

7: AllocateOnlyAllocator-unlimited/ReserveForOnlineCleanup 84029440b (80.13mb)

8: Pool/malloc/libhdbrskernel.so 44179264b (42.13mb)

9: Pool/MergeMonitor 38400384b (36.62mb)

10: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 37748736b (36mb)

11: Pool/ResourceContainer 33742312b (32.17mb)

12: Pool/FileAllocator/FileFactory[/usr/sap/HDB/SYS/global/hdb/log/mnt00001/hdb00003/] 31461392b (30mb)

13: Pool/TransactionManager 25172800b (24mb)

14: AllocateOnlyAllocator-unlimited/ReserveForOnlineCleanupSmall 16789504b (16.01mb)

15: Pool/PersistenceManager/Savepoint/Page 16781312b (16mb)

16: Pool/StatementSnapshotManager 16744640b (15.96mb)

17: Pool/Statistics 15280304b (14.57mb)

18: Pool/RowEngine/RSTempPage 12533760b (11.95mb)

19: Pool/PersistenceManager/PersistentSpace(0)/RowStoreLPA 10486112b (10mb)

20: Pool/PersistenceManager/LogBuffer 9439232b (9mb)

[27780]{-1}[-1/-1] 2014-03-21 23:16:58.385271 w ResMan ResourceContainer.cpp(01300) : Information about shrink at 21.03.2014 18:51:31 000 Fri: