cancel
Showing results for 
Search instead for 
Did you mean: 

After Upgrade kernel not compatible with system (rc=-100)

former_member196664
Participant
0 Kudos


Hi All,

I upgraded CRM system from 6.0 to 7.02. System is very slow and dispatcher is often getting stopped in the night time. I have posted disp trace. Please help ASAP. Kernel 7.21_EXT patch 400. SQL 2008/Windows 2008. Solman MOPZ calculated the kernel target. I also checked in PAM, that kernel release is fine for the target. "stoc check failed kernel not compatible with system (rc=-100)". I have also attached dev_ms.

---------------------------------------------------


trc file: "dev_disp", trc level: 1, release: "721"


---------------------------------------------------


sysno      00

sid        CRD


systemid   562 (PC with Windows NT)


relno      7210

patchlevel 0


patchno    400


intno      20020600


make multithreaded, Unicode, 64 bit, optimized


profile    F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


pid        3664


kernel runs with dp version 139000(ext=121000) (@(#) DPLIB-INT-VERSION-139000-UC)

length of sys_adm_ext is 592 bytes


*** SWITCH TRC-HIDE on ***


***LOG Q00=> DpSapEnvInit, DPStart (00 3664) [dpxxdisp.c   1323]


shared lib "dw_xml.dll" version 400 successfully loaded

shared lib "dw_xtc.dll" version 400 successfully loaded

shared lib "dw_stl.dll" version 400 successfully loaded

shared lib "dw_gui.dll" version 400 successfully loaded

shared lib "dw_mdm.dll" version 400 successfully loaded

shared lib "dw_rndrt.dll" version 400 successfully loaded

shared lib "dw_abp.dll" version 400 successfully loaded

shared lib "dw_sym.dll" version 400 successfully loaded

shared lib "dw_aci.dll" version 400 successfully loaded

rdisp/softcancel_sequence :  -> 0,5,-1


use internal message server connection to port 3900


rdisp/dynamic_wp_check : 1


rdisp/calculateLoadAverage : 1


Tue Jan 13 08:39:57 2015


*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds


***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c  6525]

MtxInit: 30000 0 0


DpSysAdmExtInit: ABAP is active

DpSysAdmExtInit: VMC (JAVA VM in WP) is active


display_tcp_ip: 0


Set Check Logoff Flags to 0x0


DpIPCInit2: write dp-profile-values into sys_adm_ext


DpIPCInit2: start server >SAPCRMDEV_CRD_00                        <


DpShMCreate: sizeof(wp_adm) 49632 (2256)


DpShMCreate: sizeof(tm_adm) 5517056 (27448)


DpShMCreate: sizeof(wp_ca_adm) 64000 (64)


DpShMCreate: sizeof(appc_ca_adm) 128000 (64)


DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1392064/1392080


DpShMCreate: sizeof(comm_adm) 1392080 (2768)


DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=512/48/163904/96560/260512


DpShMCreate: sizeof(slock_adm) 260512 (296)


DpFileTableSize: max/headSize/ftSize/tableSize=4400/16/422464/422480


DpShMCreate: sizeof(file_adm) 422480 (80)


DpShMCreate: sizeof(vmc_adm) 47520 (2160)


DpShMCreate: sizeof(wall_adm) (41664/42896/64/192)

DpShMCreate: sizeof(gw_adm) 48


DpShMCreate: sizeof(j2ee_adm) 3952


DpShMCreate: SHM_DP_ADM_KEY (addr: 00000000076C0050, size: 7984112)

DpShMCreate: allocated sys_adm at 00000000076C0060

DpShMCreate: allocated wp_adm_list at 00000000076C30B0

DpShMCreate: allocated wp_adm at 00000000076C32A0


DpShMCreate: allocated tm_adm_list at 00000000076CF490


DpShMCreate: allocated tm_adm at 00000000076CF4E0


DpShMCreate: allocated wp_ca_adm at 0000000007C123F0


DpShMCreate: allocated appc_ca_adm at 0000000007C21E00


DpShMCreate: allocated comm_adm at 0000000007C41210


DpShMCreate: allocated slock_adm at 0000000007D94FF0


DpShMCreate: allocated file_adm at 0000000007DD49A0


DpShMCreate: allocated vmc_adm_list at 0000000007E3BC00


DpShMCreate: allocated vmc_adm at 0000000007E3BCB0


DpShMCreate: allocated gw_adm at 0000000007E47660


DpShMCreate: allocated j2ee_adm at 0000000007E476A0


DpShMCreate: allocated ca_info at 0000000007E48620


DpShMCreate: allocated wall_adm at 0000000007E486B0


DpCommAttachTable: attached comm table (header=0000000007C41210/ft=0000000007C41220)


DpSysAdmIntInit: initialize sys_adm


rdisp/test_roll : roll strategy is DP_NORMAL_ROLL


dia token check not active (9 token)


MBUF state OFF


DpCommInitTable: init table for 500 entries


DpFileInitTable: init table for 4400 entries


DpSesCreateTable: created session table at 00000000034B0050 (len=167472)


DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm


rdisp/queue_size_check_value :  -> on,50,30,40,500,50,500,80


EmInit: MmSetImplementation( 2 ).


MM global diagnostic options set: 0


<ES> client 0 initializing ....


<ES> EsILock: use Mutex for locking


<ES> InitFreeList


ES> block size is 4096 kByte.


Using implementation view


<EsNT> Using memory model view.


<EsNT> Memory Reset disabled as NT default


<ES> 2865 blocks reserved for free list.


ES initialized.


mm.dump: set maximum dump mem to 192 MB


mm.dump: set global maximum dump mem to 192 MB


EsRegisterEmCheck: Register EmGetEsHandles at 00000001402B4270


Es2ResCreate: Changed resource parameters to


filenamePrefix =


blockSizeKB    = 2048 KB


initialSizeMB  = 480 MB


extendSizeMB   = 240 MB


maxSizeMB      = 15840 MB


DpVmcSetActive: set vmc state DP_VMC_ENABLED


DpVmcSetActive: set vmc state DP_VMC_ACTIVE


DpVmcInit2: o.k.

MPI: dynamic quotas disabled.


MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%


*** WARNING => DpJ2eeStart: profile parameter 'system/type' not correctly set to start J2EE; use 'J2EE' for Java only and 'DS'  for double stack instance (info=0x301) [dpxxj2ee.c   343]


J2EE server info


  start = TRUE


  state = STARTED


  pid = 4888


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=65000


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework


DpJ2eeStart: j2ee state = STARTED


rdisp/http_min_wait_dia_wp : 1 -> 1


**LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c   1709]


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]


MBUF state LOADING


DpStartStopMsg: send start message (myname is >SAPCRMDEV_CRD_00                        <)


DpStartStopMsg: start msg sent



Tue Jan 13 08:39:58 2015


CCMS: Initialized monitoring segment of size 60000000.


CCMS: Initialized CCMS Headers in the shared monitoring segment.


CCMS: Checking Downtime Configuration of Monitoring Segment.


CCMS: AlMsUpload called by wp 1024.



Tue Jan 13 08:39:59 2015


CCMS: AlMsUpload successful for F:\usr\sap\CRD\DVEBMGS00\log\ALMTTREE.DAT (2844 MTEs).



Tue Jan 13 08:40:05 2015


CCMS: start to initalize 3.X shared alert area (first segment).


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


DpWpBlksLow: max wp blocks in queue is 800 (80 %)


MBUF component UP


DpMsgProcess: 1 server in MBUF

DpAppcBlksLow: max appc blocks in queue is 1000 (50 %)



Tue Jan 13 08:40:37 2015


*** ERROR => [DpProcDied] Process died  (PID:4888  HANDLE:952  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:40:57 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 3752


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64998


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED


Tue Jan 13 08:40:59 2015


DpModState: change server state from STARTING to ACTIVE


Tue Jan 13 08:41:17 2015


*** ERROR => [DpProcDied] Process died  (PID:3752  HANDLE:1308  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN


Tue Jan 13 08:41:37 2015


J2EE server info

  start = TRUE


  state = STARTED


  pid = 2496


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64997


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework


DpJ2eeStart: j2ee state = STARTED


Tue Jan 13 08:41:57 2015


*** ERROR => [DpProcDied] Process died  (PID:2496  HANDLE:1320  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN


Tue Jan 13 08:42:17 2015


J2EE server info

  start = TRUE


  state = STARTED


  pid = 2120


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64996


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED


Tue Jan 13 08:42:37 2015


*** ERROR => [DpProcDied] Process died  (PID:2120  HANDLE:1328  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:42:58 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 6024


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64995


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED


Tue Jan 13 08:43:18 2015


*** ERROR => [DpProcDied] Process died  (PID:6024  HANDLE:656  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN


Tue Jan 13 08:43:38 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 5728


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64994


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED



Tue Jan 13 08:43:58 2015


*** ERROR => [DpProcDied] Process died  (PID:5728  HANDLE:484  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:44:07 2015


*** ERROR => DpMsgProcess: MsReceive () -> MSEPARTNERUNKNOWN, partner: -                    [dpxxdisp.c   2601]



Tue Jan 13 08:44:18 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 6524


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64992


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED



Tue Jan 13 08:44:38 2015


*** ERROR => [DpProcDied] Process died  (PID:6524  HANDLE:944  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:44:58 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 6692


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64991


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED



Tue Jan 13 08:45:18 2015


*** ERROR => [DpProcDied] Process died  (PID:6692  HANDLE:484  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:45:21 2015


*** ERROR => DpMsgProcess: MsReceive () -> MSEPARTNERUNKNOWN, partner: -                    [dpxxdisp.c   2601]



Tue Jan 13 08:45:38 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 6856


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64990


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED



Tue Jan 13 08:45:58 2015


*** ERROR => [DpProcDied] Process died  (PID:6856  HANDLE:656  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:46:18 2015


J2EE server info


  start = TRUE


  state = STARTED


  pid = 7152


  argv[0] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[1] = F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.EXE


  argv[2] = pf=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[3] = -DSAPSTART=1


  argv[4] = -DCONNECT_PORT=64989


  argv[5] = -DSAPSYSTEM=00


  argv[6] = -DSAPSYSTEMNAME=CRD


  argv[7] = -DSAPMYNAME=SAPCRMDEV_CRD_00


  argv[8] = -DSAPPROFILE=F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV


  argv[9] = -DFRFC_FALLBACK=ON


  argv[10] = -DFRFC_FALLBACK_HOST=localhost


  start_lazy = 0


  start_control = SAP J2EE startup framework



DpJ2eeStart: j2ee state = STARTED



Tue Jan 13 08:46:38 2015


*** ERROR => [DpProcDied] Process died  (PID:7152  HANDLE:1308  rc:0x0) [dpnttool2.c  147]


DpJ2eeEmergencyShutdown: j2ee state = SHUTDOWN



Tue Jan 13 08:46:58 2015


DpEnvCheckJ2ee: switch off j2ee start flag



Tue Jan 13 09:35:56 2015


*** ERROR => 13.01.2015 14:35:56.034 PID=3664, TID=4196 SapTimer error. SapTime differs from Windows time. ERRORS: 0, SapTime-WindowsUtcTime=-85 seconds. TotalTimeCorrection=+0 sec, QpcTime-WindowsUtcTime=-85 sec (*). [dpxxdisp.c   21913]


13.01.2015 14:35:56.034 PID=3664, TID=4196 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime:  -86300 millisec (max +0). Slept 1000 millisec, Qpc elapsed: +999 millisec, WindowsUtc elapsed +87299 millisec. SapTime-WindowsUtcTime=-85 seconds. TotalTimeCorrection=+0 millisec, QpcTime-WindowsUtcTime=-85736 millisec (*).



Tue Jan 13 09:40:24 2015


13.01.2015 14:40:24.480 PID=3664, TID=4196 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime:  -87997 millisec (max -86300). Slept 1000 millisec, Qpc elapsed: +1000 millisec, WindowsUtc elapsed +88997 millisec. SapTime-WindowsUtcTime=-88 seconds. TotalTimeCorrection=+85786 millisec, QpcTime-WindowsUtcTime=-173903 millisec (*).



Tue Jan 13 09:51:17 2015


SoftCancel request for T30 U812 M0 received from IC_MAN



Tue Jan 13 09:56:48 2015


SoftCancel request for T37 U978 M0 received from IC_MAN



Tue Jan 13 09:57:59 2015


SoftCancel request for T37 U994 M0 received from IC_MAN



Tue Jan 13 10:19:25 2015


DpHdlDeadWp: W0 (pid=3736) terminated automatically



Tue Jan 13 10:45:23 2015


SoftCancel request for T33 U1231 M0 received from IC_MAN



Tue Jan 13 10:45:47 2015


DpHdlDeadWp: W0 (pid=6728) terminated automatically



Tue Jan 13 10:47:29 2015


*** WARNING => DpEnvCheck: no answer from msg server since 33 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Tue Jan 13 10:51:18 2015


*** ERROR => 13.01.2015 15:51:18.702 PID=3664, TID=4196 SapTimer warning. Took time keeping ownership. PID=3664, old owner PID:2268 [dpxxdisp.c   21913]



Tue Jan 13 10:52:29 2015


*** WARNING => DpEnvCheck: no answer from msg server since 33 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Tue Jan 13 10:52:50 2015


*** WARNING => DpEnvCheck: no answer from msg server since 54 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Tue Jan 13 10:53:11 2015


*** WARNING => DpEnvCheck: no answer from msg server since 75 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Tue Jan 13 10:53:32 2015


*** WARNING => DpEnvCheck: no answer from msg server since 96 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Tue Jan 13 11:10:33 2015


DpHdlDeadWp: W6 (pid=4084) terminated automatically


DpHdlDeadWp: W7 (pid=3220) terminated automatically



Tue Jan 13 11:49:36 2015


SoftCancel request for T33 U1879 M0 received from IC_MAN



Tue Jan 13 11:50:34 2015


DpHdlDeadWp: W2 (pid=4288) terminated automatically



Tue Jan 13 11:52:32 2015


*** ERROR => V3 already deleted [dpxxdisp.c   8106]


***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  33    wp_ca_blk   90      wp_id 0


-IN-- action    SEND_TO_WP        uid  2141   appc_ca_blk -1      type  DIA


-IN-- new_stat  NO_CHANGE         mode 0     len         1108    rq_id 20595


-IN-- req_info  LOGIN,MSG_WITH_REQ_BUF,MSG_WITH_OH


-IN-- vmcHdl 3 sesTab -1  vmcControlsCheckedCount 0


-IN-- msg_name  SAPCRMDEV_CRD_00



Tue Jan 13 12:03:55 2015


DpHdlDeadWp: W3 (pid=4356) terminated automatically



Tue Jan 13 15:50:00 2015


SoftCancel request for T31 U2842 M0 received from IC_MAN



Tue Jan 13 15:58:03 2015


SoftCancel request for T27 U4211 M0 received from IC_MAN


SoftCancel request for T29 U4212 M0 received from IC_MAN



Tue Jan 13 17:39:04 2015


SoftCancel request for T37 U5251 M0 received from IC_MAN


SoftCancel request for T23 U5252 M0 received from IC_MAN



Tue Jan 13 17:41:53 2015


SoftCancel request for T31 U5306 M0 received from IC_MAN


SoftCancel request for T22 U5307 M0 received from IC_MAN



Tue Jan 13 20:47:22 2015


DpHdlDeadWp: W7 (pid=6344) terminated automatically



Tue Jan 13 20:58:21 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:65029: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 25/sock 968


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:65029) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Tue Jan 13 20:58:21 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     8744


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Tue Jan 13 20:58:26 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 20:58:38 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 20:59:03 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]



Tue Jan 13 20:59:11 2015


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Tue Jan 13 21:00:43 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49645: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 984


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49645) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Tue Jan 13 21:00:43 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     8761


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Tue Jan 13 21:00:48 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 21:00:54 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 21:01:09 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]



Tue Jan 13 21:01:12 2015


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]



Tue Jan 13 21:01:26 2015


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


***LOG Q0I=> NiIRead: P=127.0.0.1:49717; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 27/sock 968


    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:49717; L=127.0.0.1:3200) [nixxi.cpp    5082]


DpMsgProcess: 1 server in MBUF



Tue Jan 13 21:32:09 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49716: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1340


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49716) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Tue Jan 13 21:32:09 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     9088


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Tue Jan 13 21:32:14 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]


*** ERROR => MsISnd2: not_attached [msxxi.c      1510]


***LOG Q0N=> DpRqNoWpHandle, MsSndAdmin () [dpxxdisp.c   5590]



Tue Jan 13 21:32:17 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]


MBUF state LOADING


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF



Tue Jan 13 22:41:28 2015


***LOG Q0I=> NiIRead: P=127.0.0.1:52332; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 67/sock 944


    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:52332; L=127.0.0.1:3200) [nixxi.cpp    5082]



Tue Jan 13 22:41:42 2015


SoftCancel request for T42 U8108 M0 received from IC_MAN



Tue Jan 13 22:41:59 2015


SoftCancel request for T32 U8113 M0 received from IC_MAN


SoftCancel request for T22 U8114 M0 received from IC_MAN



Tue Jan 13 22:42:09 2015


SoftCancel request for T26 U8115 M0 received from IC_MAN



Tue Jan 13 22:43:05 2015


DpHdlDeadWp: W0 (pid=5292) terminated automatically


DpHdlDeadWp: W3 (pid=4252) terminated automatically


DpHdlDeadWp: W4 (pid=1528) terminated automatically



Tue Jan 13 22:54:19 2015


SoftCancel request for T22 U8171 M0 received from IC_MAN



Tue Jan 13 22:55:05 2015


DpHdlDeadWp: W6 (pid=6644) terminated automatically



Tue Jan 13 23:10:45 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50690: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 1000


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50690) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Tue Jan 13 23:10:45 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     10145


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Tue Jan 13 23:10:50 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 23:10:56 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 23:11:20 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Tue Jan 13 23:11:35 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]



Tue Jan 13 23:11:42 2015


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Tue Jan 13 23:59:59 2015


SoftCancel request for T31 U8648 M0 received from IC_MAN



Wed Jan 14 00:00:00 2015


SoftCancel request for T28 U8649 M0 received from IC_MAN



Wed Jan 14 00:00:04 2015


DpRqCheck: mode 0 in status CANCEL


SoftCancel request for T31 U8648 M0 received from IC_MAN


*** ERROR => DpRqCheck: T31 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  31    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  8648   appc_ca_blk -1      type  DIA


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 12995


-IN-- prio      HIGH


-IN-- req_info  DP_SOFT_CANCEL


-IN-- msg_name  -               



Wed Jan 14 00:00:31 2015


***LOG Q0I=> NiIRead: P=192.168.101.90:2900; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 32/sock 988


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:2900; L=192.168.101.63:3200) [nixxi.cpp    5082]


DpRTmPrepareReq: network error of client T25, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3



Wed Jan 14 00:00:40 2015


*** WARNING => DpAddrToHost: NiAddrToHost took 8422 ms [dpxxdisp.c   20855]


*** WARNING => DpAddrToHost: Disabling reverse name lookup [dpxxdisp.c   20856]


DpRTmPrepareReq: client address of T25 is 192.168.101.90(TERMSRVCRESTRON)


***LOG Q04=> DpRTmPrep, NiBufReceive (5728 SRASTOGI 25 TermSrvCrest) [dpxxdisp.c   12573]


RM-T25, U5728, 100     SRASTOGI, TermSrvCrestron, 23:59:55, M0, W5, SE16, 2/2


***LOG Q0I=> NiIRead: P=192.168.101.90:4082; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 90/sock 944


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4082; L=192.168.101.63:3200) [nixxi.cpp    5082]


***LOG Q0I=> NiIRead: P=192.168.101.90:4068; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 66/sock 1308


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4068; L=192.168.101.63:3200) [nixxi.cpp    5082]


***LOG Q0I=> NiIRead: P=192.168.101.90:4061; L=192.168.101.63:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 30/sock 984


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.90:4061; L=192.168.101.63:3200) [nixxi.cpp    5082]


DpRTmPrepareReq: network error of client T34, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3


DpRTmPrepareReq: client address of T34 is 192.168.101.90(192.168.101.90)


***LOG Q04=> DpRTmPrep, NiBufReceive (8644 34 TermSrvCrest) [dpxxdisp.c   12573]


RM-T34, U8644, 000             , TermSrvCrestron, 23:59:53, M0, W8, SESS, 2/0


DpRTmPrepareReq: network error of client T23, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3


DpRTmPrepareReq: client address of T23 is 192.168.101.90(192.168.101.90)


***LOG Q04=> DpRTmPrep, NiBufReceive (8646 23 TermSrvCrest) [dpxxdisp.c   12573]


RM-T23, U8646, 000             , TermSrvCrestron, 23:59:54, M0, W4, SESS, 2/0


DpRTmPrepareReq: network error of client T42, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3


DpRTmPrepareReq: client address of T42 is 192.168.101.90(192.168.101.90)


***LOG Q04=> DpRTmPrep, NiBufReceive (8647 42 TermSrvCrest) [dpxxdisp.c   12573]


RM-T42, U8647, 000             , TermSrvCrestron, 23:59:56, M0, W0, SESS, 2/0



Wed Jan 14 00:00:41 2015


DpHdlDeadWp: W1 (pid=4608) terminated automatically



Wed Jan 14 00:49:19 2015


*** WARNING => DpITimeCheck: suspicious forward jump of current time 1421214559, last time 1421211655 [dpxxdisp.c   9205]



Wed Jan 14 00:49:20 2015


DpITimeCheck: new time seems to be OK now, I accept 1421214560


*** WARNING => DpTimeCheck: huge diff between current 1421211676 and next schedule (act_time 1421214560 delay 120 [dpxxdisp.c   9379]


DpRqCheck: mode 0 in status CANCEL


*** WARNING => DpHdlSoftCancel: T28/M0 still active in W1 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c   18360]



Wed Jan 14 00:49:22 2015


SoftCancel request for T26 U8655 M0 received from REMOTE_TERMINAL



Wed Jan 14 01:08:41 2015


DpHdlDeadWp: W7 (pid=5988) terminated automatically



Wed Jan 14 01:15:12 2015


SoftCancel request for T42 U8862 M0 received from IC_MAN


SoftCancel request for T25 U8859 M0 received from IC_MAN



Wed Jan 14 01:15:30 2015


DpRqCheck: mode 0 in status CANCEL


SoftCancel request for T42 U8862 M0 received from IC_MAN


*** ERROR => DpRqCheck: T42 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  42    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  8862   appc_ca_blk -1      type  DIA


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 21027


-IN-- prio      HIGH


-IN-- req_info  DP_SOFT_CANCEL


-IN-- msg_name  -               


***LOG Q0I=> NiIRead: P=127.0.0.1:55047; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 86/sock 1304


    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:55047; L=127.0.0.1:3200) [nixxi.cpp    5082]



Wed Jan 14 01:15:35 2015


DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)



Wed Jan 14 01:15:40 2015


DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)



Wed Jan 14 01:15:45 2015


DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)



Wed Jan 14 01:15:50 2015


DpHdlSoftCancelTimeout: previous kill in wp 0 not yet handled, reschedule softcancel (level 2)



Wed Jan 14 01:16:00 2015


DpRqCheck: mode 0 in status CANCEL


SoftCancel request for T25 U8859 M0 received from IC_MAN


*** ERROR => DpRqCheck: T25 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  25    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  8859   appc_ca_blk -1      type  DIA


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 21060


-IN-- prio      HIGH


-IN-- req_info  DP_SOFT_CANCEL


-IN-- msg_name  -               



Wed Jan 14 01:20:56 2015


SoftCancel request for T22 U8887 M0 received from IC_MAN


SoftCancel request for T35 U8885 M0 received from IC_MAN



Wed Jan 14 01:21:10 2015


DpHdlDeadWp: W1 (pid=5524) terminated automatically



Wed Jan 14 01:21:24 2015


DpHdlDeadWp: W8 (pid=564) terminated automatically


***LOG Q0I=> NiIRead: P=127.0.0.1:55171; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 102/sock 1016


    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:55171; L=127.0.0.1:3200) [nixxi.cpp    5082]



Wed Jan 14 01:22:16 2015


DpRqCheck: mode 0 in status CANCEL


*** WARNING => DpHdlSoftCancel: T35/M0 still active in W1 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c   18360]



Wed Jan 14 01:22:44 2015


DpRqCheck: mode 0 in status CANCEL



Wed Jan 14 01:22:49 2015


SoftCancel request for T22 U8887 M0 received from IC_MAN


*** ERROR => DpRqCheck: T22 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  22    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  8887   appc_ca_blk -1      type  DIA


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 21394


-IN-- prio      HIGH


-IN-- req_info  DP_SOFT_CANCEL


-IN-- msg_name  -               



Wed Jan 14 01:25:19 2015


SoftCancel request for T35 U8951 M0 received from IC_MAN



Wed Jan 14 01:25:20 2015


SoftCancel request for T22 U8954 M0 received from IC_MAN



Wed Jan 14 01:25:35 2015


DpRqCheck: mode 0 in status CANCEL


***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]


-IN-- sender_id IC_MAN            tid  24    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  8823   appc_ca_blk -1      type  NOWP


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 21226


-IN-- req_info  DP_ICM_EVENT


-IN-- mpi read    <f9d>  mpi write    <f9e>   prot   1  obj fd    -1   rq_id  0


-IN-- icm_opcode: ICM_PLUGIN_REQUEST   virt_host_idx: 0


-IN-- dtrace {root-id=00505682314E1EE4A6F674B291BD340B}_{conn-id=00505682314E1EE4A6F69543410C740B}_1


-IN-- msg_name  -               



Wed Jan 14 01:52:12 2015


DpHdlDeadWp: W7 (pid=6772) terminated automatically



Wed Jan 14 01:54:51 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:53037: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 22/sock 1324


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:53037) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Wed Jan 14 01:54:51 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     11354


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Wed Jan 14 01:54:57 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]



Wed Jan 14 01:55:08 2015


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Wed Jan 14 02:07:58 2015


*** WARNING => DpTimeCheck: skipped 1 buffer refresh requests because one is in the queue [dpxxdisp.c   9397]



Wed Jan 14 02:12:07 2015


SoftCancel request for T26 U9125 M0 received from IC_MAN



Wed Jan 14 02:41:40 2015


DpHdlDeadWp: W7 (pid=7164) terminated automatically



Wed Jan 14 03:33:27 2015


DpHdlDeadWp: W7 (pid=3892) terminated automatically



Wed Jan 14 03:44:27 2015


DpHdlDeadWp: W4 (pid=6928) terminated automatically



Wed Jan 14 03:47:47 2015


*** WARNING => DpEnvCheck: no answer from msg server since 20 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Wed Jan 14 03:48:07 2015


*** WARNING => DpEnvCheck: no answer from msg server since 40 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8538]



Wed Jan 14 03:48:25 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:55941: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1348


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:55941) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Wed Jan 14 03:48:25 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     12537


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Wed Jan 14 03:48:30 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 03:48:40 2015


*** ERROR => MsISnd2: not_attached [msxxi.c      1510]


***LOG Q0N=> DpRqNoWpHandle, MsSndAdmin () [dpxxdisp.c   5590]



Wed Jan 14 03:48:46 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]



Wed Jan 14 03:48:53 2015


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Wed Jan 14 03:49:15 2015


*** WARNING => DpHdlSoftCancel: T27/M0 still active in W4 with state WP_NEW, can't deligate SOFT_CANCEL [dpxxdisp.c   18360]



Wed Jan 14 03:51:40 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:58283: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 984


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:58283) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Wed Jan 14 03:51:40 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     12564


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]


SoftCancel request for T27 U10155 M0 received from IC_MAN



Wed Jan 14 03:51:45 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 03:52:20 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]


***LOG Q0I=> NiIWrite: P=127.0.0.1:58350; L=127.0.0.1:3200: WSASend (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 4599]


*** ERROR => NiIWrite: SiSendV failed for hdl 33/sock 1348


    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:58350; L=127.0.0.1:3200) [nixxi.cpp    4599]


*** ERROR => NiICheck: NiBufIWriteUserMsg failed (rc=-6) [nixxi.cpp    6096]


*** ERROR => NiBufIProcMsg: NiICheck failed (rc=-6) [nibuf.cpp    2027]


*** ERROR => NiBufIIn: NiBufIProcMsg failed (rc=-6) [nibuf.cpp    2951]


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Wed Jan 14 04:18:03 2015


DpHdlDeadWp: W8 (pid=6688) terminated automatically



Wed Jan 14 04:32:45 2015


SoftCancel request for T34 U10386 M0 received from IC_MAN



Wed Jan 14 04:39:05 2015


DpHdlDeadWp: W0 (pid=6904) terminated automatically



Wed Jan 14 04:56:46 2015


DpHdlDeadWp: W1 (pid=7028) terminated automatically



Wed Jan 14 05:01:47 2015


DpHdlDeadWp: W8 (pid=5276) terminated automatically



Wed Jan 14 05:18:06 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:58369: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 19/sock 984


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:58369) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Wed Jan 14 05:18:06 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     13511


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Wed Jan 14 05:18:11 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 05:18:17 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 05:18:37 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 05:18:57 2015


***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]


DpMBufHwIdSet: set Hardware-ID


***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]


MBUF state ACTIVE


MBUF component UP


DpMsgProcess: 1 server in MBUF


DpMsgProcess: 1 server in MBUF



Wed Jan 14 05:53:07 2015


DpHdlDeadWp: W8 (pid=6128) terminated automatically



Wed Jan 14 06:08:11 2015


DpRqCheck: mode 0 in status CANCEL


***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]


-IN-- sender_id IC_MAN            tid  29    wp_ca_blk   -1      wp_id -1


-IN-- action    SEND_TO_WP        uid  11175   appc_ca_blk -1      type  NOWP


-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 45714


-IN-- req_info  DP_ICM_EVENT


-IN-- mpi read    <19e1>  mpi write    <19e2>   prot   1  obj fd    -1   rq_id  0


-IN-- icm_opcode: ICM_PLUGIN_REQUEST   virt_host_idx: 0


-IN-- dtrace {root-id=00505682314E1EE4A6FB44610803940B}_{conn-id=00505682314E1EE4A6FB8DA63301940B}_1


-IN-- msg_name  -               


*** ERROR => DpRqCheck: T29 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  29    wp_ca_blk   226     wp_id 3


-IN-- action    SEND_TO_WP        uid  11175   appc_ca_blk -1      type  NOWP


-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 45832


-IN-- req_info  CANCELMODE


-IN-- msg_name  -               



Wed Jan 14 06:12:28 2015


DpHdlDeadWp: W1 (pid=6508) terminated automatically



Wed Jan 14 06:32:38 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:60578: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]



Wed Jan 14 06:32:39 2015


*** ERROR => NiIRead: SiRecv failed for hdl 23/sock 1344


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:60578) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)


*              failed.


*              Please check the trace file of the message server.


*


*  TIME        Wed Jan 14 06:32:39 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -36


*  MODULE      msxxi.c


*  LINE        2852


*  COUNTER     14236


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


MBUF state OFF


MBUF component DOWN


*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]


*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]



Wed Jan 14 06:32:44 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 06:32:49 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 06:32:59 2015


*** ERROR => DpRqCheck: T38 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  38    wp_ca_blk   272     wp_id 5


-IN-- action    SEND_TO_WP        uid  11318   appc_ca_blk -1      type  NOWP


-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 48480


-IN-- req_info  CANCELMODE


-IN-- msg_name  -               


*** ERROR => DpRqCheck: T38 in state TM_SLOT_FREE [dpxxdisp.c   8135]


***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]


*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]


-IN-- sender_id DISPATCHER        tid  38    wp_ca_blk   451     wp_id 5


-IN-- action    SEND_TO_WP        uid  11318   appc_ca_blk -1      type  NOWP


-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 48482


-IN-- req_info  CANCELMODE


-IN-- msg_name  -               



Wed Jan 14 06:33:04 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 06:33:25 2015


*** ERROR => MsISnd2: not_attached [msxxi.c      1510]


***LOG Q0N=> DpRqNoWpHandle, MsSnd2Type () [dpxxdisp.c   5557]



Wed Jan 14 06:33:30 2015


*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]



Wed Jan 14 06:34:22 2015


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:62123: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1356


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:62123) [nixxi.cpp    5082]


***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:62123: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]


*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1356


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:62123) [nixxi.cpp    5082]


*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]



*****************************************************************************


*


*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV


*  ERROR       The connection to the specified message server


*              (SAPCRMDEV / 3900) failed. Please check the trace file


*              of the message server. If a network filter has


*              been activated (see message server parameters


*              ms/acl_file_...) please check the configuration


*


*  TIME        Wed Jan 14 06:34:22 2015


*  RELEASE     721


*  COMPONENT   MS (message handling interface, multithreaded)


*  VERSION     4


*  RC          -35


*  MODULE      msxxi.c


*  LINE        2861


*  COUNTER     14239


*


*****************************************************************************



*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]


***LOG Q1K=> MsIAttachEx: StoC check failed, Kernel not compatible with system (rc=-100) [msxxi.c      820]


*** ERROR => Kernel incompatible to already connected instances (see dev_ms for details) [dpxxdisp.c   12889]


DpHalt: shutdown server >SAPCRMDEV_CRD_00                        < (normal)


DpHalt: stop work processes



Wed Jan 14 06:34:28 2015

SoftCancel request for T0 U1 M0 received from DISPATCHER


softCancel request for T22 U10574 M3 received from DISPATCHER

SoftCancel request for T39 U11246 M0 received from DISPATCHER


*** ERROR => DpHdlSoftCancel: terminal has token [dpxxdisp.c   18317]


RM-T22, U10574, 100     DKASHYAP, TermSrvCrestron, 06:32:50, M3, W1, SE24, 4/1


SoftCancel request for T35 U11340 M0 received from DISPATCHER


SoftCancel request for T15 U16 M0 received from DISPATCHER



Wed Jan 14 06:35:29 2015

*** ERROR => DpWpKill(5956, SIGUSR2): kill failed [dpxxtool.c   2792]

DpHalt: stop gateway

DpHalt: stop icman

DpHalt: terminate gui connections


DpHalt: wait for end of work processes


DpHalt: waiting for termination of work processes ...



Wed Jan 14 06:35:30 2015


DpHalt: waiting for termination of work processes ...


Wed Jan 14 06:35:31 2015

DpHalt: waiting for termination of work processes ...


Wed Jan 14 06:35:32 2015

DpHalt: waiting for termination of work processes ...


Wed Jan 14 06:35:33 2015


DpHalt: waiting for termination of work processes ...


Wed Jan 14 06:35:34 2015


pHalt: wait for end of gateway


*** ERROR => [DpProcDied] Process died  (PID:880  HANDLE:948  rc:0x0) [dpnttool2.c  147]


DpHalt: wait for end of icman


*** ERROR => [DpProcDied] Process died  (PID:4132  HANDLE:956  rc:0x0) [dpnttool2.c  147]


DpHalt: not attached to the message server


DpHalt: cleanup EM


Wed Jan 14 06:35:35 2015


***LOG Q05=> DpHalt, DPStop ( 3664) [dpxxdisp.c   11819]

Thanks,
Kavitha Rajan.

Accepted Solutions (1)

Accepted Solutions (1)

Reagan
Advisor
Advisor
0 Kudos

[Thr 4916] Tue Jan 13 08:44:07 2015

[Thr 4916] *** ERROR => MsSClientHandle: C1 (SAPCRMDEV_CRD_00) send inv_name sapcrmdev_CRD_00 [msxxserv.c   6196]

[Thr 4916] ------------------------------------- dump of MSADM table ------------------------------------

[Thr 4916] # NR  ADDR    >server name                             < TYP STA FTNR    FIHDL REQLSTREPLST

[Thr 4916] ----------------------------------------------------------------------------------------------

[Thr 4916] #  0 bad0690 > SAPCRMDEV_CRD_00                           3 191    1   102080 (nil) (nil)

[Thr 4916] #  1 bad00d0 > -                                          3   0    0        0 (nil) (nil)

[Thr 4916] #entries = 2

The problem looks like the system is considering sapcrmdev_CRD_00 as another instance (If I've understood it correctly) and it is taking SAPCRMDEV_CRD_00 as the real instance which is weird. They both have the same system number and the only difference is that one is in upper case and the other is in lower case which shouldn't be a problem as this is a Windows server.

Check this KBA as it is addressing this issue.

1974139 - Message server disconnection error while system works fine.

Go to the profile directory and keep only the correct profiles there and move the old profiles (if any) to a differnt location.

Call Tx RZ10 and delete the profiles there (IMPORTANT: DO NOT confirm deletion of profiles at the OS level) and import the profiles from the OS level.

Check Tx RZ04 and check whether the instance profile is correct there.

Restart SAP and check the trace files.

Cheers

RB

former_member196664
Participant
0 Kudos

Hi Reagan/All,

I did as per your advise. I also checked Note 1713986. Monitored couple of days. Still several issue keep occurring. But at least dispatcher it is not stopping. All RFCs stopped working repeatedly with CPIC CALL error. Once restarted, it goes away, but again occurring. I have latest patch 400 already. Please advise if I have to revert back to 7.20_EXT? Also, just to let you all know, I did not have all old kernel files mixed up. Completely replaced 7.20_EXT to 7.21_EXT. 2 kernel sar files, igs, igshelper, cryptolib are thf iles I extracted and used.

http://scn.sap.com/message/15668109 is my other thread for RFCs. As per the thread, the issue was resolved but keep occurring and going away after restart. Please help ASAP.

Thanks,

Kavitha Rajan.

Matt_Fraser
Active Contributor
0 Kudos

You shouldn't have had to explicitly replace the cryptolib, as it is now included in the kernel patches. However, I doubt this is related to the issues.

Did you run the vcredist.msi that comes with the new kernel, as per Note 1553465? The symptoms you'd experience if you hadn't are a bit different than you describe, but still it's something to check. This is specific for the EXT kernel on Windows. This one has bitten me in the past.

Besides S Sriram's recommended Note, the other one with important information for upgrading to 721 kernel is 1716826. Still, I don't think this specifically addresses your issue; just dotting the i's.

It may be worthwhile to drop back to 720_EXT as an experiment to see what it does. If the problem goes away, that pretty much isolates it to something wrong with the kernel. If it doesn't, then something else is the cause. The 721_EXT kernel should work fine for you, though, and is generally recommended.

Regards,

Matt

former_member196664
Participant
0 Kudos

Matt,

Your reply is very helpful. I started looking outside of kernel except one question, vcredist.msi is already done with 7.20_EXT. Now again run with 7.21_EXT? I will share my findings here ASAP and look for all your advise.

Thanks,
Kavitha Rajan.

Matt_Fraser
Active Contributor
0 Kudos

If you ran it before with the 720_EXT, you shouldn't have to do it again. However, it's unlikely to hurt anything to run it again, and the 721_EXT may have a slightly newer version.

former_member196664
Participant
0 Kudos

Thanks Matt. I re-ran the vcredist.msi.

Hi All,

I found the following in the logs that I suspect.

First.

Fri Jan 16 08:19:29 2015

*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Fri Jan 16 08:19:55 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]
Fri Jan 16 08:20:15 2015

*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Fri Jan 16 08:21:08 2015

*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Second:

Fri Jan 16 08:22:23 2015

16.01.2015 13:22:23.254 PID=5260, TID=4888 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime:  -56291 millisec (max +0). Slept 1000 millisec, Qpc elapsed: +1000 millisec, WindowsUtc elapsed +57291 millisec. SapTime-WindowsUtcTime=-57 seconds. TotalTimeCorrection=+2133 millisec, QpcTime-WindowsUtcTime=-60036 millisec (*).

Fri Jan 16 08:22:25 2015

*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

With IT, I have fixed the 7 minutes difference in windows and the correct EST time, just now. I hope it is in sync with SAP. Also will restart SAP ASAP.

Third, is the parameters after dual stack split.

Do I have to remove all these parameters from ABAP CRM. (Separate DB for ABAP and Java)

jstartup/trimming_properties

jstartup/protocol

jstartup/vm/home

jstartup/max_caches

jstartup/release

jstartup/instance_properties

j2ee/dbdriver

A Check on the profile gives below error.

jstartup/trimming_properties                off

W:Unknown parameter jstartup/trimming_properties , a check cannot be performed

jstartup/protocol                           on

jstartup/vm/home                            $(DIR_SAPJVM)

W:Unknown parameter jstartup/vm/home , a check cannot be performed

jstartup/max_caches                         500

W:Unknown parameter jstartup/max_caches , a check cannot be performed

jstartup/release                            700

jstartup/instance_properties                $(jstartup/j2ee_properties);$(jstartup/sdm_properties)

E:Cannot open jstartup/instance_properties file

E:  F:\usr\sap\CRD\DVEBMGS00\SDM\program\config\sdm_jstartup.properties

eg2/max_extend_size_MB                      4

W:Unknown parameter eg2/max_extend_size_MB , a check cannot be performed

vmcj/option/gc/sharedGCPrecision            1000n

W:Unknown parameter vmcj/option/gc/sharedGCPrecision , a check cannot be

W:  performed

vmcj/sgc_options                            TEMPLATE_CLEANUP

vmcj/memory_isolation                       0

W:Unknown parameter vmcj/memory_isolation , a check cannot be performed

vmcj/option/extMTX                          share

W:Unknown parameter vmcj/option/extMTX , a check cannot be performed

vmcj/option/internLock                      1

W:Unknown parameter vmcj/option/internLock , a check cannot be performed

vmcj/option/scnLock                         3

W:Unknown parameter vmcj/option/scnLock , a check cannot be performed

vmcj/option/tmpHashtable                    1

W:Unknown parameter vmcj/option/tmpHashtable , a check cannot be performed

DIR_SAPJVM                                  $(DIR_EXECUTABLE)$(DIR_SEP)sapjvm_4

W:Unknown parameter DIR_SAPJVM , a check cannot be performed

jstartup/signal_handling                    0

W:Unknown parameter jstartup/signal_handling , a check cannot be performed

SAPJVM_VERSION                              4.1.047

W:Unknown parameter SAPJVM_VERSION , a check cannot be performed

Thanks,

Kavitha Rajan.

Sriram2009
Active Contributor
0 Kudos

Hi

Is this firewall or windows uac on?

BR

ss

former_member196664
Participant
0 Kudos

Also

DIR_SAPJVM                                  $(DIR_EXECUTABLE)$(DIR_SEP)sapjvm_4

W:Unknown parameter DIR_SAPJVM , a check cannot be performed

jstartup/signal_handling                    0

W:Unknown parameter jstartup/signal_handling , a check cannot be performed

exe/icmbnd                                  $(DIR_CT_RUN)/icmbnd

E:Cannot open exe/icmbnd file f:\usr\sap\crd\sys\exe\uc\ntamd64/icmbnd for

E:  reading

former_member196664
Participant
0 Kudos

I think windows uac.

Sriram2009
Active Contributor
0 Kudos

Hi

Check the sap jvm settings in sapadm user environment variable?

BR

ss

former_member196664
Participant
0 Kudos

I have removed all java parameters from the ABAP system. After dual split, these does not exist in DEV, but somehow in this system. I have now removed them. Restarted SAP. Still little conflict with windows time. All RFCs working at this time. I will start monitoring if the port issue or sap shutdown or RFC error occur again.

Thanks,

Kavitha Rajan.

Sriram2009
Active Contributor
0 Kudos

Hi Kaviytha

1.  Is this any time zone is mismatch?

2. Check the Windows UAC on or off ?

BR

former_member196664
Participant
0 Kudos

Sriram/All,

Time mismatch is corrected, in one of master VM. SAP/DB restarted for the change to correct 3 minjtes diff. It is all fine now. But the issue still occurs.

On the java side, Server0 keep getting stopped with -11113 (Core service or additional service start fails: "HaltOnAdditionalServiceFailure=true"(service manager)). I checked in config tool, however the value is showing as FALSE.

Below is the dev_disp on the abap side. Please help. This is occurring in both SBX and DEV after dual stack split followed by upgrade crm 6.0 to 7.02, kernel from 7.20_ext to 7.21_ext.


---------------------------------------------------
trc file: "dev_disp", trc level: 1, release: "721"
---------------------------------------------------
sysno      00
sid        CRD
systemid   562 (PC with Windows NT)
relno      7210
patchlevel 0
patchno    400
intno      20020600
make       multithreaded, Unicode, 64 bit, optimized
profile    F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
pid        189176

kernel runs with dp version 139000(ext=121000) (@(#) DPLIB-INT-VERSION-139000-UC)
length of sys_adm_ext is 592 bytes
*** SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 189176) [dpxxdisp.c   1323]
shared lib "dw_xml.dll" version 400 successfully loaded
shared lib "dw_xtc.dll" version 400 successfully loaded
shared lib "dw_stl.dll" version 400 successfully loaded
shared lib "dw_gui.dll" version 400 successfully loaded
shared lib "dw_mdm.dll" version 400 successfully loaded
shared lib "dw_rndrt.dll" version 400 successfully loaded
shared lib "dw_abp.dll" version 400 successfully loaded
shared lib "dw_sym.dll" version 400 successfully loaded
shared lib "dw_aci.dll" version 400 successfully loaded
rdisp/softcancel_sequence :  -> 0,5,-1
use internal message server connection to port 3900
rdisp/dynamic_wp_check : 1
rdisp/calculateLoadAverage : 1

Mon Jan 19 08:53:26 2015
*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c  6525]
MtxInit: 30000 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: VMC (JAVA VM in WP) is active
display_tcp_ip: 0
Set Check Logoff Flags to 0x0
DpIPCInit2: write dp-profile-values into sys_adm_ext
DpIPCInit2: start server >SAPCRMDEV_CRD_00                        <
DpShMCreate: sizeof(wp_adm)  63168 (2256)
DpShMCreate: sizeof(tm_adm)  5517056 (27448)
DpShMCreate: sizeof(wp_ca_adm)  64000 (64)
DpShMCreate: sizeof(appc_ca_adm) 128000 (64)
DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1392064/1392080
DpShMCreate: sizeof(comm_adm)  1392080 (2768)
DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=512/48/163904/96560/260512
DpShMCreate: sizeof(slock_adm)  260512 (296)
DpFileTableSize: max/headSize/ftSize/tableSize=5600/16/537664/537680
DpShMCreate: sizeof(file_adm)  537680 (80)
DpShMCreate: sizeof(vmc_adm)  60480 (2160)
DpShMCreate: sizeof(wall_adm)  (41664/42896/64/192)
DpShMCreate: sizeof(gw_adm) 48
DpShMCreate: sizeof(j2ee_adm) 3952
DpShMCreate: SHM_DP_ADM_KEY  (addr: 000000000E2E0050, size: 8125808)
DpShMCreate: allocated sys_adm at 000000000E2E0060
DpShMCreate: allocated wp_adm_list at 000000000E2E30B0
DpShMCreate: allocated wp_adm at 000000000E2E32A0
DpShMCreate: allocated tm_adm_list at 000000000E2F2970
DpShMCreate: allocated tm_adm at 000000000E2F29C0
DpShMCreate: allocated wp_ca_adm at 000000000E8358D0
DpShMCreate: allocated appc_ca_adm at 000000000E8452E0
DpShMCreate: allocated comm_adm at 000000000E8646F0
DpShMCreate: allocated slock_adm at 000000000E9B84D0
DpShMCreate: allocated file_adm at 000000000E9F7E80
DpShMCreate: allocated vmc_adm_list at 000000000EA7B2E0
DpShMCreate: allocated vmc_adm at 000000000EA7B390
DpShMCreate: allocated gw_adm at 000000000EA89FE0
DpShMCreate: allocated j2ee_adm at 000000000EA8A020
DpShMCreate: allocated ca_info at 000000000EA8AFA0
DpShMCreate: allocated wall_adm at 000000000EA8B030
DpCommAttachTable: attached comm table (header=000000000E8646F0/ft=000000000E864700)
DpSysAdmIntInit: initialize sys_adm
rdisp/test_roll : roll strategy is DP_NORMAL_ROLL
dia token check not active (15 token)
MBUF state OFF
DpCommInitTable: init table for 500 entries
DpFileInitTable: init table for 5600 entries
DpSesCreateTable: created session table at 0000000002D20050 (len=167472)
DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm
rdisp/queue_size_check_value :  -> on,50,30,40,500,50,500,80
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0
<ES> client 0 initializing ....
<ES> EsILock: use Mutex for locking
<ES> InitFreeList
<ES> block size is 4096 kByte.
Using implementation view
<EsNT> Using memory model view.
<EsNT> Memory Reset disabled as NT default
<ES> 2865 blocks reserved for free list.
ES initialized.
mm.dump: set maximum dump mem to 192 MB
mm.dump: set global maximum dump mem to 192 MB
EsRegisterEmCheck: Register EmGetEsHandles at 0000000140744270
Es2ResCreate: Changed resource parameters to
filenamePrefix =
blockSizeKB    = 2048 KB
initialSizeMB  = 480 MB
extendSizeMB   = 240 MB
maxSizeMB      = 19680 MB
DpVmcSetActive: set vmc state DP_VMC_ENABLED
DpVmcSetActive: set vmc state DP_VMC_ACTIVE
DpVmcInit2: o.k.
MPI: dynamic quotas disabled.
MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c   1709]
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12944]
MBUF state LOADING
DpStartStopMsg: send start message (myname is >SAPCRMDEV_CRD_00                        <)
DpStartStopMsg: start msg sent
CCMS: Initialized monitoring segment of size 60000000.
CCMS: Initialized CCMS Headers in the shared monitoring segment.
CCMS: Checking Downtime Configuration of Monitoring Segment.
CCMS: AlMsUpload called by wp 1024.

Mon Jan 19 08:53:28 2015
CCMS: AlMsUpload successful for F:\usr\sap\CRD\DVEBMGS00\log\ALMTTREE.DAT (3204 MTEs).

Mon Jan 19 08:53:31 2015
CCMS: start to initalize 3.X shared alert area (first segment).
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
MBUF state ACTIVE
DpWpBlksLow: max wp blocks in queue is 800 (80 %)
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpAppcBlksLow: max appc blocks in queue is 1000 (50 %)

Mon Jan 19 08:53:39 2015
DpModState: change server state from STARTING to ACTIVE

Mon Jan 19 08:59:46 2015
DpWpConf: requested operation mode switch:
DIA 15->17 BTC 6->4 UPD 2->2 UP2 1->1 ENQ 1->1 SPO 1->1 RESTR 0->0 DYN 2->2
***LOG Q0V=> DpWpConf, mod_req (018BTCDIA) [dpxxdisp.c   16041]
***LOG Q0V=> DpWpConf, mod_req (023BTCDIA) [dpxxdisp.c   16041]
DpAdaptWppriv_max_no : 10 -> 12
DpWpConf: operation mode switch completed
DIA water mark reached (>30)
DIA water mark underrun (<=30)

Mon Jan 19 10:52:36 2015
SoftCancel request for T30 U678 M0 received from REMOTE_TERMINAL

Mon Jan 19 10:53:37 2015
SoftCancel request for T29 U791 M2 received from REMOTE_TERMINAL
SoftCancel request for T31 U940 M0 received from APPC_SERVER
SoftCancel request for T41 U934 M0 received from APPC_SERVER
SoftCancel request for T37 U961 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER       tid  37    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  961   appc_ca_blk 72      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         80      rq_id 19495
-IN-- msg_name  -                  
SoftCancel request for T33 U976 M0 received from APPC_SERVER
-IN-- sender_id APPC_SERVER       tid  41    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  934   appc_ca_blk 75      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         80      rq_id 19510
-IN-- msg_name  -                 

Mon Jan 19 10:53:47 2015
DpHdlDeadWp: W4 (pid=189384) terminated automatically

Mon Jan 19 10:53:48 2015
*** ERROR => DpRqCheck: T33 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id APPC_SERVER       tid  33    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  976   appc_ca_blk 71      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         80      rq_id 19568
-IN-- msg_name  -                 

Mon Jan 19 12:29:48 2015
*** ERROR => DpRqCheck: T37 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4231): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  37    wp_ca_blk   350     wp_id 3
-IN-- action    SEND_TO_WP        uid  1613   appc_ca_blk -1      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 27301
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                 

Mon Jan 19 14:24:27 2015
***LOG Q0I=> NiIRead: P=172.17.10.4:50358; L=192.168.200.142:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 63/sock 388
    (SI_ECONN_BROKEN/10054; I4; ST; P=172.17.10.4:50358; L=192.168.200.142:3200) [nixxi.cpp    5082]
DpRTmPrepareReq: network error of client T29, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3

Mon Jan 19 14:24:32 2015
*** WARNING => DpAddrToHost: NiAddrToHost took 4586 ms [dpxxdisp.c   20855]
*** WARNING => DpAddrToHost: Disabling reverse name lookup [dpxxdisp.c   20856]
DpRTmPrepareReq: client address of T29 is 172.17.10.4(172.17.10.4)
***LOG Q04=> DpRTmPrep, NiBufReceive (791 SRASTOGI 29 SATYAVRATRA1) [dpxxdisp.c   12573]
RM-T29, U791, 100     SRASTOGI, SATYAVRATRA1D18, 14:04:02, M3, W23, BSP_, 5/2

Mon Jan 19 19:02:31 2015
***LOG Q0I=> NiIRead: P=172.17.10.134:52265; L=192.168.200.142:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 28/sock 1388
    (SI_ECONN_BROKEN/10054; I4; ST; P=172.17.10.134:52265; L=192.168.200.142:3200) [nixxi.cpp    5082]
DpRTmPrepareReq: network error of client T68, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
DpRTmPrepareReq: client address of T68 is 172.17.10.134(172.17.10.134)
***LOG Q04=> DpRTmPrep, NiBufReceive (4680 SRASTOGI 68 SATYAVRATRA1) [dpxxdisp.c   12573]
RM-T68, U4680, 100     SRASTOGI, SATYAVRATRA1D18, 18:42:01, M1, W2,     , 3/2

Mon Jan 19 20:00:48 2015
DpWpConf: requested operation mode switch:
DIA 17->15 BTC 4->6 UPD 2->2 UP2 1->1 ENQ 1->1 SPO 1->1 RESTR 0->0 DYN 2->2
***LOG Q0V=> DpWpConf, mod_req (023DIABTC) [dpxxdisp.c   16041]
***LOG Q0V=> DpWpConf, mod_req (004DIABTC) [dpxxdisp.c   16041]
DpAdaptWppriv_max_no : 12 -> 10
DpWpConf: operation mode switch completed

Mon Jan 19 20:08:31 2015
***LOG Q0I=> NiIRead: P=172.17.10.53:52663; L=192.168.200.142:3200: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 26/sock 292
    (SI_ECONN_BROKEN/10054; I4; ST; P=172.17.10.53:52663; L=192.168.200.142:3200) [nixxi.cpp    5082]
DpRTmPrepareReq: network error of client T69, NiBufReceive (-6: NIECONN_BROKEN), dp_tm_status=3
DpRTmPrepareReq: client address of T69 is 172.17.10.53(172.17.10.53)
***LOG Q04=> DpRTmPrep, NiBufReceive (5786 SRASTOGI 69 SATYAVRATRA1) [dpxxdisp.c   12573]
RM-T69, U5786, 100     SRASTOGI, SATYAVRATRA1D18, 19:47:58, M0, W0, SE16, 2/2

Tue Jan 20 00:12:36 2015
SoftCancel request for T29 U8195 M0 received from IC_MAN
SoftCancel request for T28 U8197 M0 received from IC_MAN
SoftCancel request for T55 U8198 M0 received from IC_MAN

Tue Jan 20 00:41:46 2015
*** WARNING => DpTimeCheck: skipped 1 buffer refresh requests because one is in the queue [dpxxdisp.c   9397]

Tue Jan 20 00:42:27 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 00:45:46 2015
*** WARNING => DpTimeCheck: skipped 1 buffer refresh requests because one is in the queue [dpxxdisp.c   9397]

Tue Jan 20 00:47:46 2015
*** WARNING => DpTimeCheck: skipped 2 buffer refresh requests because one is in the queue [dpxxdisp.c   9397]

Tue Jan 20 00:48:39 2015
SoftCancel request for T66 U6995 M0 received from REMOTE_TERMINAL
DIA water mark reached (>30)

Tue Jan 20 00:49:13 2015
DpEnvCheck: set dp_high_prio_state to DP_HPRIO_TIMEOUT (1 rqs)

Tue Jan 20 00:49:30 2015
DIA water mark underrun (<=30)

Tue Jan 20 00:49:31 2015
DIA water mark reached (>30)

Tue Jan 20 00:49:32 2015
DIA water mark underrun (<=30)

Tue Jan 20 00:49:48 2015
DIA water mark reached (>30)

Tue Jan 20 00:49:50 2015
DIA water mark underrun (<=30)
DIA water mark reached (>30)

Tue Jan 20 00:49:52 2015
DIA water mark underrun (<=30)

Tue Jan 20 00:50:49 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 00:50:54 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 00:53:48 2015
DIA water mark reached (>30)

Tue Jan 20 00:54:07 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 00:54:08 2015
DIA water mark underrun (<=30)
DIA water mark reached (>30)
DIA water mark underrun (<=30)
DIA water mark reached (>30)
*** ERROR => DpRqCheck: T40 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  40    wp_ca_blk   495     wp_id 12
-IN-- action    SEND_TO_WP        uid  8255   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30824
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T55 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  55    wp_ca_blk   496     wp_id 12
-IN-- action    SEND_TO_WP        uid  8248   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30826
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T73 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  73    wp_ca_blk   497     wp_id 12
-IN-- action    SEND_TO_WP        uid  8134   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30828
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T44 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  44    wp_ca_blk   498     wp_id 12
-IN-- action    SEND_TO_WP        uid  8109   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30830
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
DIA water mark underrun (<=30)

Tue Jan 20 00:54:09 2015
*** ERROR => DpRqCheck: T40 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  40    wp_ca_blk   499     wp_id 12
-IN-- action    SEND_TO_WP        uid  8255   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30834
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T55 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  55    wp_ca_blk   500     wp_id 12
-IN-- action    SEND_TO_WP        uid  8248   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30836
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T73 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  73    wp_ca_blk   501     wp_id 12
-IN-- action    SEND_TO_WP        uid  8134   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30838
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T44 in state TM_SLOT_FREE [dpxxdisp.c   8135]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6264]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7176): [dpxxdisp.c   6266]
-IN-- sender_id DISPATCHER        tid  44    wp_ca_blk   502     wp_id 12
-IN-- action    SEND_TO_WP        uid  8109   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 30840
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                 

Tue Jan 20 01:01:32 2015
***LOG Q0I=> NiIRead: P=192.168.200.142:3900; L=192.168.200.142:62298: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 952
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.200.142:3900; L=192.168.200.142:62298) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)
*              failed.
*              Please check the trace file of the message server.
*
*  TIME        Tue Jan 20 01:01:32 2015
*  RELEASE     721
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -36
*  MODULE      msxxi.c
*  LINE        2852
*  COUNTER     11585
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2601]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2672]

Tue Jan 20 01:01:37 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Tue Jan 20 01:01:43 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Tue Jan 20 01:02:03 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Tue Jan 20 01:02:37 2015
***LOG Q0I=> NiIRead: P=192.168.200.142:3900; L=192.168.200.142:59971: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 21/sock 1164
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.200.142:3900; L=192.168.200.142:59971) [nixxi.cpp    5082]
***LOG Q0I=> NiIRead: P=192.168.200.142:3900; L=192.168.200.142:59971: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 21/sock 1164
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.200.142:3900; L=192.168.200.142:59971) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       The connection to the specified message server
*              (SAPCRMDEV / 3900) failed. Please check the trace file
*              of the message server. If a network filter has
*              been activated (see message server parameters
*              ms/acl_file_...) please check the configuration
*
*  TIME        Tue Jan 20 01:02:37 2015
*  RELEASE     721
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -35
*  MODULE      msxxi.c
*  LINE        2861
*  COUNTER     11588
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
***LOG Q1K=> MsIAttachEx: StoC check failed, Kernel not compatible with system (rc=-100) [msxxi.c      820]
*** ERROR => Kernel incompatible to already connected instances (see dev_ms for details) [dpxxdisp.c   12889]
DpHalt: shutdown server >SAPCRMDEV_CRD_00                        < (normal)
DpHalt: stop work processes

Tue Jan 20 01:02:38 2015
SoftCancel request for T33 U8797 M0 received from DISPATCHER
SoftCancel request for T45 U10950 M0 received from DISPATCHER
SoftCancel request for T67 U8812 M0 received from DISPATCHER
SoftCancel request for T53 U8805 M0 received from DISPATCHER
SoftCancel request for T32 U8783 M0 received from DISPATCHER
SoftCancel request for T48 U10952 M0 received from DISPATCHER
SoftCancel request for T10 U11 M0 received from DISPATCHER
SoftCancel request for T49 U8810 M0 received from DISPATCHER
SoftCancel request for T61 U8795 M0 received from DISPATCHER
SoftCancel request for T71 U8807 M0 received from DISPATCHER

Tue Jan 20 01:03:39 2015
DpHalt: stop gateway
DpHalt: stop icman
DpHalt: terminate gui connections
DpHalt: wait for end of work processes
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:40 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:41 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:42 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:43 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:44 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:45 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:46 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:47 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:48 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:49 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:50 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:51 2015
DpHalt: waiting for termination of work processes ...
*** ERROR => 20.01.2015 06:03:51.730 PID=189176, TID=188776 SapTimer warning. Took time keeping ownership. PID=189176, old owner PID:189484 [dpxxdisp.c   21913]

Tue Jan 20 01:03:52 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:53 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:54 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:55 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:56 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:57 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:58 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:03:59 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:00 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:01 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:02 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:03 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:04 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:05 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:06 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:07 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:08 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:09 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:10 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:11 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:12 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:13 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:14 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:15 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:16 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:17 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:18 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:19 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:20 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:21 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:22 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:23 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:24 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:25 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:26 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:27 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:28 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:29 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:30 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:31 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:32 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:33 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:34 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:35 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:36 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:37 2015
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:38 2015
*** ERROR => DpHalt: W0 (pid 189416) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189416, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W1 (pid 188616) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(188616, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W2 (pid 188880) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(188880, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W3 (pid 189136) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189136, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W4 (pid 208844) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(208844, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W5 (pid 188808) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(188808, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W6 (pid 189320) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189320, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W8 (pid 189452) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189452, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W9 (pid 189468) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189468, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W10 (pid 189484) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189484, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W11 (pid 189500) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189500, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W12 (pid 189516) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189516, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W13 (pid 189532) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189532, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W14 (pid 189548) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189548, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W18 (pid 189612) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189612, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W19 (pid 189628) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189628, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W22 (pid 189676) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189676, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W23 (pid 189692) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189692, SIGKILL): kill failed [dpxxtool.c   2756]
*** ERROR => DpHalt: W24 (pid 189708) still alive ... [dpxxdisp.c   11537]
*** ERROR => DpWpKill(189708, SIGKILL): kill failed [dpxxtool.c   2756]
DpHalt: waiting for termination of work processes ...

Tue Jan 20 01:04:39 2015
DpHalt: wait for end of gateway
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:40 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:41 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:42 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:43 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:44 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:45 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:46 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:47 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:48 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:49 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:50 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:51 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:52 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:53 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:54 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:55 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:56 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:57 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:58 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:04:59 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:00 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:01 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:02 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:03 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:04 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:05 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:06 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:07 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:08 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:09 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:10 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:11 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:12 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:13 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:14 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:15 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:16 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:17 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:18 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:19 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:20 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:21 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:22 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:23 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:24 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:25 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:26 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:27 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:28 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:29 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:30 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:31 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:32 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:33 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:34 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:35 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:36 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:37 2015
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:38 2015
*** ERROR => DpHalt: gateway (pid 189168) still alive ... [dpxxdisp.c   11581]
*** ERROR => DpProcKill: kill failed [dpntdisp.c   385]
DpHalt: waiting for termination of gateway ...

Tue Jan 20 01:05:39 2015
DpHalt: wait for end of icman
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:40 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:41 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:42 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:43 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:44 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:45 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:46 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:47 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:48 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:49 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:50 2015
DpHalt: waiting for termination of icman ...

Tue Jan 20 01:05:51 2015
*** ERROR => [DpProcDied] Process died  (PID:189280  HANDLE:964  rc:0x0) [dpnttool2.c  147]
DpHalt: not attached to the message server
DpHalt: cleanup EM
***LOG Q05=> DpHalt, DPStop ( 189176) [dpxxdisp.c   11819]

hemanth2
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Kavitha,


Hope you are doing good. The issue is with the kernel that you are using (obviously ).

I would advise that you follow all the steps of note 1636252 and then try. If the issue persists, downgrade the kernel to a older version and then check.
Alse the kernel version in SCS/exe and the run folder should be the same; if both are of different versions, this issue can occur.

____________
Kind Regards,
Hemanth
SAP AGS
 

former_member196664
Participant
0 Kudos

Hemanth,

Yes, I have done the same, reverted back to 7.20_EXT. Let me watch couple of days if the issue re-occurs or not. Only MOPZ calculated 7.21_EXT, but looks like still it is not compatible.

Thanks,
Kavitha Rajan.

hemanth2
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Kavitha,

Yes, thats the best option. I would suggest that you raise an incident in the BC-CST-STS component so that this can be checked.
In the meantime, just use the manual kernel update and use a older stable kernel.


Regards,
Hemanth

former_member196664
Participant
0 Kudos

Hi All,

Even after reverting the kernel, the issue still exists. I cannot move to QA upgrade until I fix this in SBX and DEV. Pls help.

Disp+work on ABAP side stopped. In the java system server0 stopped with-11113 (Core service or additional service start fails: "HaltOnAdditionalServiceFailure=true"(service manager)).

---------------------------------------------------
trc file: "dev_disp", trc level: 1, release: "720"
---------------------------------------------------
sysno      00
sid        CRD
systemid   562 (PC with Windows NT)
relno      7200
patchlevel 0
patchno    700
intno      20020600
make       multithreaded, Unicode, 64 bit, optimized
profile    F:\usr\sap\CRD\SYS\profile\CRD_DVEBMGS00_SAPCRMDEV
pid        6404

kernel runs with dp version 140000(ext=121000) (@(#) DPLIB-INT-VERSION-140000-UC)
length of sys_adm_ext is 592 bytes
*** SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 6404) [dpxxdisp.c   1326]
shared lib "dw_xml.dll" version 700 successfully loaded
shared lib "dw_xtc.dll" version 700 successfully loaded
shared lib "dw_stl.dll" version 700 successfully loaded
shared lib "dw_gui.dll" version 700 successfully loaded
shared lib "dw_mdm.dll" version 700 successfully loaded
shared lib "dw_rndrt.dll" version 700 successfully loaded
shared lib "dw_abp.dll" version 700 successfully loaded
shared lib "dw_sym.dll" version 700 successfully loaded
shared lib "dw_aci.dll" version 700 successfully loaded
rdisp/softcancel_sequence :  -> 0,5,-1
use internal message server connection to port 3900
rdisp/dynamic_wp_check : 1
rdisp/calculateLoadAverage : 1

Tue Jan 20 11:45:08 2015
*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c  6492]
MtxInit: 30000 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: VMC (JAVA VM in WP) is active
display_tcp_ip: 0
DpIPCInit2: write dp-profile-values into sys_adm_ext
DpIPCInit2: start server >SAPCRMDEV_CRD_00                        <
DpShMCreate: sizeof(wp_adm)  63168 (2256)
DpShMCreate: sizeof(tm_adm)  5517056 (27448)
DpShMCreate: sizeof(wp_ca_adm)  64000 (64)
DpShMCreate: sizeof(appc_ca_adm) 128000 (64)
DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1384064/1384080
DpShMCreate: sizeof(comm_adm)  1384080 (2744)
DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=512/48/163904/96560/260512
DpShMCreate: sizeof(slock_adm)  260512 (296)
DpFileTableSize: max/headSize/ftSize/tableSize=5600/16/537664/537680
DpShMCreate: sizeof(file_adm)  537680 (80)
DpShMCreate: sizeof(vmc_adm)  60480 (2160)
DpShMCreate: sizeof(wall_adm)  (41664/42896/64/192)
DpShMCreate: sizeof(gw_adm) 48
DpShMCreate: sizeof(j2ee_adm) 3952
DpShMCreate: SHM_DP_ADM_KEY  (addr: 0000000007780050, size: 8117840)
DpShMCreate: allocated sys_adm at 0000000007780060
DpShMCreate: allocated wp_adm_list at 00000000077830D0
DpShMCreate: allocated wp_adm at 00000000077832C0
DpShMCreate: allocated tm_adm_list at 0000000007792990
DpShMCreate: allocated tm_adm at 00000000077929E0
DpShMCreate: allocated wp_ca_adm at 0000000007CD58F0
DpShMCreate: allocated appc_ca_adm at 0000000007CE5300
DpShMCreate: allocated comm_adm at 0000000007D04710
DpShMCreate: allocated slock_adm at 0000000007E565B0
DpShMCreate: allocated file_adm at 0000000007E95F60
DpShMCreate: allocated vmc_adm_list at 0000000007F193C0
DpShMCreate: allocated vmc_adm at 0000000007F19470
DpShMCreate: allocated gw_adm at 0000000007F280C0
DpShMCreate: allocated j2ee_adm at 0000000007F28100
DpShMCreate: allocated ca_info at 0000000007F29080
DpShMCreate: allocated wall_adm at 0000000007F29110
DpCommAttachTable: attached comm table (header=0000000007D04710/ft=0000000007D04720)
DpSysAdmIntInit: initialize sys_adm
rdisp/test_roll : roll strategy is DP_NORMAL_ROLL
dia token check not active (15 token)
MBUF state OFF
DpCommInitTable: init table for 500 entries
DpFileInitTable: init table for 5600 entries
DpSesCreateTable: created session table at 0000000002E90050 (len=167472)
DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm
rdisp/queue_size_check_value :  -> on,50,30,40,500,50,500,80
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0
<ES> client 0 initializing ....
<ES> EsILock: use spinlock for locking
<ES> InitFreeList
<ES> block size is 4096 kByte.
Using implementation view
<EsNT> Using memory model view.
<EsNT> Memory Reset disabled as NT default
<ES> 2865 blocks reserved for free list.
ES initialized.
mm.dump: set maximum dump mem to 192 MB
mm.dump: set global maximum dump mem to 192 MB
EsRegisterEmCheck: Register EmGetEsHandles at 0000000140A25A70
Es2ResCreate: Changed resource parameters to
filenamePrefix =
blockSizeKB    = 2048 KB
initialSizeMB  = 480 MB
extendSizeMB   = 240 MB
maxSizeMB      = 19680 MB
DpVmcSetActive: set vmc state DP_VMC_ENABLED
DpVmcSetActive: set vmc state DP_VMC_ACTIVE
DpVmcInit2: o.k.
MPI: dynamic quotas disabled.
MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c   1712]
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12865]
MBUF state LOADING
DpStartStopMsg: send start message (myname is >SAPCRMDEV_CRD_00                        <)
DpStartStopMsg: start msg sent
CCMS: Initialized monitoring segment of size 60000000.
CCMS: Initialized CCMS Headers in the shared monitoring segment.
CCMS: Checking Downtime Configuration of Monitoring Segment.
CCMS: AlMsUpload called by wp 1024.

Tue Jan 20 11:45:09 2015
CCMS: AlMsUpload successful for F:\usr\sap\CRD\DVEBMGS00\log\ALMTTREE.DAT (2864 MTEs).

Tue Jan 20 11:45:13 2015
CCMS: start to initalize 3.X shared alert area (first segment).

Tue Jan 20 11:45:15 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
MBUF state ACTIVE
DpWpBlksLow: wp blocks in queue threshold is 800 (80 %)
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpAppcBlksLow: appc blocks in queue threshold is 1000 (50 %)

Tue Jan 20 11:45:18 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
DpModState: change server state from STARTING to ACTIVE
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:20 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:21 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:22 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:23 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:24 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:25 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:26 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:27 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:28 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:29 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:30 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:32 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:33 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:34 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:35 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:36 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:43 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:44 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:45 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:46 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:45:47 2015
*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

Tue Jan 20 11:52:25 2015
DpWpConf: requested operation mode switch:
DIA 15->17 BTC 6->4 UPD 2->2 UP2 1->1 ENQ 1->1 SPO 1->1 RESTR 0->0 DYN 2->2
***LOG Q0V=> DpWpConf, mod_req (020BTCDIA) [dpxxdisp.c   15852]
***LOG Q0V=> DpWpConf, mod_req (022BTCDIA) [dpxxdisp.c   15852]
DpAdaptWppriv_max_no : 10 -> 12
DpWpConf: operation mode switch completed
DIA water mark reached (>30)
DIA water mark underrun (<=30)

Tue Jan 20 15:16:32 2015
DpHdlDeadWp: W4 (pid=5256) terminated automatically

Tue Jan 20 15:19:52 2015
DpHdlDeadWp: W6 (pid=3108) terminated automatically

Tue Jan 20 15:20:32 2015
DpHdlDeadWp: W0 (pid=5020) terminated automatically

Tue Jan 20 15:29:33 2015
DpHdlDeadWp: W7 (pid=3204) terminated automatically
DpHdlDeadWp: W9 (pid=152) terminated automatically

Tue Jan 20 16:22:08 2015
SoftCancel request for T29 U10629 M0 received from REMOTE_TERMINAL
SoftCancel request for T31 U16703 M0 received from APPC_SERVER
SoftCancel request for T53 U16704 M0 received from APPC_SERVER

Tue Jan 20 16:22:35 2015
DpHdlDeadWp: W11 (pid=4088) terminated automatically

Tue Jan 20 16:22:39 2015
*** ERROR => DpRqCheck: T53 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7118): [dpxxdisp.c   6208]
-IN-- sender_id APPC_SERVER       tid  53    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  16704   appc_ca_blk 80      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         80      rq_id 60903
-IN-- msg_name  -                  

Tue Jan 20 16:24:25 2015
SoftCancel request for T31 U18443 M0 received from IC_MAN
SoftCancel request for T53 U18444 M0 received from IC_MAN
DpRqCheck: mode 0 in status CANCEL

Tue Jan 20 16:24:46 2015
SoftCancel request for T31 U18492 M0 received from IC_MAN
SoftCancel request for T54 U18484 M0 received from IC_MAN

Tue Jan 20 16:24:50 2015
SoftCancel request for T53 U18498 M0 received from IC_MAN
SoftCancel request for T32 U18496 M0 received from IC_MAN
SoftCancel request for T54 U18495 M0 received from IC_MAN
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T53 U18498 M0 received from IC_MAN
*** ERROR => DpRqCheck: T53 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7118): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  53    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  18498   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 61436
-IN-- prio      HIGH
-IN-- req_info  DP_SOFT_CANCEL
-IN-- msg_name  -                  
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T32 U18496 M0 received from IC_MAN
*** ERROR => DpRqCheck: T32 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7118): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  32    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  18496   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 61439
-IN-- prio      HIGH
-IN-- req_info  DP_SOFT_CANCEL
-IN-- msg_name  -                  

Tue Jan 20 16:25:00 2015
SoftCancel request for T32 U18518 M0 received from IC_MAN
SoftCancel request for T31 U18516 M0 received from IC_MAN
DpRqCheck: mode 0 in status CANCEL
SoftCancel request for T32 U18518 M0 received from IC_MAN
*** ERROR => DpRqCheck: T32 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req ( DIA) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 7118): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  32    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_WP        uid  18518   appc_ca_blk -1      type  DIA 
-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 61498
-IN-- prio      HIGH
-IN-- req_info  DP_SOFT_CANCEL
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T31 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4234): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  31    wp_ca_blk   140     wp_id 2
-IN-- action    SEND_TO_WP        uid  18516   appc_ca_blk -1      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 61489
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  

Tue Jan 20 16:25:09 2015
SoftCancel request for T31 U18532 M0 received from IC_MAN
DpRqCheck: mode 0 in status CANCEL

Tue Jan 20 16:25:10 2015
SoftCancel request for T31 U18536 M0 received from IC_MAN
SoftCancel request for T32 U18534 M0 received from IC_MAN
*** ERROR => DpRqCheck: T32 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 3943): [dpxxdisp.c   6208]
-IN-- sender_id IC_MAN            tid  32    wp_ca_blk   -1      wp_id -1
-IN-- action    SEND_TO_DP        uid  18534   appc_ca_blk -1      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         0       rq_id 61554
-IN-- req_info  DP_SOFT_CANCEL,DP_ICM_EVENT
-IN-- msg_name  -                  

Tue Jan 20 16:25:23 2015
*** ERROR => DpRqCheck: T54 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4234): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  54    wp_ca_blk   455     wp_id 0
-IN-- action    SEND_TO_WP        uid  18544   appc_ca_blk -1      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 61607
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  
*** ERROR => DpRqCheck: T54 in state TM_SLOT_FREE [dpxxdisp.c   8087]
***LOG Q0G=> DpRqBadHandle, bad_req (NOWP) [dpxxdisp.c   6206]
*** ERROR => DpRqBadHandle: BAD REQUEST - Reason: DpRqCheck failed (line 4234): [dpxxdisp.c   6208]
-IN-- sender_id DISPATCHER        tid  54    wp_ca_blk   456     wp_id 0
-IN-- action    SEND_TO_WP        uid  18544   appc_ca_blk -1      type  NOWP
-IN-- new_stat  NO_CHANGE         mode 0     len         8       rq_id 61620
-IN-- req_info  CANCELMODE
-IN-- msg_name  -                  

Tue Jan 20 16:31:35 2015
DpHdlDeadWp: W20 (pid=5076) terminated automatically

Tue Jan 20 16:32:35 2015
DpHdlDeadWp: W6 (pid=1552) terminated automatically
DpHdlDeadWp: W7 (pid=8028) terminated automatically

Tue Jan 20 16:32:55 2015
DpHdlDeadWp: W1 (pid=6044) terminated automatically
DpHdlDeadWp: W10 (pid=5420) terminated automatically

Tue Jan 20 16:39:15 2015
DpHdlDeadWp: W9 (pid=8160) terminated automatically

Tue Jan 20 16:39:35 2015
DpHdlDeadWp: W12 (pid=5432) terminated automatically

Tue Jan 20 16:39:55 2015
DpHdlDeadWp: W8 (pid=5096) terminated automatically

Tue Jan 20 16:40:35 2015
DpHdlDeadWp: W22 (pid=1804) terminated automatically

Tue Jan 20 17:10:36 2015
DpHdlDeadWp: W5 (pid=2452) terminated automatically

Tue Jan 20 17:29:16 2015
DpHdlDeadWp: W8 (pid=8120) terminated automatically

Tue Jan 20 17:31:56 2015
DpHdlDeadWp: W11 (pid=2856) terminated automatically

Tue Jan 20 17:32:16 2015
DpHdlDeadWp: W1 (pid=740) terminated automatically

Tue Jan 20 17:42:17 2015
DpHdlDeadWp: W14 (pid=5928) terminated automatically

Tue Jan 20 17:42:37 2015
DpHdlDeadWp: W3 (pid=3684) terminated automatically

Tue Jan 20 20:00:37 2015
DpWpConf: requested operation mode switch:
DIA 17->15 BTC 4->6 UPD 2->2 UP2 1->1 ENQ 1->1 SPO 1->1 RESTR 0->0 DYN 2->2
***LOG Q0V=> DpWpConf, mod_req (014DIABTC) [dpxxdisp.c   15852]
***LOG Q0V=> DpWpConf, mod_req (004DIABTC) [dpxxdisp.c   15852]
DpAdaptWppriv_max_no : 12 -> 10
DpWpConf: operation mode switch completed

Tue Jan 20 21:06:07 2015
DpHdlDeadWp: W22 (pid=6096) terminated automatically

Tue Jan 20 21:48:48 2015
DpHdlDeadWp: W22 (pid=1164) terminated automatically

Tue Jan 20 22:22:10 2015
***LOG Q0I=> NiIRead: P=127.0.0.1:64745; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 63/sock 996
    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:64745; L=127.0.0.1:3200) [nixxi.cpp    5082]

Tue Jan 20 23:25:09 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49596: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 960
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49596) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

Tue Jan 20 23:25:32 2015

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)
*              failed.
*              Please check the trace file of the message server.
*
*  TIME        Tue Jan 20 23:25:32 2015
*  RELEASE     720
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -36
*  MODULE      msxxi.c
*  LINE        2852
*  COUNTER     8024
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2604]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2675]

Tue Jan 20 23:25:46 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12865]
MBUF state LOADING
***LOG Q0I=> NiIRead: P=127.0.0.1:49788; L=127.0.0.1:3200: recv (10053: WSAECONNABORTED: Software caused connection abort) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 57/sock 1296
    (SI_ECONN_BROKEN/10053; I4; ST; P=127.0.0.1:49788; L=127.0.0.1:3200) [nixxi.cpp    5082]
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF

Tue Jan 20 23:51:32 2015
*** WARNING => DpEnvCheck: no answer from msg server since 20 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8486]

Tue Jan 20 23:51:52 2015
*** WARNING => DpEnvCheck: no answer from msg server since 40 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8486]

Tue Jan 20 23:52:12 2015
*** WARNING => DpEnvCheck: no answer from msg server since 60 secs, but dp_ms_keepalive_timeout(300 secs) not reached [dpxxdisp.c   8486]

Tue Jan 20 23:52:13 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:49789: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 18/sock 1000
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:49789) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)
*              failed.
*              Please check the trace file of the message server.
*
*  TIME        Tue Jan 20 23:52:13 2015
*  RELEASE     720
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -36
*  MODULE      msxxi.c
*  LINE        2852
*  COUNTER     8335
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2604]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2675]

Tue Jan 20 23:52:17 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12865]

Tue Jan 20 23:52:31 2015
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF

Wed Jan 21 00:08:32 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50407: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 19/sock 1016
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50407) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)
*              failed.
*              Please check the trace file of the message server.
*
*  TIME        Wed Jan 21 00:08:32 2015
*  RELEASE     720
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -36
*  MODULE      msxxi.c
*  LINE        2852
*  COUNTER     8474
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2604]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2675]

Wed Jan 21 00:08:44 2015
***LOG Q0K=> DpMsAttach, mscon ( SAPCRMDEV) [dpxxdisp.c   12865]

Wed Jan 21 00:08:57 2015
DpMBufHwIdSet: set Hardware-ID

Wed Jan 21 00:09:04 2015
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
MBUF state ACTIVE
MBUF component UP
DpMsgProcess: 1 server in MBUF
DpMsgProcess: 1 server in MBUF

Wed Jan 21 00:38:15 2015
*** WARNING => DpITimeCheck: suspicious forward jump of current time 1421818695, last time 1421817252 [dpxxdisp.c   9153]
DpITimeCheck: new time seems to be OK now, I accept 1421818695
*** WARNING => DpTimeCheck: huge diff between current 1421817312 and next schedule (act_time 1421818695 delay 120 [dpxxdisp.c   9326]

Wed Jan 21 00:38:34 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50773: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1292
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50773) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       Reading data from the message server (SAPCRMDEV / 3900)
*              failed.
*              Please check the trace file of the message server.
*
*  TIME        Wed Jan 21 00:38:34 2015
*  RELEASE     720
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -36
*  MODULE      msxxi.c
*  LINE        2852
*  COUNTER     8507
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
MBUF state OFF
MBUF component DOWN
*** ERROR => DpMsgProcess: MsReceive () -> MSENILAYER, partner:                      [dpxxdisp.c   2604]
*** ERROR => DpMsgProcess: disconnect from msg_server [dpxxdisp.c   2675]
*** ERROR => MsISnd2: not_attached [msxxi.c      1510]
***LOG Q0N=> DpRqNoWpHandle, MsSnd2Type () [dpxxdisp.c   5557]

Wed Jan 21 00:38:39 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:38:45 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:39:05 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:39:25 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:39:50 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:40:15 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:40:35 2015
*** ERROR => MsIAttachEx: NiBufConnect to SAPCRMDEV/3900 failed (rc=NIECONN_PENDING) [msxxi.c      718]

Wed Jan 21 00:59:28 2015
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50894: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1292
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50894) [nixxi.cpp    5082]
***LOG Q0I=> NiIRead: P=192.168.101.63:3900; L=192.168.101.63:50894: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5082]
*** ERROR => NiIRead: SiRecv failed for hdl 20/sock 1292
    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:50894) [nixxi.cpp    5082]
*** ERROR => MsINiRead: NiBufReceive failed (NIECONN_BROKEN) [msxxi.c      2833]

*****************************************************************************
*
*  LOCATION    SAP-Dispatcher SAPCRMDEV_CRD_00 on host SAPCRMDEV
*  ERROR       The connection to the specified message server
*              (SAPCRMDEV / 3900) failed. Please check the trace file
*              of the message server. If a network filter has
*              been activated (see message server parameters
*              ms/acl_file_...) please check the configuration
*
*  TIME        Wed Jan 21 00:59:28 2015
*  RELEASE     720
*  COMPONENT   MS (message handling interface, multithreaded)
*  VERSION     4
*  RC          -35
*  MODULE      msxxi.c
*  LINE        2861
*  COUNTER     8510
*
*****************************************************************************

*** ERROR => MsIReadFromHdl: NiRead (rc=NIECONN_BROKEN) [msxxi.c      1867]
***LOG Q1K=> MsIAttachEx: StoC check failed, Kernel not compatible with system (rc=-100) [msxxi.c      820]
*** ERROR => Kernel incompatible to already connected instances (see dev_ms for details) [dpxxdisp.c   12810]

Wed Jan 21 00:59:41 2015
DpHalt: shutdown server >SAPCRMDEV_CRD_00                        < (normal)
DpHalt: stop work processes

Wed Jan 21 00:59:42 2015
SoftCancel request for T39 U19749 M0 received from DISPATCHER
DpHdlSoftCancel: wp 0 inside critical section, do not interrupt now
SoftCancel request for T11 U12 M0 received from DISPATCHER
SoftCancel request for T18 U19 M0 received from DISPATCHER
SoftCancel request for T21 U22 M0 received from DISPATCHER
SoftCancel request for T23 U24 M0 received from DISPATCHER

Wed Jan 21 01:01:06 2015
*** ERROR => DpWpKill(7968, SIGUSR2): kill failed [dpxxtool.c   2792]
*** ERROR => DpWpKill(584, SIGUSR2): kill failed [dpxxtool.c   2792]
*** ERROR => DpWpKill(3088, SIGUSR2): kill failed [dpxxtool.c   2792]
DpHalt: stop gateway
DpHalt: stop icman
DpHalt: terminate gui connections
DpHalt: wait for end of work processes
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:07 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:09 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:10 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:12 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:13 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:14 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:16 2015
DpHalt: waiting for termination of work processes ...

Wed Jan 21 01:01:17 2015
DpHalt: wait for end of gateway
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:18 2015
*** ERROR => 21.01.2015 06:01:18.185 PID=6404, TID=5220 SapTimer warning. Took time keeping ownership. PID=6404, old owner PID:5888 [dpxxdisp.c   21728]
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:19 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:20 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:21 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:22 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:24 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:26 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:27 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:28 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:29 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:30 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:32 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:33 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:34 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:36 2015
DpHalt: waiting for termination of gateway ...

Wed Jan 21 01:01:37 2015
*** ERROR => [DpProcDied] Process died  (PID:3924  HANDLE:964  rc:0x0) [dpnttool2.c  147]
DpHalt: wait for end of icman
*** ERROR => [DpProcDied] Process died  (PID:7016  HANDLE:972  rc:0x0) [dpnttool2.c  147]
DpHalt: not attached to the message server
DpHalt: cleanup EM
***LOG Q05=> DpHalt, DPStop ( 6404) [dpxxdisp.c   11745]

former_member182657
Active Contributor
0 Kudos

Hi,

Could you share your system host file and are you able to resolve host name of the system.

Regards,

former_member196664
Participant
0 Kudos

Gaurav/All,

I found why java Server0 is shutting. It is because of UME service failure from the ABAP side. If disp+work is not shutting down on ABAP side, then Server0 should be fine in java side. I have also attached hosts file from C:\Windows\System32\drivers\etc. Pls help.


com.sap.engine.frame.ServiceException: Start of UME service failed. Check help topic "Start of UME Service Failed". Technical details: Connect to SAP gateway failed
Connection parameters: TYPE=A DEST=<Java Rfc client> ASHOST=SAPCRMDEV SYSNR=00 PCS=1

LOCATION    CPIC (TCP/IP) on local host SAPCRMDEV with Unicode
ERROR       partner '192.168.101.63:3300' not reached
TIME        Wed Jan 21 01:02:46 2015
RELEASE     720
COMPONENT   NI (network interface)
VERSION     40
RC          -10
MODULE      nixxi.cpp
LINE        3283
DETAIL      NiPConnect2: 192.168.101.63:3300
SYSTEM CALL connect
ERRNO       10061
ERRNO TEXT  WSAECONNREFUSED: Connection refused
COUNTER     2

at com.sap.security.core.server.ume.service.UMEServiceFrame.start(UMEServiceFrame.java:407)
at com.sap.engine.frame.ApplicationFrameAdaptor.start(ApplicationFrameAdaptor.java:31)
at com.sap.engine.core.service630.container.ServiceRunner.startApplicationServiceFrame(ServiceRunner.java:214)
at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:144)
at com.sap.engine.frame.core.thread.Task.run(Task.java:64)
at com.sap.engine.core.thread.impl5.SingleThread.execute(SingleThread.java:83)
at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:109)
Caused by: com.sap.security.core.persistence.datasource.PersistenceException: Connect to SAP gateway failed
Connection parameters: TYPE=A DEST=<Java Rfc client> ASHOST=SAPCRMDEV SYSNR=00 PCS=1

LOCATION    CPIC (TCP/IP) on local host SAPCRMDEV with Unicode
ERROR       partner '192.168.101.63:3300' not reached
TIME        Wed Jan 21 01:02:46 2015
RELEASE     720
COMPONENT   NI (network interface)
VERSION     40
RC          -10
MODULE      nixxi.cpp
LINE        3283
DETAIL      NiPConnect2: 192.168.101.63:3300
SYSTEM CALL connect
ERRNO       10061
ERRNO TEXT  WSAECONNREFUSED: Connection refused
COUNTER     2

at com.sap.security.core.persistence.datasource.imp.R3PersistenceBase.newPersistenceException(R3PersistenceBase.java:236)
at com.sap.security.core.persistence.datasource.imp.R3PersistenceBase.init(R3PersistenceBase.java:491)
at com.sap.security.core.persistence.imp.PrincipalDatabagFactoryInstance.<init>(PrincipalDatabagFactoryInstance.java:447)
at com.sap.security.core.persistence.imp.PrincipalDatabagFactory.newInstance(PrincipalDatabagFactory.java:164)
at com.sap.security.core.persistence.imp.PrincipalDatabagFactory.getInstance(PrincipalDatabagFactory.java:117)
at com.sap.security.core.persistence.imp.PrincipalDatabagFactory.getInstance(PrincipalDatabagFactory.java:63)
at com.sap.security.core.InternalUMFactory.initializeUME(InternalUMFactory.java:221)
at com.sap.security.core.server.ume.service.UMEServiceFrame.start(UMEServiceFrame.java:288)
... 6 more

[Framework -> criticalShutdown] Core service com.sap.security.core.ume.service failed. J2EE Engine cannot be started.
Jan 21, 2015 1:02:46 AM             com.sap.engine.core.Framework [SAPEngine_System_Thread[impl:5]_70] Fatal: Critical shutdown was invoked. Reason is: Core service com.sap.security.core.ume.service failed. J2EE Engine cannot be started.
Heap
par new generation   reserved 614400K, committed 614400K, used 39570K [0x000000007fff0000, 0x00000000a57f0000, 0x00000000a57f0000)
  eden space 409600K,   1% used [0x000000007fff0000, 0x00000000806be6a8, 0x0000000098ff0000)
  from space 102400K,  31% used [0x000000009f3f0000, 0x00000000a13c6350, 0x00000000a57f0000)
  to   space 102400K,   0% used [0x0000000098ff0000, 0x0000000098ff0000, 0x000000009f3f0000)
concurrent mark-sweep generation reserved 1482752K, committed 1482752K, used 0K [0x00000000a57f0000, 0x00000000ffff0000, 0x00000000ffff0000)
concurrent-mark-sweep perm gen reserved 524288K, committed 524288K, used 34760K [0x00000000ffff0000, 0x000000011fff0000, 0x000000011fff0000)

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
# 127.0.0.1       localhost
# ::1             localhost

Thanks,

Kavitha Rajan.

former_member196664
Participant
0 Kudos

Just want to add that both ABAP and Java (CRD and CDJ) are in the same server with separate DB.

former_member182657
Active Contributor
0 Kudos

Hi Kavitha,


# localhost name resolution is handled within DNS itself.

# 127.0.0.1       localhost

# ::1             localhost

Could you edit above and make it as like

127.0.0.1  localhost.localdomain  localhost

<Server IP>   <Hostname>

save the entries and try to start the server again.

Regards,

Matt_Fraser
Active Contributor
0 Kudos

I suspect part of the root of the problem may be a lack of available dialog work processes on the ABAP system. You are getting the error:

*** WARNING => DpRqServiceQueue: timeout of HIGH PRIO msg, return DP_CANT_HANDLE_REQ

This happens very close to the beginning of the dispatcher trace. What is probably happening here is the system is trying to generate programs during startup (which is common when importing a new version of the kernel -- or reverting to an older version), for which it needs available dialog WPs, but all the WPs are busy. Make sure that you have at least as many DIA WPs as non-DIA WPs configured (i.e., sum up the BTC, ENQ, VB, VB2, and SPO work processes, and set DIA to that number (or higher)).

I also see the setting:

rdisp/dynamic_wp_check : 1

This should be ok with NetWeaver 7.02, but just to be safe, you might want to revert to the 7.01 behavior, which is to disallow dynamic work processes (at least while troubleshooting -- once the system starts up in a stable manner, you can experiment with re-enabling them). To do this, set the following parameters in your instance profile:

rdisp/wp_no_restricted = 0

rdisp./configurable_wp_no = 0

rdisp/dynamic_wp_check = FALSE

Also make sure that rdisp/wp_max_no is set to a number equal to the sum of all the other rdisp/wp_no_* settings (dia, vb2, btc, vb, spo, and enq), and that rdisp/wp_no_dia is equal to or greater than the sum of the "non-dia" settings.

Regards,

Matt

hemanth2
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Kavitha,

UME is a core service and unless it starts, the J2EE server nodes wil not reach the running state. So you need to make sure that the ABAP part is fully functional.


Apart from the point Matt mentioned, check if the same stack SAPEXE* and SAPEXEDB* files were used when you did the kernel upgrade.

Check the output of:

sapcontrol -nr 00 -function GetVersionInfo

The versions should be same for all files.

regards,
Hemanth

former_member196664
Participant
0 Kudos

Hi All,

I changed the hosts file as recommended by Gaurav and monitored couple of days, system is continuously up now. No shutdown in the non-business hours. Reverting kernel and host file changes, fixed my issue of dispatcher shutting down. Thank you ALL for helping me in resolving the issues.

Regards,

Kavitha Rajan.


hemanth2
Product and Topic Expert
Product and Topic Expert
0 Kudos

Great news Kavitha .

Answers (3)

Answers (3)

former_member182657
Active Contributor
0 Kudos

Oh Great to know your issue has been resolved now !!

Nice experience for me also.

Regards,

Gaurav

Matt_Fraser
Active Contributor
0 Kudos

Have you confirmed that the same 721_EXT patch 400 kernel is being used for the message server and the dispatcher instance? The error seems to be complaining about a kernel mismatch between components.

I think the dispatcher is crashing because it's losing its connection to the message server. So, probably the message server is the first piece to troubleshoot.

Regards,

Matt

former_member196664
Participant
0 Kudos

Hi Matt,

I replaced entire kernel under usr/sap/SID/sys/exe. Issue still occurred. This is CRM abap. Please let me know where to check version mismatch?

Thanks,
Kavitha Rajan.

Reagan
Advisor
Advisor
0 Kudos

Connect as crdadm and run this command.

sapcontrol -nr 00 -function GetVersionInfo

former_member196664
Participant
0 Kudos

Hi Reagan,

C:\Users\crdadm>sapcontrol -nr 00 -function GetVersionInfo

14.01.2015 11:33:48
GetVersionInfo
OK
Filename, VersionInfo, Time
F:\usr\sap\CRD\DVEBMGS00\exe\sapstartsrv.exe, 721, patch 400, changelist 1542710
, optU (EXT), NTAMD64, 2014 12 07 17:58:06
F:\usr\sap\CRD\DVEBMGS00\exe\disp+work.exe, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:05:54
F:\usr\sap\CRD\DVEBMGS00\exe\gwrd.exe, 721, patch 400, changelist 1542710, optU
(EXT), NTAMD64, 2014 12 07 17:52:08
F:\usr\sap\CRD\DVEBMGS00\exe\msg_server.exe, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 17:54:36
F:\usr\sap\CRD\DVEBMGS00\exe\dboraslib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\dbmssslib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\dbdb2slib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\dbdb4slib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\dbdb6slib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\dbsybslib.dll, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 18:25:28
F:\usr\sap\CRD\DVEBMGS00\exe\enserver.exe, 721, patch 400, changelist 1542710, o
ptU (EXT), NTAMD64, 2014 12 07 15:16:37
F:\usr\sap\CRD\DVEBMGS00\exe\icman.exe, 721, patch 400, changelist 1542710, optU
(EXT), NTAMD64, 2014 12 07 14:10:17
F:\usr\sap\CRD\DVEBMGS00\exe\sapwebdisp.exe, 721, patch 400, changelist 1542710,
optU (EXT), NTAMD64, 2014 12 07 14:11:29
F:\usr\sap\CRD\DVEBMGS00\exe\jcontrol.exe, 721, patch 400, changelist 1542710, o
ptU (EXT), NTAMD64, 2014 12 07 16:18:01
F:\usr\sap\CRD\DVEBMGS00\exe\jlaunch.exe, 721, patch 400, changelist 1542710, op
tU (EXT), NTAMD64, 2014 12 07 16:19:05
F:\usr\sap\CRD\DVEBMGS00\exe\jstart.exe, 721, patch 400, changelist 1542710, opt
U (EXT), NTAMD64, 2014 12 07 16:16:48

C:\Users\crdadm>

Thanks,

Kavitha Rajan.

Sriram2009
Active Contributor
0 Kudos

Hi Kavitha

Could you refer the SAP Notes  point number "5. Additional steps before starting the system with the new kernel"

1713986 - Installation of kernel 721 (EXT)


BR

SS

former_member182657
Active Contributor
0 Kudos

Hi,


*** ERROR => NiIRead: SiRecv failed for hdl 17/sock 984


    (SI_ECONN_BROKEN/10054; I4; ST; P=192.168.101.63:3900; L=192.168.101.63:58283) [nixxi.cpp    5082]

Issue seems with network connections at your end,kindly refer SAP Notes  1478133 - MS SQL Server: How to analyze connection problems to database server

  413330 - Network timeouts

Also check for port entry of 3900 under services.If possible share trans.log by executing R3trans -d

Regards,

Gaurav