SAP CI instance cannot start when ECC6.0 HA Installation

[Former Member]

Hi, Experts,

I am installing ERP 6.0 - EHP4 Ready SR1 on AIX HA, ABAP only, and the AIX version is 6.1.4 SP4 and with oracle database.

I followed the sap instruction install central services instance with virtul hostname on node A and install Database on node A,

erverything is ok, and when I install CI instance on Node A, the step start CI instance is failed. I manual start CI, messager server cannot start, dispatch shutdown.

I check the following log, including dev_disp, and find the error.

MtxInit: 30000 0 0

• ERROR => shmget(10010,160000000,8672) (22: Invalid argument) [shmux.c 1

556]

• ERROR => shmget(10010,160000000,8672) (22: Invalid argument) [shmux.c 1

556]

• ERROR => Shm Pool Create Error. PoolKey=10, Key=1. [shmux.c 2079]

• ERROR => DpSysAdmExtCreate: ShmCreate (ATTACH) failed (3) [dpxxtool2.c 652]

• ERROR => DpIPCInit2: DpSysAdmExtCreate [dpxxtool2.c 369]

• DP_FATAL_ERROR => DpSapEnvInit: DpIPCInit2

• DISPATCHER EMERGENCY SHUTDOWN ***

I stop all instance and run cleanipc 00 remove, and restart it, the error is same and also I restart the server, the error is same.

I checked the profile with ./sappfpar check pf=/usr/sap/PRD/SYS/profile/PRD_DVEBMGS00_SAPPRD1

and found just some warning without error.

could you give me some suggestion for this , thanks a lot, below is dev_disp and sppfpar check log for your reference.

[former_member189546]

Hello,

***LOG Q0I=> NiIBindSocket: bind (67: Address already in use) http://nixxi.cpp3239

• ERROR => NiIBindSocket: SiBind failed for hdl 1 / sock 8

(SI_EPORT_INUSE/67; I4; ST; 0.0.0.0:3200) http://nixxi.cpp 3239

Please check the note 164937 and use netstat to check the availability

of the services.

164937 - NiPBind: service 'sap????' in use

As the error messages ishow, port 3200 is used by an other

application, therefore MS cannot bind to this port

Check services filke

#540379 - Ports and services used by SAP

Please also check

http://www-03.ibm.com/systems/p/security/feature/enhancements.html

regards,

John Feely

[markus_doehr2]

What's the output of

ipcs -m

Markus

[Former Member]

T ID KEY MODE OWNER GROUP

Shared Memory:

m 1048576 0x900dc0de rw----- root system

m 1048577 0x7800006f --rw-rw-rw- root system

m 1048578 0x78000052 --rw-rw-rw- root system

m 3 0x670030c3 rw-rr-- root system

m 4 0x680030c3 rw-rr-- root system

m 5 0x700030c3 rw----- root system

m 6 0xffffffff rw-rw-- root haemrm

m 26214500 0xffffffff --rw-rw-rw- prdadm sapsys

m 25165931 0x0000271a rw-r--- prdadm sapsys

m 14680176 0x6f9a1cb0 rw-r--- oraprd dba

m 1048689 0x00004dbe --rw-rw-rw- root sapsys

m 17825906 0xffffffff --rw-rw-rw- prdadm sapsys

m 10486681 0x0382be84 --rw-rw-rw- prdadm sapsys

[markus_doehr2]

So it seems that there are still fragments of shared memory allocated.

Can you try

cleanipc 00 rm (as user 'root')
shlibclean

and then again

ipcs -m

Markus

[Former Member]

Hi, Markrus,

We are starting to reinstall os and SAP, if the issue still persist, I will update it.

Thanks again for your help.

Will

[Former Member]

I reinstall OS and restart SAP, ASCS and DB instance is ok, but when CI instance installation, the error happened again in step "start CI instance", I found the CI instance cannot start, check dev_disp log, got info as below

error:

***LOG Q0I=> NiIBindSocket: bind (67: Address already in use) [nixxi.cpp3239]

• ERROR => NiIBindSocket: SiBind failed for hdl 1 / sock 8

(SI_EPORT_INUSE/67; I4; ST; 0.0.0.0:3200) [nixxi.cpp 3239]

• ERROR => DpCommInit: NiBufListen (rc=-4) [dpxxdisp.c 10088]

• DP_FATAL_ERROR => DpSapEnvInit: DpCommInit

• DISPATCHER EMERGENCY SHUTDOWN ***

[markus_doehr2]

Are you shell limits (for <sid>adm) too low?

Markus

[Former Member]

Hi, Markrus,

Thanks for your quick response and I have already set all users shell limits to -1 including root , prdadm and oraprd.

and restart OS, but error still exist.

[former_member189546]

Hello,

Perform command to clear any faulty shared memory at OS level:

cleanipc <instance number> remove

i.e cleanipc 00 remove (if the instance number is 00)

If possible reboot OS

There maybe a problem with pool 10

On os level run

sappfpar pf=<instance-profile> check

Check for errors with pool 10.

regards,

John Feely

[Former Member]

hi, John,

I cleanipc and also reboot the server, But the error still exist.

also I check the profile pool without error.

my server phsical memory is 64GB and I set swap 50GB.

I attached the latest dev_disp log and profile check log as below for reference

Will

[Former Member]

---

trc file: "dev_disp.new", trc level: 1, release: "701"

-

---

sysno 00

sid PRD

systemid 324 (IBM RS/6000 with AIX)

relno 7010

patchlevel 0

patchno 32

intno 20020600

make: single threaded, Unicode, 64 bit, optimized

pid 123734

Sat Jun 12 22:49:48 2010

kernel runs with dp version 241000(ext=110000) (@(#) DPLIB-INT-VERSION-241000-UC)

length of sys_adm_ext is 576 bytes

• SWITCH TRC-HIDE on ***

***LOG Q00=> DpSapEnvInit, DPStart (00 123734) [dpxxdisp.c 1286]

shared lib "dw_xml.so" version 32 successfully loaded

shared lib "dw_xtc.so" version 32 successfully loaded

shared lib "dw_stl.so" version 32 successfully loaded

shared lib "dw_gui.so" version 32 successfully loaded

shared lib "dw_mdm.so" version 32 successfully loaded

rdisp/softcancel_sequence : -> 0,5,-1

use internal message server connection to port 3900

MtxInit: 30000 0 0

• ERROR => shmget(10010,160000000,8672) (22: Invalid argument) [shmux.c 1556]

• ERROR => shmget(10010,160000000,8672) (22: Invalid argument) [shmux.c 1556]

• ERROR => Shm Pool Create Error. PoolKey=10, Key=1. [shmux.c 2079]

• ERROR => DpSysAdmExtCreate: ShmCreate (ATTACH) failed (3) [dpxxtool2.c 652]

• ERROR => DpIPCInit2: DpSysAdmExtCreate [dpxxtool2.c 369]

• DP_FATAL_ERROR => DpSapEnvInit: DpIPCInit2

• DISPATCHER EMERGENCY SHUTDOWN ***

NiWait: sleep (5000ms) ...

NiISelect: timeout 5000ms

NiISelect: maximum fd=1

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Sat Jun 12 22:49:53 2010

NiISelect: TIMEOUT occured (5000ms)

• WARNING => DpHalt: sys_adm has not been initialized! [dpxxdisp.c 10373]

DpHalt: shutdown server >< (normal)

DpJ2eeDisableRestart

Switch off Shared memory profiling

ShmProtect( 57, 3 )

ShmProtect( key 57 valid )

ShmProtect( slot Index 56 )

ShmProtect: Key=57 not attached

• ERROR => ShmProtect(SHM_PROFILE [dpxxprf.c 486]

DpShmPrfSwitch : State already set (OFF)

ShmProtect( 57, 1 )

ShmProtect( key 57 valid )

ShmProtect( slot Index 56 )

ShmProtect: Key=57 not attached

• ERROR => ShmProtect(SHM_PROFILE [dpxxprf.c 494]

SigIGenAction: caught signal 11 (pid=123734)

SigIRegisterRoutine: handler for signal 14 installed (DpSigAlrm)

SigIUnblock: signal 11 was in UNBLOCKED mode

-

---

Signal 11 encountered.

=> 64 bit R/3 Kernel

=> 64 bit AIX Kernel

=> Heap limit = unlimited

=> Stack limit = 4294967296

=> Core limit = unlimited

=> File size limit = unlimited

=> Heap address = 0x0x113207740

=> Stack address = 0xfffffffffffc810

=> Stack low = 0xfffffffffff9f20

=> Stack high = 0xffffffffffff880

-

---

=> Stack Trace:

DpVmcCheckDebugProxy() at 0x1015681e0

DpHalt() at 0x101599e40

DpFatalErr() at 0x10159c69c

DpSapEnvInit() at 0x101598584

DpMain() at 0x10159ce14

nlsui_main() at 0x1019ec510

-

---

=> CPU Registers:

msr = 0xa00000000000d032 iar = 0x00000001015681e0

ctr = 0x0900000000594700 lr = 0x0000000101599e60

xer = 0x0000000020000037 cr = 0x000000002252422a

r00 = 0x0000000000000001 r01 = 0x0fffffffffffc810

r02 = 0x00000001104a17f8 r03 = 0x0000000110c39e78

r04 = 0x0000000000000000 r05 = 0x0000000000000008

r06 = 0xffffffffffff800a r07 = 0x0000000000000000

r08 = 0x00000000100423c1 r09 = 0x00000000100423c1

r10 = 0x0000000000000000 r11 = 0x0000000000000000

r12 = 0x000000002252522a r13 = 0x0000000112d7add8

r14 = 0x0000000000000001 r15 = 0x0ffffffffffff900

r16 = 0x0ffffffffffff918 r17 = 0x0000000110463c30

r18 = 0x0ffffffffffffed0 r19 = 0x09fffffff000c4a8

r20 = 0xbadc0ffee0ddf00d r21 = 0xbadc0ffee0ddf00d

r22 = 0xbadc0ffee0ddf00d r23 = 0x0000000110001428

r24 = 0x0000000110c39e98 r25 = 0x0000000110c3e440

r26 = 0x000000000000003c r27 = 0x0000000110c3b680

r28 = 0x0000000110354278 r29 = 0x0000000110001428

r30 = 0x000000011001b1c8 r31 = 0x0000000000000000

-

---

***LOG Q0E=> SigIGenAction, signal ( 11) [sigux.c 1067]

SigIGenAction: call exithandler DpHalt(FALSE)

• WARNING => DpHalt: sys_adm has not been initialized! [dpxxdisp.c 10373]

DpHalt: shutdown server >< (normal)

DpJ2eeDisableRestart

Switch off Shared memory profiling

ShmProtect( 57, 3 )

ShmProtect( key 57 valid )

ShmProtect( slot Index 56 )

ShmProtect: Key=57 not attached

• ERROR => ShmProtect(SHM_PROFILE [dpxxprf.c 486]

DpShmPrfSwitch : State already set (OFF)

ShmProtect( 57, 1 )

ShmProtect( key 57 valid )

ShmProtect( slot Index 56 )

ShmProtect: Key=57 not attached

• ERROR => ShmProtect(SHM_PROFILE [dpxxprf.c 494]

[Former Member]

================================================================================

== Checking profile: /usr/sap/PRD/SYS/profile/PRD_DVEBMGS00_SAPPRD1

================================================================================

***WARNING: Unexpected parameter: j2ee/dbhost =SAPPRD

***WARNING: Unexpected parameter: j2ee/dbname =PRD

***WARNING: Unexpected parameter: j2ee/dbtype =ora

Shared memory disposition overview

================================================================

Shared memory pools

Key: 10 Pool

Size configured.....: 160000000 ( 152.6 MB)

Size min. estimated.: 157422202 ( 150.1 MB)

Advised Size........: 160000000 ( 152.6 MB)

Key: 40 Pool for database buffers

Size configured.....: 112000000 ( 106.8 MB)

Size min. estimated.: 108975040 ( 103.9 MB)

Advised Size........: 112000000 ( 106.8 MB)

Shared memories inside of pool 10

Key: 1 Size: 2500 ( 0.0 MB) System administration

Key: 2 Size: 6161540 ( 5.9 MB) Disp. administration tables

Key: 11 Size: 500000 ( 0.5 MB) Factory calender buffer

Key: 12 Size: 6000000 ( 5.7 MB) TemSe Char-Code convert Buf.

Key: 13 Size: 60500000 ( 57.7 MB) Alert Area

Key: 14 Size: 4400000 ( 4.2 MB) Presentation buffer

Key: 16 Size: 22400 ( 0.0 MB) Semaphore activity monitoring

Key: 17 Size: 2672386 ( 2.5 MB) Roll administration

Key: 30 Size: 17408 ( 0.0 MB) Taskhandler runtime admin.

Key: 31 Size: 4206000 ( 4.0 MB) Dispatcher request queue

Key: 33 Size: 10240000 ( 9.8 MB) Table buffer, part.buffering

Key: 34 Size: 4194304 ( 4.0 MB) Enqueue table

Key: 51 Size: 3200000 ( 3.1 MB) Extended memory admin.

Key: 52 Size: 40000 ( 0.0 MB) Message Server buffer

Key: 54 Size: 4202496 ( 4.0 MB) Export/Import buffer

Key: 55 Size: 8192 ( 0.0 MB) Spool local printer+joblist

Key: 57 Size: 1048576 ( 1.0 MB) Profilparameter in shared mem

Key: 58 Size: 4096 ( 0.0 MB) Enqueue ID for reset

Shared memories inside of pool 40

Key: 42 Size: 7232992 ( 6.9 MB) DB TTAB buffer

Key: 43 Size: 32534392 ( 31.0 MB) DB FTAB buffer

Key: 44 Size: 7958392 ( 7.6 MB) DB IREC buffer

Key: 45 Size: 4886392 ( 4.7 MB) DB short nametab buffer

Key: 46 Size: 20480 ( 0.0 MB) DB sync table

Key: 47 Size: 3073024 ( 2.9 MB) DB CUA buffer

Key: 48 Size: 300000 ( 0.3 MB) Number range buffer

Key: 49 Size: 2968344 ( 2.8 MB) Spool admin (SpoolWP+DiaWP)

Shared memories outside of pools

Key: 3 Size: 13714400 ( 13.1 MB) Disp. communication areas

Key: 4 Size: 511648 ( 0.5 MB) statistic area

Key: 6 Size: 319488000 ( 304.7 MB) ABAP program buffer

Key: 7 Size: 14838 ( 0.0 MB) Update task administration

Key: 8 Size: 67108964 ( 64.0 MB) Paging buffer

Key: 9 Size: 134217828 ( 128.0 MB) Roll buffer

Key: 18 Size: 917604 ( 0.9 MB) Paging adminitration

Key: 19 Size: 30000000 ( 28.6 MB) Table-buffer

Key: 41 Size: 25010000 ( 23.9 MB) DB statistics buffer

Key: 62 Size: 85983232 ( 82.0 MB) Memory pipes

Key: 63 Size: 409600 ( 0.4 MB) ICMAN shared memory

Key: 64 Size: 4202496 ( 4.0 MB) Online Text Repository Buf.

Key: 65 Size: 4202496 ( 4.0 MB) Export/Import Shared Memory

Key: 1002 Size: 400000 ( 0.4 MB) Performance monitoring V01.0

Key: 58900100 Size: 4096 ( 0.0 MB) SCSA area

Nr of operating system shared memory segments: 17

Shared memory resource requirements estimated

================================================================

Total Nr of shared segments required.....: 17

System-imposed number of shared memories.: 1000

Shared memory segment size required min..: 319488000 ( 304.7 MB)

System-imposed maximum segment size......: 35184372088832 (33554432.0 MB)

Swap space requirements estimated

================================================

Shared memory....................: 915.7 MB

..in pool 10 150.1 MB, 98% used

..in pool 40 103.9 MB, 97% used

..not in pool: 654.4 MB

Processes........................: 165.2 MB

Extended Memory .................: 4092.0 MB

-

---

Total, minimum requirement.......: 5172.9 MB

Process local heaps, worst case..: 1907.3 MB

Total, worst case requirement....: 7080.2 MB

Errors detected..................: 0

Warnings detected................: 3