on 04-15-2014 6:48 PM
Hi All,
We are facing issues with ICM only in one of the app servers. It is taking more load and going to emergency mode only in one app. I looked at ICM services in both instances
No. Protocol Service Name/Port Host Name Keep Alive Proc.Timeo Actv External Bind
1 HTTP 8000 Host2 9,999,999 1-
2 SMTP 0 Host2 30 9,999,999
icm/server_port_0 = PROT=HTTP,PORT=8000,PROCTIMEOUT=-1
No. Protocol Service Name/Port Host Name Keep Alive Proc.Timeo Actv External Bind
1 HTTP 8001 Host1 30 9,999,999
2 SMTP 0 Host1 30 9,999,999
icm/server_port_0 = PROT=HTTP,PORT=8001
Problem is with Host2. I see differences here. Please help fixing the issue. Which timeout parameter should I modify? If I just remove Proctimeout=-1, will it fix issue. Production is getting affected. PLs advise ASAP.
Thanks,
Kavitha Rajan.
Kavitha,
What are the configuration settings for the Web Dispatcher in front of this ABAP stack? Ideally these are stand alone Web Dispatchers.
How are the users hitting the ITS here? Directly thru a browser? Thru a portal? Just trying to get an idea of the access flow here to understand why one server seems to be hit more than the others.
It sounds like you're seeing an unusually large load on one of the servers - it does not sound like they are being balanced correctly.
Thanks,
Tim
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Tim,
Users are login using the url http://host1/sap/bc/bsp/sap/crm_ui_frame/default.htm
Host1 is CI. Host 2 is App server which has issue. We noticed the ICM connections spiking to 5000+ and it still remains 5000+ even after 2 days. CI has max 50 connections only. After implementing above mentioned 7 parameters in CI and APP instance profile, now users are able to login even when the connections are 5000+. But still why App server only taking the load? Why the connections are not reducing when there is no business activities. Below are the parameter settings for web dispatcher. Please help.
ms/http_port = 8101
icm/max_conn = 2000
icm/max_sockets = 1024
icm/req_queue_len = 2000
icm/min_threads = 50
icm/max_threads = 120
mpi/total_size_MB = 100
#-----------------------------------------------------------------------
# SAP Web Dispatcher Ports
#-----------------------------------------------------------------------
icm/server_port_0 = PROT=HTTP,PORT=80
icm/server_port_1 = PROT=ROUTER,PORT=443
#icm/server_port_2 = PROT=HTTP,PORT=81$$
wdisp/HTTP/max_pooled_con = 2000
wdisp/HTTPS/max_pooled_con = 2000
wdisp/auto_refresh = 120
wdisp/HTTP/esid_support = 1
mpi/max_pipes = 4000
mpi/buffer_size = 65536
icm/max_sockets = 4000
icm/max_services = 100
icm/listen_queue_len = 512
icm/min_spare_threads = 3
icm/log_level = 0
icm/accept_remote_trace_level = 0
icm/ccms_monitoring = 1
icm/conn_timeout = 5000
icm/keep_alive_timeout = 600
rdisp/plugin_auto_logout = 0
icm/ccms_refresh_rate = 30
icm/HTTP/max_request_size_KB = 102400
Tahnsk,
Kavitha Rajan.
Kavitha,
You answered your own question 😉 You're funneling everyone thru that one server.
If you have multiple application servers like this and they are all suposed to render ITS services, I would very seriously explore the use of a dedicated Web Dispatcher server. It's really a service installed on an OS that has the ITS functions you reference as well as load balancing capabilities for ABAP (and J2EE stacks).
Check out this link: SAP Web Dispatcher - SAP Library - this should provide some decent guidance for you. I would suggest building out two web dispatchers for redundancy. If your network team can provide you a single web address like http://erp.domain.com then users can use this for everything in terms of access and the details are handled underneath automatically.
The help page covers architecture, sizing and SSL should you desire that. It's very lightweight from a server perspective - a single CPU with 2 or 4 GB of memory and can be configured to service many different backend systems at the same time.
I hope this is helpful!
Tim
Ok cool. Then if so, please post the content of the Web Dispatcher's instance profile, specfically the following sections:
#-----------------------------------------------------------------------
# Accesssability of Message Server
#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
# SAP Web Dispatcher Ports
#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
# SAP Web Dispatcher Redirects
#-----------------------------------------------------------------------
Here's an example of one from a previous life:
#-----------------------------------------------------------------------
# Accesssability of Message Server
#-----------------------------------------------------------------------
#rdisp/mshost = dw1.domain.com
#ms/http_port = 8101
wdisp/system_0 = SID=ABC,MSHOST=abc.domain.com,MSPORT=8101,SRCSRV=alias1.domain.com:80
wdisp/system_1 = SID=DEF,MSHOST=def.domain.com,MSPORT=8101,SRCSRV=alias2.domain.com:80
wdisp/system_2 = SID=GHI,MSHOST=ghi.domain.com,MSPORT=8100,SRCSRV=alias3.domain.com:80
wdisp/system_3 = SID=JKL,MSHOST=jkl.domain.com,MSPORT=8100,SRCSRV=alias4.domain.com:80
#-----------------------------------------------------------------------
# Configuration for medium scenario
#-----------------------------------------------------------------------
icm/max_conn = 500
icm/max_sockets = 1024
icm/req_queue_len = 500
icm/min_threads = 10
icm/max_threads = 50
mpi/total_size_MB = 80
#-----------------------------------------------------------------------
# SAP Web Dispatcher Ports
#-----------------------------------------------------------------------
icm/server_port_0 = PROT=HTTP,PORT=80,EXTBIND=1,PROCTIMEOUT=3600,HOST=abc.domain.com
icm/server_port_1 = PROT=HTTP,PORT=80,EXTBIND=1,PROCTIMEOUT=3600,HOST=def.domain.com
icm/server_port_2 = PROT=HTTP,PORT=80,EXTBIND=1,PROCTIMEOUT=3600,HOST=ghi.domain.com
icm/server_port_3 = PROT=HTTP,PORT=80,EXTBIND=1,PROCTIMEOUT=3600,HOST=jkl.domain.com
#-----------------------------------------------------------------------
# SAP Web Dispatcher Redirects
#-----------------------------------------------------------------------
# icm/HTTP/redirect_0 = PREFIX=/, TO=/irj/portal
# icm/HTTP/redirect_1 = PREFIX=/sap, TO=/sap/bc/nwbc
icm/HTTP/redirect_0 = PREFIX=/, FOR=abc.domain.com, TO=/sap/bc/nwbc, HOST=abc.domain.com
icm/HTTP/redirect_1 = PREFIX=/, FOR=def.domain.com, TO=/sap/bc/nwbc, HOST=def.domain.com
icm/HTTP/redirect_2 = PREFIX=/, FOR=ghi.domain.com, TO=/sap/bc/nwbc, HOST=ghi.domain.com
icm/HTTP/redirect_3 = PREFIX=/, FOR=jkl.domain.com, TO=/sap/bc/nwbc, HOST=jkl.domain.com
Please keep in mind that every ABAP stack and J2EE system have ITS elements, but that does not make them a Web Dispatcher. The config you detailed above seems like it came from an ABAP server - a web dispatcher builds on these core elements and adds items like redirects and load balancing (thru message server identifucation).
Tim
Hi Tim,
Below are the web-dispatcher Instance profile parameters.
SAPSYSTEMNAME = XXX
SAPGLOBALHOST = Host1
SAPSYSTEM = 02
INSTANCE_NAME = W02
DIR_CT_RUN = $(DIR_EXE_ROOT)\$(OS_UNICODE)\NTAMD64
DIR_EXECUTABLE = $(DIR_CT_RUN)
#-----------------------------------------------------------------------
# Accesssability of Message Server
#-----------------------------------------------------------------------
rdisp/mshost = Host1
ms/http_port = 8101
#-----------------------------------------------------------------------
# Configuration for medium scenario
#-----------------------------------------------------------------------
icm/max_conn = 2000
icm/max_sockets = 1024
icm/req_queue_len = 2000
icm/min_threads = 50
icm/max_threads = 120
mpi/total_size_MB = 100
#-----------------------------------------------------------------------
# SAP Web Dispatcher Ports
#-----------------------------------------------------------------------
icm/server_port_0 = PROT=HTTP,PORT=80
icm/server_port_1 = PROT=ROUTER,PORT=443
#icm/server_port_2 = PROT=HTTP,PORT=81$$
wdisp/HTTP/max_pooled_con = 2000
wdisp/HTTPS/max_pooled_con = 2000
wdisp/auto_refresh = 120
wdisp/HTTP/esid_support = 1
mpi/max_pipes = 4000
mpi/buffer_size = 65536
icm/max_sockets = 4000
icm/max_services = 100
icm/listen_queue_len = 512
icm/min_spare_threads = 3
icm/log_level = 0
icm/accept_remote_trace_level = 0
icm/ccms_monitoring = 1
icm/conn_timeout = 5000
icm/keep_alive_timeout = 600
rdisp/plugin_auto_logout = 0
icm/ccms_refresh_rate = 30
icm/HTTP/max_request_size_KB = 102400
Thanks,
Kavitha Rajan.
Great - sorry I misunderstood you at first.
So the users come into the webdispatcher on the address -> http://host1/sap/bc/bsp/sap/crm_ui_frame/default.htm -> this represents the address of the Web Dispatcher. The dispatcher however, seems to be over loading one of the application servers on the ABAP backend.
A few items I'd check then:
1) Logon to the admin console for the Web Dispatcher itself. Check to ensure that it is indeed seeing all of the logon groups configured in the ABAP backend.
2) Check the configurations of the ABAP logon groups for any issues - i.e. they are excluding servers.
3) Check the configuration of the ICM parameters in each of the ABAP servers - they should all be identicial unless you have exceptions noted.
4) For the Web Dispatcher itself, you may wish to set it up for a 'large system' scenerio with values like this:
icm/max_conn | icm/max_sockets | icm/req_queue_len | icm/min_threads | icm/max_threads | mpi/total_size_MB | mpi/max_pipes |
16384 | 16384 | 6000 | 100 | 250 | 500 | 21000 |
The Web Dispatcher uses the configuration under "Accesssability of Message Server" to determine the location of a ABAP / J2EE message server and in turn read back the available logon groups. If the Web Dispatcher is seeing the logon groups correctly, then it's something to do with either the ICM settings on each app server or the logon group configurations itself.
The only thing I see odd here is that the servername for the Web Dispatcher is "HOST 1" but under the instance profile you have:
#-----------------------------------------------------------------------
# Accesssability of Message Server
#------------------------------------- ----------------------------------
rdisp/mshost = Host1
ms/http_port = 8101
So unless that's a typo, are the WD and ABAP stack on the same server? "rdisp/mshost" should be pointing to the servername where the message server service (CI typically) is running for the ABAP stack.
Hope this helps.
Tim
Care to explain what you mean by emergency mode? The ICM timeout settings shouldn't affect load balancing so I suspect your problem is somewhere else. Especially with that configuration, processes can run without interruption for a very long time on both hosts. Can you share the process configuration of both instances and the logon group configuration? How are you balancing the load? Using logon groups? Are you using SAP Web Dispatcher or something similar? If there are many inactive sessions in the system, set rdisp/plugin_auto_logout accordingly. I think the space would have been more suitable for this discussion thread.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Samuli,
We have set rdisp/plugin_auto_logout = 28800 currently. Problem is, users started complaining sometime in the peak business time, that they cannot login. When I looked into the system, there is issue in SMICM. I was not able to refresh, showing emergency mode, then later after few times if trying 'leave emergency mode", then it shows more # of connections used for ex: 1800 than ususal max 250 connections. This happens only in one app server not the entral instance. After restrating ICM this issue getting resolved. We are using web-dispatcher. We have configured load balancing in SMLG. I have also used load balancing in the SICF service for the web-ui. I used OSS Note 737625 - Parameter recommendations for the ICM. Changed parameters for 7.00 version.
icm/max_conn = 10000
icm/max_sockets = 16384
icm/req_queue_len = 6000
icm/min_threads = 100
icm/max_threads = 250
mpi/total_size_MB = 500
mpi/max_pipes = 20500
When I compared between the servers, I am only seeing above timeout parameter difference between the 2 servers. Problem is only with app server. I am able to use SMICM in the central instance without any issue. It is fine after refreshing unlike the app server. Also showing around only 50 connections approx. Please help ASAP to resolve this production issue. I have posted here as it has started with performance issue which lead me to look into ICM.
Thanks,
Kavitha Rajan.
User | Count |
---|---|
88 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.