cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Internal server error during WebService request

Former Member

on ‎05-06-2014 2:40 PM

0 Kudos

Hi,

In one of our production environments running SAP ME 6.0.4.22, we have the latest month occasionally
had issues with WebService calls returning "Internal server error" to the client. We have noticed it can
happen for both me-ws and papi-ws and for different types of web-service methods. It is nothing wrong
with the web-service calls XML content, because the same type of message can pass without failure if
we run it again.

Today I trapped a situation from my own Client, so I was hoping if someone can look at the trace logs and
maybe help me with further trouble-shooting.

Client Web-service call is attached to this message if someone wants to see:

Client gets this response:

2014-05-06 14:29:06 SAP ME WS ERROR: Message=*FAIL*<html><head><title>Application Server Error</title>

<style type="text/css">

body { font-family: arial, sans-serif;}

</style>

</head>

<BODY><table width=800>

<tr><td width=50 nowrap>

</td><td>

<H2><b>500 Internal Server Error</b></H2><br><hr>

<table border="0">

<tr><td>Error:</td><td>-1</td></tr>

<tr><td>Version:</td><td>7200</td></tr>

<tr><td>Component:</td><td>ICM</td></tr>

<tr><td>Date/Time:</td><td>Tue May 06 14:29:07 2014

</td></tr>

<tr><td>Module:</td><td>icxxthrio.c</td></tr>

<tr><td>Line:</td><td>1978</td></tr>

<tr><td>Server:</td><td>SEVA129_VA5_00</td></tr>

<tr><td>Error Tag:</td><td>{0026b095}</td></tr>

<tr><td>Detail:</td><td>IcmHandleNetWrite(id=38/700565): No data from server received (rc = -1)</td></tr>

</table>

<p></p>

</td></tr></table>

<table><tr><td width=50 nowrap></td>

<td nowrap><FONT SIZE=-1><a href="http://www.sap.com/">&copy; 2001-2009, SAP AG</a></FONT>

</td></tr></table>

</body></html>

Contents of D:\usr\sap\VA5\J00\j2ee\cluster\server0\log\system\server_00.1

#2.0 #2014 05 06 14:36:29:040#+0200#Warning#/System/Server#
com.sap.ASJ.http.000083#BC-JAS-WEB#http#C0000AD30320473C00000000000012D4#6383550000000003##com.sap.engine.services.httpserver#Guest#0##05A89321D51B11E3874B0000006167BE#05a89321d51b11e3874b0000006167be#05a89321d51b11e3874b0000006167be#0#Thread[HTTP Worker [@82160863],5,Dedicated_Application_Thread]#Plain#com.sap.engine.services.monitor.mbeans.MonitorResourceBundle#
Cannot parse an http request. Http error response [400 Bad Request] will be returned.IP address
#

#2.0 #2014 05 06 14:36:29:041#+0200#Warning#/System/Server#
com.sap.ASJ.http.000071#BC-JAS-WEB#http#C0000AD30320473C00000001000012D4#6383550000000003##com.sap.engine.services.httpserver#Guest#0##05A89321D51B11E3874B0000006167BE#05a89321d51b11e3874b0000006167be#05a89321d51b11e3874b0000006167be#0#Thread[HTTP Worker [@82160863],5,Dedicated_Application_Thread]#Plain#com.sap.engine.services.monitor.mbeans.MonitorResourceBundle#
Data sending failed.IP address 10.XXX.XX.XX
#

Contents of D:\usr\sap\VA5\J00\j2ee\cluster\server0\log\DefaultTrace
500 Internal Server Error is returned for HTTP request [http://seva129:50000/manufacturing-services/SapErrorHandlerServlet]:
  component [com.sap.me.productdefinition.ws.WorkInstructionService],
  web module [manufacturing-services],
  application [sap.com/me~ws],
  DC name [],
  CSN component[],
  problem categorization [],
  internal categorization [-111158633].

[EXCEPTION]
com.sap.bc.proj.jstartup.fca.FCAException: Output broken (-7).
at com.sap.bc.proj.jstartup.fca.impl2.FCAConnection.getOutBuf3(Native Method)
at com.sap.bc.proj.jstartup.fca.impl2.FCAConnection.getOutBuf(FCAConnection.java:302)
at com.sap.bc.proj.jstartup.fca.impl2.FCAOutputStream.write(FCAOutputStream.java:93)
at com.sap.engine.services.httpserver.server.Client.send(Client.java:216)
at com.sap.engine.services.httpserver.server.ResponseImpl.sendResponse(ResponseImpl.java:226)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseBase.sendToClient(HttpServletResponseBase.java:1558)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseBase.writeStatusLine(HttpServletResponseBase.java:1313)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseBase.commit(HttpServletResponseBase.java:1198)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseFacade.commit(HttpServletResponseFacade.java:1442)
at com.sap.engine.services.servlets_jsp.server.runtime.client.ServletOutputStreamImpl.commit(ServletOutputStreamImpl.java:309)
at com.sap.engine.services.servlets_jsp.server.runtime.client.ServletOutputStreamImpl.sendToClient(ServletOutputStreamImpl.java:244)
at com.sap.engine.services.servlets_jsp.server.runtime.client.ServletOutputStreamImpl.finish(ServletOutputStreamImpl.java:290)
at com.sap.engine.services.servlets_jsp.server.runtime.client.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:346)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseFacade.close(HttpServletResponseFacade.java:573)
at com.sap.engine.services.servlets_jsp.server.runtime.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:498)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseFacade.doError(HttpServletResponseFacade.java:1339)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseFacade.sendError(HttpServletResponseFacade.java:473)
at com.sap.engine.services.servlets_jsp.server.runtime.client.HttpServletResponseFacade.sendError(HttpServletResponseFacade.java:360)
at com.sap.engine.services.webservices.runtime.servlet.ServletDispatcherImpl.doPostWOLogging(ServletDispatcherImpl.java:207)
at com.sap.engine.services.webservices.runtime.servlet.ServletDispatcherImpl.doPost(ServletDispatcherImpl.java:65)
at com.sap.engine.services.webservices.servlet.SoapServlet.doPost(SoapServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at com.sap.engine.services.servlets_jsp.server.runtime.FilterChainImpl.runServlet(FilterChainImpl.java:202)
at com.sap.engine.services.servlets_jsp.server.runtime.FilterChainImpl.doFilter(FilterChainImpl.java:103)
at com.sap.me.webservice.ClearServiceContextFilter.doFilter(ClearServiceContextFilter.java:28)
at com.sap.engine.services.servlets_jsp.server.runtime.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.runServlet(HttpHandlerImpl.java:432)
at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.handleRequest(HttpHandlerImpl.java:210)
at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(RequestAnalizer.java:441)
at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(RequestAnalizer.java:430)
at com.sap.engine.services.servlets_jsp.filters.DSRWebContainerFilter.process(DSRWebContainerFilter.java:38)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.servlets_jsp.filters.ServletSelector.process(ServletSelector.java:81)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.servlets_jsp.filters.ApplicationSelector.process(ApplicationSelector.java:276)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.WebContainerInvoker.process(WebContainerInvoker.java:81)
at com.sap.engine.services.httpserver.chain.HostFilter.process(HostFilter.java:9)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.ResponseLogWriter.process(ResponseLogWriter.java:60)
at com.sap.engine.services.httpserver.chain.HostFilter.process(HostFilter.java:9)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.DefineHostFilter.process(DefineHostFilter.java:27)
at com.sap.engine.services.httpserver.chain.ServerFilter.process(ServerFilter.java:12)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.MonitoringFilter.process(MonitoringFilter.java:29)
at com.sap.engine.services.httpserver.chain.ServerFilter.process(ServerFilter.java:12)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.SessionSizeFilter.process(SessionSizeFilter.java:26)
at com.sap.engine.services.httpserver.chain.ServerFilter.process(ServerFilter.java:12)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.MemoryStatisticFilter.process(MemoryStatisticFilter.java:57)
at com.sap.engine.services.httpserver.chain.ServerFilter.process(ServerFilter.java:12)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.filters.DSRHttpFilter.process(DSRHttpFilter.java:43)
at com.sap.engine.services.httpserver.chain.ServerFilter.process(ServerFilter.java:12)
at com.sap.engine.services.httpserver.chain.AbstractChain.process(AbstractChain.java:78)
at com.sap.engine.services.httpserver.server.Processor.chainedRequest(Processor.java:475)
at com.sap.engine.services.httpserver.server.Processor$FCAProcessorThread.process(Processor.java:269)
at com.sap.engine.services.httpserver.server.rcm.RequestProcessorThread.run(RequestProcessorThread.java:56)
at com.sap.engine.core.thread.execution.Executable.run(Executable.java:122)
at com.sap.engine.core.thread.execution.Executable.run(Executable.java:101)
at com.sap.engine.core.thread.execution.CentralExecutor$SingleThread.run(CentralExecutor.java:328)

Best regards,

Johan

webservice_message.txt.zip
Show replies
Show replies
Answer

Accepted Solutions (0)

Answers (3)

Answers (3)

alexander_teslyuk
Advisor
Advisor
‎05-22-2014 1:50 PM
0 Kudos

Hi Carsten,

It makes sense to me to align all abovemntioned timeout parameters, including SAP_JTATimeout.

I've contacted the responsible person to release the Note already.

Br, Alex.

Show replies
Show replies
former_member210324
Participant
‎05-08-2014 8:54 AM
0 Kudos

Hi Johan,

did you check the settings of Netweaver. Typically you only have one AS Java node running, which by default settings is only handling 5 parallel requests.

When these are allocated to some other requests, you will get errors.

Regards,

Christoph

Show replies
Show replies
Former Member
‎05-08-2014 9:03 AM
0 Kudos

Hi Christoph,

I have checked the settings that the SAP ME Installation guide says you should do in Netweaver configTool.

Are there any other settings that you are referring to and do you know how I can monitor them?

From what I can see when looking in Netweaver and ICM the system is running ok, but from time to time I still get this "Internal server error" response in the clients sending web HTTP requests.

Best regards,

Johan

sergiy_katerinich
Advisor
Advisor
‎05-08-2014 12:14 PM
0 Kudos

I guess Christoph is talking about FCA count which is set to 50 as per the instructions of section 4.1.2 of SAP ME 6.1 Component Installation Guide. So, I doubt this is a root cause.

former_member182330
Active Contributor
‎05-06-2014 3:12 PM
0 Kudos

HI Johan,

Check note 1620740. 2 option: increase ICM timeout as per note; AND/OR increase increase client side timeout.

Br,
KC

Show replies
Show replies
Former Member
‎05-06-2014 3:49 PM
0 Kudos

Hi Konstantin,

I previously also thought that it could have something to do with timeouts, but now I could actually see for myself since it was my own client.

From my client application I have the timout set to 600 seconds.

The Web-service request was sent from client to server and the timing of the error response in Server is the same second as the client sent the request.

My feeling is that this has something to do with a break in TCP/IP connection between client and server and I wonder if there is a way from SAP ME server to trouble-shoot the network connections.

Best regards,

Johan

sergiy_katerinich
Advisor
Advisor
‎05-06-2014 3:56 PM
0 Kudos

I would think about connection issue too.

There is NIPING tool for this - see SAP Note 500235.

Regards,

Sergiy

former_member196557
Active Contributor
‎05-06-2014 4:05 PM
0 Kudos

Hi Johan,

What client are you using to transact the Web Service request with ME?

Regards, Steve

Former Member
‎05-06-2014 4:13 PM
0 Kudos

Hi Steve,

The client was an own written application in Visual Studio 2010 using HttpWebRequest.

Best regards,

Johan

Former Member
‎05-06-2014 4:14 PM
0 Kudos

Hi Sergiy,

Thank you. I will check the SAP Note out and try to replicate the issue.

Best regards,

Johan

former_member196557
Active Contributor
‎05-06-2014 5:21 PM
0 Kudos

Hi Johan,

Our current connectivity product, SAP Plant Connectivity (PCo 2.3 SP4 ), executes web service requests using Microsoft.Net 3.5SP1 Windows Communication Foundation.  Internally, the HttpWebRequest object is configured to use HTTP1.1, and KeepAlive is enabled.

It is also possible that you may be short of ICM connections or threads available to handle the number of outstanding connections.  Do you know what your range of Web Service request counts are for the ME instance?

Regards, Steve

Former Member
‎05-07-2014 7:09 AM
0 Kudos

Hi Steve,

I don't know my range of Web Service requests. Is this something I can monitor from Netweaver or ICM?

I have noticed an increase of Internal Server errors during full production in daytime, which might indicate issues with resources in our server.

Best regards,

Johan

sergiy_katerinich
Advisor
Advisor
‎05-07-2014 7:17 AM
0 Kudos

I guess this is about ICM admin interface accessible at http://server:port/sap/admin

Former Member
‎05-07-2014 7:26 AM
0 Kudos

Hi Sergiy,

Yes, I am checking it now. But I don't know exactly what to look for.
Currently everything seems to be running ok.

I have in the trace log for ICM from my error yesterday.

These was during the time I ran my client application and the IP indicates that it is my client machine that is the remote host.

Br,

Johan

[Thr 4760] Tue May 06 14:29:38 2014

[Thr 4760] {0026b096} Traffic Control: Nettimeout (30) exceeded by peer: 10.211.12.31:52822 [icxxthr.c 4285]

[Thr 4760] CONNECTION (id=38/700566):

used: 1, type: default, role: Server(1), stateful: 0

NI_HDL: 311, protocol: HTTP(1)

local host: 10.211.3.32:50000 ()

remote host: 10.211.12.31:52822 ()

status: READ_REQUEST

connect time: 06.05.2014 14:29:07

MPI request: <2a14c5> MPI response: <2a14c6>

request_buf_size: 0 response_buf_size: 0

request_buf_used: 0 response_buf_used: 0

request_buf_offset: 0 response_buf_offset: 0

[Thr 4760] Tue May 06 14:31:22 2014

[Thr 4760] IcmCreateWorkerThreads: created worker thread 10

[Thr 4760] IcmCreateWorkerThreads: created worker thread 11

[Thr 4776] Tue May 06 14:36:29 2014

[Thr 4776] *** ERROR => handle not open for reading 27 [mpixx.c 2030]

[Thr 4724] *** ERROR => IcmReadFromPartner(id=27/701237): No data from server received (rc=MPI_ESTALE: outdated MPI handle/14) [icxxthrio.c 2305]

[Thr 6040] IcmWorkerThread: end worker thread 11

[Thr 6304] Tue May 06 14:37:32 2014

[Thr 6304] IcmWorkerThread: end worker thread 10

sergiy_katerinich
Advisor
Advisor
‎05-07-2014 8:07 AM
0 Kudos

Hi Johan,

That's a hard nut for sure. If you can reproduce this issue on more or less regular basis, I believe it is worth a deep investigation by BC-CST-IC or BC-JAS-COR support team.

In old versions of kernal there were some fixes, but I guess they are in place in your release.
So far I have not seen a confirmed solution for this issue. Either because of customers quitting their original intention to chase the root cause or because of disappearance of symptoms.

However, some recommendations below:

SAP Note 1629748 about "Nettimeout (30) exceeded by peer"
SAP Note 824554 - ICM and SAP Web Dispatcher Timeout Parameter
SAP Note 737625 to adjust parameters mpi/total_size_MB and mpi/buffer_size.
SAP Note 1441273 - AS Java runs out of MPI buffers
mpi/total_size_mb = (concurrent_conn * mpi_buffer_size) / (1024 * 1024)

Regards,

Sergiy

sergiy_katerinich
Advisor
Advisor
‎05-07-2014 8:10 AM
0 Kudos

As concerns ICM interface, the first thing to check is Peak values under Monitor link that should be lower than Maximum values.

Former Member
‎05-07-2014 3:35 PM
0 Kudos

Hi Sergiy,

Now it is the end of the working day.
From the log files I can see that some people have received internal server error
for the same reasons as yesterday.

This is the current ICM monitor values:

currentpeak maximumtotal
Created Threads:1016250125
Connections used:1841500720632
Queue entries used:11010002243797

It seems not to be a problem with threads or connections.

I will try to run my client a little more often and try to stress the system.

Best regards,

Johan

sergiy_katerinich
Advisor
Advisor
‎05-08-2014 12:27 PM
0 Kudos

Hitting Max values is quite a rare case, especially for test systems.

If it is not about lost packages that might be found out by NIPING, then I would think about NW issue. And then you'll need help of NW support team.

Former Member
‎05-08-2014 1:02 PM
0 Kudos

Hi Sergiy,

This is not a test system, but a system running in production.

It is currently in ramp-up with about 50% load.

I am concerned since we have not seen these errors until a couple of weeks ago and I am afraid that it might be because of some NW parameters.

Best regards,

Johan

sergiy_katerinich
Advisor
Advisor
‎05-08-2014 1:23 PM
0 Kudos


If those numbers are for 50% loaded produciton system, IMHO it looks quite good, and I doubt those peak values could have cuased the issue.

Thus I would recommend that you either post your question to NW forum or submit a support ticket against BC-CST-IC or BC-JAS-COR component.

Ask a Question