SAP BI 4.0 SP4 Patch 7 - Tomcat Errors - End of File/Socket Exceptions

[former_member358000]

Hi Experts,

We have a load balanced distributed environment (SAP BI 4.0 SP4 Patch 7 - Windows 2008 R2 x64) with a separate web tier (tomcat) and app tier boxes (2 wide). Users can access the BusinessObjects through the load balancer URL and get redirected to either one of the web tier app tier box stacks based on the LB policy. Earlier everything was running fine, but now all of a sudden all user sessions are abrubtly being terminated when they are trying to run any report with moderate complexity. Anytime they try to run a report which runs for more than 5 - 7 minutes, the session gets terminated even without reaching the timeout value - but they same report runs perfectly fine when scheduled. This means the Business Objects services are running perfectly fine but it is an issue with the web tier.

Also, even if they are just browsing through the Public folders their session gets kicked off. It happens randomly but on a regular basis in the last couple of days.

I have attached a pic of the error i see on the report panel and the corresponding messages on tomcat logs have been included below. We also get occasional http 500 errors.

I have tried the following with no success - we still keep getting these errors

- restarting tomcat
- recreating the catalina localhost directory
- redeploying BOE on tomcat

- tried increasing connection timeout to 25 seconds on tomcat server.xml

We have allocated sufficient heap on tomcat servers and dont see any memory exceptions

I thought it was issues with our IE proxy settings and have also eliminated that issue - anything that is running within our domain is exempted from the proxy rules - so we are good on that front

Any help or suggestions would greatly be appreciated - My next step is to run reports through the rich client and see if i get the errors but most of the developers use thin client for developing/running reports.

Caused by: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:273)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:676)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:619)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 48 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:741)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:765)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:574)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
... 56 more
com.businessobjects.sdk.core.server.CommunicationException$UnexpectedServerException: java.net.SocketException: Unexpected end of file from server
                at com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilde[/size]r.java:144)
                at com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilder.java:101)
                at com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:183)
                at com.businessobjects.sdk.core.server.internal.AbstractServer.process(AbstractServer.java:133)
                at com.businessobjects.sdk.core.server.internal.InstanceServer.process(InstanceServer.java:94)
                at com.sap.sl.sdk.services.util.ServerRequestProcessor.processServerRequest(ServerRequestProcessor.java:49)
                at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.processDataProviderCommand(DataProviderServiceImpl.java:441)
                at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.processDataProviderCommand(DataProviderServiceImpl.java:453)
                at com.sap.sl.sdk.dataprovider.service.DataProviderServiceImpl.run(DataProviderServiceImpl.java:213)
                at com.sap.sl.sdk.workspace.service.WorkspaceServiceImpl.run(WorkspaceServiceImpl.java:330)
                at com.sap.webi.ui.tasks.workflows.RunWorkspaceTask.doIt(RunWorkspaceTask.java:80)
                at com.sap.webi.ui.tasks.workflows.RunWorkspaceTask.doIt(RunWorkspaceTask.java:16)
                at com.sap.webi.toolkit.ui.tasks.WebITask$PrivateWorker.doInBackground(WebITask.java:337)
                at javax.swing.SwingWorker$1.call(Unknown Source)
                at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
                at java.util.concurrent.FutureTask.run(Unknown Source)
                at javax.swing.SwingWorker.run(Unknown Source)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                at java.lang.Thread.run(Unknown Source)
Caused by: javax.xml.ws.WebServiceException: java.net.SocketException: Unexpected end of file from server
                at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(Unknown Source)
                at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source)
                at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source)
                at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
                at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
                at $Proxy41.processRequest(Unknown Source)
                at com.businessobjects.sdk.core.server.internal.http.HTTPServerImpl.doProcess(HTTPServerImpl.java:184)
                at com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:171)
                ... 17 more
Caused by: java.net.SocketException: Unexpected end of file from server
                at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
                at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
                at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
                at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
                at java.net.HttpURLConnection.getResponseCode(Unknown Source)
                ... 33 more

org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet equinoxbridgeservlet threw exception
java.lang.RuntimeException: org.apache.jasper.JasperException: java.lang.NullPointerException
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:254)
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.service(BundlePathAwareServiceHandler.java:197)
at com.businessobjects.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.businessobjects.pinger.TimeoutManagerFilter.doFilter(TimeoutManagerFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:421)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.eclipse.equinox.jsp.jasper.JspServlet.service(JspServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.businessobjects.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:110)
at com.businessobjects.http.servlet.internal.ServletLastFilterChainElement.service(ServletLastFilterChainElement.java:30)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:46)
at com.businessobjects.webutil.boetrustguard.BOETrustPrepareFilter.doFilter(BOETrustPrepareFilter.java:32)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.swd.shared.tracelog.TraceLogScopeFilter.doFilter(TraceLogScopeFilter.java:38)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.sdk.actionfilter.WorkflowFilter.doFilter(WorkflowFilter.java:45)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.swd.appcontext.RequestInitFilter.doFilter(RequestInitFilter.java:26)
at com.businessobjects.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:72)
at com.businessobjects.http.servlet.internal.filter.FilterChainImpl.doFilter(FilterChainImpl.java:43)
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.serviceHelper(BundlePathAwareServiceHandler.java:235)
... 20 more
Caused by: java.lang.NullPointerException
at com.businessobjects.webutil.clientaction.query.ActionQueryHelper.getObjectViewerActions(ActionQueryHelper.java:87)
at org.apache.jsp.jsp.common.appletactions_jsp._jspService(appletactions_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
... 41 more

[former_member202789 · Accepted solution]

Hi,

Open Tomcat Manager and check/Compare the session timeout  value (In minutes) that might have got updated upon patch installation. Check/compare for BOE and other application

Have you applied Session FO by applying <distributable>true</distributable> in Web.xml of BOE web

application. Just check whether it has got removed upon patch installation.

Http 500 is completely tomcat error and first thing to fix it is restart of Tomcat Server.

Troubleshooting: Apart from above given points if you have configured Web Server LB or DNS Alias or Hardaware or Software Loadbalancer, in any of the situation connect directly to both the tomcats with thier hostnames and check whether you still get error from both the tomcats individually.

Regards,

Animesh Kumar

[former_member358000 · Accepted solution]

Hi Animesh,

Thank you for your reply.

We have compared the timeout in all application's web.xml - they have been set to the same value.

Regarding session FO -  we have not set that yet - my session is still being served by the same application server that is hit initially and there is no reason for failover to take place - the application server on which serves the session is running fine - Valid point regarding distributable tag but i doubt if it is because of FO. We have shutdown the second application server and have the LB constantly hit the lone application server - will see if there are any improvements. I am using fiddler to track my sessions. One thing i notice on fiddler is i get a lot of 404 errors when reports are being opened but the reports do open and run fine at times inspite of all the 404 errors.

HTTP 500 - we have restarted and regenerated tomcat localhost directory several times - it still occurs randomly.

We have a hardware LB in place -  i am going test if we encounter the same issues if we hit the individual stacks directly bypassing the LB.

Thanks,
Raghav

[Former Member · Accepted solution]

Hi Ragha,

Did you get any break on that issue, i am also facing the same issue, i have installed BO 4.1 SP5, but when i click on preferences from java infoview, it gives me the same error,

i ahve restarted the tomcat server, but its not helping .

thanks

lakshman

[former_member358000 · Accepted solution]

Do you have a hardware load balancer as well? It came down to a configuration setting up sticky sessions within our f5 load balancer.

[Former Member · Accepted solution]

Wow, thanks Raghav! Our setup and error behavior are remarkably similar to yours. We have an F5 load balancer as well, and our in-session Webi queries time out at around 5-7 minutes.

I forwarded this to our network administrator, but I'm anticipating his next question: "what is the sticky session configuration setting?"

Do you have any more detail on which setting, in F5, was modified? Thank you!

Carlos

[former_member358000 · Accepted solution]

Hi Carlos,

Its been a while since we configured the same - Sorry, i was not clear in my earlier response;

Ask the admin to look into Source IP persistence profile timeout settings - i believe by default it is set to a low value.

AskF5 | Manual Chapter: Enabling Session Persistence

Hope the above link is helpful!

Raghav

[Former Member]

Hi Raghav,

Did you find a solution for the issue? we are also facing a simillar issue. If you have find solution, can you please share with me?

Thanks,

Bilal