Web Intelligence Processing Server Stops

[Former Member]

We are having an issue with BOBJ XI 3.1 where the Web Intelligence Processing Servers just stop on their own and are stuck in the "starting" state. We can't seem to find a pattern to when or why this is happening. The only way to get the server back running is to Force Termination and then restart.

Anyone else having these problems? Anyone have any ideas on a possible fix?

Thanks in advance.

[Former Member]

I've also encountered this issue with XI 3.0 and 3.1. The solution I found was to disable memory analysis and to increase the "Maximum documents before recycling" setting from 50 (default) to 250.

Depending on your usage level, the document threshold may need some experimentation.

We recently had to rebuild our PROD box after which the services started hanging again. I reimplemented the settings above and all was well. We run for a good week or so before having any need to restart services.

At the risk of stating the obvious, I think there are issues with the recycling "features" that attempt to restart services automatically based on some threshold. Memory, docs processed, etc.

I hope this helps.

John

[Former Member]

We had encountered the same problem with the 'starting' server status, modified settings as John suggested and the problem didn't occur ever since.

Thanks!

[denis_konovalov]

Webi recycling issues should be fixed in Fp1.3 and up.

[Former Member]

Denis,

I can tell you that FP 1.3 has not addressed the "stuck in starting". We have 5 customers who continue to see the behavior and all are on 1.3. What is the Adapt# of the fix you are referring to? The FP 1.3 fixes doc has no occurrence of "recycle" or "recycling" and a serach on " Web Intelligence" does not turn up anything that addresses the behavior either.

We will likely be logging a new case or perhaps re-opening an old one. But it is almost a gurantee that if you use SAP Universes, you will run into this error. Perhaps a coincidence...but it seems odd that only our SAP shops are seeing this to a degree where it interupts progress.

[denis_konovalov]

It must be SAP specific, because I can see better memory and recycling in Fp1.3 compared to RTM.

Unfortunately I do not have an ADAPT number to show for it. You just have to trust me

What I see in RTM - with big reports/complex calculations webi might use too much memory, get unresponsive, do not accept any new requests and become stale.... Only restart will clear it.

in Fp1.3 and up in same workflow, webi will kick error message to user running offending report, recycle memory back to low level, continue accepting oter requests, no need to restart.

So, if you still see hanging webi servers and it happens with SAP based universes - got to be SAP specific.

[Former Member]

hi!

maybe it is really SAP specific because i am encountering it too.

so what can be the solution/workaround to solve this issue?

thanks a lot!

[Former Member]

I got the same problem.

It was running BOMM 3.0 which refresh the metadata of webi reports. There are 3 webi processing server and one of them hangs at starting status.

The BOMM job is also hangs at running status, i think it is waiting for the Webi processing server.

I have not found any solution yet.

[Former Member]

Hi all,

I have also experienced the same problem. Searches of sites pointed to the memory analysis, so I tried it with this setting off. I also noticed that the behaviour was much better when I had 2 web processing servers on the same node. Finally, though I had never deployed it myself, there is a script floating around to start and stop the webi servers at planned times which goes a long way to ensuring it is available when you want.

http://centerofspotlight.com/2008/08/11/business-objects-xi-r3-server-stuck-in-starting-stage/

Regards

Alan

[denis_konovalov]

There is only one way to really know what's going on and it is to log a support Incident.

You can also start tracing on the webi report servers and see what that will tell you.

[Former Member]

We have also seen the exact behavior that Rodney describes, specifically w/ our SAP customers. The angle I am pursuing right now is WebI Processing servers can't hanle SAP queries (BEx) effciently. I will likely be logging a case with SAP Support to see if our traces indicate anything of value.

[Former Member]

All,

After changing the default AutoSave time to 120 minutes, we have not seen this problem in over a week. We don't think that this is the correct solution but it seems to be getting us by for now.

We tried this change based on OSS note 1297132.

We are also not sure why this works....Although, we had thought that the problem occurred when the Java client timed out. Maybe it is related to the Java client timing out and the AutoSave then trying to execute which was causing the issue??

Rodney

[Former Member]

Rodney...interested to know if you have had any WebI Processing Server stopages since your last post. I am actually looking at Virtual Memory settings. One of of our customers is seeing the stopages even with Auto-Save settings increased. In the WebI process trace files I am noticing numeorus errors such as

**ERROR:SRM:An internal error occured while dgSerializeManagerImpl is calling ibo_idgSrmStore->PublishToCorporate [kdgSerializeManager.cpp;2365]

Reseraching issues associated with dgSerializeManagerImpl, seems to suggest that the server is running out of Virtual Memory space. Going to run some tests, but I wanted to add this to the thread for any feedback.

[denis_konovalov]

Kevin,

Remember in the good old 6.5 days we had recomendation to increase Desktop Heap size to 1024 in the registry.

I wonder if this is related to that. Try increasing Desktop heap and let us know if webi works any better !

Tnx,

Denis

[Former Member]

Denis,

OK...so...

Should it better standard practice for all BOXI installs to modify the reg key

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

From:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows

SharedSection=1024,3072,512 Windows=On SubSystemType=Windows

ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3

ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off

MaxRequestThreads=16

To:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows

SharedSection=1024,3072,1024 Windows=On SubSystemType=Windows

ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3

ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off

MaxRequestThreads=16

Also, is there any Desktop Heap advantage to running the SIA under a dedicated admin account vs. Local System?

Thanks.

KMS

[denis_konovalov]

Kevin,

I think it should. I'm not sure why we stopped publishing Recomended Settings guides, but if we were - I'm sure Desktop heap would be there

Domain account vs Local System just gives you more reach on the network. Sometimes report servers can't access data from DB's if they run under local accounts, but work fine with Domain accounts.

It's really based on the security you have on your netowork and access restrictions.

[former_member212749]

Hi Rodney,

I guess you need to increase the memory of the system which is running Business Objects as there are many users getting connected to the system through WebI at one perticular time this happens i guess.

Could you please check by increasing the load to the BO server.It will turn off the services.

Regards

Prashant

[Former Member]

Check for any errors in Event Viewer as well when the Processing server goes down.

Regards,

Anisa

[Former Member]

Thanks Anisa,

We have checked and there are no errors in Event Viewer. We have tried several things to resolve this issue. For example, we saw a forum post referring to multiple NICs on the server. So we changed the servers to use a specific NIC but that did not work. We have also tried to turn on "trace" for the server process and this does not give us enough information to figure out the issue.

I also spoke with another BOBJ customer using XI 3.1 and they were having the same issue. They believe that this issue was related to specific users that were logged in to Webi. They had an OSS message open with SAP to try to solve the issue.

I was reaching out in this forum to see if anyone else had the issue and had figured out any more info on the issue.

Thanks.

[Former Member]

Are you trying to run or refresh any WEBI document when you observe this issue?

[Former Member]

It is hard to tell. Sometimes, it seems as though we get the error when the Java client times out. We have changed almost all of the time-out intervals to longer to try to resolve the issue.

Most of the time, it seems like it happens after developers have been in Webi for extended periods of time.... Then, when the system is idle, the Webi processing service tries to restart itself and gets hung up. We have multiple Webi processing services on the server and not all of them always get hung up. It seems to be a fairly inconsistent issue.

[Former Member]

We have observed that when we refresh the WEBI Documents using RowIndex function, the server hangs and we need to restart the server or SIA.

It seems the RowIndex() funciton is broken in XI3.1

Regards,

Anisa

[Former Member]

We have also observed that when the WEBI document is left idle in edit mode for more than default session timeout period, the server hangs and you see errors in Event viewer "Faulting error w3wp.exe"

I shal let you know if we find something more in this issue.

Regards,

Anisa