Suddenly Crystal Jobs went to permanent running mode

[former_member208901]

Hi,

Our environment is cluster of 3 nodes and each node has 1 job server. Luckily there is maximum concurrent jobs set to 5 on each job server, this morning suddenly all jobs started on  job server on node1 went to permanent running mode and all subsequent job requests were run on other 2 nodes job servers.

I couldn't able to find what happened in the server application log! I don't see any errors leading to this problem. Everything is normal and jobs were run successful on the same job server, But I see lot of errors on the same server after jobs went to permanent running state for "crproc" error as seen below

A failure occurred while the server was processing report 'REPORT NAME' (id=159605) for user 5109497 (RCIRAS0567)

and few crcahe Timeout. (RCIRAS0244) errors.

Attached, snapshot of application log. Highlighted jobs are the jobs went to permanent running mode.

Can you please help us, where could I able to find possible cause?

Thanks!

[Former Member]

hi,

Dont worry about the crproc and crcache errors in the event viewer since they are not part of scheduling a job.

What was supposed to be the next run time of these jobs and at what time did they run? Also what is the local timezone on the server? boe version?

-Nikhil Nair

[former_member208901]

Time zone is EDT and BOE is 4.1 SP03 Patch 02.

They started at their original run time 5:41AM to 5:45AM but didn't finish. Out of those five, three reports run daily around 5:41AM and  two reports run every 5 minutes.

Thanks!

[Former Member]

its hard to conclude the cause without logs etc. however some probable causes are when job server and the its child process loose communication due to some network issue or if the child process is hung, process waiting for thread to disconnect from database etc. etc.

I am assuming this never happened before at your end however if this is a regular issue, you might require a support case for logs investigation etc.

-Nikhil Nair