Crystal runtime 13.0.5 64 bit Access Violation Errors

[paul_marks]

Dear Forum

We are using Crystal runtime 13.0.5 64 bit but we are receiving Access Violation Errors in our live environment which is a website hosted on Windows Server 2008 servers.

We are attempting to use the Windows Debug Diag 2.0 tool to find more data about the error.

The tool requires the Crystal symbol files:

WARNING - DebugDiag was not able to locate debug symbols for \crpe32.dll, so the information below may be incomplete.

In WER56C1.tmp.hdmp the assembly instruction at crpe32+3ac12f in C:\Program Files (x86)\SAP BusinessObjects\Crystal Reports for .NET Framework 4.0\Common\SAP BusinessObjects Enterprise XI 4.0\win64_x64\crpe32.dll has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000 on thread 54

I understand from reading the SAP web site and forums that these files are not publically available on your web sites

Please can these be provided so we can continue debugging this error.

With Kind Regards

Paul Marks

[former_member183750 · Accepted solution]

Hi Paul

First thing you want to do is get to SP8:

If that does not help, it would be nice to have more info on "we are receiving Access Violation Errors".

E.g.; Win app / Web app?

What would I have to do to get the Violation? E.g.; does this occur after processing one report, two reports, more?

Are you viewing, exporting, printing?

Etc., etc. My belief is that at this point DebugDiag is an overkill and it will not lead to a solution.

- Ludek

Senior Support Engineer AGS Product Support, Global Support Center Canada

Follow us on Twitter

[paul_marks · Accepted solution]

Ludek

The web application is within a load balanced environment (8 servers) and we upgraded one of these servers to operate with SP8 – once a user has connected to a server, they remain on that server for the duration of their browser session.

Each web server is Windows Server 2008 R2, 64-bit with 8GB RAM, all connecting to a single backend server running SQL Server 2008 (with 64GB RAM).

The access violations were still happening on SP8, with an entry from the Windows Application Event Log as follows:-

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

Faulting module name: crpe32.dll, version: 13.0.8.1216, time stamp: 0x52a99c89

Exception code: 0xc0000005

Fault offset: 0x00000000003abf8f

Faulting process id: 0x110c

Faulting application start time: 0x01cf48de5ae06d78

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Program Files (x86)\SAP BusinessObjects\Crystal Reports for .NET Framework 4.0\Common\SAP BusinessObjects Enterprise XI 4.0\win64_x64\crpe32.dll

Report Id: 433af760-b4e5-11e3-9e77-f4ce46b6fcc6

The Access Violations are happening sporadically and are not consistent in nature – a server can have no access violations on a given day, but then have 3 or 4 the following day.  We tested our internal environment looking at concurrent prints with a team running a range of reports, printing, etc… for a period of 1 hour+.  No issues were experienced.

The only way we have been able to trigger any form of Access Violation on a VM image of our production environment is when an invalid parameter (e.g. empty string) is passed to a Stored Procedure that is used as a sub-report for the header, which causes the SProc to enter an infinite loop.  When we kill the process through SQL Server, we have seen it cause an APPCRASH on the web server although this again has been inconsistent (e.g. if the users web session has timed out at 20 minutes, it does not crash IIS).

We are making amendments to the SProcs that can potentially enter an infinite loop to prevent such behaviour, although a SQL Profiler trace on one of the affected stored procedures yesterday used in the majority of reports did not show any evidence of a user getting into an infinite loop.  We extended the trace last night to include the others and will be monitoring results today.

User experience is that when attempting to run a report, the browser sits there indefinitely.

With Kind Regards

Paul

[former_member11696 · Accepted solution]

Hi Paul,

Just to confirm, you are using your WEB servers AppPool in 64 bit mode and using CR for VS 64 bit runtime?

Since it is Parameter related SQL 2008 only has partial support when using the MDAC OLE DB Provider or ODBC using the SQL Server driver. It could be also the report when designed used the 32 bit client and when run in the WEB farm in 64 bit mode there may be issues between the drivers. Nothing we can do about that.... May have to get Microsoft involved but update to Native 10 and see it that helps.

First, install MS SQL Server Client tools so you get the SQL Native 10 client and use that driver in both OLE DB and ODBC on all of your IIS Servers and App server if you have one.

Second, then update your reports to use that driver, it could be the client on IIS Server having issues with Parameters, could be sharing or resource issue causing crpe32 to crash due to the missing or invalid SP parameter... CR MUST connect to the DB first when using SP's.

Third, there is a known issue with our third party Image Handler Library that was introduced in SP 5 and still partially an issue in SP 8. SP 9 ( due out very soon ) should resolve this issue.

To Do:

Install MS SQL Server 2008 Client tools on each Web Server and Report Designer's PC.

Update ALL reports to use the new client and verify each and then update your report repository with the new reports.

See if the reports causing the issue have images in them, if they do wait for SP 9 and test again.

PS - SP 5 has a lot on known issues with IIS - if you can go back to SP 4 and see if that IIS server has any issue just to confirm if it is images. If you don't use images then not the problem.

Check your code, and use a Try/Catch block around the Logon routine. Report should not be able to continue if the log on fails. something like this:

catch (Exception ex)

{

     MessageBox.Show("ERROR: " + ex.Message + " ;" + ex.InnerException.Message);

}

Also, the crash may happen in crpe32 but it could be due to some other component or third party dependency, it's just crpe32 that's catching the error.

Finally, it could be you have too many open connects to SQL Server and that could cause an issue also. Check with your DBA or MS to see how many open connections are allowed.

CR open a connection for the main report and then for each subreport.

Don

[paul_marks · Accepted solution]

Don

Just to confirm, you are using your WEB servers AppPool in 64 bit mode and using CR for VS 64 bit runtime?
Yes, web servers are 64-bit, with 64-bit app pools and CR for VS 64-bit runtime.

Since it is Parameter related SQL 2008 only has partial support when using the MDAC OLE DB Provider or ODBC using the SQL Server driver. It could be also the report when designed used the 32 bit client and when run in the WEB farm in 64 bit mode there may be issues between the drivers. Nothing we can do about that.... May have to get Microsoft involved but update to Native 10 and see it that helps.

The 80+ main & sub reports most likely were designed with 32-bit client although, if this was the cause, would it not be more consistent in its nature (i.e. all reports would cause an APPCRASH on all web servers in the farm)?  Usage / pages served (according to IIS logs, including navigation within a report) show 250-300+ per hour consistently across all 8 web servers – sometimes double this.

First, install MS SQL Server Client tools so you get the SQL Native 10 client and use that driver in both OLE DB and ODBC on all of your IIS Servers and App server if you have one.

Second, then update your reports to use that driver, it could be the client on IIS Server having issues with Parameters, could be sharing or resource issue causing crpe32 to crash due to the missing or invalid SP parameter... CR MUST connect to the DB first when using SP's.

Third, there is a known issue with our third party Image Handler Library that was introduced in SP 5 and still partially an issue in SP 8. SP 9 ( due out very soon ) should resolve this issue.

We do use the image handler for charting.

To Do:

Install MS SQL Server 2008 Client tools on each Web Server and Report Designer's PC.

Changes to our production environment are stringently controlled by another team.  We have not been able to replicate the issue in our development environment through normal use, only if we kill the SQL process in SSMS when we are trying to execute a report.  This can trigger an APPCRASH on the web server.

Update ALL reports to use the new client and verify each and then update your report repository with the new reports.

We can undertake this step although comment about consistency of errors makes us question if this will help.

See if the reports causing the issue have images in them, if they do wait for SP 9 and test again.

We’ve been testing SP9 in development, although we can’t recreate the issue consistently enough to be able to confirm that SP9 will resolve the problem.

PS - SP 5 has a lot on known issues with IIS - if you can go back to SP 4 and see if that IIS server has any issue just to confirm if it is images. If you don't use images then not the problem.

Check your code, and use a Try/Catch block around the Logon routine. Report should not be able to continue if the log on fails. something like this:

catch (Exception ex)

{

     MessageBox.Show("ERROR: " + ex.Message + " ;" + ex.InnerException.Message);

}

We’ve checked and Try/Catch blocks are already around the logon routine.

Also, the crash may happen in crpe32 but it could be due to some other component or third party dependency, it's just crpe32 that's catching the error.

We’re looking for a needle in a haystack and someone may have just moved the haystack!

Finally, it could be you have too many open connects to SQL Server and that could cause an issue also. Check with your DBA or MS to see how many open connections are allowed.

CR open a connection for the main report and then for each subreport.

SQL Server default max connections at 32,767.

Other lines of investigation include:-

1.     Anti-virus (we use Sophos) – we have a correlation between scan failures of files in C:\Windows\Temp from the SAV log and an APPCRASH taking place.

2.     Anti-virus: Setting Exclusions for scanning (what are the recommendations for this?).

3.     Windows DEP: Servers are set to use DEP for everything unless excepted (no exceptions currently in place).

We witnessed a strange behaviour last Friday (11^th), where 4 separate servers experienced the APPCRASH at 02:00hrs when there wasn’t any traffic coming to the server and no users were logged on.  In each instance, it reported crpe32.dll as the faulting module.  The only thing that happens at this time is a scheduled restart of the web server.  We’ve tried to be very careful about ensuring that objects are closed and disposed, with global.asax also having a check on Session_End (which theoretically should have kicked in before the scheduled restart due to timeout / inactivity).

Is it possible to get the Symbols please, so that we can analyse several dump files?

Additionally, we are looking at raising a paid support ticket.

With Kind Regards

Paul

[former_member11696 · Accepted solution]

Hi Paul,

Interesting about the scan in the \temp folder.

If that's the case them exclude the \temp folder from the scan. CR may not be releasing it's temp files even if there are no reports in session so excluding that folder be be part of the answer.

SP 9 is available now, interested in putting that on one of the WEB Servers to see if it helps?

The PDB's are over 4 gig. Need to know which ones you need. Likely crpe32 and maybe the crdb_ado or ODBC pdb.

SP 5 has known issues with WEB App's you should get them all updated also.

A phone case may be the answer, refer to this forum post also so there is more history and info.

One other thing, your WEB farm and load balancer must use Sticky Sessions. I assume you are using them but jsut in case...

Don

[paul_marks · Accepted solution]

Don

The PDB requested is the one for crpe32 please

With Kind Regards

Paul

[former_member11696 · Accepted solution]

What is the exact version of crpe32.dll?

Don

[former_member11696 · Accepted solution]

Sent SP 8 and 9 PDB's through FTP server. If you need SP 5 let me know although you REALLY should get all of the installs to the same patch level.

Don