'System.OutOfMemoryException' when exporting to XML format

[Former Member]

Hi,

I try to export a huge report and it failed at the end with a out of memory exception. It work fine in PDF format, but XML format crash.

[former_member208657 · Accepted solution]

Huge reports and exporting don't always get along. The reason it works with PDF is that the Adobe format is paginated. While exporting we'll create the end result page by page. But with the XML format there is no pagination and huge reports can cause memory errors.

Remember that Crystal Reports was initially designed to create human readable reports. These reports are generally small enough to be read/consumed by a person. Don't try and use Crystal Reports to create XML or CSV archives of your datasource. It simply wasn't designed to do that job.

[Former Member · Accepted solution]

Info : CR 2008 SP2 (with fix) + VS.NET 2005

We export the report in XML and after that, we take XML and generate Excel reports. We do our own Excel transformation.You will ask why not using CR to generate Excel reports. Because, we want good looking and usable reports. CR put white space and merge cells and we cannot sort row; it's not usable.

[Former Member · Accepted solution]

... and when we generate huge XML (the servers had a lot of ressource by the way) and we're getting System.OutOfMemoryException ...

[former_member183750 · Accepted solution]

"with fix" - what fix? FP2.1, 2.2, 2.3, 2.4, 2.5, 2.6? Please, go to the CR designer, click on the Help menu. Select About Crystal Reports. Let me know the exact version of CR reported here.

For the moment, let's take the dev environment out of the picture. How does the export work from the designer?

When you say huge, how huge? How many pages in the report file its self? At what "hugeness" does it work?

On another tack. You say that "CR put white space and merge cells and we cannot sort row...".

Have you tried all the XLS export formats? - In the designer first, then at runtime. E.g.; which format are you using; Excel or Excel Data-Only. It may be a good idea to discuss the export issue with the CR design team and see if they can help you in creating an export that will go straight to XLS without the XML intermediate step.

Another question; When exactly do you get the error? When generating the XML, or when generating the XLS?

Ludek

[Former Member · Accepted solution]

fix -> 2.6

Never try to export a huge XML with designer. Not enough data! The error come from production server. I generate the reports in pdf format and it produce over 4000 pages!

The error occur when generatin the XML.

When i monitorize the process and the temp directory, when the app begin to write the file, the process begin to grow and at 300Mb it stop. That's the case when we've got the error. When we try to get a smaller one, the process work fine and grow near 800Mb.

Is it an issue with the number of rows when CR initiate is XML transormation. In the second case when it work, I produce a 45000 rows Excel document.

XLS export formats? yes i try the 2 formats but it's not what the client expect!

Where I can contact DR design team and discuss my issue?

[former_member183750 · Accepted solution]

"Never try to export a huge XML with designer. "

Somehow, you will want to test this with the CR designer. I'm not sure why it is not possible to connect CR designer to your production data(?). Reason this is important is that if this does not work in the designer, it will not work at runtime. And thus, the 1st place to get this resolved will be in the design, be it another way of approaching the issue or creating a fix escalation.

However, you may want to consider the possibility that you have reached the product's limit... Possibly limiting the size of the export by use of selection formulas will be a way to go, then append the resulting XML files?

Re. Design forum:

Ludek

[Former Member · Accepted solution]

ok, i've made a test with the designer connect to the production server... and it work! what the difference with designer engine?

[former_member183750 · Accepted solution]

That is a rather broad question. I suspect the report engine manages memory differently.

Here are our options:

1) I need to reproduce the issue. Thus I need accurate information on how to reproduce it. E.g.; export huge amount of data will not do it. Once I have reproduced this, I can escalate the issue to QA for a resolution. Typical time frame for a fix is 6+ months...

2) Post to the Report Design forum and see if they can help you do the export straight to the XLS file format. If they have a solution, it should take less than 6 months...

Ludek

[Former Member · Accepted solution]

>

> 1) I need to reproduce the issue. Thus I need accurate information on how to reproduce it. E.g.; export huge amount of data will not do it. Once I have reproduced this, I can escalate the issue to QA for a resolution. Typical time frame for a fix is 6+ months... 

ok, here it's what I've got from designer :

+90000 rows

~35 columns

xml file = 230Mb

this case crash on prod server!

[Former Member · Accepted solution]

hi,

have you try to reproduce my case?

Best regards!

[former_member208657 · Accepted solution]

I was able to crash the CR .NET engine in a similar manner as you described when using a large amount of data. This is a known issue as I mentioned before. The Crystal Reports .NET engine doesn't handle the non-paginated formats well when there is a huge number of records. The memory management we use runs out of space and the app crashes. I've seen it do this with exporting to CSV and text formats too.

We have some open escalations with our product team on this issue so we'll see what they say. For now we have no solution for you though. I'd recommend using smaller sets of data.

An upgrade to Crystal Reports Server might help in this case. When you schedule a report in CR Server the exporting works in a similar manner to the Crystal Reports Designer. Keep in mind though, the upgrade is not free.

[Former Member · Accepted solution]

thanks a lot!

[former_member183750 · Accepted solution]

The issue is now being tracked under TE5000219776

Ludek

[former_member208657 · Accepted solution]

There has been lots of internal discussion re: this issue. It has already been tracked in our database under ADAPT01320923.

This problem occurs because of the way we transfer data from one service to another when using the RAS or PageServer. In this case multiple services are not involved, but the same technology is used in the reporting engine. When exporting to .csv, .xml, or text the data is written as a large binary blob. This causes problems because there is no way to know how much contiguous memory is available to the OS. That's why this issue can be difficult to pin down on different servers. If enough contiguous memory isn't available an error is thrown.

If you have Crystal Reports Server or BusinessObjects Enterprise you can get around this problem by scheduling the report to your desired format, instead of viewing it on demand. The scheduling works because the report job server and output file servers were designed to allow for these large exported files. Unfortunately this is not an option for you if you are working with Crystal Reports Server Embedded or the Crystal Reports .NET SDK. Neither of these products allow you to schedule.

At this time it doesn't look like this will be fixed in an upcoming fix pack. There is too much code to change and the risk is too high that potentially more damage may be done than what would be fixed. Discussions are still ongoing regarding this problem so we'll post more info when we have it.

[former_member208657 · Accepted solution]

There has been a lot of discussion re: this issue among support and the developers on our product team. The problem occurs because of the way we allocate memory for the exported file in one large blob. When a report is small this isn't an issue because there is enough contiguous memory space to complete the operation. When a report is large there is a chance that there isn't enough contiguous space and an error occurs. I say chance because the space available will differ from system to system depending on how much total memory the system has and how long the server has been running. Less contiguous space will be available when a server has been running for a long time. This is what makes this problem difficult to pin down - each server will have a different amount of space available.

What can be done about this? If you are using our Crystal Reports Server or BusinessObjects Enterprise products, you can choose to schedule the report to be exported to your desired format. The report job server and the output file repository are designed for these large files. Once the job is done you can retrieve it from the system with no problems. The Crystal Reports and RAS SDK simply weren't designed to handle these large reports. They were designed for small/agile reports that are returned in a matter of seconds, not for reports that take minutes to complete.

Another technique you can use is to use one of our paginated export formats. They seem to perform better for very large reports. I can't guarantee these will work in all cases, but you might be able to export a little more with these formats

Does Support Pagination

CrystalReport

Excel

HTML32

HTML40

PortableDocFormat

WordForWindows

Does NOT Support Pagination

CharacterSeparatedValues

ExcelRecord

EditableRTF

RichText

TabSeperatedText

Text

Xml

At this time we don't have a solution for clients using the CR .NET SDK or RAS SDK. Discussions are ongoing with the product managers and developers to see what we can do for future service packs and product releases. I don't expect to see any fix for this soon in any fix pack.

Sorry I don't have better news but felt it best to give you the latest info.

[former_member183750]

A good place to start is with the basics.

See the [Rules of Engagement|https://www.sdn.sap.com/irj/sdn/wiki?path=/display/home/rulesofEngagement]

Step 2 Asking Your Question; Provide Enough Information

Also, see the blog [What do I need to do to get the fastest issue resolution?|https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/12045] [original link is broken] [original link is broken] [original link is broken];

Ludek

Follow us on Twitter http://twitter.com/SAPCRNetSup