XML File generation-Performance Issue

[Former Member]

Hi , Has anyone generated large XML files using DataServices? We are trying to generate an XML file from a table having around 3.1 Million records. The job is running for around 1 hour and successfully creating the XML file of size 480 MB. but  when the job running the memory consumption is almost 100%.(Around 16 GB). No Cartesian products in monitor log,The max number in the monitor log is the number of records in source which is 3.1 Million. Also another strange thing noticed during job run is ,if we start the job at 9 AM it completes writing to the xml file by 9.30, but the job(dataflow) is running until 10 AM without doing anything. 9.30 is the last updated timestamp for XML file and if I open it at  the same time I can see the end XML tag. Completely clue less on what the job is doing for rest of 30 minutes after file generation. We don't have the validate xml option checked. Any suggestions on improving the performance are appreciated. DS version is 4.2.4 Thanks Vams.

[Former Member]

Hi Vamsi,

May be some BODS reconciling after the actual task done as flat data to XML conversion needs lots of resources,

Try splitting the whole data in to multiple chunks with parallel transformations apply same logic in all parallel paths and see, It may bring down the reconciling time for BODS after actual job done

Thanks,