Access Violation - SAP Data Services 4.2

[former_member1095969]

Hi,

We are getting access violation on table comparisons within SAP Data Services 4.2 since upgrading (datastores are SQL Server 2012, repo on SQL Server 2008). This is a simple table comparison to help generate delta tables and load full history table after extracting into staging area.

We find playing around with the degree of parallelism (DOP) has an effect, but we don't understand why these are appearing in first place as this is migrated code and worked fine before. WFs and DFs all run in parallel during this part of the job.

Re-running or changing DOP normally resolves issue.

Error Log:

(14.2) 03-25-14 08:18:54 (R) (3772:7324) SYS-170101: |Session JOB_Live_Insight_CRM_EXTRACTS_ONLY|Work flow WF_EXTRACTS|Work flow WF_EXTRACTS_1|Work flow New_WorkFlow3|Work flow WF_EXTRACTS_1A|Work flow WF_EXTRACTS_EVENTDB_CRM|Work flow WF_EXTRACT_EVENTDB|Work flow WF_EXTRACT_EVENTDB_DELTA|Work flow WF_Extract_EventDB_EU_Delta|Data flow DF_Extract_EventDB_EU_Events_Delta

                                                    System Exception <ACCESS_VIOLATION> occurred. Process dump option is off. Process is not dumped.

                                                    Call stack:

                                                    0x00000000801CEAFF, XLink_desc::getNewRow()+0015 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\core\xlink.cpp, line 0536

                                                    0x00000000801C06B2, XPort_desc::get_new_row()+0018 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\core\xport.cpp, line 0539

                                                    0x00000000811A4B61, XTran_mergesplit::split()+0129 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\xform\tran_mergesplit.cpp, line 0164

                                                    0x00000000811A4A7F, XTran_mergesplit::execute()+0095 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\xform\tran_mergesplit.cpp, line 0136+0015 byte(s)

                                                    0x00000000801A0040, XTran_desc::execute()+0448 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\core\xtran.cpp, line 0799

                                                    0x0000000080FE3D39, Rww_thread::main()+0249 byte(s),

                                                    d:\im_ds_4.2_sp_rel\src\dataservices\dataintegrator\codeline\code\src\rww\rww.cpp, line 0451

                                                    0x0000000000A1438E, RWThreadFunctionImp::run()+0126 byte(s)

                                                    0x00000000009FC184, RWRunnableImp::exec()+0372 byte(s)

                                                    0x0000000000A14643, RWThreadImp::exec()+0051 byte(s)

                                                    0x0000000000A15F59, RWThreadImp::_setTimeSliceQuantum()+0169 byte(s)

                                                    0x00000000749337D7, endthreadex()+0071 byte(s)

                                                    0x0000000074933894, endthreadex()+0260 byte(s)

                                                    0x0000000076CD652D, BaseThreadInitThunk()+0013 byte(s)

                                                    0x000000007717C541, RtlUserThreadStart()+0033 byte(s)

                                                    Registers:

                                                    RAX=0000000000000000  RBX=0000000005B0E9C0  RCX=0000000000000000  RDX=0000000000000009  RSI=0000000000000000

                                                    RDI=000000000A90F850  RBP=0000000002F41270  RSP=000000000D8CF590  RIP=00000000801CEAFF  FLG=0000000000010206

                                                    R8=0000000000000000  R9=000000000A93A7D0  R10=0000000000000016  R11=000000000D8CF540  R12=0000000000000001

                                                    R13=0000000005B0E978  R14=000000000A47B730  R15=000000001522C938

                                                    Exception code: C0000005 ACCESS_VIOLATION

                                                    Fault address:  00000001801CEAFF 01:00000000001CDAFF D:\BODS\Business Objects\Data Services\bin\acta.dll

                                                    ==========================================================

                                                    Collect the following and send to Customer Support:

                                                    1. Log files(error_*, monitor_*, trace_*) associated with this failed job.

                                                    2. Exported ATL file of this failed job.

                                                    3. DDL statements of tables referenced in this failed job.

                                                    4. Data to populate the tables referenced in the failed job. If not possible, get the last few rows (or sample of them) when

                                                    the job failed.

                                                    5. Core dump, if any, generated from this failed job.

                                                    ==========================================================

(14.2) 03-25-14 08:18:54 (E) (3772:7324) SYS-170101: |Session JOB_Live_Insight_CRM_EXTRACTS_ONLY|Work flow WF_EXTRACTS|Work flow WF_EXTRACTS_1|Work flow New_WorkFlow3|Work flow WF_EXTRACTS_1A|Work flow WF_EXTRACTS_EVENTDB_CRM|Work flow WF_EXTRACT_EVENTDB|Work flow WF_EXTRACT_EVENTDB_DELTA|Work flow WF_Extract_EventDB_EU_Delta|Data flow DF_Extract_EventDB_EU_Events_Delta

                                                    桔⁥祳瑳浥攠据畯瑮牥摥愠硥散瑰潩湡⁤慣湮瑯瀠潲散獳琠楨⁳捡楴湯‮桔獩洠獥慳敧挠湯

                                                    慴湩⁳潳敭椠瑮牥慮祳瑳浥搠瑥楡獬眠楨档栠癡⁥敢湥栠摩敤潦⁲敳畣楲祴‮晉礠畯渠敥⁤

                                                    潴猠敥琠敨映汵潣瑮湥獴漠⁦桴⁥牯杩湩污洠獥慳敧‬獡潹牵愠浤湩獩牴瑡牯琠獡楳湧愠

                                                    摤瑩潩慮牰癩汩来獥琠潹牵愠捣畯瑮.

Any suggestions welcome....

Regards,

John

[neil_phillips2]

Had same issue after upgrading from 3.2 to  4.2 , resolved by switching to In-Memory as recommended in this thread .

[Former Member]

No specific solution for this, try each suggestion provided by Dirk earlier. I had few symptoms of Oracle client crashing and upgrading it to 11.2.0.3 helped. Apart from disabling stats, audit etc. DOP is still 2 and cache is memory.

Try last suggestion cross you finger for sure with each change

Regards

Manish Singh

[former_member187605]

As a summary:

1. Set DOP to 1. On an individual level, for every data flow. Or at once, by setting GLOBAL_DOP = 1 in dsconfig.txt
2. Don't collect nor use statistics
3. Disable auditing
4. Set data flow Cache Type to In-memory
5. Uncheck all advanced options in Query transforms
6. Remove blanks from all DS object names
7. Touch wood and cross your fingers

[former_member213365]

We're trying something not mentioned yet.

In the DSConfig.txt file we have set the MAX_64BIT_PROCESS_VM_IN_MB to 8192 from the default of 4096.  I have verified in the trace log that the value is being picked up by the job server.

Given that this problem sometimes appears to be related to memory, increasing the amount of RAM the Dataflow has access to _may_ help.  We've only had this active for a week, so the jury is still out.

We're working on a Linux system with 64 GB of physical RAM.  We frequently run multiple Dataflows in parallel, however, we are still using the default setting for MAX_NO_OF_PROCESSES.  So no more than 7 Dataflows are running at a time.

[former_member187605]

Note that this setting only appies when Cache Type is set to Pageable: when a dataflow needs more than MAX_64BIT_PROCESS_VM_IN_MB of memory, it will start using the Pageable cache.

[former_member213365]

True.  We're thinking that if there is more available RAM to start out with then perhaps it won't have to go to pageable memory at all.

[former_member187605]

You're absolutely right. That's why I always set data flow Cache Type to In-memory. The data flow will take all the RAM it needs.

[former_member294082]

Hi all,

We had the same issue and I tried all the settings mentioned above but the job would fail with Process Dump Error.

Finally figured out the issue with the DataFlow which was throwing this error.

The SQL in this dataflow was not getting pushed down to the database because there was the CAST function in the query. Removed the CAST function and the SQL was getting pushed down to the database and the job ran fine after this change.

This particular dataflow is working fine in our production environment with DS 4.0.

We were testing DS 4.2 upgrade in our test environment.

We have not got any process dump issue after that with this particular dataflow after this fix.

[Former Member]

Got a same issue , resolved by changing DataFlow properties  cache type to In-Memory

[Former Member]

Hi all

We are also facing similar issue and getting different error msg every time. For the same job one job execution gives bad msg error , sometimes named pipe error and in few executions its access violation errors. We do not have the collect stats enabled and global dop is 2. Restarting the OS seems to help but not always.

It's all on VM and memory/CPU does not go very high. Any more  suggestions.

[former_member187605]

Have you tried with GLOBAL_DOP=1? That sometiems helps.

[singcheong]

I wonder anyone check the OS resource contention, such as low disk space, high memory usage, and all swap space used.

Firstly, I believe there is a possibility of bug in any BODS version, but there are some users who ignore about the OS resource contention, and immediately jump into product issue, which I think they should share the finding to eliminate it before zooming into a specific area.

Let me explain why OS resource could cause an intermittent issue

1. Any data warehouse operation needs memory

2. The more data pull into Job Server, the more memory used

3. The more parallel threads execution, the data data pull into Job Server, and more memory used

4. The more expansive the operation, such as table compare, the more memory used

5. Sorting and group by are not only CPU intensive, but used memory too

If you are able to consistently reproduce, and you are not hitting memory bottleneck, then you can stop reading, and raise a bug with SAP Support.

Second, let me suggest other approach in addition to what other users have shared

1. Reboot OS of Job Server, and try again to see whether it gives better success

1.1. This will free up the swap/pagefile, if they ever allocated

1.2. This might work better in Windows since there are some background programs that will slowly used up more memory, such as background Windows Update, and lots of new Windows daemons/features

2. Capture historical CPU and memory usage - Windows can uses PerfMon, and Linux can uses sar

2.1. Includes pagefile.sys (Windows) or swap size (Linux), see whether it grows

2.2. Personally I do not recommend the used of swap (that mean should be 0 swap activity) for performance reason, although the degradation is less if the storage media is SSD.  Nevertheless, swap is a very slow operation, and 10 - 10000 times slower than physical RAM

3. When redoing the same operation, restore the database for all Datastores (source and destination) to the previous stage, in order to narrow down the issue, regardless it is BODS bug, data issue, or OS resource.  This sets the control and measure, especially SAP Support is going to ask you to repeat the operation, or you would like to test some workaround posted by others in this thread

[former_member106536]

Access violations are caused by software attempting to touch protected memory.  Not that the server is ever running out of memory...

[Former Member]

Still experiencing issues with this one. Am using 4.2.6 on a dedicated machine. The ONLY thing that corrects the issues (which is of no help at all) is to not capture statistics. Seeing as I am wanting to capture validation statistics this is not really an option that I want to be forced into. Are SAP ever going to fix this issue?

[Former Member]

I totally agree with you. I just wonder how SAP covers this issue when a customer creates an incident and demands support...

[Former Member]

I must admit I'm tempted to do this.... Will feedback when I raise the energy and time to pursue this!

[Former Member]

Hi Guys,

We had the same issue in our environment. If you do a googling for the issue you can find different people got rid of this issue in different ways ie, the root cause for this error is not same..or BODS is throwing these errors when its not able to identify the exact issue..

Some of the solutions (referring to comments in bobj and scn forum) are

1. Changing the cache of Data Flow to "In  Memory" from Pagable

2. Removing Spaces in WF/DF/Stage names (Funny??  its been reported in few forums that the error got disappeared after changing name)

3. Changing DOP to 1

4. Changing Dump option in DSConfig.txt

Finally, What resolved my error..

Its was NULL which caused the issue in my job ...nullable column was included in least() function and when NULL came through, the job gone mad

Regards,

Manu

[Taeket]

Hi all,

Had the same problem as described above, in version 4.2 SP5.  The Job has Table Comparison.  In our case, the issue was bypassed by changing the DOP for the particular Workflow from 8 to 1.  We left the "central" DOP as it was.

[Former Member]

In my case (I had this issue with several customers) it helped to set the job options to not to "collect statistics for optimization" and not to "use statistics".

Best regards

Marco

[Former Member]

We are also experiencing this issue once in a month.
Did Anybody retrieve a decent solution already?

thanks in advance.

[former_member1095969]

Thanks All,

It looks like we have a few tricks up our sleeves to get around the issue, but what annoys me the most is a lack of a SAP fix - it's been over a year now with them, and still haven't heard anything since it went into developement for a fix.

We shouldn't have to drastically change our codeset, as they worked fine on previous versions.

Radio Silence from SAP.

[former_member429424]

Hi,

We have fixed our Access Violation issues by setting the global job server parameter (DSConfig.txt) :

Global_DOP = 0  

Our environment has been much more stable since applying this change and we have only had 1 access violation error in a month now. 

We were also getting the following intermittent error:

[FROMDBLOVER] RWDecimal: overflow converting from double (0)

It was been fixed by adding the following parameter:

Use_Decimal_Mutex = 1

[former_member213365]

Running DS 4.1 on Linux were were getting random failures on a small set of Dataflows.  Desperate to stop the bleeding I changed the Cache type to In-memory and haven't heard a peep out of them since then.  I suspect that the ETL statistics process is the root of the issue.  My "solution" isn't going to work on every Dataflow.  Just the ones you look at and say, "That shouldn't use much memory."

I opened an issue with SAP Tech Support and it went on for weeks with them asking random questions and basically not really trying to help.

[former_member106536]

What we just found was that if you print all trace messages, we no longer received our random access violation errors.  Tried this with a couple jobs already and it has worked for both of them.

[Former Member]

Did anybody found a solution to this? We are getting the same error now and I have no idea how to resolve this.

Thanks

[former_member1095969]

As you can see, raised this formally on 02/04/2014 - it took over a month to convince SAP it was a bug with all developers raising and making noise about it.

Then month later was "In Processing".

Keep asking for updates but nothing since last time I asked.

It's gone stone silent, usual SAP approach to any major bugs......waste of time raising it.

[former_member429424]

John,

Have SAP come back to you about this one yet??  I'm getting the same issue on our new 14.2.3 (latest support pack and patch) windows 2012 environment.  It seems to be random and impossible for us to reproduce here.  Lots of people are discussing it on various forums and i can't see any answers yet!

I'm going to raise this officially with SAP also. 

It's possibly going to delay us moving on to this new platform because at present its too flakey for a production environment.

[former_member1095969]

You can code around it, by decreasing degrees of parallelism to 1 (instead of default 0, which normally is 2 via ds_config.txt). These are normally on any DFs that have table comparisons in.

You'll find DFs with case statements and other operations will fail due to this error - it seems random but the more it happens over a period time, you'll start to narrow it all down and see the same old DFs that could potentially fail to it, will at some point during the weeks/months. We had to sometimes either split DFs or add a data transfer to avoid access violations cropping up.

It took us a couple of months to fully stabilise our code set again - it was very painful and no help from SAP whatsoever - it took them a while to even admit the problem.

We had to upgrade for various reasons, but it wasn't a pleasant experience.

[former_member429424]

Thanks John, this is really useful to know.  I did see a post on a forum somewhere about your DOP suggestion so will have a go at that for the DFs that fail.  I'm keeping a log of all DF that do fail with this error so can go back and change the DOP to 1.  I've just left our DSConfig with the defaults.

Luckily we don't have too much pressure to move onto this new env just (yet!) so can try and stabilise what we have. 

I'll update this post next month with an update.

Thanks...

[Former Member]

HI John,

We have a similar situation. Can you please let us know if your issue was resolved?

Thanks.

[Former Member]

Hi All,

I have the similar situation/error when am loading the data to hana from the flatfile (txt file with Tab delimited).

The file i have is having bad data like extra blank columns.

So i have tried with "Adaptable Schema = YES". to ignore the blank columns and continue the load.

I still received the same error.

Then i saved the txt file as .xls and removed the extra columns and saved back to txt file and then ran the job. This time the job was successful.

But, i don't think this is the exact solution which cannot be done all the time.

If there is any fixed solution for this. Please keep it posted so that it will be very helpful.

If any information needed. Please let me know.

Thanks,

VijayVanamala.

[former_member1095969]

Thanks everyone for replying and making me feel sane - the way SAP treats this is like it is unique to our own environments and they can't replicate the issue.

The more we all chip in and colloborate, the bigger the weight we can use to say please fix this in BODS.

I have sent back my incident to be reviewed again, it still hasn't been escalated and is languishing on the "cannot replicate" point.

[Former Member]

Hi John,

My only advice is to push hard to have it escalated. I was treated much the same way by the support consultant and was quite blunt in the end that "it may be environmental but its your product failing where as it was working fine before and we therefore need your developers to  understand what is going on!'.

Its disappointing as we are almost at 3 weeks escalated and no further feedback. More money on back end support and less marketing to me about how wonderful the products are.

[Former Member]

Hi John,

not much help but we are also experiencing a lot of access violations since upgrading to 4.2 and its happening on multiple environments.  New code seems to work fine however existing jobs that are having to use caching to disk are failing.

We have experience this issue with group by, sort and lookups that are paged and very frustrated at this point. Funnily enough the quickest solution to fix 99% of the issues is to turn off the statistics options. Other solutions have been to put in a Data_Transfer prior to the group/sort.

I have opened a case with SAP but as per usual we have to wade through the typical questions before getting to the meat of the problem.

Looking at the note posted by Sebastian I'm hoping the fix indicated will be out very soon as we are losing a lot of confidence in the products having also encountered significant bugs in our recent BI 4 upgrades.

Cheers

Glenn

[former_member232794]

Hi John Lowth,

i have experience the same problem with  2 customer: we have upgrade DS 4.0 to DS 4.2 SP1 Fix3 (last update) and the same problem occurs.

In this evening i try to implement 2 workaorund:

1) turn off the option "collect statistics"

After that DS experienze Freeze before complex Stage with multiple qwery trasform/R3_Df before a MERGE operation or Befor TABLE COMPARISON or MAP_OPERATION.

2) I place before the complex stage (merge,map etc ) a DX TRANSFER for every input.

This workaround seems to solve the Access Violation and Freeze after disable collect statistics

[Former Member]

Hi Glenn,

Did you find any solution. We are getting same error.

Thanks,

Nishith

[Former Member]

Hi Nishith,

Unfortunately we are still having to use our workarounds.

We have raised a call with SAP support and its now into its 5th week. 3 weeks of that was disappointing backwards and forwards with the support rep telling us that because she couldn't replicate it she wouldn't escalate to the developers even though we had provided a lot of information up front including dmp files.

We finally had to work with our account rep and support advisory person to try and move this forward and has been escalated for a week and a half now to the developers. I now have a few people who have contacted me with the issue as well which is comforting as the support rep wasn't overly helpful and was putting the onus on me to find the problem rather than helping us determine what's going on.

I will update here if/when we getting further responses from the developers.

cheers

Glenn

[Former Member]

Did you ever get a resolution to this issue?

I am experiencing this on a relatively basic dataflow too. I have tried all the solutions here but none of them worked.

The bug report linked by is for DS4.1, but we are on 4.2.2 right now.

[former_member136842]

Hi,

i think you are adressed by this Bug:

http://service.sap.com/sap/support/notes/1932983

Maybe you want to create an Incident with the SAP Support to validate this.

Regards

-Seb.