Adaptive Server Enterprise system exception (0xc0000005) generated by a storage access violation

[former_member117942]

Hi,

I've have installed SAP ERP 6.0 EHP6 on windows /sybase ASE 15.7.0.041 and system was running well from several days.

Tomorrow I've got this  unexpected error in the ASE log file:

##################

00:0006:00000:00076:2013/04/03 21:02:50.61 server  Increase the config parameter 'number of open partitions' to avoid descriptor reuse. Reuse may result in performance degradation.

00:0008:00000:00072:2013/04/04 09:03:24.49 server  Increase the config parameter 'number of open partitions' to avoid descriptor reuse. Reuse may result in performance degradation.

00:0006:00000:00191:2013/04/04 11:37:09.70 kernel  attempted read from memory location FFFFFFFFFFFFFFE4 from instruction at location 0x00000001404B2D60

00:0006:00000:00191:2013/04/04 11:37:09.78 kernel  Adaptive Server Enterprise system exception (0xc0000005) generated by a storage access violation.

00:0006:00000:00191:2013/04/04 11:37:11.42 kernel  Symbolic stack trace information is successfully loaded

00:0006:00000:00191:2013/04/04 11:37:13.84 kernel  pc: 0x00000001404B2D60 ssql__choose_victim+ 0x50 (0x000000000000000E, 0x0000000000000000, 0x0000000053B79A50, 0x000000006F046A1D)

00:0006:00000:00191:2013/04/04 11:37:13.84 kernel  pc: 0x00000001404B6BD5 ssql_save_SQL+ 0xa45 (0x0000000027DA0081,

##############

Sybase ASE has stopped working, SAP processes has changed their status in reconnected mode.

After 20-30 minutes of Sybase activity (sqlserver.exe process was running with high CPU consumption - see log file reported below), I've reboot the server (the only way to restart application) and now SAP System is online. ASE has made recovery with no problem.

Windows system is virtualized with VMware; no error was reported in the Event System of windows server.

No error was reported by VMware event log.

Do you have any idea what might have happened ?

Thanks

  Maurizio Manera

LED - ASE LOG FILE.txt.zip

[Former Member]

FYI, these errors are not gone yet as of Adaptive Server Enterprise 15.7 SP102. Just had it today.

2014/04/08 13:27:23.76 kernel  attempted read from memory location FFFFFFFFFFFFFFE4 from instruction at location 0x0000000000ED23E0

2014/04/08 13:27:23.82 kernel  Adaptive Server Enterprise system exception (0xc0000005) generated by a storage access violation.

2014/04/08 13:27:26.09 kernel  Symbolic stack trace information is successfully loaded

There are three sapnotes on similar Access violation errors relevant to earlier releases.

[madhvi_pai]

Hi Alex,

Please see SAP Note 1925953 which likely applies in this case.

Thanks,

Madhvi Pai
SAP AGS

[Former Member]

Maurizio

What is the end of this story? Did you resolve the issue?

[former_member117942]

Hi,

I've checked all parameters as indicated in oss note and the problem has disappeared.

I'm not sure if this is the solution, but i advise  you to upgrade sybase to the last version.

There are many bugs that have been corrected  with last patches.

We are waiting for a new fix from Sap for another customer where sybase on Windows sometimes crashes.

So the rule is to install the last patch of sybase.

Regards.

Maurizio

[Former Member]

Hi Maurizio,

ASE on Windows crashes due to system exception (0xc0000005) generated by a storage access violation with error message "attempted write to memory location 0000000000000000 from instruction at location 0x0000000000000000".

Solution

apply on your server the EBF 21519: 15.7 SP102

Regards,

César Fonseca

[sap_mk]

Hi Maurizio,

In addition to Bret's reply, please check the size of your statement cache. Per Note 1539125, it needs to be at least 100MB. However, to prevent frequent flushing, you may need a 2x or 3x larger size.

The high CPU following the error is due to the other processes trying to get a spinlock. We can see this in the following messages:

timeslice -1001, current process infected at 00000001408BD743 (spinobtain+ 0x13 )

Regards,

Mark Kusma

[tilman_model-bosch]

Dear Maurizio

Adding to Bret and Mark's reply :

It is indeed very unlikely that it is related to a vmware issue - looks like an spot in ASE code with potential for improvement.

Please configure shared memory dumps in the system as suggested by Bret .

In addition set a CSMD for 'panic' condition

exec sp_shmdumpconfig 'add', panic, NULL, 1, "<DRIVE>:\fullpathtodumpdirectory", null

go

Once the issue reproduced and shred memory dump has been collected please open a support ticket at SAP.

Rgds

Tilman

[former_member188883]

Hi Maurizio,

To avoid unnecesary performance penalities in sybase ase 15, you need to set the configuration parameter 'number of open partitions' high enough. In Sybase ASE 15 each table and index consists of atleast one partition even when you don't use semantic data partitioning (optional licensed feature).

Sybase recommends using sp_countmetadata to estimate the required setting of this parameter and sp_monitorconfig to monitor its usage.

The default run value is 500. If this number is insufficient, Adaptive Server displays a message after trying to reuse active partition descriptors. You must adjust this value.

Estimate number of open partitions setting for your sybase server. 

To set the number of open partitions parameter optimally:

1. Use sp_countmetadata to find the total number of open partitions. For example:

sp_countmetadata "open partitions"

The best time to run sp_countmetadata is when there is little activity in the server. Running sp_countmetadata during a peak time can cause contention with other processes.

Suppose Adaptive Server reports the following information:

  There are 42 user partitions in all database(s), requiring 109 Kbytes of memory. The 'open partitions' configuration parameter is currently set to 110.

2. Configure number of open partitions to that value, as follows:

  sp_configure "number of open partitions", 110

3.During a peak period, find the number of active metadata cache descriptors, for example:

  sp_monitorconfig "open partitions"

Usage information at date and time: Apr 02 2013 2:49PM.

Name              num_free num_active pct_act        Max_Used    Reused

--------------          --------      ---------       --------            --------------    --------------------

number of open          160          357        71.40              397               No

In this example, 397 is the maximum number of partition descriptors that have been used since the server was last started.

4. Configure the number of open partitions to 397, plus 10 percent (40), for a total of 437:

  sp_configure "number of open partitions", 437

Hope this resolves the issue.

Regards,

Deepak Kori

[former_member117942]

Hi Kori,

I think this is not my problem.

The error reported by ASE was:

00:0006:00000:00191:2013/04/04 11:37:09.70 kernel  attempted read from memory location FFFFFFFFFFFFFFE4 from instruction at location 0x00000001404B2D60

00:0006:00000:00191:2013/04/04 11:37:09.78 kernel  Adaptive Server Enterprise system exception (0xc0000005) generated by a storage access violation.

This message before error:

00:0006:00000:00076:2013/04/03 21:02:50.61 server  Increase the config parameter 'number of open partitions' to avoid descriptor reuse. Reuse may result in performance degradation.

is only an information to increase a config parameter.

I've also execute a check of this parameter. Here the output message:

1> sp_countmetadata "open partitions"
2> go
There are 183509 user partitions in all database(s), requiring 250925 Kbytes of
memory. The 'open partitions' configuration parameter is currently set to 50000.
(return status = 0)

1> sp_monitorconfig "open partitions"
2> go
Usage information at date and time: Apr  4 2013  2:56PM.

Name                      Num_free    Num_active  Pct_act Max_Used
         Reuse_cnt   Instance_Name
------------------------- ----------- ----------- ------- -----------
         ----------- ------------------------------
number of open partitions       40162        9838  19.68         9838
                   0 NULL

(1 row affected)
(return status = 0)

Thanks.

Maurizio

[former_member188958]

Hi Maurizio,

I don't think the messages about open partitions are related to the storage access violation.

The function call on the stack, ssql__choose_victim, has to do with ASE trying to find unused query plans in statement cache to free in order to make room for a new statement.

There are no known bugs or even reported incidents involving this function in ASE 15.7.x

To help diagnose this issue, Support would be interested in seeing a configured shared memory dump on this error, with procedure cache included.  Data cache should not be needed.

The error type would be "signal" (storage access violation is equivalent to an "infected with signal 11" error on UNIX systems.

http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc36273.1572/html/sprocs/BEIGIFCI.htmhttp://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc00641.1572/html/new_features/new_fea...

If you are willing and able to configure a dump, the commands to do so are:

sp_configure "enable dump conditions", 1

go

declare @sig int
select @sig=hextoint("0xc0000005")
exec sp_shmdumpconfig 'add', signal, @sig, 1, "full_pathname_to_dump_directory", null
go

-bret