ORA-03113: end-of-file on communication channel - Central System

[former_member189462]

Hi experts,

I’m going to describe our current scenario as detailed as possible.

The following dump is being constantly obtained in our ERP 6.0 EHP7 system (Windows 2012/Oracle 11.2.03):

Category               Installation Errors

Runtime Errors         DBSQL_SQL_ERROR

1. Except.                CX_SY_OPEN_SQL_DB

Date and Time          12.04.2015 12:47:04

----------------------------------------------------------------------------------------------------

|Short Text |

|    SQL error "SQL code: 3113" occurred while accessing table "TSPOPTIONS".                       |

----------------------------------------------------------------------------------------------------

|What happened? |

|    Database error text: "SQL message: ORA-03113: end-of-file on communication                    |

|     channel

Reviewing the developer trace in addition to the above oracle error the ora-03114 and ora-03135 are also present. Please see below>

C  *** ERROR => ORA-03135 occurred at SQL stmt (parse error offset=0)

B  ***LOG BY0=> ORA-03135: connection lost contact

C     ORA-03135: connection lost contact

C     ORA-03114: not connected to ORACLE

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-12571: TNS:packet writer failure

C     ORA-01041: internal error. hostdef extension doesn't exist

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03113: end-of-file on communication channel

C     ORA-01041: internal error. hostdef extension doesn't exist

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03135: connection lost contact

C     ORA-01041: internal error. hostdef extension doesn't exist

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03135: connection lost contact

C     ORA-01041: internal error. hostdef extension doesn't exist

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03114: not connected to ORACLE

C  DB instance  is running on  with ORACLE version 11.2.0.3.0 since APR 04, 2015, 16:57:44

C     ORA-03135: connection lost contact

Even when it’s seems to be a very common issue, almost all notes and SCN questions point me to a solution for distributed system but in my case this is a central one.

The following note describes the scenario exactly as the one we have, but again it seems to apply for distributed systems: 2064591 - work processes reconnect to Oracle database with error ORA-03135. The proposed solution has been tested but with no lack. Also the solution presented in the bellow links but with no result either:

http://www.oracle-experts.info/ora-03135-connection-lost-contact

http://www.inthebasis.com/solve-ora-03135-connection-lost-contact/

Just in case the Windows Firewall has been completely turned off and the NIPING self test is OK.

In addition, the data base library (lib_dbsl_215-10012499.sar) has been updated and we are planning to update the whole kernel and oracle instant client.

Could you please suggest what to check or what would be the recommended troubleshooting?

Thanks

[Former Member]

Dear Diego,

did you solve your problem?

We face some similar errors and maybe you have a solution or a little hint.

Thanks

Kind regards

Martin

[former_member189462]

Hi Martin,

In our case, the problem was in connection with the network card used in VMware. After performing some analysis it was noticed the ocurrence of the following warning in Windows event viewer:

Log Name:      System

Source:        e1iexpress

Event ID:      27

Task Category: None

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      --- censored by me! ---

Description:

Intel(R) 82574L Gigabit Network Connection

Network link is disconnected.

After comparing, it was noticed that system dumps were obtained just after the occurrence of the above events.

Notes 2161991 and 2015392, pointed me to the right solution:

“We recommend you choose VMXNET 3 virtual NICs for your latency-sensitive or otherwise performance-critical VMs.”

“Use vmxnet3 network cards to reach best network performance. For details, see VMware KB 1001805:http://kb.vmware.com/kb/1001805.”

After replace Intel(R) 82574L Gigabit Network Connection with VMXNET 3, the OS warnings stopped happening and system dumps as well.

Hope this helps you.

Regards

[Former Member]

Hi Diego,

thank you very much for your answer.

I will give your information to my colleagues, maybe it helps.

Kind regards

Martin

[former_member182657]

Hi,

You could also refer SAP notes   1672954 - Oracle 11g: Usage of hugepages on Linux

&   335199 - ora-3113 collective note

Regards,

[JamesZ]

Dear Diego,

Indeed this is a very common issue.
Please upload the oracle alert log to this discussion.

Best regards,
James

[former_member189462]

Hi James,

Please find attached the most recent alert.log file.

I was however not able to find any ORA error or message in connection with the ones described in dumps files or dev_trace. Dumps are still being generated.

Please let me know if something else is required.

Regards

alert.txt.zip

[JamesZ]

Dear Diego,

This alert log is xml version, please attach the alert_<sid>.log from saptrace directory.

Best regards,
James

[Sriram2009]

Hi

Could you refer the SAP KBA error message "ORA-03113: end-of-file on communication channel"

2152158 - Intermittent Oracle connectivity issue

1678268 - ORA-03113 on shutdown

BR

SS

[former_member189462]

Hi James,

Find attached the requested file. Please note that due its size the file was compressed and its extension changed in order to upload it. Please change file extension back to .rar before open it.

Regards

alert_SID.rar.txt.zip

[former_member189462]

Hi SS,

Note 2152158 - Intermittent Oracle connectivity issue applies for Linux platforms or RAC installation. We are on Windows and it's not a RAC installation.

Note 1678268 - ORA-03113 on shutdown is for AIX environments.

Thansk for your reply.

Regards

[JamesZ]

Hi Diego,

Just like I suspected, there are full of network errors in the alert log.

First you have to let your network vendor to check if there is network issue. Only if network is realy fine. Then you need to set oracle sqlnet trace as per SAP note 562403. Please set parameter DIAG_ADR_ENABLED=OFF in sqlnet.ora and then other parameters can work. See
link:

Parameters for the sqlnet.ora File

Best regards,
James