LiveCache complete data backup on FILE is failing.

[former_member270282]

Hello Experts.

liveCache Version         KERNEL    7.7.04   BUILD 029-123-196-543

DBMServer Version         DBMServer 7.7.04   Build 029-123-196-543

Operating System          Linux

SCM/APO 7.0

I wanted to keep AUTOLOG mode on. so, when i was scheduling, it has mentioned that "complete data backup" is missing. which i found is the right away.

so I have set the media backup as "FILE". on my file system. my run directory is /sapdb/wrk/data/<SID>.

overwritable - NO

Type - FILE

Backup Template - DATA

it failed. here is the log from DB13.

Hello Experts.

I wanted to keep AUTOLOG mode on. so, when i was scheduling, it has mentioned that "complete data backup" is missing. which i found is the right away.

so I have set the media backup as "FILE". on my file system. my run directory is /sapdb/wrk/data/<SID>.

overwritable - NO

Type - FILE

Backup Template - DATA

it failed. here is the log from DB13.

Job started

Step 001 started (program RSDBAJOB, variant &0000000000016, user ID )

Execute logical command DBMCLI On host helsid01

Parameters:-d SID -n helsid01 -uUTL -tpi SID -tpp no_longer_used -tpc LCA backup_start COM RECOVERY DATA AUTOIG

NORE

ERR

-24988,ERR_SQL: SQL error

-903,Host file I/O error

3,Data backup failed

Servertask Info: because Error in backup task occured

Job 1 (Backup / Restore Medium Task) [executing] WaitingT152 Result=3700

Error in backup task occured, Error code 3700 "hostfile_error"

Backupmedium #1 (SID_COM) cannot open xparam file

Backup error occured, Error code 3700 "hostfile_error"

External program terminated with exit code 1

DBMCLI returned error status E

Job finished

this is from OS level

more /sapdb/wrk/data/SID/dbahist/dba20130303090443.sda
*** DBA Action starts:
Timestamp: 20130303090443   Function: sda   Object: DATA
*** SAVE/RESTORE request:
SAVE DATA QUICK TO 'SID_COM' FILE BLOCKSIZE 8 NO CHECKPOINT MEDIANAME 'COM'
*** SAVE/RESTORE request accepted:
OK

*** SAVE/RESTORE result:
ERR
-24988,ERR_SQL: SQL error
-903,Host file I/O error
3,Data backup failed
Servertask Info: because Error in backup task occured
Job 1 (Backup / Restore Medium Task) [executing] WaitingT152 Result=3700
Error in backup task occured, Error code 3700 "hostfile_error"
Backupmedium #1 (SID_COM) cannot open xparam file
Backup error occured, Error code 3700 "hostfile_error"

*** SAVE/RESTORE request:
SAVE DATA CANCEL
*** SAVE/RESTORE request accepted:
OK

*** SAVE/RESTORE result:
ERR
-24988,ERR_SQL: SQL error
-104,DBM command impossible at this time
3,Data backup failed
Servertask Info: because Error in backup task occured
Job 1 (Backup / Restore Medium Task) [executing] WaitingT152 Result=3700
Error in backup task occured, Error code 3700 "hostfile_error"
Backupmedium #1 (SID_COM) cannot open xparam file
Backup error occured, Error code 3700 "hostfile_error"

*** DBA Action ends:
Timestamp: 20130303090944   State:    Failure

tail -100 /sapdb/wrk/data/SID/dbm.prt

2013-03-03 09:02:31 0x00004ed8 INF        283 DBMSrv   Command 'backup_start COM RECOVERY DATA AUTOIGNORE' is being executed.

2013-03-03 09:02:31 0x00004ed8 ERR     -24580 DBMSrv   ERR_COMMAND_FAILED: Command 'backup_start' has ended and failed with return code -24958.

                    0x00004ed8 ERR     -24958 DBMSrv   ERR_BAOVERWRITE: file already exists, cannot overwrite

2013-03-03 09:02:31 0x00004ed8 INF          2 DBMSrvCo A DBM Server client connection was released (client process has process ID 8404 on computer helsid01.fiskars.com).

2013-03-03 09:04:43 0x00004f29 INF          1 DBMSrvCo A DBM Server client connection was established at 2013-03-03 09:04:43 (client process has process ID 9391 on computer .

2013-03-03 09:04:43 0x00004f29 INF        283 DBMSrv   Command 'backup_start COM RECOVERY DATA AUTOIGNORE' is being executed.

2013-03-03 09:04:43 0x00004f29 INF          1 DBMKnl   Sending an administrative statement to the database

                    0x00004f29 INF          8 DBMKnl   Statement:  SAVE DATA QUICK TO 'SID_COM' FILE BLOCKSIZE 8 NO CHECKPOINT MEDIANAME 'COM'

2013-03-03 09:09:44 0x00004f29 ERR          3 DBMKnl   Received the result of an administrative statement from the database

                    0x00004f29 ERR          9 DBMKnl   Statement:  SAVE DATA QUICK TO 'SID_COM' FILE BLOCKSIZE 8 NO CHECKPOINT MEDIANAME 'COM'

                    0x00004f29 ERR         10 DBMKnl   Returncode: -903

                    0x00004f29 ERR          5 DBMKnl   Errortext:  Host file I/O error

2013-03-03 09:09:44 0x00004f29 ERR     -24580 DBMSrv   ERR_COMMAND_FAILED: Command 'backup_start' has ended and failed with return code -24988.

                    0x00004f29 ERR     -24988 DBMSrv   ERR_SQL: SQL error

                    0x00004f29 ERR     -24778 DBMSrv   -903,Host file I/O error

                    0x00004f29 ERR     -24778 DBMSrv   3,Data backup failed

                    0x00004f29 ERR     -24778 DBMSrv   Servertask Info: because Error in backup task occured

                    0x00004f29 ERR     -24778 DBMSrv   Job 1 (Backup / Restore Medium Task) [executing] WaitingT152 Result=3700

                    0x00004f29 ERR     -24778 DBMSrv   Error in backup task occured, Error code 3700 "hostfile_error"

                    0x00004f29 ERR     -24778 DBMSrv   Backupmedium #1 (SID_COM) cannot open xparam file

                    0x00004f29 ERR     -24778 DBMSrv   Backup error occured, Error code 3700 "hostfile_error"

2013-03-03 09:09:44 0x00004f29 INF          2 DBMSrvCo A DBM Server client connection was released (client process has process ID 9391 on computer ).

2013-03-03 09:09:44 0x00004f29 INF          1 DBMKnl   Sending an administrative statement to the database

                    0x00004f29 INF          8 DBMKnl   Statement:  SAVE DATA CANCEL

2013-03-03 09:09:44 0x00004f29 ERR          3 DBMKnl   Received the result of an administrative statement from the database

                    0x00004f29 ERR          9 DBMKnl   Statement:  SAVE DATA CANCEL

                    0x00004f29 ERR         10 DBMKnl   Returncode: -104

                    0x00004f29 ERR          5 DBMKnl   Errortext:  DBM command impossible at this time

2013-03-03 10:33:00 0x00006404 INF        283 DBMSrv   Command 'dban_stop' is being executed.

2013-03-03 10:33:06 0x00006404 INF        419 DBMSrv   Command 'dban_stop' has ended with return code 0.

2013-03-03 10:33:07 0x00005b36 INF          1 DBMSrvCo A DBM Server client connection was established at 2013-03-03 10:33:07 (client process has process ID 23295 on computer .

so backup is failing because "cannot open xparam file". i have searched the whole system. couldn't fine anything releant there. i have searched service marketplace and understood that X_PARAM is used for APO 1.1. this in SCM 7.0 . so not relevant.

google is also failed for this.

so this is the last resort. please help.

BR

[former_member229109 · Accepted solution]

Hello,

could you run the following commands on the liveCache server as OS user in sdba group:

dbmcli -d <SID> -u control,<pwd>

<enter>

param_checkall

<enter>

db_admin

<enter>

medium_getall

<enter>

medium_put Test  Compete FILE DATA 0 8 NO

<enter>

db_connect

<enter>

backup_start Test RECOVERY DATA AUTOIGNORE

<enter>

exit

ls -l /sapdb/wrk/data | grep SID

ls -l /sapdb/wrk/data/<SID> | grep Complete

What is the liveCache name?

Did you create SAP message where you reported this issue?

Backup Data document in MAXD library:

http://maxdb.sap.com/doc/7_7/44/bbddac91407006e10000000a155369/frameset.htm

Regards, Natalia Khlopina

[former_member270282 · Accepted solution]

Hello Natalia,

Thank you for your reply. Here is the data you have asked for.

i have replaced livecache name here with SID just to keep my project confidential. please let me know if i really need to disclose

. no, i haven't opened the SAP message yet. i have received so many solutions from SCN and especially from your posts. so thought to post here first.

more /etc/group | grep sdba
sdba:x:901:sidadm

helsid01:sidadm 123> dbmcli -d SID -u control,password

dbmcli on SID>param_checkall
OK

---
dbmcli on SID>db_admin
OK

---
dbmcli on SID>medium_getall
OK
COMM    SID_COMM        FILE    AUTO    0       8       YES     NO              20130101220529  20130101220529          NONE    0
COM     SID_COM FILE    DATA    0       8       YES     NO              20130303131527  20130303131527          NONE    0

---
dbmcli on SID>medium_put Test  Compete FILE DATA 0 8 NO
OK

---
dbmcli on SID>db_connect
OK

---
dbmcli on SID>backup_start Test RECOVERY DATA AUTOIGNORE
ERR
-24988,ERR_SQL: SQL error
-903,Host file I/O error
3,Data backup failed
Servertask Info: because Error in backup task occured
Job 1 (Backup / Restore Medium Task) [executing] WaitingT266 Result=3700
Error in backup task occured, Error code 3700 "hostfile_error"
Backupmedium #1 (Compete) cannot open xparam file
Backup error occured, Error code 3700 "hostfile_error"

---
dbmcli on SID>

exit

ls -l /sapdb/data/wrk | grep SID
i didn't find wrk/data under /sapdb. but i found other way around. /sapdb/data/work so here the output for that.

[root@helsid01 sapdb]# ls -l /sapdb/data/wrk | grep SID
drwxrwx--- 6 sdb    sdba      4096 Mar  5 07:50 SID
-rw-rw---- 1 sdb    sdba      2601 Mar  5 07:55 SID.dbm.shi
-rw-rw---- 1 sdb    sdba    102401 Mar  5 07:55 SID.dbm.shm
-rw-rw---- 1 sdb    sdba      8513 Mar  3 09:09 SID.knl

ls -l /sapdb/data/wrk/SID | grep Complete
Nothing here.

BR

[former_member270282 · Accepted solution]

Hello Live Cache Experts,

Hoping to receive some help.

BR

[former_member229109 · Accepted solution]

Hello BR,

we help on forum volunteerely, so if you need help urgently => create SAP message & get SAP support, it will be also easy to do analysis of the issue on your system.

-> First, please don't use the RUNDIRECTORY for the databackup and autosavebackups writing.

   If the FS will be full, the databackup of big size and/or you forgot to archive the logbackups files, the database logs will be not written, and the confif files could be corrupted.

-> What are ulimits of sidadm OS user?

    How much data you have in liveCache? < data usage >

-> Post output:

     xinstinfo <SID>

     ls -l /sapdb/data/config

     ls -l /sapdb/data/wrk/<SID>

Regards, Natalia Khlopina

[former_member270282 · Accepted solution]

Hello Natalia,

Thanks for your reply.

not urgently, i thought that this blog might have been overlooked. so asked. anyway, appreciate your help.

-->  sure, i will change the backup location. is there a parameter?

-->  What are ulimits of sidadm OS user? sidadm is under sapsys group.

i tried ulimit -a but didnt work for RHEL 5.6

so here is the output of /etc/security/limits.conf

@sapsys    hard    nofile    32800

@sapsys    soft    nofile    32800

@dba       hard    nofile    32800

@dba       soft    nofile    32800

-->  How much data you have in liveCache? 30 GB in last 6 months.

sid01:sidadm 124> xinstinfo SID
IndepData           : /sapdb/data
IndepPrograms       : /sapdb/programs
InstallationPath    : /sapdb/SID/db
Kernelversion       : KERNEL    7.7.04   BUILD 029-123-196-543
Rundirectory        : /sapdb/data/wrk/SID

sid01:sidadm 125> ls -l /sapdb/data/config
total 304
-rw-rw---- 1 sdb  sdba        54 Mar 21  2012 Databases.ini
-r--r----- 1 sdb  sdba     22230 Feb 21 10:46 SID
-rw-r--r-- 1 sdb  sdba     22229 Sep 18 20:23 SID_18sept
-rw-rw---- 1 sdb  sdba        52 Mar  3 08:56 SID.cfg
-rw-rw---- 1 sdb  sdba       542 Mar  5 07:50 SID.mmm
-rw-rw---- 1 sdb  sdba    226050 Feb 21 10:46 SID.pah
-rw-rw---- 1 sdb  sdba      2560 Mar  3 08:56 SID.upc
drwxr-xr-x 2 root sapinst   4096 Mar 21  2012 install
-r--r--r-- 1 sdb  sdba        73 Mar 21  2012 Installations.ini
-r--r--r-- 1 sdb  sdba        48 Mar 21  2012 Runtimes.ini

sid01:sidadm 126> ls -l /sapdb/data/wrk/SID
total 19992
drwxrwx--- 71 sdb sdba     4096 Mar  9 00:13 analyzer
-rw-rw----  1 sdb sdba    65536 Mar  5 07:50 Compete
drwxrwx---  2 sdb sdba     4096 Mar  5 07:50 dbahist
-rw-rw----  1 sdb sdba       52 Mar  3 08:56 dbm.cfg
-rw-rw----  1 sdb sdba   844655 Mar  3 08:56 dbm.ins
-rw-rw----  1 sdb sdba     9080 Mar  5 07:55 dbm.knl
-rw-rw----  1 sdb sdba      542 Mar  5 07:50 dbm.mmm
-rw-rw----  1 sdb sdba   350378 Mar  5 10:06 dbm.prt
-rw-rw----  1 sdb sdba     2560 Mar  3 08:56 dbm.upc
drwxrwx---  5 sdb sdba     4096 Mar  5 07:48 DIAGHISTORY
-rw-rw----  1 sdb sdba    65536 Mar  3 09:04 SID_COM
-rw-rw----  1 sdb sdba    65536 Feb 24 20:51 SID_COM.old
-rw-rw----  1 sdb sdba  1151030 Mar  9 11:11 KnlMsg
-rw-rw----  1 sdb sdba   225280 Mar  5 07:55 KnlMsgArchive
-rw-rw----  1 sdb sdba   351415 Mar  5 07:48 KnlMsg.old
-rw-rw----  1 sdb sdba 16728064 Mar  5 07:48 knltrace
-r--r-----  1 sdb sdba   173802 Mar  3 08:47 lcapo_com_trace_01.txt
-r--r-----  1 sdb sdba     2918 Mar  3 08:47 lc_apo_plvlog.txt
-r--r-----  1 sdb sdba     1653 Mar  5 07:49 lc_apo_version.txt
-rw-rw----  1 sdb sdba   309133 Mar  5 07:49 lcinit.his
-rw-rw----  1 sdb sdba    14755 Mar  5 07:49 lcinit.log
drwxr-x---  2 sdb sdba     4096 Mar  8 19:27 rtedump_dir

BR

SAMIRR

[former_member270282 · Accepted solution]

Hello Natalia,

Found the way around to check the ulimit of SIDADM. i think its unlimited.

sidadm 124> bash -c "ulimit -a"

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 139264

max locked memory       (kbytes, -l) 32

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32800

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) 139264

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

[Former Member · Accepted solution]

Hi BASIS,

I faced this issue as well.

go to /sapdb/data/wrk. please check the permission of niserverxxxxxxxxxxxxfiles.

they should be sdb:sdba

BR

Samirr DeSai.

[former_member246694]

Hi

Problem has been resolved as /usr/sap/SID got fulled. After making this directory free, data backup has finished successfully.

Thanks

[former_member246694]

Hi Samil

Permissions of niserverxxxxxxxxxxxxfiles are sdb:sdba only.

But still backup has been cancelled with error

Job started

Step 001 started (program RSDBAJOB, variant &0000000000438, user ID SAP*)

Execute logical command DBMCLI On host pspdscmq

Parameters:-d LC2 -n pspdlcq -uUTL -tpi SCQ -tpp no_longer_used -tpc LCA backup_start Back_up RECOVERY DATA AUT

OIGNORE

Please check file DBAHIST.PRT --> No header?

Error! Connection failed to node pspdlcq for database LC2:

tp error:

External program terminated with exit code 2

DBMCLI returned error status E

Job finished

Please help.

Thanks