Backup failing insecure ssfs

[Former Member]

When I start a back up (whether file or backint) I am getting the following error in HANA Studio:

SAVE DATA finished with error: [447] backup could not be completed, [1000013] Inconsistent SSFS!

I also get a failure when attempting a backup from the portal.

There is apparently a discussion on this subject in " SAP Note 2097613" but I can not seem to access it with my scn credentials

Thank you

Bob

[sadanand_bhat · Accepted solution]

Below are the note contents. However be aware that if you are generating a new key there could be loss of data.

Symptom

You have one or all of the following symptoms in your database:

• Alert "Your database is currently running with an inconsistent secure storage file system (SSFS). Access to encrypted data and encryption of data are limited."
• SYS.M_SECURESTORE contains information about an inconsistency of the SSFS
• Backup is failing with error message: "Inconsistent SSFS!"
• When Data Volume Encryption is enabled, system startup fails with name server crash
• Secure Internal Credential Store: SQL CREATE CREDENTIAL or CREATE REMOTE SOURCE fails with "encryption failure"
• XS Secure Store: Reading or writing from an XS Secure Store fails with "encryption failure"

Applies for Revision 90 and above.

Other Terms

HANA, Data Volume Encryption, Remote Data Source, Secure Internal Credential Store, XS Secure Store, Remote Destinations, security, hdbnsutil, encryption, administration, SSFS, inconsistent

Reason and Prerequisites

Background:

To check if the SSFS is consistent and in sync with the database, internal consistency information is stored in the persistence of the database. This consistency information is contained in the backup and restored during a recovery.

An inconsistent or corrupt SSFS can lead to data loss (all encrypted data cannot be decrypted). Encrypted data can mean

• Data volumes in the file system
• Credentials in the Secure Internal Credential Store
• SAP HANA XS HTTP destinations
• SAP HANA XS Secure Store data

To detect the inconsistency as early as possible a monitoring view and an alert (queried every hour) is provided.To prevent inconsistent backups, any kind of backup is blocked if the SSFS is inconsistent.

Normally, an administrator or user should never be confronted with an inconsistent SSFS.

Reasons for an inconsistent SSFS:

• Incomplete manual backup/recovery by copying the data volume.
• hdbnsutil -generateRootKeys without the subsequent SQL ALTER SYSTEM APPLICATION ENCRYPTION CREATE NEW KEY 
• Recovery of an inconsistent backup that was taken with a revision < 90 (backup of an inconsistent database, that got inconsistent in one of the above ways)

Check SSFS consistency:

Query the monitoring view "SYS.M_SECURESTORE". A consistent database will deliver the following output:

+---------------+---------------+-------------+ |   KEY_TYPE    | IS_CONSISTENT | RESET_COUNT | +---------------+---------------+-------------+ |               |               |             | | "PERSISTENCE" | "TRUE"        | 0           | |               |               |             | | "DPAPI"       | "TRUE"        | 0           | +---------------+---------------+-------------+

Solution

A) Preferred Solution: Restore of an inconsistent SSFS

The recommended way to restore the SSFS is to recover from a consistent backup. However, there is no functionality available to check whether a backup is consistent or not before performing the recovery. If no consistent backup is available you can try to restore the SSFS manually (procedure see below). The original SSFS file can be found in different ways depending on your scenario:

• If you manually copied the data volume from A to B, try to find the SSFS of A and copy it with the data volume to B. The SSFS is located at /usr/sap/<SID>/SYS/global/hdb/security/ssfs/SSFS_<SID>.DAT.
  Note that copying copy data volumes manually isn't offically supported: Please use backup and recovery.
• If you called hdbnsutil -generateRootKeys, you can find the old SSFS next to the newly generated one: The command creates backups of the old SSFS named SSFS_<SID>.DAT_<root key type>_<seconds since 1970>.sav.
• If you recovered an inconsistent backup, try to find the reason for the inconsistency of the source database and get the original SSFS from there.

In case you have changed the master key of your SSFS file using rsecssfx generatekey, the resulting key file must be copied from the original location as well. The default location of the key file SSFS_<SID>.KEY is next to the SSFS_<SID>.DAT file.

IMPORTANT NOTE:

When the original SSFS file is restored in the file system, no loss of credentials occurs.
However, all XS Secure Store data that has been created while the inconsistent SSFS was active will be lost. With revisions > 90, creation of XS Secure Stores will be blocked when the SSFS is inconsistent.

Check for potential XS Secure Store data loss:

• Query the internal table SYS.P_DPAPI_KEY_ (currently visible for user SYSTEM, only). A database not containing any XS Secure Store data will deliver no record for CALLER "XsEngine".

Manual Restore Procedure:

1. Stop the SAP HANA system.
2. Replace the current SSFS file /usr/sap/<SID>/SYS/global/hdb/security/ssfs/SSFS_<SID>.DAT by the original file.
3. Copy the original master key file to /usr/sap/<SID>/SYS/global/hdb/security/ssfs/ or the location you have configured.
4. Start the SAP HANA system.

B) Repair Inconsistent SSFS: Hard reset of consistency information
In case solution A) cannot be applied, the only way to make the system functional again is to reset the internal consistency information.

WARNING: This procedure should be the last option and should only be used if every attempt to restore the SSFS is failing or not possible. ALL credentials and XS Secure Store data stored within your database will be lost!

Check for potential data loss:

1. Credential Store: Query the public view PUBLIC.CREDENTIALS. A database not containing any credentials in the Credential Store will deliver an empty result.
2. XS Secure Store: Query the internal table SYS.P_DPAPI_KEY_ (currently visible for user SYSTEM, only). A database not containing any XS Secure Store data will deliver no record for CALLER "XsEngine".

B.1) Reset consistency information

Reset consistency information by using the HANA tool hdbcons

$ hdbcons "crypto ssfs resetConsistency"

SAP HANA DB Management Client Console (type '\?' to get help for client commands)

Try to open connection to server process 'hdbindexserver' on system 'SFX', instance '02'

SAP HANA DB Management Server Console (type 'help' to get help for server commands)

Executable: hdbindexserver (PID: 15420)

[OK]

--

This command will completely rewrite the consistency informations about the ssfs,

please ensure that this is the last resort for fixing the ssfs

and you did all means to restore the consistent ssfs.

If you are sure about this, call the command again within 20s to rewrite ssfs consistency information.

[OK]

--

[EXIT]

--

[BYE]

$ hdbcons "crypto ssfs resetConsistency"

SAP HANA DB Management Client Console (type '\?' to get help for client commands)

Try to open connection to server process 'hdbindexserver' on system 'SFX', instance '02'

SAP HANA DB Management Server Console (type 'help' to get help for server commands)

Executable: hdbindexserver (PID: 15420)

[OK]

--

rewriting ssfs consistency information...

done.

[OK]

--

[EXIT]

--

[BYE]

B.2) Create new application keys
Create new keys for the Credential Store and all individual XS Secure Stores at once:

ALTER SYSTEM APPLICATION ENCRYPTION CREATE NEW KEY

[Former Member · Accepted solution]

Thank you, this appears to be the issue.  Although I am not sure why the secure file system would be inconsistent IMMEDIATELY upon creation of the Amazon HANA instance.  Maybe a bad image was delivered to Amazon

[Former Member]

can you take a restart of the DB and check once,

[Former Member]

Restarting the database had no effect, but thank you!