SPS10 streaming data - disk full!

[Former Member]

Hi all,

we accidentally left a streaming project running over a few days, and our ES Server log and data directories are now full up! HANA has ground to a halt as a result.

I'm on SPS10 Dev Edition on AWS, so my ES data and log files exist here: /hana/shared/HDB/data_es/ and /hana/shared/HDB/log_es/. How can I cut these things down to size? Is it as simple as stopping the instance and rm'ing the log files? Or will that leave me in an unstable state?

Many thanks,

Hagen

[RobertWaywell · Accepted solution]

Hi Hagen,

As a housekeeping item, can you confirm that your streaming project is publishing data to extended tables in Dynamic Tiering?

Are you sure you are getting a disk full error as opposed to the Dynamic Tiering dbspace being full? The HANA Dev Edition instance does have limited disk space by default, however I would expect you to run into the limit of the 10GB dbspace created for Dynamic Tiering well before you would start running out of disk space. Note that Dynamic Tiering does not dynamically grow the disk storage space, rather the database administrator is required to extend the dbspace size as needed.

If the disk/storage volume isn't full, then you can extend the size of the dbspace used by Dynamic Tiering. Extending the size of the dbspace is explained in the Lesson 5 of the . For the Dev Edition the following SQL statement can be used to add another 5GB to the dbspace:

ALTER EXTENDED STORAGE ALTER DBSPACE ES_USER ADD FILE

"ES_USER_FILE1" 'HA0ESDB_usr_1.es' SIZE 5000 MB RESERVE 250 MB;

If the disk/storage volume is full, then the 'sledge hammer' approach would be to drop and recreate the extended storage. The drop command would be:

DROP EXTENDED STORAGE CASCADE;

To recreate the extended storage with the same parameters originally used when the Dev Edition template was created, you would execute:

CREATE EXTENDED STORAGE AT 'vhcalhdbdb' SIZE 10 GB ENABLE DELTA;

Note: Dropping extended storage will drop all of the extended tables and you will need to recreate the tables after recreating extended storage.

[Former Member · Accepted solution]

Robert, I could kiss you. Or perhaps just a manly hug. Up to you.

It was indeed a Dynamic Tiering issue. So, because extended storage (ES - ha, I get it!!!) creates a partition on the volume, the volume always appears full to the file system. But in my case, the partition had also been filled up completely.

Now that I understand the mechanics a little better, I'm clear on what needed to be done. In my case, because the partition was full, I had to use your sledge hammer approach and drop the extended storage (I couldn't bring esserver up to do the extend). Accordingly, I noted the user file in /hana/shared/HDB/data_es/HDB/mnt00001/es/user was removed. That allowed me to start the esserver process. I then recreated the extended storage, again, as per your instructions, and the file was recreated. The file, although empty, is sized to the entire partition (this I now know), much like a database storage physical file.

From now on, I will monitor the extended storage dbspace, and extend as necessary.

The one caveat, as you mention Robert, is that I need to recreate all the DB artifacts. For me, no dramas - I didn't have that much in there to begin with - but for anyone else, just bear this in mind if you have to drop your extended storage: you will lose your user schemas, and any other schemas (and schema objects) when dropping extended storage.

And now I start Wednesday with a spring in my step. Boing!

Thank you again Robert,

Hagen

[Former Member · Accepted solution]

Hi Robert,

so I ran into this issue again this morning. It baffles me, but the ES storage fills up incredibly quick, and I'm barely doing anything with it. However, the ES server was again down this morning, so I cascade dropped the storage and brought the server back up.

I then attempted to recreate the storage partition via. SQL console, as we did last time. However, I received an error indicating there was now insufficient room for the partition. There is something else taking up space on this disk. No matter, I'll make the partition smaller.

I went to the Fiori LP tile app to do it this time, just because. When creating the new partition for 5GB, the app ran... and ran... and ran. In fact, it never stopped loading. So I refreshed the page. The ES server is now stuck in a state of 'Starting'. I cannot DROP, and I cannot CREATE. I continually get this message trying to perform either of these functions:

I get the same message at the SQL console.

     SAP DBTech JDBC: [471]: invalid data source name: __esstore$

Do you have any advice for this situation?

Many thanks,

Hagen

[Former Member · Accepted solution]

Robert,

I was able to ALTER extended storage, and add some more, following which I performed a restart of HANA. ES server is now up and Dynamic Tiering is running.

Cheers,

Hagen