"FILLING_FAILED-->CHECK LOG" error on BIA index creation (RSD_TREX 100)

[rick_markowski]

I have searched everywhere in SDN and SAP Marketplace and have not been able to find anyone else with the same problem we are having building BIA indexes on a few selected cubes. I would greatly appreciate any assistance someone could offer on this.

I am attempting to build a BIA index and the background job completes successfully but an error message at the end of the background job log states:

FILLING_FAILED-->CHECK LOG!!!!! | RSD_TREX | 100 | S

In SLG1-Application Log, I see the following errors which are highlighted in red (last few lines of the log):

| |Read from 'F-' fact table | |

| |Batch parallelization: START | |

| |Backgr. parallelization of index fill process (number of processes:'3) | |

| |Index filling process in main process: | |

| |WHERE clause: OPT ='BT', LOW = '', HIGH = '0000756991' (field: 'SID_0REQUID') | |

| |Parallel index filling process in background job 'TRP_4B1FBXJ955869RDJNQMXTXCK3' scheduled | |

| |WHERE clause: OPT ='BT', LOW = '', HIGH = '0000760564' (field: 'SID_0REQUID') | |

| |Parallel index filling process in background job 'TRP_4B1FBXJ955869RDJNQMXTXCK3' scheduled | |

| |WHERE clause: OPT ='BT', LOW = '', HIGH = '0000764133' (field: 'SID_0REQUID') | |

| |Batch parallelization: END | |

| |Background process '1' complete: number of indexed records '86022' | |

| |* Batch process with ID TRP_4B1FBXJ955869RDJNQMXTXCK3 number 1 and process type TREXINDEXP started | |

| |* Process running in work process 20 with WP-PID 8392784 on server sapbwpdbg_BWP_00 and host sapbwpdbg | |

| |* Start function module RSDDTREX_INDEX_WRITE_PARALLEL at (time stamp) 20080903155635 | |

| |* End of processing of function module RSDDTREX_INDEX_WRITE_PARALLEL at (time stamp) 20080903155637 | |

| |* No messages tranferred to batch manager - Subrc = '0'. | |

| |* Batch process with ID TRP_4B1FBXJ955869RDJNQMXTXCK3 Number 1 and process type TREXINDEXP ended | |

| |Background process '2' complete: number of indexed records '38874' | |

| |* Batch process with ID TRP_4B1FBXJ955869RDJNQMXTXCK3 number 2 and process type TREXINDEXP started | |

| |* Process running in work process 12 with WP-PID 160148 on server sapbwpap5g_BWP_00 and host sapbwpap5g| |

| |* Start function module RSDDTREX_INDEX_WRITE_PARALLEL at (time stamp) 20080903155634 | |

| |* End of processing of function module RSDDTREX_INDEX_WRITE_PARALLEL at (time stamp) 20080903155635 | |

| |* No messages tranferred to batch manager - Subrc = '0'. | |

| |* Batch process with ID TRP_4B1FBXJ955869RDJNQMXTXCK3 Number 2 and process type TREXINDEXP ended | |

| |Index 'BWP_BIC:FBCEBIIP' for BIA index filled (written records '9160701'): | |+

| |Prepare optimize for BIA subindex 'BWP_BIC:FBCEBIIP': | |

| |Commit optimize for BIA subindex 'BWP_BIC:FBCEBIIP': | |

| |Not all data was indexed for index 'BWP_BIC:FBCEBIIP' ('9160711' for '9160701') | | | |Fatal error during indexing; rebuild required | | | |BIA-Index of InfoCube 'BCEBIIP' set to status 'INA' | | | |Creation of the BIA index for InfoCube 'BCEBIIP' terminated while filling | | | |Creation of the BIA index for InfoCube 'BCEBIIP' terminated while filling | | | |Creation of the BIA index for InfoCube 'BCEBIIP' terminated while filling | |

| |BI support package and BI accelerator revision are compatible

Complete background job log:

Job log overview for job: BW_TR_RSDDTREX_AGGREGATES_FILL / 10531400

----

---

|Date |Time |Message text |Message class|Message no.|Message type|

----

---

|09/03/2008|10:53:14|Job started | 00 | 516 | S |

|09/03/2008|10:53:14|Step 001 started (program RSDDTREX_AGGREGATES_FILL, variant &0000000000072, user ID RMARKOWS) | 00 | 550 | S |

|09/03/2008|10:53:14|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:16|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:16|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:16|INFOCUBE: BCEBIIP | RSD_TREX | 100 | S |

|09/03/2008|10:53:16|Statistics UID of indexing job: '4B1FB6FQ92N5K8RJ2ODMCZVMB' (RSDDSTATTREX/RSDDSTATTREXSERV) | RSD_TREX | 084 | S |

|09/03/2008|10:53:17|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:18|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:18|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:18|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:18|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:19|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:19|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:19|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:22|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:23|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:30|Loading data to index BWP_BIC:SBBNAME (records '0000000519', job '0' ) | RSD_TREX | 211 | I |

|09/03/2008|10:53:34|Total number of indexed reocrds in index 'BWP_BIC:SBBNAME' after COMMIT: '519' | RSD_TREX | 218 | I |

|09/03/2008|10:53:34|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:35|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:35|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:35|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:35|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:36|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:36|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:36|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:36|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:37|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:37|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:37|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:37|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:37|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:38|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:38|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:38|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:38|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:38|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:39|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:39|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:39|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:39|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:40|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:41|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:41|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:53:41|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:54:04|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:54:04|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:54:47|Loading data to index BWP_BI0:SCUST_SALES (records '0000000015', job '0' ) | RSD_TREX | 211 | I |

|09/03/2008|10:54:57|Total number of indexed reocrds in index 'BWP_BI0:SCUST_SALES' after COMMIT: '15' | RSD_TREX | 218 | I |

|09/03/2008|10:55:02|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:55:03|BCEBIIP OFF | RSD_TREX | 100 | S |

|09/03/2008|10:55:03|0OPA_C11 ACT | RSD_TREX | 100 | S |

|09/03/2008|10:55:04|0OPA_C11 ACT | RSD_TREX | 100 | S |

|09/03/2008|10:55:04|Parameter BATCHPARA during indexing: '3' | RSD_TREX | 126 | S |

|09/03/2008|10:55:04|'22' different values in column 'SID_0CALMONTH' of table '/BIC/DBCEBIIPT' | RSD_TREX | 127 | S |

|09/03/2008|10:55:04|WHERE clause: OPT ='BT', LOW = '', HIGH = '0000200802' (field: 'SID_0CALMONTH') | RSD_TREX | 077 | S |

|09/03/2008|10:55:04|Program RSBATCH_EXECUTE_PROZESS successfully scheduled as job BITRP_4B1FBUZ2TLZO28W5PPUUH9RYR with ID 10550400| RSM | 703 | S |

|09/03/2008|10:55:04|WHERE clause: OPT ='BT', LOW = '', HIGH = '0000200809' (field: 'SID_0CALMONTH') | RSD_TREX | 077 | S |

|09/03/2008|10:55:04|Program RSBATCH_EXECUTE_PROZESS successfully scheduled as job BITRP_4B1FBUZ2TLZO28W5PPUUH9RYR with ID 10550401| RSM | 703 | S |

|09/03/2008|10:55:29|Loading data to index BWP_BIC:FBCEBIIP (records '0001315789', job '1' ) | RSD_TREX | 211 | I |

|09/03/2008|10:55:53|Loading data to index BWP_BIC:FBCEBIIP (records '0002631578', job '2' ) | RSD_TREX | 211 | I |

|09/03/2008|10:55:55|Loading data to index BWP_BIC:FBCEBIIP (records '0002717362', job '0' ) | RSD_TREX | 211 | I |

|09/03/2008|10:56:19|Background process '2' complete: number of indexed records '2,631,256' | RSD_TREX | 080 | I |

|09/03/2008|10:56:34|Background process '1' complete: number of indexed records '3,654,139' | RSD_TREX | 080 | I |

|09/03/2008|10:56:34|Parallel background process for indexing table '/BIC/FBCEBIIP' complete ('6') | RSD_TREX | 078 | S |

|09/03/2008|10:56:34|Total number of indexed reocrds in index 'BWP_BIC:FBCEBIIP' after COMMIT: '9,002,757' | RSD_TREX | 218 | I |

|09/03/2008|10:56:34|Parameter BATCHPARA during indexing: '3' | RSD_TREX | 126 | S |

|09/03/2008|10:56:34|'12' different values in column 'SID_0REQUID' of table '/BIC/DBCEBIIPP' | RSD_TREX | 127 | S |

|09/03/2008|10:56:34|WHERE clause: OPT ='BT', LOW = '', HIGH = '0000760564' (field: 'SID_0REQUID') | RSD_TREX | 077 | S |

|09/03/2008|10:56:34|Program RSBATCH_EXECUTE_PROZESS successfully scheduled as job BITRP_4B1FBXJ955869RDJNQMXTXCK3 with ID 10563400| RSM | 703 | S |

|09/03/2008|10:56:34|WHERE clause: OPT ='BT', LOW = '', HIGH = '0000764133' (field: 'SID_0REQUID') | RSD_TREX | 077 | S |

|09/03/2008|10:56:34|Program RSBATCH_EXECUTE_PROZESS successfully scheduled as job BITRP_4B1FBXJ955869RDJNQMXTXCK3 with ID 10563401| RSM | 703 | S |

|09/03/2008|10:56:34|Loading data to index BWP_BIC:FBCEBIIP (records '0000033048', job '0' ) | RSD_TREX | 211 | I |

|09/03/2008|10:56:40|Background process '1' complete: number of indexed records '86,022' | RSD_TREX | 080 | I |

|09/03/2008|10:56:40|Background process '2' complete: number of indexed records '38,874' | RSD_TREX | 080 | I |

|09/03/2008|10:56:40|Parallel background process for indexing table '/BIC/FBCEBIIP' complete ('2') | RSD_TREX | 078 | S |

|09/03/2008|10:56:40|Total number of indexed reocrds in index 'BWP_BIC:FBCEBIIP' after COMMIT: '9,160,701' | RSD_TREX | 218 | I |

|09/03/2008|10:57:10|FILLING_FAILED-->CHECK LOG!!!!! | RSD_TREX | 100 | S |

|09/03/2008|10:57:10|Job finished | 00 | 517 | S |

----

---

[Former Member · Accepted solution]

You haven't mention the origin of infocubes. I wonder aren't they realtime?

[rick_markowski · Accepted solution]

No, these are standard cubes.

[Vitaliy-R · Accepted solution]

Hi Richard,

Does the problem still exist or did you resolve it with SAP support? Any news on root-cause of this?

Thanks,

-Vitaliy

[rick_markowski · Accepted solution]

This problem still exists. We have an open message with SAP but I am still waiting on our BASIS team to be able to provide the connection parameters to them.

[Former Member · Accepted solution]

We are having this exact same problem in our new BIA installation. We are testing BIA in our staging tier before moving this to Production in the next few weeks but this is currently a show-stopper.

What's interesting is that the variance between the expected record count and the number of records committed is always 28 in our case for the 4 InfoCubes that are exhibiting this behavior. One is around 12,000,000 records and the others are around 23,000,000 records, but the difference is always 28 records. They are all similarly structured (and exist under the same MultiProvider), so I wonder if it's an issue with something in one of the characteristic SID tables or something else. We've tried RSRV checks and cleaned up warnings we find, but nothing has helped to-date.

We also have a customer message open and have not yet found a resolution.

We are also on Rev 47.

Please let me know if you find a resolution and we'll do the same.

[rick_markowski · Accepted solution]

Aaron, thanks for your input. I suppose misery enjoys company on this one. We had some issues with approval for access to the TREX server, and I just created a new OSS message myself today to get that rolling again. We went from 2 cubes to 6 cubes now with this problem. In our case, the counts seem to be off by 10 on everything.

Likewise, I will also keep you in the loop if I hear anything back before you do. Good luck.

[Former Member · Accepted solution]

I believe we've found the solution. Our on-site SAP Basis consultant found the following Note which we've applied and subsequently filled 2 indexes successfully (one >12,000,000 records and the other >23,000,000 records). Both of these indexes had been consistenly failing before the Note was applied.

Note 1232308 - BIA 7.0: Backup blade not used for all indexes

This is a patch to BIA Rev 47.

We're continuing our testing but so far, so great!

[Vitaliy-R · Accepted solution]

Aaron, Richard,

Could you share your BIA configuration details? There should be something in common.

This would help other customers to detect similar situation upfront.

Thank you,

-Vitaliy

[rick_markowski · Accepted solution]

Aaron, thanks for the update. I've also just got a response from SAP to apply the exact same note. I've forwarded to our UNIX team. I'll let you know how it goes. Glad to hear that you are having success with it!

[rick_markowski · Accepted solution]

Aaron, do you know if you had to apply this to only the primary server or to any secondary servers as well?

[Former Member · Accepted solution]

Hi Richard,

We only have one BIA server and this was applied there.

It wasn't clear to me from the text of the Note whether this applied to the BI app servers or the BIA appliance, but our Basis consultant directed us to put it on the BIA appliance itself.

It also says to reorg the indexes, but since ours had failed to fill anyway we just dropped and rebuilt them after the Note was applied. It wouldn't let us trigger a reorg.

Hope that helps!

Aaron

[Former Member · Accepted solution]

Hi Vitaliy,

I believe the common contributors to this issue are as follows:

1. BIA Rev 47

2. At least one backup blade in the appliance

For completion, here's additional details of our configuration:

BI app servers (BI 7.0 Support Pack 16) on z10 mainframe running on zLinux virtual hosts. Database is DB2 v9. BIA configuration is X number of blades with I believe 1 fail-over. We have 3 app servers connected to our BIA appliance through a single RFC logon group configured on the central instance.

Hopefully that's enough information, but I think the real culprit is BIA Rev 47 + a backup blade.

Aaron

[rick_markowski · Accepted solution]

Well, we finally got our issue solved. Unfortunately there is no traceable reason for it (I hate when that happens). Anyways, I am attaching the last response from SAP on this. Thanks to all who participated and hopefully this thread helps others that might run into the same problem.

Dear customer,

I was able to create BIA indexes for mentioned cubes

BCECOPA07

BCECOPA08

BCGMKCU78

BCGBIL07L

BCGBIL08L

BCGBIL08S

without any issues . Unfortunately it's not clear from the available

logs what was the reason of the problem in the past. As I saw you

have restarted TREX/BIA so I suppose that the load statistic in the

python TREXAdmin isn't available anymore for the time when the issue

occured (unfortunately I couldn't logon into the TREX blades as the

provided in the secure area logon information isn't valid).

Anyway as I already meant the problem seems to be resolved now.

Best regards,

TREX Development Support

[rick_markowski]

Thanks for the input Marc. I appreciate you taking the time to quickly answer this. I also saw that note which was the closest I could find to our issue, but this is not a delta index (it's happening on the initial fill) and we are on BI 7.0 SP17 and BIA revision 7.00.47.00.

[Vitaliy-R]

Hi Richard,

You brought couple of good points about BIA logging that SAP developers could look into:

1/ I've seen this several times, when BIA job was finished Ok (green), but there were error messages inside.

2/ Job log (SM37) provides quite a good level of information comparable to application from (SLG1), but only SLG1 contains more details of the error.

3/ SLG1 itself is not convenient because there are no timestamps (at least I do not see them).

Regarding your failure: is it every time for this InfoCube, or only once? the reason for my question is that it seems to find mismatch between number of records in the cube and number of records indexed, which is either an issue with data changed in the cube during indexing (although it should be locked for changes), or one of those unpredictable internal errors.

Regards,

-Vitaliy

[rick_markowski]

Thanks for your comments Vitaliy. This happens every time I try to create the BIA index. I also notice the disconnect between the record counts. I've tried deleting and recreating the indexes, but to no avail. This is happening on two other cubes as well.

[former_member93896]

Hi Richard,

please see SAP note <a href="http://service.sap.com/sap/support/notes/1164381">1164381</a>.

If it does not fix the issue, please open a customer message so SAP support can analyze the issue.

Regards,

Marc

SAP NetWeaver RIG