Automation of Delta Queues monitoring

[Former Member]

Data Sources will fetch data to BW system from various application systems based on delta method or full load. Delta data load will capture the changed records from the time previous data load completes based on the logic derived and filters applied in initialization of delta in BW.

        Some times Manual checking of delta queues is necessary because of some delays occurs when there is a huge amount of records found in the delta queue.  Conventional Process involves manual checks of delta queues to avoid the delays in regular data loads. Manual checking of each delta queue separately will consume lot of effort. Proposed Approach addresses the above gaps and gives the flexibility to select the type of application system or given data sources. Once the data extracted, automatic e-mail alerts will be sent to support team through a programming logic.

Solution:

The Custom data source has been implemented based on extractor Function module to get the number of records for all the delta enabled data sources. Data source will fetch the information about all delta enabled data sources names and its number of records (will load in next load). This data source runs couple of hours before the regular data load starts followed by a program which will check  the number of records for each delta enabled data source and compare with threshold limit and trigger an e-mail to IT Ops support team if more number of records found in any Data source than usual record count. (A custom data source providing the information about list of all other delta enabled data sources (Standard or custom) and their number of records will delta get in the next data load)

To automate the regular monitoring of the delta loads, need to perform the below steps:

1. Need to Create a generic Data source with function module with following steps in application System.
   

             1. Data source with structure contains Data source name, number of records and number of LUWs.

             2. Create a Function module to find the delta queue length including number of records and number of LUWs for each data source.  Use standard function module  'RSC1_TRFC_QUEUE_READ'  to fetch the number of records and number of LUWs for all delta enabled data sources in the given application system.
2. Replicate the generic data source in BW system.
3. Create a info package and add it in the process chain.
4. After loading data to PSA; need to run a program to check the length of each data source available in PSA table.
5. Create a Program to check the loaded data in the PSA and compare it with  threshold limits for each data source.
6. Need to set the threshold limit for each data source based on past data loads.
7. Program will check for length of an each data source in terms of number of records, if any data source length found greater than threshold limit, send a mail to group of members in the list.
8. Need to write another program with following steps: (Program is for safety purpose).
9. Second Program will wait for 20mins then check for the status of a variant for data source data loading in step 1 in Process Chain table.
10. If the status is in not in green status(G) then send a mail to group of members in the list.

Usually delay occurs whenever there is a failure or huge number of records found.

  11.  Finally, Set Global variables through transaction STVARVC to set the waiting time or to set the threshold limit for data sources instead of hard coding. This global variable can be modified at any point of time as per the requirement in production environment.