siegfried.szameitat

    11 Posts

    Coming to the end of the little blog series (The diary of a BW 7.3 Ramp-Up project (Part 1), The diary of a BW 7.3 Ramp-Up project (Part 2), The diary of a BW 7.3 Ramp-Up project (Part 3),
    The Diary of a BW 7.3 Ramp-Up project (Part 4), 'Force Delta' in BW Dataflows) I will shortly return to the technical stuff. As I know many of
    you face the same issues when it comes to process chain scheduling. Of course,
    there are tools available such as Control-M, UC4 ... just to name two of them,
    but in case you don't have these tools implemented and you have to schedule the chains directly in your system, you will have the same issues as we have. E.g. a chain should only run on the 6th and 9th workday of a month, a chain should run every 2 hours, but not between 1:00pm and 7:00pm, a chain should run daily but not on saturdays.... You can achieve this by creating a separate factory calendar for each requirement and scheduling the chains based on the special factory calendar. To achieve all this plus some more options, we created our own process chain scheduling program. This program  als has the advantage that, in case of a system down time, you only stop the program and no chain will run unless the program gets restarted.

    But now, coming to the conclusion, I would say that BW 7.3 is a pretty stable system, even as we had some trouble after installing SP03. BW 7.3 comes with a lot of new features in the admin workbench which makes life a lot easier. We didn't use all of them, but in our first step we started with the data flow copy functionality, the SPOs and the graphical dataflow design tool. All of them are working pretty well. Only the SPOs might work faster and more flexible when it comes to the dtp generation.

    As a resume I would recommend using BW 7.3 and look forward to implement BO 4.0.

    For all of you interested in some more, there will be a webinar in the next weeks
    and in addition we can talk at TechEd Madrid which I plan to attend.

    After publishing my The Diary of a BW 7.3 Ramp-Up project (Part 4), I was asked about the 'force delta' process which was recommended by SAP. This process isn't a new feature at all, it is just another way of getting a delta from a DSO. One should always implement this process in case there are two or more DataSources which should be combined into one Object
    using a different key. All datasources have their own delta and all data has to be enhanced by data from the other DataSources. In the target object the most actual data has to be stored.
    Here an example:

    DataSource 1:
    Field A (Key)
    Field B
    Field C
    Field D
    Field E

    DataSource 2:
    Field B (Key)
    Field C
    Field F
    Field G
    Field H
     
    Target-DSO or Characteristic:
    Field C (Key)
    Field A
    Field B
    Field D
    Field E
    Field F
    Field G
    Field H

    Now you build one DSO (DSO1) for DataSource 1 and another one (DSO2) for Datasource 2. But instead of heaving a flow (transformation) from
    both the DSOs to the Target DSO selecting the data of the other DSO in a start-/end/expert routine you create only one flow from one of the
    DSOs to the target where you implement the logic. In my example add a keyfigure to DSO2 (I call it 'force delta'). Additionally
    create a flow from DataSource 1 to DSO2, just updating the key and the keyfigure which should be set to addition and a constant value of 1.
    This would force a delta from DSO2 to the Target for all affected keys and you need to implement the necessary logic only once in one flow.
    That's basically it. So no big deal, but it really saves some coding.

    With this type of flow you achieve a delta going to DSO1 having the most actual data here. Additionally a delta going to DSO2 having the most actual data here also.
    And in Addition with each independant delta of one of the DataSources there will be a delta going from DSO2 to the Target DSO so that you also have
    the most actual data in your Target.

    Here a short picture:

     Forced Delta

    Back from vacation, I follow up my blog series (The diary of a BW 7.3 Ramp-Up project (Part 1), The diary of a BW 7.3 Ramp-Up project (Part 2) and The diary of a BW 7.3 Ramp-Up project (Part 3)).

    Within this blog I will come to the probably most interesting part of the project. The technical details. Of course, you will not see any code here, but I will talk a bit about what we did at which step and how we realized it.
    I will start with the extraction. The first step was to take out the coding in the includes zxrsau*. Therefore we enhanced the extractors as far as possible with rsa6 and added all possible fields. If there was a necessity to get some other data we activated the appropriate standard extractor or created our own generic extractor for the data.
    Posting the data to the corporate memory we added the request date and the request time to the data fields. There is one corporate memory object for each datasource and each corporate memory object contains all fields a datasource delivers.

    Up to the propagation layer we added one central method to the start routine of each transformation going to a propagation layer object. This method contains the logic to do upper/lower case conversions, to switch alpha conversion on/off, to check for not allowed characters, to check the date format, to check the format of numeric fields and to do some more project specific things. Via settings
    in customizing tables it is possible to switch these actions on and off. No entry in the customizing tables for a transformation means, the method in the start routine will not be processed. All actions will be logged in the dtp monitor.
    From the propagation layer to the reporting layer we have two methods which have to be added to each transformation. One in the start routine and one in the end routine. Within these methods, we implemented again the upper/lower case conversion and the switch alpha conversion on/off. Additionally there is one central method to read data from other objects (p-tables, DSOs) just via customizing thru some table entries. You have to specify the table to be read, the where clause as well as the fields you want to read and how they should be mapped to the target fields. For very specific derivations we also added the functionality of 'exit'-methods. For this we added the call of methods with a predefined interface.
    The 'exit'-methods can be called row wise or data packet wise in the start or the end routine. The name of the methods will be maintained in customizing tables and here they can be switched on or off addtionally.
    For loading files, we implemented a logic to check the file content before even starting to load the file. This logic consists of checking the number of fields in a file record, checking the length of a field, checking the delimiter, checking the number format, checking the date format and of course project specific checks.
    On top we created an abap to check all project conventions and added this functionality to the change and transport system. So, in our system it is not possible to release a task or order which contains objects that doesn't fullfil our conventions such as naming conventions and the correct implementation of our methods in the transformations.

    As you know specially in case of file data you have to carry out a selective deletion of incorrect data. The possibilities to carry out the selective deletion are pretty well described Selective Deletion scenarios in SAP BW  and possible solutions. We implemented approach 3 with a little change. Instead of using an abap and DSO2 we are just using a transformation from DSO1 to DSO1 setting the recordmode to 'deletion' in the transformation.

    We also use SPOs. To setup the SPOs system wide with the same settings we used the SAP NetWeaver BW 7.30: Semantically Partitioned Objects (SPOs) built from BAdI – Consistent, Rule Based Modeling of Logical Partitions. The settings are maintained in customizing tables.

    Another story is the implementation of our 'force delta' process which was recommended by SAP. This process is used when you have 2 or more datasources which data needs to be combined in one target. Imagine you have DSO1 and DSO2 and you need to combine the data (the keyfields are different) into DSO3. So instead of doing a lookup from DSO1 to DSO3 and from DSO2 to DSO3, you just do one of the lookups. The other one gets replaces by an additional transformation from datasource1 to DSO3 changing a dummy field in DSO2 which will force a delta going from DSO2 to DSO3 doing the lookup again. This process eliminates a lot of coding.

    With this I will end my blog of the technical details. In case anybody is interested I would love to talk about it at Teched 2011 in Madrid which I am planning to attend.

    In this blog I will talk a bit about the cooperation with SAP during the project. In the process of The diary of a BW 7.3 Ramp-Up project (Part 1) and The diary of a BW 7.3 Ramp-Up project (Part 2) the project we also had some consultants from SAP on board. One to review the architecture and a so called Ramp-Up Coach. The first one was only there for one day and we had a good discussion explaining out thoughts and getting back some pretty good ideas of which one (force deltas from one to another dso while data should be consolidated into a third dso) was adopted and implemented a few times. I will talk about this one when I come to the technical details in the next blog. The Ramp-Up Coach was mainly focused on getting us in contact with the development department in case of finding the best solution for a task as well as for priorizing our error messages in the service market place. In our project he did nothing else than priorizing our messages. This isn't meant negative in any way and our Ramp-Up Coach was very successful doing this because most of our error messages were solved within a few days.

    Our experience with the new system was quite good. Of course the were some issues, but no show stoppers. Most of the issues were just handling things. May be others had more errors, specially if they did an upgrade from 7.x or earlier. So they are invited to tell their experience here also. I need to say that SAP did a great job and delivered a pretty usable product.

    As already mentioned, my next blog will be about some of the technical details of the project --> what we did for the file interface, which methods we used in the routines of the different layers and how every developed object gets checked automatically before it gets transported. Unfortunately this blog will take some time, as I am on vacation for the next 3 weeks. But anyway: anybody will be invited to post his experience, good or bad, with BW 7.3 here.

    As I mentioned in the last The diary of a BW 7.3 Ramp-Up project (Part 1), we started with a complete new system, so with a green field, just knowing what applications to move from the old to the new system. There were different naming conventions for objects, no coding standardization, different architectures for different applications.... So the first discussions we had was about the new architecture. Here we followed the guidelines of the newest lsa concept and adjusted it to our specific needs.

    So we have a corporate memory that contains all data delivered by the source system without any changes in write optimized DSOs. Here you find almost everything a datasource delivers (--> following the 'support the unknown' concept). This Layer should move to a NLS in future. The second layer will be the quality and harmonization layer. No data is stored here(well of course there might be exceptions) but the transformations from datasource to the propagation layer objects will be found in this layer. In the propagation layer the data will be stored in a consolidated view in standard DSOs and ready to use for the applications. These levels build the EDW, the Enterprise DataWarehouse.

    Now, the ADM, the Archtected DataMarts, come on top. Here you find the business and transformation layer. Basically we have only transformations here to the reporting layer objects, but again, exceptions might be possible. The reporting layer contains our InfoCubes and SPOs and on top, we have the virtualization layer where we have the MultiCubes on which the reports are created.

    After being sure about our architecture, we had to define new naming conventions. Well I will not stress this too much, as it will be different in each project, and not that relevant for standardization. But surely the naming conventions should follow one rule all over the system, maybe slightly different for the different layers, so that others who join the project can easily adopt them. The only thing we had to take care of was about the new SPOs as the technical name for a SPO is 2 characters less than a normal Cube or DSO.

    The nicest thing we did was about the coding for start/end/infoobject routines. We created just a few methods which have to be added to each and every transformation we have in the system. There is one method in the endroutine of all corporate memory objects. There is one method in the startroutine of all propagation layer objects.... And there are some customizing tables to enable the functionality in these methods. The so to enable functions are e.g. conversion to upper case, conversion to lower case, switch alpha on/off, select data from other objects and map correctly.... And for really specific functions we also implemented the call of 'Exit-Methods'.

    At the end we decided to document everything in a wiki, with links to the metadata repository, the online help and ofcourse with more detailed information for the users.

    I will talk about the rampup process and some technical details in the next blogs.

    Have you ever had the opportunity to work in a ramp-up project? Well I know that there are a lot of people out there who think one should wait some time after a new release is out before doing an upgrade. 

    The company I am working for at the moment planned an upgrade to 7.0 first, but for some reasons (some of the new features which were already introduced at teched 2009) we decided to go with the brand new system and tried to join the ramp-up process. Fortunately we succeeded. 

    But before I start reporting about the whole process, I will tell you a bit about the history of our BW.

    The development/implementation of the whole system started with 1.2B up to the 3.5 of today. As the system was used for 2 companies of which one separated about 2 years ago, our system was left with a lot of unnecessary stuff we cannot get rid of because of all the well known reasons you can imagine. Another thing is the use of yearly cubes to limit the data of the cubes in order to have a better reporting performance. I know this is pretty common in all BW implementations and of course, sometimes you have other criterias to separate the data in different cubes. So this was the point were the idea was born to use the new SPOs and to summarize -- the idea was born to redevelop the whole system from scratch.

    This meant, that we had to go thru the complete process of blue printing and defining specs as well as an overall develpment manual. The discussions we had as well as the questions that came up during the process while defining the architecture as well as the conventions (standardization) will be described in the next blog.

    Siegfried Szameitat

    Celebrating BW

    Posted by Siegfried Szameitat Apr 4, 2006

    A few times back, Roberto posted his weblog:

    SDN Success Story - Blue shooting stars from BW sky !


    And I think, now there is some more to celebrate.
    If you take a look at the top contributor list for all forums (as of 4th of April 2006) you will find at least 5 BWers in the top ten and 3 among the top 5 with more than 10k of points.

    image

    So feel free to celebrate Roberto, Bhanu, A.H.P, Eugene and Dinesh for their excellent work.
    May be Roberto's dream of having 3 BWers on top will come true soon.
    How is it possible to restart a process chain at a failed step/request?
    Sometimes, it doesn't help to just set a request to green status in order to run the process chain from that step on to the end.
    You need to set the failed request/step to green in the database as well as you need to raise the event that will force the process chain to run to the end from the next request/step on.

    Therefore you need to open the messages of a failed step by right clicking on it and selecting 'display messages'.
    In the opened popup click on the tab 'Chain'.
    In a parallel session goto transaction se16 for table rspcprocesslog and display the entries with the following selections:
    1. copy the variant from the popup to the variante of table rspcprocesslog
    2. copy the instance from the popup to the instance of table rspcprocesslog
    3. copy the start date from the popup to the batchdate of table rspcprocesslog
    Press F8 to display the entries of table rspcprocesslog.
    Now open another session and goto transaction se37. Enter RSPC_PROCESS_FINISH as the name of the function module and run the fm in test mode.
    Now copy the entries of table rspcprocesslog to the input parameters of the function module like described as follows:
    1. rspcprocesslog-log_id -> i_logid
    2. rspcprocesslog-type -> i_type
    3. rspcprocesslog-variante -> i_variant
    4. rspcprocesslog-instance -> i_instance
    5. enter 'G' for parameter i_state (sets the status to green).
    Now press F8 to run the fm.
    Now the actual process will be set to green and the following process in the chain will be started and the chain can run to the end.


    Of course you can also set the state of a specific step in the chain to any other possible value like 'R' = ended with errors, 'F' = finished, 'X' = cancelled ....
    Check out the value help on field rspcprocesslog-state in transaction se16 for the possible values.
    How does the extraction api work in case you are extracting data using a self defined function module with generic delta?
    Sometimes, if your application or requirement makes it necessary, you have to create your own function module to extract the data from whatever sap system to bw.
    Therefore you normally make a copy of the standard fm 'RSAX_BIW_GET_DATA_SIMPLE'. To make this new function work for you, you have to realize the following.

    1.) the fm gets called by the extraction api for at least 2 times. First time is for initialization and from the 2nd time on it is for data extraction.
    2.) right after the last data package (somehow you need to make sure that it is the last one) you have to raise the exception no_more_data.

    Example with lots of comments


    FUNCTION RSAX_BIW_GET_DATA_SIMPLE.
    *"--------------------------------------------------------------------------
    *"*"Lokale Schnittstelle:
    *" IMPORTING
    *" VALUE(I_REQUNR) TYPE SRSC_S_IF_SIMPLE-REQUNR
    *" VALUE(I_DSOURCE) TYPE SRSC_S_IF_SIMPLE-DSOURCE OPTIONAL
    *" VALUE(I_MAXSIZE) TYPE SRSC_S_IF_SIMPLE-MAXSIZE OPTIONAL
    *" VALUE(I_INITFLAG) TYPE SRSC_S_IF_SIMPLE-INITFLAG OPTIONAL
    *" VALUE(I_READ_ONLY) TYPE SRSC_S_IF_SIMPLE-READONLY OPTIONAL
    *" TABLES
    *" I_T_SELECT TYPE SRSC_S_IF_SIMPLE-T_SELECT OPTIONAL
    *" I_T_FIELDS TYPE SRSC_S_IF_SIMPLE-T_FIELDS OPTIONAL
    *" E_T_DATA STRUCTURE SFLIGHT OPTIONAL
    *" EXCEPTIONS
    *" NO_MORE_DATA
    *" ERROR_PASSED_TO_MESS_HANDLER
    *"--------------------------------------------------------------------------

    * Auxiliary Selection criteria structure
    DATA: L_S_SELECT TYPE SRSC_S_SELECT.

    * Maximum number of lines for DB table
    STATICS: S_S_IF TYPE SRSC_S_IF_SIMPLE,

    * counter
    S_COUNTER_DATAPAKID LIKE SY-TABIX,

    * cursor
    S_CURSOR TYPE CURSOR.

    * Initialization mode (first call by SAPI) or data transfer mode
    IF I_INITFLAG = SBIWA_C_FLAG_ON.
    * the coding here will be processed the first time the fm gets called.
    * it is used to populate static or global variables and to check wether
    * it's called by the right datasource or not.
    ELSE. "Initialization mode or data extraction ?
    * this part will be executed from the 2nd call on
    * First data package -> OPEN CURSOR
    IF S_COUNTER_DATAPAKID = 0.
    * in case it is for the first data package, range tabs gets filled, a cursor will
    * be opened or a initial dataset will be read from database into a global internal table
    * this has to be done here, with the first data package, to avoid getting the same data with
    * each call of the fm.
    * additionally you can check wether you get select-options for your 'generic delta'-field or not,
    * and, if it is provided, you are able to set a flag in order to do different selections
    * based on this flag for delta or full load.
    ENDIF. "First data package ?
    * from now on records gets fetched from the database or gets read from the internal table
    * and the return table e_t_data gets populated with the number of rows given by parameter s_s_if-maxsize.
    * if the last record is populated to table e_t_data you need to raise the exeption no_more_data
    * and you need to close the cursor, if you opened one.
    * with each data package you also have to increase s_counter_datapakid.
    ENDIF. "Initialization mode or data extraction ?

    ENDFUNCTION.


    If you forget to raise no_more_data, the extraction api will call the fm forever.
    Problems during the data load occur more or less frequently. The cause is differently. Sometimes it is an issue with disk space, sometimes there is no memory available, sometimes there are timeouts;.
    Here some examples for the error messages:

    Data load

    Data loading error- URGENT



    As we are all dealing with BW, we only see the monitor entries, and mostly it is giving us the impression, the load is still running, but in fact the load already terminated for whatever reason. Here I will describe some things to check and the resulting action to fix the issue.
    Of course, the dataflow has to be set up correctly. The datasource has to be active and replicated. The mapping from datasource to infosource has to be maintained and activated and/or the update rules to the datatargets as well.

    Loading from R/3:
    Here it depends on how you load your data. If you load it using the PSA, you have to check in the SourceSystem if the data doesn't arrive in the PSA. If it is already posted to the PSA, you have to check your BW itself.
    If you don't use PSA for your loads, well then you have to check both sides. R/3 for the extractor related issues and BW for the transfer or update rules related issues.
    1. Actions in the SourceSystem:
    a) goto transaction sm37 and check for a job named BI_REQU....(REQU.... will be the request number in BW) for your communication user (usually it will be ALEREMOTE). If the job doesn't exist, check with your basis. There might be a delay time until the job will be created. If it exists, check whether it is running or not. If it is running goto b), if not, your load just finished or terminated. Check the job log. If it finished successfully ok, if not, goto c)
    b) goto transaction sm50 (may be better sm51 if you have more than one application server) to check for running processes for your communication user. Here it is also possible to check what a process is doing at the moment and you can start debugging the process. If there is no process, the load just finished or you need to goto c)
    c) goto transaction sm21 and choose 'System-Log->Choose->All remote system logs' to display the log entries for all servers and not only for the one you are currently logged on to. Check for entries related to your communication user in the relevant time. If there are errors, check with your basis. Additionally goto d)
    d) goto transaction st22 to check for dumps for your communication user in the relevant time. Analyze the dumps. There might be a program error -> check with a developer to fix the program error, there might be disc space, memory or other parameter errors -> check with your basis and/or a developer to get the issue fixed.
    2. Actions in BW:
    The actions here are almost the same as described for R/3, but just for completion, here the steps once again.
    The difference here is the user you have to check. If you scheduled the load in background it is again the communication user (mostly ALEREMOTE). If you scheduled the load directly, the communication user is your user name.
    a) goto transaction sm50 (may be better sm51 if you have more than one application server) to check for running processes for your communication user. Here it is also possible to check what a process is doing at the moment and you can start debugging the process. If there is no process, the load just finished or you need to goto b)
    b) goto transaction sm21 and choose 'System-Log->Choose->All remote system logs' to display the log entries for all servers and not only for the one you are currently logged on to. Check for entries related to your communication user in the relevant time. If there are errors, check with your basis. Additionally goto c)
    c) goto transaction st22 to check for dumps for your communication user in the relevant time. Analyze the dumps. There might be a program error -> check with a developer to fix the program error, there might be disc space, memory or other parameter errors -> check with your basis and/or a developer to get the issue fixed.


    Loading from BW itself:
    If you are loading data within BW, eg. from ods to cube, things are mostly a bit easier except you have lots of coding in all the routines in the transfer or update rules. Without the coding you normally get a message in the monitor for the error.
    The checks to do are again the same as described before in 'Actions in BW' and I will not repeat them here.

    Loading from Flatfiles/Legacy systems:
    As there is no control over the legacy system while loading data, we also need to concentrate on checking BW for errors. The actions in the legacy system or the file (system) will be more or less concentrated on checking the authorities on files, tables or views. For other errors, there is a error message available in the monitor and if not, goto 'Actions in BW'.

    Possible actions to solve the issues:
    If you are dealing with program errors, ok, it is pretty easy, talk to a developer and correct the error. Additionally it might be necessary to increase the performance of the program by using another logic.
    If you are dealing with system error, talk to your basis. In this case you might need to add some disk space, enhance table spaces, increase the size of some system parameters (for program runtime, for memory management ...). The basis needs to do some fixes.
    If nothing solves your issue, then open a new thread or search for a thread in SDN or open or search for a note in the

    OSS


    In the BW forums quite often questions are popping up about errors while loading texts. Here some examples:

     

     

    Re: Error when uplaoding data from R/3

    Re: Invalid characters in Characterstics

    Data load error

     

     

    There are a lot more and this is not really representative, but I think it already shows the issue.

     

    As you can see, basically there are two standard solutions to get these fixed.

     

    Solution 1:
    add the not allowed characters to table rsallowedchar using transaction RSKC.
    This works very often but still it is limited, because it not possible to add more than 72 characters.

     

    Solution 2:
    in the maintenance screnn of the infoobject in question, click the checkbox for lower case letters.
    This is also very helpful and in combination with solution 1 you will be able to load almost every kind of texts to BW.

     

    Now I will introduce another solution which might be of some interest for some implementations.

     

    Solution 3:
    For this I will take the text as a description. Here now the step by step explanation.
    a) identify the object, the source field e.g. from R/3 vbak-bstnk which is the order number customer in SD.
    b) create a infoObject for it e.g. ZBSTNK of type numc, length 10 with long text, without master data. In the Bex settings set it to display as text.
    c) create a table in BW with the following fields: zzbstnk_num of type numc and length 10, zzbstnk_txt of type char and lenght 60.
    d) create a number range object in transaction SNRO
    e) add your new infoobject ZBSTNK to your infosource
    f) create a routine for your new infoobject in your transfer rules doing the following:
    - check your coming text (bstnk) against your table
    - if the text already exists in the table take the number and assign it to your result field.
    - if the text doesn't exsit in the table get a new number from your number range object, post that number into your result field and insert this number together with the text into your table.
    g) add your new infoobject (ZBSTNK) to your datatargets.
    h) always post the new assigned number to this object.
    i) create a generic extractor on your table
    j) replicate the datasource
    k) add your infoobject as infosource and assign the new datasource to it.
    l) map field zzbstnk_num to ZBSTNK and zzbstnk_txt to 0txtlg
    m) create a infopackage and load the data

     


    As you are now just dealing with a new object and a loaded text, you can assign whatever text you like to it without any issues.
    Another tricky thing with this solutions is that you will be able to do some checks on the data. vbak-bstnk can contain lots of different informations. As it is a free data entry in R/3 without a check table behind, sometimes it is filled with the order number with or without leading zeroes or it is filled with some other information. Now you enhance step f) by adding leading zeroes to a number or by translating the information differently by changing abrieviations to their long text and other possible translations. Doing this might also keep your amount of data small and additionally the dimensions.

     

    to be continued...
    (with long texts)

    Filter Blog

    By date:
    By tag: