1 2 Previous Next

juergen.haupt

17 Posts

/LSA300/ LSA Tactical Scalability - Flow Split using Semantic Partitioning

 

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA300 that handles situations where a single InfoProvider cannot guarantee the required service levels.

For general information about LSA Data Flow Templates please refer to

Description

LSA300 illustrates how to deal with situations where a single InfoProvider cannot guarantee the required service levels. Violation of service levels may be caused by the following:

  • Poor load performance (-> report availability)
  • Poor query performance with ROLAP
  • Long recovery time (-> report availability)
  • Lack of robustness (-> report availability)
  • Master data consistency issues (-> referential integrity)
  • Overall manageability issues of large InfoProviders
  • Other reasons 

The LSA suggest using the best practice modeling pattern ‘semantic partitioning’ of an InfoProvider if it is difficult to achieve a certain service level like ‘report availability at 8h local time’. Semantic partitioning is a very popular modeling pattern in BW. Instead of storing data in a single InfoCube or DSO, it means distributing the data into several identical InfoCubes or DataStore objects, thus splitting the affected flow. Either organizational or time characteristics serve as semantic partitioning criteria. 

Note: Semantic partitioning with a time characteristic does not normally speed up data loads.

With BW 7.30, the new Semantically Partitioned Object (SPO) supports the semantic partitioning of InfoProviders. Template LSA300 uses the new SPO feature, but the flow logic can also be used with older BW releases.

 

 

 

LSA300-conceptual

Picture 1: LSA300 conceptual view

LSA300-EDW-BW

Picture 2: LSA300 EDW Layers in BW 7.30

LSA300-DM-BW

Picture 3: LSA300 Data Mart Layers in BW 7.30

 

LSA300 assumes that service level issues occur only with specific InfoProviders.  The LSA calls a semantic partitioning ‘tactical’ if only a dedicated InfoProvider is semantically partitioned. Unlike this ‘tactical’ semantic partitioning the LSA Data Domains define a ‘strategic’ semantic partitioning, which applies to all or a large range of the data flows.

Note: Semantic partitioning does not solve modeling and implementation issues! 

Target Group

LSA300 can be used for all InfoProviders

  • Where service level achievement is a problem
  • Where service level achievement issues are restricted to a specific scenario (tactical level).

 

Note:  For large/global BWs, the tactical semantic partitioning approach is not sufficient.

Instead, the BW (the transactional InfoProviders) should be structured in terms of a common ‘strategic’ semantic partitioning approach defined by the LSA data domains (see LSA400)

 

Implementation Details

Acquisition Layer

Note: For more details, see the previous LSA data flow templates

Harmonization & Quality Layer

  • Tactical semantic partitioning can be modeled for any InfoProvider on any layer.
  • Tactical semantic partitioning is normally performed at reporting layer level. 

Note: For more details, see the previous LSA data flow templates

Corporate Memory Layer (CM)

The corporate memory DataStore objects are normally not semantically partitioned. If you are confronted with volume issues because of a missing data life cycle strategy, you could think about a semantic partitioning with a time characteristic.

Note: For more details, see the previous LSA data flow templates 

Propagation Layer

  • Tactical semantic partitioning can be modeled for any InfoProvider on any layer.
  • Tactical semantic partitioning is normally performed at reporting layer level. 

Note: For more details, see the previous LSA data flow templates 

Business Transformation Layer

  • Tactical semantic partitioning can be modeled for any InfoProvider on any layer.
  • Tactical semantic partitioning is normally performed at reporting layer level.

Note: For more details, see the previous LSA data flow templates

Reporting Layer

Tactical semantic partitioning can be modeled for any InfoProvider on any layer.

Tactical semantic partitioning is normally performed at reporting layer level.  Note the following:

  • Only parts of the flow are split
  • The condition (log. partitioning by time or organizational criteria, or both) that defines the split of an InfoProvider (how the data is distributed across multiple identical InfoProviders) is defined according to the required service target. It might therefore differ from scenario to scenario and from data flow to data flow.
  • BW 7.30 offers the semantically partitioned object (SPO) as functionality to set up semantically partitioned InfoCubes or DataStore objects.

For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA310/ LSA Scalability – Flow Split and InfoSources

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA310 that illustrates the power of InfoSources increasing transparency as Transformation rules are always located between InfoSources.

For general information about LSA Data Flow Templates please refer to

Description

For the sake of transparency and flexibility, the LSA advocates shielding all persistent InfoProviders with InfoSources. Using InfoSources increases transparency as Transformation rules are always located between InfoSources. Using InfoSources increases flexibility by introducing semantic partitioning or making it possible to add semantic partitions later on. It also reduces redundancy (of coding & transformation) as transformations for all semantic partitioned InfoProviders are stored in one location.

LSA310 is based on LSA300 and illustrates the most consistent way of using InfoSources. In LSA300 already, it can be seen that the SPO automatically generates an InfoSource as Inbound InfoSource (and an Outbound InfoSource, which is not shown in the graphic).

Using the InfoSource – especially in the early stages of an implementation - might seem a bit excessive. Experience shows that the abstraction from persistent InfoProviders using InfoSources is worthwhile in the long run, as it reduces maintenance costs and increasing overall flexibility.  More pragmatic use of InfoSources is shown in the LSA315 flow. A further extension of InfoSource usage dealing with multiple sources is provided in LSA410.

 

LSA310_conceptual

Picture 1: LSA310 conceptual view

LSA310-EDW-BW

Picture 2: LSA310 - EDW Layers in BW 7.30

LSA310-DM-BW

Picture 3: LSA310 continued - Data Mart Layers in BW 7.30

Target Group

Large BW implementations built for flexibility, maintainability and growth.

Implementation Details

Acquisition Layer

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Harmonization & Quality Layer

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Corporate Memory Layer (CM)

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Propagation Layer

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Business Transformation Layer

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Reporting Layer

  • Inbound and Outbound InfoSources are used for all persistent InfoProviders of any layer.
  • Inbound and Outbound InfoSources are generated automatically using the BW 7.30 SPO feature.

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates 

 

/LSA315/ LSA Scalability InfoSources & Single ERP Source

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA315 that illustrates, compared to LSA310, a simplified usage of InfoSources for high quality data from SAP application sources.

For general information about LSA Data Flow Templates please refer to

Description

LSA315 is a simplified template based on LSA310. We can often simplify the early stages without losing the advantages of LSA, especially with high quality loaded data from a single SAP ERP system.  The data can then be accepted more or less directly from the source into the PSA.

 

Picture 1: LSA315 conceptual view

Picture 2: LSA315 - EDW Layers in BW 7.30

Picture 3: LSA315 continued - Data Mart Layers in BW 7.30

Target Group

BW implementations that receive high quality data from a single SAP ERP.

 

Implementation Details

Acquisition Layer

Note: For more details, see the previous LSA data flow templates 

Harmonization & Quality Layer

Useful information for administration is added to data.

Note: For more details, see the previous LSA data flow templates 

Corporate Memory Layer (CM)

In general,  an InfoSource is not required in front of a CM DSO, as semantic partitioning of CM DataStore objects is not recommended.

Note: For more details, see the previous LSA data flow templates

Propagation Layer

Note: For more details, see the previous LSA data flow templates

Business Transformation Layer

Note: For more details, see the previous LSA data flow templates

Reporting Layer

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA320/ LSA Scalability Entire Data Flow Split

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.  

This blog describes the LSA Data Flow Template LSA320 that illustrates how to deal with situations where load performance issues are caused by a single data flow. Guaranteeing report availability is the main challenge.

For general information about LSA Data Flow Templates please refer to

Description

LSA320 illustrates how to deal with situations where load performance issues are caused by a single data flow. Guaranteeing report availability is the main challenge.

If load performance issues occur for a specific data flow, the LSA suggests splitting the entire flow at the earliest possible point in time - on top of the PSAs. All InfoProviders in the flow are homogeneously semantically partitioned using the same organizational criteria, 0COMP_CODE for example.

As the semantic partitioning applies only to a specific data flow, the LSA also refers to this as a tactical semantic partitioning, unlike to the strategic, general semantic partitioning defined by the LSA Data Domains for ‘all’ flows (see LSA 400).

Picture 1: LSA320 conceptual view

Picture 2: LSA320 - EDW Layers in BW 7.30

Picture 3: LSA320 continued - Data Mart Layers in BW 7.30

Note: The corporate memory DataStore objects are usually not semantic partitioned!

For a better understanding another more conceptual representation:

 

Picture 4: LSA320 entire data flow split 

 

Note:  The data transfers are split - not the logic! The logic is bundled using Transformation Rules that are only located between InfoSources!

Target Group

 LSA320 addresses dedicated (high volume) data flows causing load performance issues

Please note:

If it is difficult in general to achieve service levels (report availability, performance or robustness). A strategic semantic partitioning approach implementing LSA domains should be considered (see LSA400).

 

Implementation Details

Acquisition Layer

Note: For more details, see the previous LSA data flow templates

Harmonization & Quality Layer

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • The semantic partitioning is performed using organizational criteria  – not a time characteristic

Note: For more details, see the previous LSA data flow templates

Corporate Memory Layer (CM)

In general, an InfoSource is not required in front of a CM DSO, as semantic partitioning of CM DataStore objects is not recommended.

Note: For more details, see the previous LSA data flow templates

Propagation Layer

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • The semantic partitioning is performed using organizational criteria  – not a time characteristic

Note: For more details, see the previous LSA data flow templates

Business Transformation Layer

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • The semantic partitioning is performed using organizational criteria  – not a time characteristic

Note: For more details, see the previous LSA data flow templates

Reporting Layer

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • The semantic partitioning is performed using organizational criteria  – not a time characteristic

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA330/ LSA Scalability Data Flow Split Using a Pass Thru DataStore object

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA330 that illustrates the role of Pass Thru DSOs

For general information about LSA Data Flow Templates please refer to

 

Description

LSA330 is very similar to LSA320.

The Pass Thru DataStore object may serve to bundle an early error handling. Instead of having ‘n’ error DTPs to the ‘n’ partitions of a propagation layer DataStore object, there is just one error DTP from PSA to the Pass Thru DataStore object.

 The Pass Thru DataStore object makes split/semantic partitioning handling easier. Instead of being performed on the PSA element, the split is performed on the Pass Thru DataStore object, which makes it possible to enrich the PSA data for easier DTP selections. (Merge (transform) the records to a common characteristic (e.g. company code to markets) before loading into the Pass Thru DataStore object. It is then possible to transfer the data to the partitioned Propagation Layer DataStore object simply by filtering the DTPs on a few market values. Loading data directly from the PSA element might mean filtering on a significant number of company codes.)

 

Picture 1: LSA330 conceptual view

 

Picture 2: LSA330 - EDW Layers in BW 7.30

 

Picture 3: LSA330 continued - Data Mart Layers in BW 7.30

 

Target Group

Like LSA320, LSA330 addresses dedicated (high volume) data flows that cause load performance issues

LSA330 also addresses the following:

  • The need to investigate loaded records for consistency at an early stage
  • Ease of operation  (as little contact as possible with PSA)
  • Pass Thru DataStore objects  allow making  semantic partitioning of subsequent InfoProviders easier and transparent, if you map the DataSource specific partition criteria & values (like company code, controlling area…) in a transformation rule  to a standard criteria like country. This standard criterion serves as a standard partition and selection criteria on top of the Pass Thru DataStore object for all data flows and InfoProviders. In BW 7.3, the semantically partitioned object in conjunction with a BAdI implementation support creating partitioning templates which can be assigned to multiple partitions. This makes Pass Thru DataStore objects from transparency and ease of implementation perspective obsolete. The other above mentioned aspects remain valid.

 

Implementation Details

Acquisition Layer

Take a look at the Harmonization & Quality Layer.

Note: For more details, see the previous LSA data flow templates 

Harmonization & Quality Layer

  • A Pass Thru DataStore object is of type ‘write optimized’. 
  • The data in a Pass Thru DataStore object should be deleted regularly if a corresponding CM DataStore object exists. 
  • From a modeling perspective, the Pass Thru DataStore object replaces the harmonization layer InfoSource.
  • We recommend naming Pass Thru DataStore objects with a dedicated layer qualifier (T for example)

Note: For more details, see the previous LSA data flow templates 

Corporate Memory Layer (CM)

Pass Thru & Corporate Memory:

  • A Pass Thru DataStore object  has to be cleaned up regularly if a CM DataStore object  exists
  • The Pass Thru DataStore object  normally has the same InfoObjects as the CM DataStore object 
  • You might consider using the Pass Thru DataStore object as your CM DataStore object.  As the volume of the Pass Thru will quickly grow, the CM considerations then apply to the Pass Thru DataStore object (see above). The Pass Thru DataStore object then ceases to be a temporary data store!

Note: Making the Pass Thru DataStore object a CM puts the CM DataStore object directly into the data flow to the Data Marts. This makes the architecture less robust.

Note: For more details, see the previous LSA data flow templates 

Propagation Layer

Note: For more details, see the previous LSA data flow templates 

Business Transformation Layer

Note: For more details, see the previous LSA data flow templates

Reporting Layer

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA400/ LSA Scalability & Data Domains – Strategic Semantic Partitioning of Entire BW (Split)

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA400 that addresses challenges across the DWH (e.g, with global BWs) by introducing Data Domains. Simply speaking, this means that we do not mix American transactional data with European or Asian transactional data.

For general information about LSA Data Flow Templates please refer to

 

Description

For large/global BWs, keeping to service level agreements (report availability, time-zones, robustness of operations etc.) is generally something of a challenge. For global BW, you have to be aware that there is no maintenance window (24x365) applicable to all data.

The LSA addresses these challenges by introducing Data Domains. Simply speaking, this means that we do not mix American transactional data with European or Asian transactional data. From a modeling perspective, defining data domains means providing a standard (strategic) concept for semantically partitioning the InfoProviders across your BW:

  • Which domains? e.g. APJ, EMEA, AMS, US and so on 
  • What are the standard criteria for semantically partitioning the data? e.g. 0COMP_CODE for sales data, 0CO_AREA for controlling data and so on
  • What criteria values define the domains? i.e. assigning 0COMP_CODE values to APJ or EMEA or AMS and so on

In addition to the layers, LSA domains achieve clear structuring of the transactional data in your BW. As the semantic partitioning applies to all or a wide range of data flows and InfoProviders, the LSA calls this way of using the semantic partitioning pattern ‘strategic’.

Consistent distribution of transactional data is crucial, and can be highly automated using SPO functionality. 

If we look at a single data flow, we observe that the visualization is the same as the tactical semantic partitioning introduced in LSA320 - Entire Data Flow Split.

Note that LSA domains do not apply in the corporate memory layer!

 

Picture Picture 1: LSA400 conceptual view

 

Picture 2: LSA400 - EDW Layers in BW 7.30

 

Picture 3: LSA400 continued - Data Mart Layers in BW 7.30

A conceptual representation shows how data domains work filled by a split for all SD flows using the same criteria:

Picture 4: LSA400 - Extract Once - Deploy Many

 

Target Group

 LSA400 is a standard flow template for all larger BWs that typically have a large geographical spread

Implementation Details

Acquisition Layer

The same implementation details apply as for LSA320.

Note: For more details, see the previous LSA data flow templates 

Harmonization & Quality Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders in the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic) 

Note: For more details, see the previous LSA data flow templates 

Corporate Memory Layer (CM)

The same implementation details apply as for LSA320. The customer LSA Domain concept does not apply to CM DataStore objects.

Note: For more details, see the previous LSA data flow templates 

Propagation Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders in the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic

Note: For more details, see the previous LSA data flow templates

Business Transformation Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders in the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic

Note: For more details, see the previous LSA data flow templates

Reporting Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders in the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA410/ LSA Scalability & Data Domains – Strategic Semantic Partitioning of Entire BW (Collect)

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA410 that addresses challenges across the DWH (e.g, with global BWs) by introducing Data Domains. Here we collect (organize) data from multiple sources.

For general information about LSA Data Flow Templates please refer to

Description

If BW has to process data from multiple source systems for the same business processes as with multiple SAP ERP sources across the globe, LSA recommends introducing data domains. These then define the criteria collecting data for the same DataSource from multiple source systems into semantically partitioned InfoProviders throughout the entire BW.

  • Data domains collecting data flows for the same DataSource but from different source systems will keep the BW manageable and transparent.  The more source systems you have, the truer this becomes.
  • Be careful and do not create bottlenecks!
  • Splitting and collecting data into domains can be performed for the same DataSource

 

Picture 1: LSA410 conceptual view

 

Picture 2: LSA410 conceptual view (DTPs & Corporate Memory)

 

Picture 3: LSA410 - EDW Layers in BW 7.30

 

Picture 4: LSA410 continued - Data Mart Layers in BW 7.30

 

Target Group

  • For all large BWs with geographically spread source systems.
  • Often relevant when multiple BWs are consolidated  into a single corporate/divisional BW

Implementation Details

Acquisition Layer

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA data flow templates 

Harmonization & Quality Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders in the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic 

Note: For more details, see the previous LSA data flow templates 

Corporate Memory Layer (CM)

The same implementation details apply as for LSA320. The customer LSA Domain concept does not apply to CM DataStore objects.

Note: For more details, see the previous LSA data flow templates 

Propagation Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic 

Note: For more details, see the previous LSA data flow templates 

Business Transformation Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic 

Note: For more details, see the previous LSA data flow templates

Reporting Layer

The same implementation details apply as for LSA320.

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA420/ LSA Scalability & Data Domains – Business Transformation Layer

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA420 that illustrates the role of the Business Transformation Layer.

For general information about LSA Data Flow Templates please refer to

 

Description

Template LSA420 is based on LSA400. LSA420 illustrates the role of the Business Transformation Layer. With complex business rules like measuring supply chain performance (Case & Line Fill scenario), it might be necessary to have an additional persistency (SPO DSO) in the Business Transformation Layer.

The semantic partitioning criteria for the Business Transformation Layer DSOs are normally inherited from the previous layer (propagation layer).

 

Picture 1: LSA420 conceptual view

 

Picture 2: LSA420 - EDW Layers in BW 7.30

 

Picture 3: LSA420 continued - Data Mart Layers in BW 7.30

 

Target Group

 LSA420 is a standard flow template for all larger BWs that typically have a large geographical spread

Implementation Details

Acquisition Layer

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA Data Flow Templates

Harmonization & Quality Layer

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA Data Flow Templates

Corporate Memory Layer (CM)

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA Data Flow Templates

Propagation Layer

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA Data Flow Templates

Business Transformation Layer

The same implementation details apply as for LSA400.

  • The semantic partitioning is modeled homogeneously for all InfoProviders of the data flow
  • In terms of the customer LSA Domain concept , the semantic partitioning is performed homogeneously by an organizational criteria (characteristic)semantic

Note: For more details, see the previous LSA Data Flow Templates

Reporting Layer

The same implementation details apply as for LSA400.

Note: For more details, see the previous LSA Data Flow Templates

Virtualization Layer

Note: For more details, see the previous LSA Data Flow Templates

/LSA110/ LSA Basic Flow Extensions - Corporate Memory (CM) & Harmonization InfoSource

 

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

This blog describes the LSA Data Flow Template LSA110 that overcomes the shortcomings of PSA elements.

For general information about LSA Data Flow Templates please refer to

Description

The LSA110 template overcomes the shortcomings of PSA elements when storing load history by introducing a dedicated Layer known as the Corporate Memory (CM).  The corporate memory provides for all unforeseeable or unlikely situations.  The CM InfoProviders offer service level like a source system for initial loads. LSA 110 shows the standard range of LSA Layers. Further layers can be added if required.

LSA110 also introduces a harmonization layer InfoSource that makes it possible to separate standard data unification transformations from (master) data integration transformations. This increases transparency.

LSA110_01_conceptual

Picture 1: LSA110 conceptual view

 

LSA110-EDW-BW

Picture 2: LSA110 EDW Layers in BW 7.30

 LSA110-DM-BW

Picture 3: LSA110 Data Mart Layers in BW 7.30                  

Target Group

LSA110 is applicable for BWs that are built with the following in mind:

 

  • Flexibility supporting unknown situations
  • Safety and robustness  – avoiding unwanted surprises
  • Managing the growth of historical data

 

Note: Introducing a CM layer should always be considered with large, global BWs. But the simple template LSA110 is normally not suitable for designing data flows for these kinds of BW implementations! You should consider using the more complex LSA data flow templates instead. 

 

Implementation Details

Acquisition Layer

  • Built using PSA elements as with LSA100
  • PSA elements can only be used for immediate inspection of data loads
  • Data from PSA elements should be deleted regularly

Note: For more details, see the previous LSA data flow templates 

 

Harmonization & Quality Layer

  • Data unification transformations are separated from flow-specific harmonization transformations using a harmonization layer InfoSource

Note: For more details, see the previous LSA data flow templates

 

Corporate Memory Layer (CM)

  • Write-optimized DataStore objects build the corporate memory layer.
  • CM DataStore objects are defined at DataSource level
  • CM DataStore objects should store all fields of a DataSource (comprehensiveness)
  • Load history from PSA is stored 1:1 in terms of content. This means that adding information and transformations like compounding keys is ok
  • CM DataStore objects and propagation DataStore objects add LSA unification information for loaded records (see LSA100)    
  • CM DataStore objects are located in addition to the daily data flows, loading InfoCubes (branch out services) to prevent load impacts and reduce human access and errors.
  • Data from CM DataStore objects should be stored  on NLS (near-line storage)

Propagation Layer

Note: For more details, see the previous LSA data flow templates

Business Transformation Layer

Note: For more details, see the previous LSA data flow templates

Reporting Layer

Note: For more details, see the previous LSA data flow templates

Virtualization Layer

Note: For more details, see the previous LSA data flow templates

/LSA100/ LSA Basic Flow –Layering Data & Logic

 

Background

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

 

This blog describes the LSA Data Flow Template LSA100 that offers basic yet general layering of data and flow logic.

For general information about LSA Data Flow Templates please refer to

 

Description

LSA100 offers basic yet general layering of data and flow logic.  The propagation layer DataStore object provides reusable data for multiple InfoCubes, which serve different business needs. Reusability is achieved by strictly separating harmonization (e.g. for master data) and quality transformations from transformations that serve a specific business need (InfoCube).  All InfoCubes obtain data solely from the propagation layer DataStore objects and never directly from extraction.  The propagation layer therefore implements the EDW principle ‘extract once – deploy many’ and is the ‘single version of truth’ for all InfoCubes and the provided reports and analytics.

 

LSA100_01_conceptual view

Picture 1: LSA100 conceptual view

 

LSA100_BW

Picture 2: LSA100 in BW 7.30

 

Target Group

Data flows should always be layered, regardless of the expected data warehouse size. LSA100 is applicable for BWs, which can be qualified as follows:

  • Small, mid-range size
  • No general predictable volume issues
  • Limited geographical reach (time zones)
  • Pragmatic customer orientation
  • Single source system per DataSource

 

Implementation Details

  • All InfoCubes obtain data from the propagation layer DataStore objects.
  • Suitable for extractors that offer a delta load.
  • Suitable for full loads if the loaded data offers a complete business picture (also reverse and deletion records!) – Propagation Layer DataStore objects calculate the delta image
  • Transformations  rules directly between InfoProviders, therefore without using InfoSources

 

Acquisition Layer

Accept extracted data 1:1. Add additional information to each record like source system, load time, organizational data (like company code) to make standardized administration and addressing of all records possible.

  • Extracted data is stored in a PSA element
  • The PSA element is defined at DataSource and source system level
  • The DataSource should be comprehensive (offer all relevant process information)
  • The load history is stored in a PSA element. Note the following impacts:
  • No archiving/near-line storage (NLS) with PSA (volume!)
  • History is stored in the regular data provisioning flow (performance!)
  • Impossible to influence how history of loaded data is stored (1:1 from extraction)

Harmonization & Quality Layer

Apply data harmonization & data quality transformations (if any) between acquisition PSA elements and propagation layer DataStore objects.

  • Data harmonization transformations if necessary
  • Compounding of InfoObjects  to handle multiple source systems that offer data for the same DataSource (an LSA data flow template that supports domains is recommended)
  • Mapping of local keys/values to integrated keys/values (common coded)
  • Standardized  error handling/(integrity) checks via error DTP
  • Adding additional information for each record
    • 0SOURSYSTEM - Origin (possible part of Propagation Layer DSO key)
    • Load time(stamp)
    • Other technical characteristics, such as request ID  (for ease of administration)
    • Other customer defined useful, (stable) criteria to qualify/describe content of record that increases transparency for administrators and business user (assigning country/region for example)
  • LSA data unification transformations (always)

Note: No business scenario-related transformations are allowed here! (This would reduce or prevent reusability of the data in the propagation layer) 

Propagation Layer

The InfoProvider for the propagation layer is the DataStore object.

  • Normal DataStore objects are preferable due to their functionality (calculation of delta images etc.)
  • Business key is (part of) the DSO key
  • No SID generation
  • With unique business key (e.g. COPA) consider using standard DataStore objects with ‘unique data’ option. (This option can be switched on and off in process chains and thus enables single operations like on standard DataStore objects) 
  • With unique business key and large volumes write-optimized DataStore objects are another option
  • The Propagation DataStore objects are related to a DataSource (s. naming – area part)

Business Transformation Layer

Data Mart specific transformations

  • Lookups of master data/other transactional data
  • Merge of data from different propagation layer DataStore objects can result in dedicated business transformation layer DataStore objects for reasons of synchronization.

Note:  Apply business transformation rules only on top of the reusable propagation layer (DataStore Objects)

Reporting Layer

Normally InfoCubes – options:

  • Standard InfoCube data hosted on RDBMS and usage of aggregates
  • Standard InfoCube data hosted on RDBMS and BWA Index
  • BWA InfoCube only  (no RDBMS persistence – BW 7.30)

Virtualization Layer

Queries use only MultiProvider (Virtualization Layer) for reasons of transparency and flexibility

Background

This blog gives a short overview on the LSA. A basic understanding of the LSA is necessary in order to understand the LSA Data Flow Templates.

LSA Data Layer - Structuring the Data Flows

The LSA proposes a standard set of layers. The services offered by the various layers are described below. 

  • There are four EDW layers: the Data Acquisition layer, the Corporate Memory layer, the Harmonization & Quality layer and the Data Propagation layer
  • There are three Architected Data Mart related layers: the Business Transformation layer, the Reporting (Architected Data Marts) layer and the Virtualization layer

 II-01-LSA Layers

 

Depending on customer preferences and on operational conditions and the like, there might be additional layers.  Not all layers necessarily host persistent InfoProviders (InfoProviders that store data in database tables).

Note that the LSA is very strict in terms of where the necessary transformations should be performed during the staging process:

  • Between Acquisition Layer and Propagation Layer (in the Quality & Harmonization Layer) only non-business purpose specific transformations are used. These transformations convert data into the data warehouse data model. As a result, the (EDW) Propagation Layer offers data with corporate compliant semantics and values, which are reusable by different data marts.
  • Business -specific  transformations, which only serve a certain data mart scope, only take place on the (EDW) Propagation Layer  (in the Business Transformation Layer).

This applies to all LSA data flow templates offered.

The graphic below provides a rough overview of the data flow activities (from left to right):

II-02-LSA Single Point of Truth

Data Acquisition Layer

The Acquisition Layer is the Inbox of the Data Warehouse:

  • Fast inbound & outbound flow of data to targets
  • Accepts data from extraction with as little overhead as possible – no early checks, merges, transformations (1:1)
  • Adds additional information to each record, like origin (source system), load time, origin organizational data (like company code). This makes standardized administration and addressing of all records possible
  • Provides abstraction of Data Warehouse from sources of data
  • Provides short term history of extracted data for immediate/short term data inspection

Harmonization and Quality Layer

The data is passed from the Acquisition Layer to the Harmonization and Quality Layer, which is responsible for transforming the extracted data in accordance with common semantics and values. The final result is stored in Propagation Layer DataStore objects. What happens in this layer depends largely on the quality and integration level of the extracted data. There is often no explicit data storage in this layer. This is especially true with data flows that receive data from SAP DataSources, as the data derived from the SAP sources is frequently already in good shape.

Please note: No business scenario-driven transformation are allowed here. This would reduce or prevent the reusability of the data in the Propagation Layer. 

Data Propagation Layer

The Data Propagation Layer offers persistent data (stored in DataStore objects), which is compliant with the defined company quality, integration and unification standards. The Data Propagation Layer meets the ‘extract once deploy many’ and ‘single version of truth’ requirements (reusability).

Business Transformation Layer

The data mart related layers get their data from the Data Propagation Layer. All business related requirements are modeled in these layers.

In the Business Transformation Layer, data transformations take place, which serve to fulfill the Data Mart scope. Dedicated DataStore objects in the Business Transformation Layer might be necessary if we have to join or merge data from various Propagator Layer DataStore objects.

Please note:  Only apply business transformation rules on reusable Propagation Layer DataStore Objects.

Reporting layer

As the name implies, the Reporting layer contains the reporting related InfoProviders (Architected Data Marts). The Reporting Layer Objects can be implemented as InfoCubes with or without BW Accelerator or sometimes as DataStore Objects.

Virtualization Layer

To ensure greater flexibility, the queries should always be defined on a MultiProvider.

LSA Data Domains – Collect & Split Data Flows

For large BWs, especially for global ones, the LSA suggests introducing standardized semantic partitioning of transactional data on the Data Acquisition Layer throughout your BW Data Warehouse or large parts of it. Standardized means that all InfoProviders (or areas thereof) are semantically partitioned using the same partitioning strategy. The semantic partitioning of data has to be stable over time. Splitting transactional data in BW by markets or group of markets can serve as an example from the consumer product industry for a strategic semantic partitioning of a BW EDW.  The LSA calls the resulting parts of the BW Domains. Organizational characteristics, like 0COMP_CODE and 0CO_AREA, serve as semantic partitioning criteria that implement these market related Domains.

The semantic partitioning modeling pattern is supported in BW 7.3 by the Semantically Partitioned Object (SPO). SPOs allow automated definition and (to a large degree) automated maintenance of partitions and partition criteria values. More information: http://help.sap.com/saphelp_nw73/helpdata/en/d1/468956248e4d9ca351896d54ab3a78/frameset.htm 

Please note: In addition to the described SPOs derived from Domain partitioning strategy, there might be additional partitions for certain InfoProviders (by time for example).

II-03-LSA Data Domains

 

The picture shows a global BW, partitioning the transactional data into three Domains.

Depending on the sources system landscape, this results in splitting and/or collecting data flows on the Data Acquisition Layer.

Split of data flows according to LSA Domains:

II-04-LSA Data Domain split

Collect data flows according to LSA Domains:

II-05-LSA Data Domain collect

 

Please note that there are no domains in the Corporate Memory Layer.          

           

The Role of InfoSources in the LSA

InfoSources play a decisive role in the LSA. The larger a BW becomes, the more important it is to use InfoSources to ensure maintainability of the data flows and overall flexibility. Starting with BW 7.0, InfoSources are no longer required for modeling a data flow. The transformation rules can be implemented directly between InfoProviders.  With larger BWs and semantic partitioning of transactional data flows into LSA Domains as shown above, this will result in a large number of transformation rules, which can be confusing.  Any change to the semantically partitioned InfoProviders results in a change to all transformation rules. Using an inbound InfoSource in front of an InfoProvider, and an outbound InfoSource after an InfoProvider, makes it possible to bundle the flow logic. The transformation logic is always located between the outbound InfoSource of the source InfoProvider and the inbound InfoSource of the target InfoProvider. This is illustrated in the graphic below:

II-06-LSA and InfoSources

 

Note that there are never transformations between inbound InfoSource and InfoProvider and InfoProvider and outbound InfoSource. This flow logic is guaranteed and implemented automatically using SPOs.

 

LSA Template Naming Conventions

The LSA framework, defined by layer and domains, is reflected by the naming of the BW metadata, or by the technical names of the InfoProviders to be exact.

LSA Template Naming Conventions for InfoProviders

The naming of InfoProviders is limited by the DataStore object naming limitation (8 digits) and the restrictions on the SPO naming.

As far as the actual naming is concerned, customers have a significant level of freedom. The InfoProviders of LSA data flow templates are named as follows:

  • Layer: byte 1
  • 1  digit abbreviation to qualify the layer the InfoProvider is located in
  • 4 digit abbreviation to qualify the data content of the InfoProvider
    • A DataSource related identifier for all EDW layers related InfoProviders or
    • A Business scenario related identifier for all Data Mart Layers related InfoProviders
  • There might be more than one InfoProvider in a specific Layer for a specific area
  • 1  digit abbreviation to qualify the Domain the InfoProvider is located in (U for US for example)
  • 1  digit abbreviation to qualify further partitioning of an InfoProvider (1  for 2011 for example)
  • Area: byte 2 to 5
  • Sequence number: byte 6
  • Domain: byte 7
  • Sub-Partition: byte 8

Note: The naming used in the LSA data flow templates should serve as an example. Naming should always be checked in accordance with customer requirements before starting implementation.

Example:   

P

L

S

H

D

0

U

0

1

2

3

4

5

6

7

8

 

1      P           Propagation Layer InfoProvider

2-5  LSHD    Area filled from Sales Order Header DataSource

6      0           1st InfoProvider with respect to area in this layer

7      U          Domain US

8      0           No further logical/semantic partitioning

 Note:  Letters qualifying the partitions of an SPO are only enabled using the SPO BadI. For more information:  Semantically Partitioned Objects (SPOs) built from BAdI

 

LSA Template Naming Conventions for InfoSources

Two InfoSources are always generated automatically when you activate a SPO: One InfoSource in front of the SPO partitions - the inbound InfoSource - and another InfoSource to manage the data flow to the targets – the outbound InfoSource.

The InfoSource name is derived from the name of the SPO:

Example:   SPO name is PLSHD0

  • PLSHD0_I       name of InfoSource before  SPO (I - Inbound )
  • PLSHD0_O     name of InfoSource after  SPO (O -  Outbound )

LSA Template Naming Conventions for DTPs

Data Transfer Processes (DTPs) can be generated for SPOs if the transformations between source InfoProvider and SPO are active. The DTP description comprises the technical name of the source InfoProvider and the technical name of the target SPO partition, PLSHD0U0 -> DSISY002 for example.

Introduction to LSA Data Flow Templates

The Layered Scalable Architecture (LSA) is a reference architecture built on Enterprise Data Warehouse principles to ensure consistent, scalable and flexible design and implementation of SAP NetWeaver Business Warehouse (BW).

The larger a BW is or is expected to become, the more important the standardization introduced by the LSA will be.  At the same time, the LSA also provides valuable guidelines for smaller BW systems. 

The Data Flow Template is a new feature of SAP NetWeaver BW 7.3.  A data flow template makes it possible to define and document basic settings, which can be reused by real data flows.  

With the new Data Flow Template feature it is now possible describing LSA standards directly in BW.

With BW 7.30 there comes a set of 10 LSA data flow templates as an initial offering.

I-01-LSA Templates in BW

Picture 1. LSA Data Flow Templates in BW 7.30 SP3

How to Work with LSA Data Flow Templates

The target of LSA data flow templates is to facilitate the adoption of LSA in the customer community. 

For those not yet familiar with the LSA principles, a short overview of LSA basic architecture components (the building blocks) and the derived naming conventions is provided. It is important to read this before applying the delivered LSA data flow templates. The LSA data flow templates are built in such a way that higher numbered ones are based on the previous ones. Template LSA300 is therefore an extension of LSA200 for example, while LSA200 is an extension of LSA100.

Before using the LSA data flow templates, you should perform a verification and adjustment taking into account your particular requirements. You can then create your own customer LSA data flow templates based on the templates provided.

For more information about BW LSA, see:

SAP NetWeaver BW: BW Layered Scalable Architecture (LSA) / Blog Series

 

Supported Versions

The data flow template is a feature of SAP NetWeaver BW 7.3.

The LSA data flow templates are offered with SAP NetWeaver BW 7.3 SP3.

The LSA data flow templates use new SAP NetWeaver BW 7.3 features like the Semantically Partitioned Object (SPO) for automated definition of logically partitioned DataStore Objects and InfoCubes.

That does not mean that the data flows will not work for BW 7.x environments. They will even work (with certain restrictions) in BW 3.5, as customer LSA implementations show.  But the implementation effort will be far greater than with BW 7.3.

Abstract

Logical Partitioning is a very popular modeling pattern in BW. It means instead of storing data in a single InfoCube or DSO distributing the data into several identical InfoCubes or DSOs.

Fig01- LogicalModelingPattern

Fig. 1: Logical Partitioning Modeling Pattern

 

The Logical Partitioning modeling pattern is used to achieve and preserve

  • reasonable, manageable sizes of InfoProviders (load, recovery, querying)
  • independent decoupled data loads, time zones support (report availability) and robustness separating data from different organizational units  - as illustrated in the picture above          

The new Semantically Partitioned Object (SPO) feature supports the Logical Partitioning Modeling pattern. The SPO wizard automates building of consistent semantically partitioned InfoCubes and DSOs and guarantees consistent distribution of data into the PartProviders for more information please refer to BW 7.30: Semantically Partitioned Objects

Nevertheless Logical Partitioning may still be challenging even with the new SPO:

  • if we want to create multiple SPOs using the same partitioning criteria (consistency, costs)
  • if we have complex partitioning criteria and/ or partitioning criteria with numerous values
  • if partitioning criteria change over time (partitioning by time characteristics or new partitions during roll out)

  

To address these challenges the new SPO offers the possibility defining centrally the rules for partitioning i.e. partitioning criteria and values using a BAdI.

This blog gives an introduction about modeling, maintaining and managing your SPOs using the SPO BAdI.

Background - The Challenge of Volatile or Complex Logical Partitioning Conditions  

Traditionally logical partitioning modeling is done case by case (tactical logical partitioning). In other words the definition of logical partitions is done for each InfoCube and each DSO individually. Nevertheless the partition characteristic and the partition characteristic values assigned to the PartProviders of different InfoCubes/ DSOs are often identical: e.g. the Sales Order InfoCube is logical partitioned by year resulting into 3 PartProviders storing data of the last 3 years. The same partitioning could apply to purchasing order InfoCubes etc.

The logical partitioning for our little example can be modeled in various ways e.g.

  • Assigning a fix  partition characteristic values: assign a fix year for each PartProvider e.g. PartProvider 1 - for the actual year 2010, PartProvider 2- for 2009, PartProvider 3- for 2008
  • Modeling a rolling window: define a rule which assigns partition characteristic values (in the example year) in an automated way to the Partitions 

With this simple example we can learn two things (at least):

1.   We should not use different logical partitioning implementations for the same requirements of different InfoProviders as this will decrease maintainability and increase costs. Beside this we will have a lot of second best implementations like implementation 1. in the example

2.    If partitioning conditions are volatile over time we should not assign fix values but a rule based logical partitioning to remain flexible. The origin of the volatility in our example is in the expression ‘last 3 years’ – with every new year this means that the partition condition will change

If we look to the real world we can learn a third and a forth thing:

3.   Assigning partition characteristic values can be quite complex. As shown in figure 1 there is normally no partition criteria characteristic that allows addressing all American or German records directly. Instead we have characteristics like 0COMP_CODE, which serves as partition characteristic what means that we often have to assign a bunch of 0COMP_Code values selecting e.g. the American records for the American PartProvider.

4.   Building a logical partitioned InfoCube or DSO in a continuous roll-out situation not all partition characteristic values are known at definition time and often even the final number of PartProviders is not known. That means over time you have to maintain the logical Partitions again and again assigning new values to PartProviders and/ or creating new PartProviders. 

Both, 3 and 4, offer a tremendous chance to create erroneous entries and is cumbersome if you want to logical partition multiple InfoCubes and DSOs in the same way.

Again this kind of logical partitioning is rule based in the sense that you can once define centrally the mapping rule of 0COMP_CODE values to Americas and Germany etc Partitions.

We have this situation designing BWs using the Layered Scalable Architecture (LSA).

We can summarize that individual modeling of Logical Partitioned InfoCubes/ DSOs is the second best choice especially with large BWs (EDW) as we have numerous InfoProviders following the same logical partitioning conditions.

Semantically Partitioned Objects (SPOs) and Complex or Volatile Partitioning Conditions

The new SPO feature is a big step forward modeling complex partitioning conditions as the new wizard makes the assignment of partitioning characteristics and partitioning characteristic values to semantical partitions (PartProvider) transparent and consistent (DTP filters).

Nevertheless we still have to manually enter the partition characteristic(s) and partition characteristic(s) values for each SPO defining the semantical partitions. Thus with the standard SPO wizard functionality we are confronted with the same challenges mentioned before even though on a higher more comfortable level:

  • homogeneous modeling of different SPOs with identical semantical partitioning characteristic and values
  • volatile semantical partitioning by time

To overcome these challenges the SPO supports a rule based central definition of PartProviders and related DTPs offering the possibility to get all necessary criteria defining PartProviders via a BAdI implementation (BAdI RSLPO_BADI_PARTITIONING).

 

Automated SPO Partitions Modeling via BAdI Implementation – Introduction 

 

If we look closer to the SPO definition process we can differ between actively modeled and generated SPO components and assignments:

 

Fig02_SPOWizardSteps 

Fig. 2: SPO Wizard Steps

 

Actively modeled (the numbers refer to the overview picture above):  

  • The DSO / InfoCubes SPO Master – like normal DSO / InfoCube design (1)
  • The SPO Partitions – Defining partition characteristic and assigning partition characteristic values to PartProvider (2)
  • The Transformations between source and SPO InBound InfoSource (4)
  • The DTP assignment (using DTP templates)  between source and SPO PartProviders (5)

Generated:

  • SPO PartProviders (2)
  • InBound- , OutBound InfoSources (3)
  • (dummy) transformations between InBound-  and OutBound InfoSources and the SPO PartProviders (3)
  • the DTPs with Filters applying the Filters derived from PartProviders partition characteristic values (after the creation of Transformation(s) between source(s) and Inbound InfoSources) (5)      

The SPO with an implementation of the BAdI RSLPO_BADI_PARTITIONING allows to automate the SPO definition and maintenance process beyond standard wizard functionality creating all SPO PartProviders settings and the related DTPs based on central definitions (rules) thus reducing the manual design steps to a minimum.

There are two options addressing the BAdI implementation:

  • Interactively during the SPO Wizard based partitions definition and maintenance process and/ or
  • Centrally performing a mass execution of SPO partitions definition and maintenance process via transaction RSLPO_MASS_ACT

 

Interactive Wizard based SPO Modeling using BAdI Implementation  

The following picture illustrates the steps where the BAdI can actively support the SPO wizard:

fig03_InteractiveSPOmodelingusingBAdI

Fig. 3: Interactive SPO modeling using BAdI

 

Let’s walk thru the SPO using BAdI implementation definition process step by step introducing a simple example.

Rule-based Semantical Partitioning by Organizational/ Geographical Criteria

Example description:

We want to set up an SPO for a DSO that allows distributing Sales Order Header records from a single source with respect to the company which created the records, gathering companies by geographic location. Thus the mapping rule for the partitioning characteristic 0COMP_CODE could be as follows:

fig04_Example-LogicalPartitioningbyOrganization

Fig. 4: Example-Logical Partitioning by Organization

SPO Wizard – Model Master (1)

(The number refer to Figure 3)

We model the SPO Master as usual. The Characteristic 0COMP_CODE is part of the DSO key. Save it.

fig05_Example-DefineSPOMaster

Fig. 5: Example-Define SPO Master

 

SPO Wizard – Maintain Partitions –Define Partition Characteristic(s) (2)

(The number refer to Figure 3)

 

 We enter the Maintain Partitions screen and are asked to enter the partition Characteristic(s) – here 0COMP_CODE:

 

fig06_Example-DefineSPOPartitioningCharacteristic 

 Fig. 6: Example-Define SPO Partitioning Characteristic(s)

 

SPO Wizard – Maintain Partitions – Call BAdI (2)

(The number refer to Figure 3)

Entering the Partition maintenance screen you do not need to enter any partitions or partition characteristic values. These are entered via the BAdI: ‘Build Version From BAdI Implementation’:

fig07_-Example-MaintainSPOPartitionsusingBAdIimplementationscreen

Fig. 7: Example- Maintain SPO Partitions using BAdI implementation screen

The result looks as follows:

 

fig08_-Example-SPOPartitionsaftergettingdatausingBAdIscreen

Fig. 8: Example- SPO Partitions after getting data using BAdI screen

 

We observe that all SPO PartProviders are defined like given in the mapping table above: names, partition characteristic, partition characteristic values and even the sequence like the partitions are shown.

Centrally Define SPO PartProvider – Implement BAdI  RSLPO_BADI_PARTITIONING

You have to create a BAdI implementation of the SPO BAdI RSLPO_BADI_PARTITIONING using SE18/ SE19.

The interface IF_RSLPO_BADI_PARTITIONING contains five methods:

 

Fig09-SPOBAdIMethods

Fig. 9: SPO BAdI Methods

The methods GET_T_SPO, GET_T_PART, GET_T_PART_TEXT and GET_T_PART_CRIT return the parameters to define the SPO PartProviders, the method GET_T_PART_DTP returns the parameters to define the DTPs filling the SPO PartProviders.

RSLPO_BADI_PARTITIONING: Method  GET_T_SPO – returns SPOs

We have to return the (technical) names of the SPOs that shall be managed using the BAdI

 

Fig10-SPOBAdIMethodGET_T_SPO

Fig.10: SPO BAdI Method GET_T_SPO

 

Coding for our example:

method IF_RSLPO_BADI_PARTITIONING~GET_T_SPO.

  DATA:
    l_spo TYPE rslponame.

* Return a table with names of supported BAdI SPOs.

  l_spo = 'PSOHD3'.
  APPEND l_spo TO r_t_spo.
  
endmethod.

In our example the name SPO ‘PSOHD3’ is returned.

 

RSLPO_BADI_PARTITIONING: Method GET_T_PART - returns Partitions

This method is called for each SPO, which is returned by the first method GET_T_SPO and returns the Partitions and the display position of the Partition.

 

Fig11-SPOBAdIMethodGET_T_PART

Fig.11: SPO BAdI Method GET_T_PART

 

Coding for our example:

method IF_RSLPO_BADI_PARTITIONING~GET_T_PART.

  DATA:
       l_s_part       TYPE rslpo_badi_s_part.

* Return a table with the partitions of SPO I_SPO.
  IF i_spo = 'PSOHD3'.
*   Valid for SPO with name PSOHD3.

*   Partition 01:
    l_s_part-idpart = '01'.        <<<< Partition Id
    l_s_part-posit  = 1.           <<<< Position Partition
    APPEND l_s_part TO r_t_part.
*   Partition 02:
    l_s_part-idpart = '02'.
    l_s_part-posit  = 2.
    APPEND l_s_part TO r_t_part.
*   Partition 03:
    l_s_part-idpart = '03'.
    l_s_part-posit  = 3.
    APPEND l_s_part TO r_t_part.
*   Partition 04:
    l_s_part-idpart = '04'.
    l_s_part-posit  = 4.
    APPEND l_s_part TO r_t_part.
  ENDIF.

endmethod. 

The method returns for our example SPO ‘PSOHD3’ four Partitions.

RSLPO_BADI_PARTITIONING:  Method GET_T_PART_TEXT - returns description of Partitions

This method is called for each SPO, which is returned by the first method GET_T_SPO (other assignments only done here will be ignored) and returns the names / description of the Partitions. The method supports multi-languages. 

 

Fig12-SPOBAdIMethodGET_T_PART_TEXT

Fig.12: SPO BAdI Method GET_T_PART_TEXT

method IF_RSLPO_BADI_PARTITIONING~GET_T_PART_TEXT.

  DATA:
    l_text_d       TYPE string,
    l_text_e       TYPE string,
    l_s_part_text  TYPE rslpo_badi_s_part_text.

* Return a table with the partition texts of SPO I_SPO.
    IF i_spo = 'PSOHD3'.
*   Determine partition texts:
*   Partition 01:
*   German text:
    l_s_part_text-idpart = '01'.  <<<< for Partition 01
    l_s_part_text-langu = 'D'.    <<<< language German
    l_s_part_text-txtlg = 'APJ Asien, Ozeanien, Japan'.
    APPEND l_s_part_text TO r_t_part_text.
*   English text:
    l_s_part_text-langu = 'E'.
    l_s_part_text-txtlg = 'APJ Asia, Pacific, Japan'.
    APPEND l_s_part_text TO r_t_part_text.

*   Partition 02,03,04...
  ENDIF.
endmethod.
RSLPO_BADI_PARTITIONING:  Method GET_T_PART_CRIT - returns Partitioning Criteria (partition characteristic(s) & values)  

This method is called for each SPO, which is returned by the first method GET_T_SPO and returns the partitioncriteria(s) and the assigned values for partitions returned by the method GET_T_PART (other assignments onlydone here will be ignored):

 

Fig13-SPOBAdIMethodGET_T_PART_CRIT

Fig.13: SPO BAdI Method GET_T_PART_CRIT

 

Coding for our example:

method IF_RSLPO_BADI_PARTITIONING~GET_T_PART_CRIT.

  DATA:
    l_s_part_crit  TYPE rslpo_badi_s_part_crit.
* Return a table with the partitioning criteria.
   IF i_spo = 'PSOHD3'.
*   Valid for SPO with name PSOHD3.
*   Fill partitioning criteria:

    l_s_part_crit-iobjnm = '0COMP_CODE'. <<<< Partition characteristic
    l_s_part_crit-posit  = 1.            <<<< 1st Partition characteristic
    l_s_part_crit-opt    = 'EQ'.         <<<< Operator

*   Partition 01:
    l_s_part_crit-idpart = '01'.         <<<< Partition ID
    l_s_part_crit-low = 'MY02'.          <<<< Partition characteristic value
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.            
    APPEND l_s_part_crit TO r_t_part_crit.
    l_s_part_crit-low = 'MY08'.
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.
    APPEND l_s_part_crit TO r_t_part_crit.
    l_s_part_crit-low = 'MY17'.
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.
    APPEND l_s_part_crit TO r_t_part_crit.
    l_s_part_crit-low = 'JP03'.
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.
    APPEND l_s_part_crit TO r_t_part_crit.
    l_s_part_crit-low = 'CN17'.
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.
    APPEND l_s_part_crit TO r_t_part_crit.

*   Partition 02,03,04…..

  ENDIF.

endmethod.

The so far described methods return all information necessary defining the PartProviders of an SPO.

SPO Wizard – Activate (3)

(The number refer to Figure 3)

All SPO Components are generated and activated i.e. all PartProviders, In- and Outbound InfoSources and the (dummy) Transformations between these InfoSources and the SPO PartProviders (s. overview picture above).

SPO Wizard – Create Transformation between Source and In-Bound InfoSource of SPO & Activate (4)

(The number refer to Figure 3)

This part is again a ‘creative’ modeling part like the definition of the SPO Master and cannot be automated using the BAdI.

 

Fig14-Example-AssignSourceInfoProvider

Fig.14: Example - Assign Source InfoProvider

 

Fig15-Example-CreateTransformations

Fig.15: Example - Create Transformations

 

SPO Wizard – Assign Data Transfer Processes - Call BAdI (5)

(The number refer to Figure 3)

Now as the Transformation between Source and SPO InBound InfoSource is active we can create the DTPs between the Source, which is in our example a DSO, and the SPO PartProviders.

When we enter the DTP Generation screen no DTPs are assigned.  For a rule-based assignment we call again the BAdI:

Fig16-Example-AssignDTPsviaBAdI

Fig.16: Example - Assign DTPs via BAdI

As a result the BAdI returns the DTP assignments:

 

Fig17-Example-AssignDTPsviaBAdI

Fig.17: Example - Assign DTPs via BAdI

 

We observe that for each PartProvider a DTP from source DSO TSOHD100 is assigned.

Marking the DTPs and pressing ‘Generate’ leads to:

 

Fig18-Example-GenerateDTPs

Fig.18: Example - Generate DTPs

 

Now all the DTPs are generated and the correct filter values are automatically applied:

 

Fig19-ExampleAutomatedFilteringinDTPs.

Fig.19: Example – Automated Filtering in DTPs

 

RSLPO_BADI_PARTITIONING:  Method GET_T_PART_DTP - Returns DTPs with Filters

This method is called for each SPO, which is returned by the first method GET_T_SPO and returns DTPs and the correct Filter values for partitions returned by the method GET_T_PART (other assignments only done here will be ignored):

 

Fig20-SPOBAdIMethodGET_T_PART_CRIT

Fig.20: SPO BAdI Method GET_T_PART_DTP

 

Some extracts from BAdi documentation:

  • IDPART
    The field IDPART is used to assign the data transfer process to a specific partition of the SPO. The partitions of the SPO are defined in the interface method GET_T_PART.
    .
  • DTPTEMPLATE

    The field DTPTEMPLATE contains the name of a DTP template. The properties of this DTP template are applied to the new data transfer process when it is created.

    The following rules and consistency conditions apply:

1. The name of an existing DTP template must be entered.

2. The type of the specified DTP template must match the TLOGO type TLOGO of the data source. If the data source of the DTP is a DataSource, for example, the DTP template must also be of type DataSource.

  • TLOGO

    The field TLOGO contains the TLOGO type of the data source that the data transfer process is connected to.
    :
    • DataSource ('RSDS')
    • InfoObject ('IOBJ')
    • DataStore-Objekt ('ODSO')
    • InfoCube ('CUBE')
    • MultiProvider ('MPRO')
    • InfoSet ('ISET')
  • SUBTYPE

    The field SUBTYPE contains the TLOGO type of the data source that the data transfer process is connected to. The TLOGO subtype is relevant, if the data source is an InfoObject. The TLOGO subtype can be used to differentiate between the extraction of attributes, texts and hierarchies of a characteristic.
    • Attributes ('ATTR')
    • Texts ('TEXT')
    • Hierarchies ('HIER')
  • OBJNM

    The field OBJNM contains the technical name of the data source that the data transfer process is connected to.

  • LOGSYS

    The field LOGSYS contains the name of a source system. The name of the source system is relevant, if the data source is a DataSource.

Coding for our example:

method IF_RSLPO_BADI_PARTITIONING~GET_T_PART_DTP.

* Return a table with DTPs that are to be generated for new partitions.

  data l_s_part_dtp type rslpo_badi_s_part_dtp.
  case I_spo.
    
    when 'PSOHD3' .
    l_s_part_dtp-idpart = '01'. <<<< Partition ID
    l_s_part_dtp-dtptemplate = 'LSA_DSO_SRC'. <<<< DTP template name
    l_s_part_dtp-OBJNM = 'TSOHD100'. <<<< Source DSO name
    l_s_part_dtp-tlogo = 'ODSO'. <<<< Type of source (TLOGO)

    APPEND l_s_part_dtp to r_t_part_dtp.
    l_s_part_dtp-idpart = '02'.
    l_s_part_dtp-dtptemplate = 'LSA_DSO_SRC'.
    l_s_part_dtp-OBJNM = 'TSOHD100'.
    l_s_part_dtp-tlogo = 'ODSO'.

    …….
    endcase.
endmethod.

Interactive Wizard-Based SPO Modeling Using BAdI Implementation – Summary

Fig21-ExampleWorkbenchviewonBAdIcreatedPartitions

Fig.21: Example – Workbench view on BAdI created Partitions

 

We have seen that the SPO wizard in conjunction with the BAdI implementation highly automates the SPO creation process. The BAdI allows the central definition of any kind of rules that offer the parameters for SPO PartProvider and DTP assignments (DTP automated generation is planned).  This highly increases development efficiency if multiple different SPOs follow the same semantical partitioning criteria as we have to define the criteria only once. Beside this the overall consistency of our model increases considerably. This becomes evident if you structure your BW applying BW Layered Scalable Architecture (LSA) principles as semantical partitioning (Domains) is done strategically for the large parts or even the entire BW. 

With respect to our example I want to emphasize that it is just a simple one illustrating the basic behavior of the BAdI. In general it is not recommended hard coding the SPOs and the parameters in the methods! Instead you should store the parameters in central user defined tables making partitioning criteria reusable for multiple different SPOs! Doing this means just updating your central tables if the Partitions for a new SPO shall be generated using the BAdI.

SPO Modeling using BAdI Implementation and SPO Partition Maintenance 

Beside the consistent creation of SPO Partitions the BAdI helps to maintain the Partitioning if it changes over time. First of all you can change the partitioning criteria and values for a PartProvider without restrictions as long as a PartProvider is empty. Let’s have a look what it means if data have been already loaded to the PartProviders:

 

  1. New PartProvider (Partitions) – no problem as long as the partition criteria values do not overlapMake the new parameters for the BAdI available. Start SPO Wizard (Change SPO), call BAdI for PartProviders and DTPs. Activate/ generate both. That’s all if the central Transformation (3) is done generically (no fix PartProvider specific coding).
  2. New partition criteria values for existing PartProvider – no problem long as the partition criteria values do not overlapSame procedure as described in 1.
  3. Drop a PartProvider – no problem as long the partition value is not used elsewhereJust remove Partition-ID - Same procedure as described in 1.
  4. Remove partition criteria values for existing PartProvider – not supported yet. Planned: First you have to manually delete all data entries in PartProvider for partition criteria values you plan to remove.  Call BAdI – BAdI will check whether there are still data in the PartProvider for partition criteria values, which shall be removed. If no data found remove partition criteria values for PartProvider and DTP.

Last but not least the BAdI PartProvider definitions overrule all manual changes to the Partitioning done in the SPO wizard! This is by itself a very strong feature as it guarantees the consistency of all BAdI managed SPOs with respect to the central definitions throughout the BW! 

We notice that so far applying any expansion or change to existing SPO PartProviders using the BAdI is done calling the SPO Wizard for each SPO affected. This may mean still a lot of manual work in case we want to change a large number of SPOs and implies the danger that we might forget an SPO to maintain.

Thus there is a need for applying changes to a large number of BAdI managed SPOs in an automated manner and this leads us to our next chapter.

Mass Maintenance and Activation of SPOs Using BAdI via Transaction RSLPO_MASS_ACT

The Transaction RSLPO_MASS_ACT allows maintaining all BAdI managed SPOs (returned by method GET_T_SPO) centrally.

RSLPO_MASS_ACT compares the BAdI (methods GET_T_PART, GET_T_PART_TEXT, GET_T_PART_CRIT) returned SPO definition with the active version of these SPOs and lists whether there are differences:

 

Fig22-ExampleRSLPO_MASS_ACTscreen

Fig.22: Example – RSLPO_MASS_ACT screen

 

In detail the status highlights New Partitions, Deleted Partitions and Changed Partitions.

You can now select any set of BAdI managed SPOs and apply the new BAdI settings in a mass change and activation run.

RSLPO_MASS_ACT allows reducing the SPO interactive modeling to an absolute minimum for all SPOs managed via BAdI in case of

  • adding new PartProviders
  • adding new partition criteria values to existing an PartProvider
  • removing partition criteria values if no data for this values exist any longer in the PartProvider (planned)
  • dropping/ deleting PartProviders
  • automated generation of DTPs (planned) – not just the DTP assignment like with the SPO Wizard

RSLPO_MASS_ACT functionality therefore perfectly addresses the volatile logical partitioning challenges mentioned above applying changes consistently throughout your BW.

 

RSLPO_MASS_ACT - SPO Mass Changes during ‘Continuous Roll Out’

If we look to our above introduced example it fits perfect to a market by market rollout what we often find with large BWs. Going live with a new market (e.g. with Germany – s. below) may mean adding company-codes to existing PartProviders of a large number of SPOs following the same partition criteria or creating new PartProviders. Let’s assume Germany gets an own PartProvider:

 

Fig23-Exampleextension

 Fig.23: Example – extension

E.g. in our simple example the method GET_T_PART_CRIT – (returns Partitioning Criteria (partition characteristic(s) & values)) just has to be extended adding Partition 05:

method IF_RSLPO_BADI_PARTITIONING~GET_T_PART_CRIT.

  DATA:
    l_s_part_crit  TYPE rslpo_badi_s_part_crit.
* Return a table with the partitioning criteria.
   IF i_spo = 'PSOHD3'.
*   Valid for SPO with name PSOHD3.
*   Fill partitioning criteria:

    l_s_part_crit-iobjnm = '0COMP_CODE'.
    l_s_part_crit-posit  = 1.            
    l_s_part_crit-opt    = 'EQ'.         

*   Partition 01,02,03,04 unchanged
*   
*   Partition 05:
    l_s_part_crit-idpart = '05'.                   <<<<<<<< new entries
    l_s_part_crit-low = 'DE12'.                    <<<<<<<< new entries
    l_s_part_crit-posit  = 1.                      <<<<<<<< new entries          
    APPEND l_s_part_crit TO r_t_part_crit.         <<<<<<<< new entries
    l_s_part_crit-low = 'DE18'.                    <<<<<<<< new entries
    l_s_part_crit-posit  = l_s_part_crit-posit + 1.<<<<<<<< new entries
    APPEND l_s_part_crit TO r_t_part_crit.  
 
ENDIF.

endmethod.

Calling RSLPO_MASS_ACT the BAdI detects that for our example SPO ‘PSOHD3’ a Partition is returned that does not exist in the active version of ‘PSOHD3’.

Fig24-ExampleRSLPO_MASS_ACTversioncomparison

Fig.24: Example – RSLPO_MASS_ACT version comparison

 

Executing RSLPO_MASS_ACT for ‘PSOHD3’ creates the new Partition:

Fig25-ExampleCentrallyrunningRSLPO_MASS_ACT

Fig.25: Example – Centrally running RSLPO_MASS_ACT

 

The result of centrally maintaining the SPO ‘PSOHD3’ with RSLPO_MASS_ACT:

Fig26ExampleRSLPO_MASS_ACTresultnewGermanpartition

Fig.26: Example – RSLPO_MASS_ACT – new German partition as result 

 

If you imagine that you may have to add this German Partition to multiple SPOs you immediately realize the increase of quality in terms of consistent modeling and the decrease of TCD (Total Cost of Development). 


RSLPO_MASS_ACT - SPO Mass Changes and Volatile Time Criteria - Rolling Window Scenario

As described above logical partitioning by time is a frequently used modeling pattern. If we logically partition multiple InfoProviders using the same time partition criteria we can define the rule via the BAdI implementation and apply it consistently to multiple SPOs:

Let’s assume we partition InfoCubes or DSOs by time e.g. 0CALYEAR with the option e.g. keeping data only of the actual-year, actual-year-1 and actual-year-2. Data older than 3 years shall be deleted automatically. Using the BAdI we can define a simple rule directing the data of a new year to a new partition and dropping old data, which run out of scope without having more than always 4 PartProviders. The picture illustrates the basic BAdI rule settings:

Fig27-ExampleSPOBAdIandRollingWindowscenario

Fig.27: Example – SPO BAdI and Rolling Window scenario I

 

The result of running RSLPO_MASS_ACT once a year is shown in the next pictures:

Fig28-ExampleSPOBAdIandRollingWindowscenarioII

Fig.28: Example – SPO BAdI and Rolling Window scenario II

 

Fig29-ExampleSPOBAdIandRollingWindowscenarioIII

Fig.29: Example – SPO BAdI and Rolling Window scenario II

 

A SPO BAdI rule managed Rolling Window scenario resolves the tasks

  • Of getting rid of ‘old’ data (housekeeping) if you do not want to archive them.
  • Of directing the new year’s data automated to an empty PartProvider.
  • Keeping all Partition assignments stable for already loaded data
  • Allowing positioning of Partitions in the Wizard like you prefer it – either the ‘oldest’ year first or the ‘oldest’ year at last position (like in the example)
  • Making you independent form year end activities – you can run RSLPO_MASS_ACT whenever you want during a year and immediately the Partition for the next year is prepared to accept data

 

Summary

The SPO Wizard is a powerful feature modeling logically partitioned InfoCubes and DSOs. With complex and volatile partitioning conditions you should consider using the BAdI RSLPO_BADI_PARTITIONING and define the partition criteria and rules centrally. All compatible (s. above) changes to BAdI managed SPOs can then be applied by the Transaction RSLPO_MASS_ACT. This decreases TCD (Total Cost of Development) and increases the overall quality (consistency) of your BW.

The BAdI RSLPO_BADI_PARTITIONING in conjunction with the Transaction RSLPO_MASS_ACT will play a fundamental role automating the implementation and maintenance of SPOs guaranteeing consistency of the logical partitions throughout your BW.

 

Disclaimer
http://www.sdn.sap.com/irj/sdn/index?rid=/webcontent/uuid/b0b72642-0fd9-2d10-38a9-c57db30b522e
This 30 minutes Live Expert Session shall serve as an appetizer for all those of the community:
  • Who are in a situation of planning/ building a BW on corporate, divisional and/ or global scale (aka EDW)
  • Who want to get a first impression about the principles doing it right from the very beginning
  • Who want to get a first contact with the BW Layered Scalable Architecture (LSA) 
Of course as a transparent scalable design of BW always pays back – everybody should join! 
Location/ details  (Vienna)
Location/ details  (Phoenix)

For those who are already familiar with the BW LSA or those who want to learn more about it, we kindly invite you to join the following TECHED 2009 sessions:

last update June, 11th 2011 

This blog bundles the information related to the SAP NW BW Enterprise Data Warehouse (EDW) Layered Scalable Architecture (LSA) the reference architecture for large scale BW DWHs/ EDWs.image

The documents listed below describe different facets of the LSA. New documents about LSA will be added from time to time depending also on your feedback.

 

For a first rapid walk thru the LSA please refer to SAP NetWeaver BW Layered, Scalable Architecture (LSA) for BI Excellence - Webinar Presentation.

Please find the different links to the LSA documents below:

  1. Blog: SAP NetWeaver BW: What is the BW Layered Scalable Architecture (LSA) all about?
  2. Blog: SAP NetWeaver BW: BW Layered Scalable Architecture (LSA) Building Blocks
  3. Blog: SAP NetWeaver BW 7.30: SAP NetWeaver BW 7.30: LSA Data Flow Templates Series - I. Introduction

More details in my Teched 2009 presentation: The Layered Scalable Architecture for SAP NetWeaver BW on a Corporate Scale’ – lecture BI302

May be a more general presentation from SAPSKILLS 2008 is also of value for you: SAP NetWeaver BI: Data Infrastructure und BI-Maturity – Data Marts, Data Warehouses (DWH), Enterprise Data Warehouses (EDW), Appliances

(Don't become confused only the first two slides are in German :-))

Actions

Filter Blog

By date: