LSA++ Open ODS and Corp Memory - whats the difference ?

[glen_spalding]

dear all

i am trying to understand the difference between the Open ODS layer, and the Corporate Memory, because as far as i can see, they do the same job ?

thanks

g

[sander_vanwilligen]

Hi Glen,

The Corporate Memory is part of the EDW Core. EDW Core is meant for standardized template-based BI. Corporate Memory is the place for a long term persistent storage of all extracted history (often on Near-line Storage). It is not meant for querying, consider it as a BI life insurance: if anything happens to the data, then it is still available for repair actions.

The Open ODS layer is the inbound layer. It can be BW managed (i.e. data stored in BW) or externally managed (i.e. data stored on HANA). It concerns field level data which can even be real-time replicated from the source system. It can be used for operational reporting but it can at the same time provide services to EDW Core.

Last but not least, please have a look to the document LSA++ for an overview of LSA++.

Best regards,

Sander

[glen_spalding]

hi sander, i have see these, and tried to understand many more like this, but i still get confused.

lets say, i have a transaction data source coming from sap erp - 0FI_GL_XX, this is loaded into a DSO - ZFIGL.

I wish to report off this DSO as usual business throughout the years to come.

which layer would i put the DSO in?

1. Open ODS

2. EDW

3. Corp Mem

[glen_spalding]

hi sander

just as an FYI, i am not considering in the PSA in any of the LSA++ layers, unlike with the older LSA, but should i?

i believed, in the earlier LSA the PSA was in the Acquisition Layer, but now in the LSA++ we see it replaced by the Open ODS layer.

if so, does this mean that the Open ODS layer is not for DSOs, and DSOs should start either at a Corp Mem, and/or the Propagation layer.

ta

anyone else like to jump in?

[sander_vanwilligen]

Hi Glen,

The Open ODS layer can be considered as the successor of the Data Acquisition layer. For persistent storage an aDSO with field-based modeling should be used here (instead of PSA in a scenario with ODP). Next to the persistent storage, also Open ODS Views can be used. An Open ODS View enables virtual access to externally managed data.

The EDW Core will offer the EDW services which we know from classic LSA. Here you can find aDSOs in both Corporate Memory layer (probably field-based) and Propagation layer (InfoObject-based).

Best regards,

Sander

[glen_spalding]

hi sander

thanks for the communications.

so, in my scenario above, lets say i have a sap standard transaction data source coming from sap erp  - 0FI_GL_XX, this is loaded into a sap standard DSO - ZFIGL.

I wish to report off this DSO as usual business throughout the years to come.

how would it look in the LSA++? do i need any other objects to fulfil the LSA++

which layer(s) would i put the DSO in?

1. Open ODS

2. EDW

3. Corp Mem

ta

[sander_vanwilligen]

Hi Glen,

First of all, you should not use classic DSOs anymore. In most cases it can be replaced by an aDSO.

I suggest to use the following objects / layers:

• Open ODS layer: aDSO with field-based modeling;
• EDW Core layer: aDSO with InfoObject-based modeling;
• Corporate Memoy layer: aDSO with field-based modeling.

Best regards,

Sander

[glen_spalding]

Hi sander

This is great and starting to make sense now.

Let me qualify to check I have understood.

So for a single extractor in my example above, you would have 0FI_GL_XX loading into an Open ODS aDSO,  another aDSO in the EDW layer,  and also an aDSO in the Corp Mem.

Yep?

[sander_vanwilligen]

Hi Glen,

Yes, you correctly understood my proposal. I would like to share two other suggestions.

Note that you need clear architecture and development guidelines to standardize the data flow patterns. This is of course not new for LSA++. I advice you to develop data flow templates (please see my blog for an idea).

For each "use case" which requires a data flow template, you have to think well how "heavy" or "light" the data flow must become in terms of persistent storage. This means often a trade-off. On one hand we should prevent uncontrolled redundancy. On the other hand we might need some redundancy, e.g. a Corporate Memory since the data is so important. Or we have to harmonize and cleanse data which we cannot do "on the fly".

If you choose for Corporate Memory, then do I really need storage in the Open ODS layer? There are different choices you can make and to avoid a future chaos, you need guidelines and standards.

Another remark can be made for data retention periods and storage. You have to think well for which period of time you are going to store the data. It will depend on where you store data (in-memory or not) and if you store the data more than once (e.g. a Corporate Memory). You will also have to involve the Business Users to define guidelines and standards for data and their data retention periods.

Last but not least, your discussion shows that an easy question like this can have different answers. LSA++ offers quite high-level architecture guidelines which can lead to different interpretations and implementation approaches. You have to translate the LSA++ reference architecture into your own company's LSA++ architecture guidelines and development standards.

Best regards,

Sander

[glen_spalding]

Hi sander,

Now we are getting to the crux of something that has my interest.

What you termed "heavy" or "light" persistance.

It seems a full and ideal LSA++ would require heavy persistance.  Right?  As with the three aDSO's in my simple example above.

This would be more costly BW on HANA. (Without dynamic tiering or archiving).

Is it fair to say this?

As you summarised in your last paragraph, and what I am beginning to understand,  trying to articulate the LSA++ is subjective to a companies architecture and policies.

I'm not sure if that is a good thing. I think the industry needs solid examples and not too much freedom to interpret the LSA++, otherwise it will defeat its own, BW structureded and flexible, objectives. Jon and anyone else, what do you think

[glen_spalding]

hi sander, in response to this, which you proposed

I suggest to use the following objects / layers:


Open ODS layer: aDSO with field-based modeling;
EDW Core layer: aDSO with InfoObject-based modeling;
Corporate Memoy layer: aDSO with field-based modeling.

is there any reason why the Open ODS layer, can act as the Corp Mem layer, seeing that it can technically contain the same object, and business wise, the same data ?

i had seen somewhere, a diagram with the Corp Layer of the LSA++ acting as a feeder to the EDW layer, just as the Open ODS layer would be. this too, seems a little redundant.

ta

[glen_spalding]

I meant to say

"is there any reason why the Open ODS layer, CANNOT act as the Corp Mem layer"

this question bugging me for a while now

[former_member71289]

Hi Glen,

Sander already explained it very well, here an illustrative example which is available as part of the SAP HANA-optimized BI Content.

Not sure if you are aware of this content, but it actually provides you with data models that are aligned with the LSA++ architecture (Open ODS Layer, EDW Core Layer, Virtual Data Mart Layer - and also Corporate Memory).

Below is an example of the shipped content in the content area MM - Purchasing.

Here you can see that the Open ODS Layer does not necessarily need to have a persistence in BW, but it makes sense to persist the data e.g. if you use ODP based extraction without PSA and you distribute the data into more than one data targets (e.g. Reporting ADSOs) in the EDW Core Layer.

Example is the procurement document line items (data source 2LIS_02_ITM) and schedule line items (2LIS_02_SCL) that are used to load different targets, such as purchasing document data, principal agreements, and purchase order delivery data.

The definition of the advanced DSOs for the Open ODS Layer and for the Corporate Memory might be very similar, e.g. they are both field based and write optimized. But the purpose of the two ADSOs and the amount of data they store is different:

- The field based ADSOs in the Open ODS Layer (e.g. /IMO/O_PUR11) is used as kind of PSA replacement to distribute the data within BW to the different data targets, otherwise you would fetch all data first from the ODQ in the source system multiple times. Similar to a PSA you usually would delete the data from this DSO after the data was loaded succesfully into all data targets

-  The field based corporate memory ADSO (e.g. /IMO/CMPUR11) stores the whole history of data and is only used e.g. to reconstruct data e.g. if a change to  transformation rules in the EDW Core Layer is required. As the data is only required on in exceptional cases the data does not need to be hold in-memory. Therefore the flag "Inbound table as extended table" is turned on for this ADSO in the standard content.

Now when you compare the persistences and amount of data of LSA++ data flows with LSA data flows:

If you do it right, LSA++ actually means less data.

All the best,

Andreas

[glen_spalding]

hi Andreas,

i guess i still cannot get around why the Open ODS and Corp Memory cannot be combined.

you state that the Open ODS is a kind of replacement for the PSA. I thought the PSA was intended only to retrieve the data and then be deleted after a short while (month or so) when the data was QA'd in the EDW. Is this the case with the Open ODS then ? Data in this layer is not persisted for all time ?

you also confirm that the objects (aDSOs in the Open ODS and Corp Mem) are similar, are they identical. Both layers want all the data, in unconverted form, data is stored in the Corp Mem, for ever. Data in the Open ODS, i thought was also stored for ever (hence the question above).

yes, i have seen the new HANA Advanced BI Content too. but it does not answer my questions.

i hope you can, and dont mind my questions.

thanks

g

[glen_spalding]

hi andreas, and anyone

i still cannot understand why there are two flows from the 2LIS_02_ITM. one into the Open ODS, the other into the Corp Memory. when each are persisted, even by the same object, and technical settings - an write optimized aDSO.

has this been answered above, and i have missed it ?

[former_member71289]

Hi Glen,

sorry if it was not clear.

I would see aDSOs in the Open ODS layer as optional. I would only add them to the data flow if they support the data load process.

As outlined for 2LIS_02_ITM in the BW content an aDSO was added in this layer, as the data is distributed into multiple EDW core aDSOs. On this way data from the ODQ in the source system into SAP BW system needs only transferred once. But you can see that for the most of the other data flows an acqusition aDSO (Open ODS layer) was skipped.

Older requests should be deleted from these write-optimized aDSOs, similar like PSA.

The corporate memory is different as it holds the full history and as this data is only required to be accessed in exceptional cases this data does not need to be stored in-memory.

If you combine both functionality in one ADSO, this would mean that you will need to load the data in all cases sequential first into the CM / Open ODS Layer then into the ADSO in the EDW Core Layer instead of parallel loading your EDW Core Layer and your Corporate Memory.

And you would need to decide where to store this data (in-memory/extended memory).

.

All the best, Andreas

[glen_spalding]

Hi A

so when you say "I would see aDSOs in the Open ODS layer as optional. I would only add them to the data flow if they support the data load process", you would load straight into the EDW layer?  I see.

And also you delete from the Acquisition layer? Is that a suggested action of the LSA++?

Finally,  yes, loading into CM then EDW is serial, but it  avoids loading into Acquisition layer, and consequent deletion, if I understood right.

This is why I am confused

[sander_vanwilligen]

Hi Glen,

Further to the excellent replies of Andreas (on Feb 17, 2016 2:09 PM and Feb 19, 2016 11:05 AM) I would like to add some final remarks.

Implementing LSA++ has new aspects we have to deal with. An important one is less (in-memory) persistence. On the other hand, many classic LSA principles are still valid and we still have to use "common sense".

The function / service of Open ODS layer and Corporate Memory layer is totally different. That is why as a best practice, they should not be combined.

The function of Open ODS layer is facilitating the inbound daily data flow (i.e. data acquisition). With the ODP (Operational Data Provisioning) framework, PSA is optional. Therefore an aDSO (field-based and in extended memory) can be used if required (i.e. it is optional). Andreas explained it with the Purchasing example. Like before, inbound data should not be stored for a long term. PSA clean-up is normally done after an x number of data loads, e.g. max. one week. This way you keep the inbound data flow clean.

Corporate Memory has an entirely different function: long term storage of all extracted data. This is why in classic LSA, Corporate Memory is modeled as a "branch-out" data flow. It can be updated after peak-hours and it does not burden the daily data flow anymore. The aDSO (field-based and in extended memory) will grow steadily in the course of time. Only in exceptional / emergency cases it can be used to repair or reconstruct data in the daily data flow. Moreover, it is considered that the data is more safe if it is stored apart from the daily data flow.

To wrap up: Inbound ODS layer is optional. If it is used, then the data retention period is rather short (e.g. one week). It can either be modeled as PSA (w/o ODP) or a field-based aDSO (with ODP). Housekeeping should be performed on a regular basis so that the data volume stays very small.

Corporate Memory is loaded off peak hours as a branch-out data flow. It is modeled as a field-based aDSO and the full history of extracted data is stored. Consider it as a "BI life insurance", you hope you won't need it but in case you must be able to rely on it. That is why it should not be part of the daily data flow to avoid any risks.

Last but not least, thank you Glen for initiating such an interesting discussion and thank you Andreas for your valuable knowledge sharing.

Best regards,

Sander

[glen_spalding]

hi sander

well, it does seem we are going round a bit, and maybe time to wrap up, but still, i find it confusing to articulate to customers.

so to summarize my thoughts

1. LSA++ is a principle, i think we all agree there. i believe it should be detached from the interpretation to implement, because archiving, costs, memory footprint etc ... tend to complicate the idea of it. whether it stays in memory, external tables, or archive, as far as a principle is concerned, should not matter.

2. i don't think the older Acquisition layer, was ever the PSA, and comparing the newer ODS to the PSA may not be correct.

3. the existence of the aDSO for the 2LIS_02_ITM in the ODS layer, because the data is distributed into multiple EDW core aDSOs should not be why the ODS layer is used.

4. loading the data into two different places, i.e ODS and CM, i think is redundant. i still struggle with that idea, and why they cannot be merged. it can still be clean. if errors occur in the source, and need fixing in BW, that will still occur, regardless of it's destination in BW. with both layers, would require two fixes.

5. i am trying to think of a reason why we cannot load into the CM, retain the detail transactions as history, and further load into the EDW. I have seen some that suggest this, out there on the web.

6. if the ODS used because of the need to load into multiple targets. this can be replaced by using the CM.

well, maybe i'll get it eventually, but until then, i'll carry on reading.

thanks to all who participated.

if anyone shares my confusion, or wishes to comment further, please do.

until then, thanks

[glen_spalding]

hi sander,

just a quick one, i hope ...

when you state "...Inbound ODS layer is optional. If it is used, then the data retention period is rather short (e.g. one week). "

i was revisiting some documentation and here LSA++ (Layered Scalable Architecture) for SAP NetWeaver BW on SAP HANA page 40 and forward, it says some interesting things

1. Real time querying

2. Source for EDW

disregarding the virtual sources, such as smart data access, i do not read that this layer is optional. from what i can read, it is used to populate the EDW layer, and can also be queried upon.

there is nothing in that document that suggests the ODS is temporary.

thanks

glen