048 sap bw datastore

7/27/2019 048 SAP BW DataStore

http://slidepdf.com/reader/full/048-sap-bw-datastore 1/17

© 2011 SAP AG

Best-Practice Document

DataStore Objects Upload withSAP NetWeaver BW 7.x

Dietmar-Hopp-Allee 16D-69190 Walldorf

DATE

June.2011

SOLUTION MANAGEMENT PHASE SAP SOLUTION

Operations Implementation SAP NetWeaver BW

TOPIC AREA SOLUTION MANAGER AREA

Application Management System Administration




DataStore Objects Upload with SAP NetWeaver BW 7.x

© 2011 SAP AG page 2/17

Table of Contents

1 Management Summary 3

1.1 Goal of Using This Service 3

1.2 Staff and Skills Requirements 3

1.3 System Requirements 3

1.4 Duration and Timing 3

2 Best-Practice Document 4

2.1 DataStore Objects: Overview 4

2.1.1 Standard DSO 4

2.1.2 Write-Optimized DSO 4

2.1.3 Direct Update DSO 5

2.2 Data Upload into a DataStore Object 5

2.2.1 Serial and Parallel Upload in Standard DSO 5 2.3 DataStore Object Activation 7

2.4 DataStore Object: Customizing and Performance 9

2.4.1 Properties (Settings) of a Standard DataStore Object 9

2.4.2 Properties (Settings) of a Write-Optimized DataStore Object 11

2.4.3 DataStore Object Customizing 12

2.4.4 Performance 14

2.5 Further Information and References 16





© 2011 SAP AG page 3/17

1 Management Summary

DataStore objects are an essential part in the layered scalable architecture. Usually they are used in the BWinbound layer to store the data that was provided by the source systems without any transformations, or for first simple transformations. Good DataStore object design and Customizing is necessary to have satisfyingtimes for the data loads.

1.1 Goal of Using This Service

This service provides recommendations for the upload of data into DataStore objects (DSO) and the dataactivation within a DSO. It explains the DSO structure, as well as settings that help to improve theperformance of DSO data loading and activation. You also might use this service when you want to reviewyour scenario regarding performance aspects.

1.2 Staff and Skills Requirements

For optimal benefit you should make sure that you have experience in the following areas:• Practical experience with the Data Warehousing Workbench• Knowledge of upload scenarios within an SAP NetWeaver BW system• The differences between an InfoCube and a DataStore object• The use of InfoPackages, InfoSources, process chains, and other typical BW functionality

1.3 System Requirements

This document refers only to SAP NetWeaver BW 7.0x and SAP NetWeaver BW 7.3x. There is another best-practice document for SAP NetWeaver BW 3.x.

1.4 Duration and Timing

This service not only provides a simple step-by-step description, but also contains recommendations for parameter settings and for DataStore object issues. The time required to implement this best-practicedocument depends on the objects being used and the data volume being loaded.

We recommend that you use this best-practice document in two steps:- First, read the entire document to get an overview of the functions related to data upload.- Apply the knowledge obtained from it to your individual objects.

We strongly advise against direct changes in the production environment, and recommend first checking anychanges in a test system.





© 2011 SAP AG page 4/17

2 Best-Practice Document

2.1 DataStore Objects: OverviewThis section gives a brief overview of the functionality and design of DataStore objects. This description isrelatively short, as it was designed to give recommendations regarding the use of DataStore objects, andshould not be seen as documentation or a documentation enhancement. For detailed information, pleaserefer to the SAP NetWeaver BW online documentation:http://help.sap.com/saphelp_nw70/helpdata/en/f9/45503c242b4a67e10000000a114084/frameset.htm

There are different types of DataStore objects (DSO). They were designed for different use cases and differ in functionality and technical structure. The DSO types are:

1. Standard2. Write-Optimized3. Direct Update

2.1.1 Standard DSOThe standard DSO is an InfoProvider that consists, in general, of three flat tables. This means that it differsfrom an InfoCube, which is based on the star schema. Though you can load requests into a DSO, the maintable, which contains the data for reporting, does not contain a request reference. This design makes itpossible to overwrite single records in a DSO, which is not possible for an InfoCube. The content of a DSOdepends on the order of loaded requests and the order of the records within one request.

Because of this architecture, the data load into a DataStore object consists of two steps:

1. Data upload

2. Data activation

In the first step, the request data is written into the transparent table (activation queue) of the DSO object. Inthe activation queue of the DSO the data is not yet available for reporting. It needs to be activated. The DSOdata activation is the second step and makes the uploaded data available for reporting. Technically, thismeans that the data is moved into another DSO table (Active table or A-table) that contains all data availablefor reporting. This table contains as key the characteristics of the DSO (in the figure above, Document No.

was chosen as the only DSO characteristic).Within the activation process, the Change Log table is written as well. This table contains information aboutthe changes made during the DSO data activation regarding the Active table. The Change Log table has—along with the the activation queue—a technical key consisting of request, package, and record number.The change log is needed for delta loads from the DataStore object to other InfoProviders, and also if arequest will be deleted from the DataStore object. It is possible to delete the change log entries to save diskspace, but be careful, as delta and deletion of a request that no longer exists in the change log is not possibleany more.

2.1.2 Write-Optimized DSOThe write-optimized DSO is an InfoProvider that consists only of a single table with active data. In contrary tothe standard DSO, no activation of data is necessary. The write-optimized DSO is reduced to minimal

http://help.sap.com/saphelp_nw70/helpdata/en/f9/45503c242b4a67e10000000a114084/frameset.htm

http://help.sap.com/saphelp_nw70/helpdata/en/f9/45503c242b4a67e10000000a114084/frameset.htm





© 2011 SAP AG page 5/17

functionality to get the best possible performance. The active table of the DSO is technically similar to aPersistent Staging Area (PSA) table. It contains as technical key the fields request ID, data package, andrecord number. Additionally, a secondary index is created during DSO creation; it contains the semantic keyfields of the DataStore object.

The write-optimized DSO can be used in the Enterprise Data Warehouse (EDW) layer scenarios to store datain BW 1:1 like it was provided by the source system. The data is not aggregated and the history is retained.The accumulation of data can be done in subsequent InfoProviders.The write-optimized DSO is not intended for reporting. Delta handling is only possible on request level, not onrecord level like during activation in the standard DSO.

2.1.3 Direct Update DSO A direct update DSO consists only of the A-table (Active table), which contains the activated DSO data. The Active table is similar to the Active table of the standard DSO. It contains as technical key on data base thefields which were defined in the DSO definition as key fields. There is no data activation step and, therefore,

the data is immediately available for reporting.The DataStore object gets data from an external system via API. The Direct Update DSO is used for specialapplications, for example, in SAP Strategic Enterprise Management (SAP SEM). Please note that it is notpossible to build regular BEx queries on a direct update DSO, as surrogate IDs (SIDs) are not generated. If you want to report on a direct update DSO, you can use InfoSet Queries.

2.2 Data Upload into a DataStore Object

2.2.1 Serial and Parallel Upload in Standard DSO

The data upload into the activation queue of a standard DSO is handled as a regular transactional upload.You can update several requests in parallel without having a conflict. This is possible because the activationqueue, which is technically a transparent table, contains the request ID, the package ID, and the recordnumber as key fields.

Ordering of RequestsThough requests can be loaded in parallel into a DataStore object, the request order can be crucial for theDataStore data should the requests not be independent of each other. This is due to the overwritefunctionality that is used for DSOs. If a DSO is updated from various data sources that depend on each other,it has to be guaranteed by the data-loading design that the data is overwritten in the correct sequence.

Requests are considered dependent on each other if ALL of the following conditions are fulfilled:

Prerequisite: No DataSource runs in parallel with itself.1. The DataStore object is filled by more than one DataSource.2. There is at least one key figure K that is updated by at least two DataSources D1 and D2.3. At least DataSource D1 or D2 updates key figure K by update method Overwrite.





© 2011 SAP AG page 6/17

4. There is at least one record in the active table that is updated by DataSource D1 and at least onerecord updated by D2 (with regard to the key fields of the table).

The above information is very important, as otherwise it is possible that the wrong data may be entered into aDataStore object. These four conditions provide the theoretical definition of dependent requests.

Scheduling Independent Requests

The schedule order is not crucial for independent requests. The easiest way to schedule data in parallel is byusing a process chain.

Two data transfer processes (DTP) are scheduled in parallel, as shown in the figure above. This means thatthe order of the requests is not fixed; therefore, the above scenario can only be used for independentrequests.If you are still using the SAP NetWeaver BW 3.x data flow with transfer rules and update rules, you have touse InfoPackages instead of DTP.

Scheduling Dependent Requests

Dependent requests must be scheduled in the correct order. The purpose of this schedule order is that therequest with the data that should be activated first gets the lowest request ID, and so on. If you use a processchain for dependent requests, you have to execute the DTP/InfoPackages in sequence.





© 2011 SAP AG page 7/17

As an alternative to the serial execution, you can define—via an ABAP program—a waiting process toguarantee that the loading processes are started in the correct sequence.

In the example above, the sequence is guaranteed by the DSO activation because the second data load is

started 10 seconds later and has a higher request ID. Both data loads can load the activation queue inparallel, so data loading time is saved.Precondition for this alternative above: There is no shortage of background processes that result in waittimes. If the first DTP -process cannot be started because of a shortage of background processes, the job istransferred to a waiting job pool. In this pool the sequence is no longer guaranteed and it could happen thatthe second DTP is executed before the first. This risk has to be considered.

2.3 DataStore Object Activation

DataStore object data can be activated manually, automatically without a process chain, or within a processchain with the process type DataStore Data Activation. Additionally, you can choose if the data activation is to

be executed serially or in parallel.The DataStore object data activation can be restarted. Should errors occur, restart the data activationprocess records that are not yet activated.

Manual Data Activation

Within DataStore object administration, choose Activate for the popup for manual data activation.

You can select one or several requests for activation; however, the right order must be kept.





© 2011 SAP AG page 8/17

With the Do not condense requests into one request when activation takes place flag, you can control if several requests are to be activated within one change-log request or if one individual change-log request iscreated for each request. Requests activated together can only be deleted together and are updated together in subsequent data targets. We generally recommend that this flag be activated. Choose the Start button to

start the DSO activation.

The buttons Activate in Parallel and Get SIDs in parallel will be explained in the Customizing andPerformance part of this document. All other buttons, besides Process Chains, provide more monitoringinformation, the actual status of the activation job (Refresh button), application log for activation (Log button),and background job logs (Jobs button).

Automatic Data Activation Without Process Chain (Outdated SAP NetWeaver BW 2.x and 3.x Logic)

In earlier SAP NetWeaver BW releases, when no process chains existed, the only way to activate DataStoredata automatically was via the Activate Data Automatically flag in the DataStore object definition.





© 2011 SAP AG page 9/17

When a request is loaded to a DataStore object via an InfoPackage, and the InfoPackage is not part of aprocess chain, the Activate data flag is taken into account and the DataStore data is activated automatically.The activation flag is ignored for DTP requests and data loads started via process chains.

SAP recommends not using this outdated automatic activation procedure for InfoPackages any more, andswitching to BW process chains for existing scenarios.

Data Activation with Process Chain

The recommended way to activate DSO data is via BW process chains. The process chain variant of theDataStore object activation defines the way in which data activation takes place.

If only the DataStore object is referenced in this variant (and not an InfoPackage or DTP), all not-yet-activatedrequests are activated by this process chain object. This means that all the not-yet-activated requests musthave a green status in the monitor.If an InfoPackage or DTP is referenced, the system activates only the request that was loaded by thisInfoPackage/DTP. Please note that another not-activated request with a smaller request ID, loaded from

another source, will result in the activation failing.

The Do not condense requests… flag has the same effect as that described above in the Manual Data Activation section. The Parallel processing button will be explained in the Customizing and Performance partof this document.

2.4 DataStore Object: Customizing and Performance

The performance of DataStore object data activation can be influenced by the settings of a DataStore objectand by the DataStore Customizing for the activation.

2.4.1 Properties (Settings) of a Standard DataStore Object

When creating or changing a standard DataStore object, you can specify the settings.





© 2011 SAP AG page 10/17

SID Generation:If the DataStore object is not used for reporting, but only as a data container, you can choose the Never Create SIDs option for SID generation (in earlier SAP NetWeaver BW releases, this option was called BEx Reporting ).

The advantage is that no SIDs are generated, which leads to better activation performance. If your DataStoreobject reporting requirements are very simple, that is, you are only selecting a limited number of records, youcan use InfoSets for reporting on DataStore objects even if the SID-creation option is switched off. Pleasenote that the loaded values are not checked for correctness if SIDs are not generated.

As an alternative to InfoSets for sporadic reporting with few records, you can use the SID-creation optionDuring Reporting . In this case, the SIDs are created when the query is executed. This, of course, has anegative performance impact on the query execution. Therefore this option should only be chosen in case of rare or exceptional reporting on the DataStore object, for example, for sporadic data quality checks.It is possible to change the Create SIDs reporting flag even if the DataStore object already contains data. Inthis case, the system generates the SIDs. However, if you want to use the DataStore object for reporting,choose the During activation option.

Unique Data RecordsIf each key field combination is only loaded once, you can set the Unique Data Records flag, which leads to aperformance enhancement during activation. The system then does not need to check the existence of arecord in the active data table and a mass insert into the active table can be executed directly (otherwise, thesystem first checks whether the datasets are already available in the active data table, and then the dataset

is marked either for update or insert). Additionally, the data is not sorted.





© 2011 SAP AG page 11/17

Note: If the Unique Data Records flag is activated, the SAP NetWeaver BW system does not guaranteeunique records by deleting duplicates with regard to the DataStore key fields. The DataSource has to makesure that unique records are always delivered. If this is not the case, the DSO data activation will cancel.

In-MemoryThis option will be available with SAP NetWeaver BW 7.30 on in-memory databases and relates to theactivation performance. The activation and SID -generation will be done directly in the in-memory DB, whichshows better performance than the “normal” activation.

2.4.2 Properties (Settings) of a Write-Optimized DataStore Object

When creating or changing a standard DataStore object, you can specify the settings.

SID Generation

Write-optimized DataStore objects were not designed for reporting, but as an inbound layer in stagingprocesses and for fast data loading. The standard setting for write-optimized DSO is Never Create SIDs.However, it is possible to choose the option During Reporting for SID generation. In this case, the SIDs will

be created during execution of the query. But as this SID creation takes time, the query performance will notbe good. Therefore, reporting scenarios on write-optimized DSO should be avoided.

Allow Duplicate Data RecordsIf you do not set this indicator, the uniqueness of the data is checked and the system generates a secondaryindex according to the semantic key of the InfoObject. This index has the technical name KEY.If this indicator is set, the active table of the DataStore object can contain more than one record with the same key.

Regarding performance, this flag has pros and cons. The secondary index can be used to read data from thewrite-optimized DSO and will speed up data loads that use fields from the semantic key as selection criteria.However, this index has to be adapted during the data load into the write-optimized DSO, which takes time.





© 2011 SAP AG page 12/17

Check Delta HandlingWrite-optimized DSOs are often used in an enterprise data warehouse layer. Data that is loaded into theDataStore object and then fetched from the layer should be deleted after a reasonable amount of time.In other scenarios, the DataStore object is used as part of the consistency layer. In this case, data may not bedeleted if it was already recorded in a delta, or the data in the source and target could become inconsistent.Depending on the scenario, the delta consistency check can be activated and deactivated. Independent from theflag, there should be a data volume management strategy in place for the write-optimized DataStore object to limitgrowth and size. This would be also beneficial for performance when reading from this DSO.

2.4.3 DataStore Object Customizing

The functionality described here is mainly relevant for standard DataStore objects. In the DataStore objectCustomizing, you can define for the activation process whether background or dialog work processes will beused, the number of processes for activation, the number of records to be activated by one process, and themaximum waiting time of the background job that extracts the data from the activation queue. These settings

can be made via transaction code RSODSO_SETTINGS.

In Customizing, you can set the activation parameters globally (Cross-DataStore option) or DataStore-specifically. When choosing the Cross-DataStore option, you can define the default value that will be valid for all DataStore objects. It is also possible to define here the default value for a specific DataStore object. Thissetting will overrule the Cross-DataStore default value. The DataStore Default can also be overruled in theDataStore object activation variant of the process chain, or in the screen for manual activation.

In the Customizing of the sefault or specific DataStore you can define the parameters for DataStore activationand SID generation separately. Additionally, you can define the wait time for the deletion process (rollback).





© 2011 SAP AG page 13/17

Maximum Package SizeIt is not possible to give a general recommendation om how to configure this parameter. If you have enoughhardware capacity and memory, you can increase the number of records for each process. But you shouldkeep in mind that this increased memory per package during activation can influence other jobs running atthe same time on your machine.

Maximum Wait Time for ProcessThe wait time specifies how long the batch job, which extracts data from the activation queue or controls theSID generation, waits for an answer from each process that was split up to perform activation or SIDgeneration. This guarantees that the extraction job does not run forever if one of the activation processesfails. On the other hand, it is possible to control the maximum runtime of a process. A wait time that is tooshort results in a DSO data activation failure even though the processes ran successfully. Setting a wait timethat is too long means that the background process waits too long before an error message is sent and timeis lost for reacting to the problem.Recommendation:Because of other parallel jobs that run on the system, it is possible that the runtime of such processes maydiffer. Therefore, we recommend setting the wait time higher than the observed runtime of one process,especially if there is a high load on the server during the DSO data activation.Example: The maximum runtime of an observed DSO data activation process is about 10 minutes. Werecommend setting the waiting time to not less than 30 minutes.

Change Process Parameters (No. of Parallel Processes)By choosing the Change process parameters icon, you can set the number of parallel processes for DSO

activation and SID generation. You can also define whether the activation will be done in the background or dialog work processes.

The number of parallel processes for activation you can maintain from different transactions in the SAPNetWeaver BW system:

1. Default setting:Transaction code RSODSO_SETTINGS Cross DataStore

Alternatively: Transaction code RSBATCH Process types ODSACTIVAT/ODSSID2. DataStore-specific:

Transaction code RSODSO_SETTINGS DataStore specific Alternatively: During manual activation of a request in DSO, choose the Activate in parallel button.

3. Process-chain-variant specific:In a process chain variant for DataStore object activation, choose the Parallel processing button.

When performing the activation, the most granular Customizing setting is taken. This means when there is,

for example, a DSO activation step in a process chain, the system first checks for the settings in the processchain variant. If there is an entry, it will take these settings for the activation. If there is no entry, it checks for the DataStore-specific entry. If there is none, it will take the system default for activation.





© 2011 SAP AG page 14/17

Background or Dialog Work ProcessesIn earlier SAP NetWeaver BW releases, the only option for DSO activation was dialog work processes. For higher runtime security, SAP now recommends using background work processes. This has the advantage,for example, that the activation does not terminate if there is a lack of free work processes or if the dialogwork process runtime parameter (general ABAP parameter) is exceeded.Load balancing can be done for background and dialog work processes. The load balancing for background

jobs can be done via background server groups, which can be maintained in transaction code sm61. Theload balancing for dialog work processes can be done via server groups, which can be maintained viatransaction code rz12.

2.4.4 Performance

The performance of DSO data activation and uploads to and from a DataStore object can be influenced byvarious parameters and settings. The following list gives an overview of points that should be checked for

good performance.

Do not use SID generation if the standard DataStore object is not needed for reporting.If no queries are defined on the DataStore object and the object is only used as a data container, choose theoption Never Create SIDs for SID generation (see Properties [Settings] of a Standard DataStore Object). Theadvantage is that no SIDs are generated, which leads to better activation performance.

Avoid combining the activation of various DataStore objects in one process chain activation step.Often the activation of various DataStore objects is defined in one process chain step, as shown below.

This definition has the disadvantage that the activation of the DataStore objects is done serially. They areactivated one after the other, each with the number of parallel processes set in Customizing.

If your system has enough hardware resources and enough free work processes during the execution of theprocess chain, it is better to define a separate activation step for each DataStore object.

How many processes you can define in parallel must be checked with your system administrator, who knowswhat other processes are running at the same time in the system and how busy the system is.





© 2011 SAP AG page 15/17

Optimize the number of parallel processes for activation.For DataStore objects where a large number of records have to be loaded and activated, consider increasingthe number of parallel processes specific to the or in the process chain variant (see DataStore ObjectCustomizing).Define secondary indexes, if necessary, for better read performance from DSO.It is possible to define secondary indexes within the maintenance of the DataStore object.

The secondary indexes are created on the Active table of the DataStore object. In general, secondaryindexes can be used to increase the performance of specific queries. By default, only the primary key isavailable as an index that contains the key fields of the DataStore object definition. As the DSO structureconsists of a regular transparent table, you can use the SQL Trace to analyze queries on DSO objects in thesame way as for regular SAP tables.

As of SAP NetWeaver BW 3.x, these secondary indexes are transported automatically with the DSO object.Please note that you have to verify that such an index makes sense in your production environment, for example, because of different user behavior or volume of data.

Additionally, secondary indexes have a negative impact on the data activation performance because thedatabase also has to maintain these indexes during this operation. Compared with InfoCubes, there is no

functionality available that allows dropping and recreating a secondary index before/after the data activation.Therefore, you should be sure that you really need a specific secondary index. As a workaround, you canwrite a simple report that drops and creates the indexes on database level. Using a process chain, you cansimply insert the self-defined drop index report as ABAP-step before the data activation. After the successfuldata activation, you can use in the process chain the self-defined create index report.

Buffer InfoObject number ranges.

If you have data loads to standard DataStore objects with SID generation where many new records have tobe created (for example, during initial load), time in the activation is spent to determine new SIDs fromdatabase table NRIV. For better loading performance, it is recommended to buffer the number ranges of thebig master data InfoObjects before executing the data load.How to perform number range buffering and how to identify the relevant InfoObject number range object BIM*is described in detail in SAP Note 857998.Note: Never buffer the InfoObject 0REQUID!

Consider removing huge master data tables from single record buffer.

This recommendation refers to the data activation in a standard DataStore object with SID generation. In rarecases, up to 80% of processing time can be spent on buffer synchronization during the SID generation. Thisproblem occurs if the single record buffer is configured too small, that is, some master data tables are too bigfor the single record buffer.Master data tables are delivered single-record buffered by default. Huge objects (for example, documentnumbers) might cause performance problems during data activation if the single record buffer is too small. If you notice buffer swaps in transaction st02 or st10, consider increasing the single record buffer. Or,alternatively, remove some huge InfoObjects from the single record buffer. You can find further information

about how to do this and how to analyze in SAP Note 803958.





© 2011 SAP AG page 16/17

Limit the number of requests. A high number of requests in a DataStore object can lead to an administration overhead, as all requestinterdependencies are checked for each load. Avoiding a high data load frequency is recommended.In general, we recommend applying SAP Note 620361 if more than ~10000 requests are contained in aDataStore object.

Perform parallel extraction from DSO changelog.The standard behavior is that there is only one extraction job reading data from the DSO change log. With theRSADMIN parameter and coding correction of SAP Note 1570910, it is possible to extract in parallel from thechange log.

2.5 Further Information and ReferencesFurther information can be found in:

- SAP NetWeaver BW 7.x online documentation- Documentation in the SAP Developer Network (http://sdn.sap.com -> Data Warehousing) or

alternatively via SAP Service Marketplace (http://service.sap.com), alias BI.

http://sdn.sap.com/

http://sdn.sap.com/

http://sdn.sap.com/

http://sdn.sap.com/





© 2011 SAP AG. All rights reserved.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and

services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, andother Business Objects products and services mentioned herein as well as their respective logos are trademarks or registeredtrademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein aswell as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this documentserves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies (“S APGroup”) for informational purposes only, without representation or warranty of any kind, and SAP Group s hall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in theexpress warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting anadditional warranty.

048 sap bw datastore

Documents