How to move more than 10000+ objects from BO 4.1 Dev to BO 4.1 QA

[Former Member]

Dear All,

Please help me to find out way to move content from BO 4.1 Dev to BO 4.1 QA .

Pramotion mamangent tool is not allowing us to do bluck export which are more than 2000 objects.

I have requiremnt to move more than 2000 objects, lets say 4000 in single shot.

As a alternate soltion i can create two promotion jobs to  achive this.

Please sujject us , how to move all the 4000 objects in a single pramotion job insted of creating multiple pramotions.

Thanks In Advnace.

[Former Member · Accepted solution]

Hi,

In order to import many objects you need to follow those steps :

LCM Command line interface (LCM_CLI.bat or lcm_cli.sh)

The command line option of the promotion management tool allows promoting
objects from one BI platform deployment to another. You can create a batch
script for multiple jobs.  Use the command line option for jobs that contain a
large number of objects.  The promotion management tool supports the following
job promotion types from the command line:

• Export an existing promotion job template to LCMBIAR with password
  encryption
• Export an existing promotion job template to LCMBIAR without password
  encryption
• Export single or multiple platform queries
• Promote multiple platform queries
• Promote with an existing job template
• Import and promote an existing LCMBIAR file
• Perform Live-to-Live promotion

Location: C:\Program Files (x86)\SAP Business
Objects\SAP BusinessObjects Enterprise XI
4.0\win64_x64\scripts\lcm_cli.bat
Command:
lcm_cli.bat
-lcmproperty=<location\>CLI.properties
Example:  
lcm_cli.bat
-lcmproperty=C:\CLI.properties

Refer to 1634962 for more details on LCM CLI usage. A sample
CLI.properties file is attached to this kbase.

Why LCM CLI ?

LCM CLI Utility is the solution for Bulk promotion scenarios. You can

identify the objects to be promoted via CMS queries and promote them over by

using the LCM CLI solution. As discussed earlier, dependency calculation and

security calculation take the maximum time and have an impact on overall job

performance, so the best way to improve performance is to design the overall

promotion process to take advantage of the inherited Folder rights in the target

system thereby avoiding promotion of security.  Similarly, dependencies could be

identified and promoted prior to promoting the actual content thereby avoiding

dependency calculation during the job runtime.

• Prevent timeouts: The web based Promotion Manage tool in
  the CMC is bound by a web 20 minute time-out. Promotion management does a lot of
  executions on the web application server. The executions include object
  dependency and security dependency calculations.  The effort of these two main
  tasks can grow  as more contents are added to the job during creation. When
  adding objects on the Web UI, from the time you selected the contents
  to add
  to seeing these content being added to the job, promotion management has a 20
  minute window to complete the tasks of object dependency and security
  calculations.
  If the amount to time taken is greater than 20 minutes, you
  will get a time-out message, and your job creation will fail.
• Bulk promotions: Use the Command Line Interface
  (CLI) to promote larger sets of content: This feature is often overlooked but
  can get around the web 20 minute time-out by running the job creation and
  promotion from a command prompt.
  When running from the command prompt
  (Windows or UNIX), there is no worry of running into the 20 minute time-out
  issue because you are not running inside a web browser and there is no web
  application server to interact with.
  The CLI is flexible but not very user
  friendly.
• Customize:You will be able to select the objects
  you want to include in a promotion management job in the form of a SQL statement
  (BI Infostore query)
• Control: you have more control to not include
  dependencies and securities in your job, and you can promote way more objects
  than you have ever done before.
• Dependencies: to shorten the amount of time to do
  promotions, you might not need to include dependencies because you are quite
  familiar with the objects you are selecting, and you can express subsequent
  queries to select those dependencies. However, If you are not comfortable
  turning dependency calculation off, you can still have them on and let Promotion
  Management do the dependencies calculation for you, and take the hit on
  execution time.
• Security: In most cases, Security model does not
  change across systems.  Also, security is usually on the folder and not on the
  reports, so it's best to first promote the folders with security (by doing this
  first, the number of content in a job should be smaller), and then promote the
  content in the folder without security. You'll see significant time saving.
• Multiple Queries: LCM CLI allows multiple queries
  to be issued as part of the same promotion job.

For more details on the LCM performance improvements & CLI, refer to the

BI4.1 SP2 performance improvements SCN wiki blog.

Best Practices:

Here are the best practices to ensure successful promotion of a large number

of objects using LCM CLI Tool -

1. For better performance and resource utilization, when doing bulk promotions,
   Ensure that only essential processes are running [ Eg: any
   other unused processes like Webi, CR, dashboards servers etc should be turned
   off ].  You can find an overview of promotion management workflows here.
2. TEMP Space: Ensure that the system running LCM_CLI and the
   Adaptive Job Server (AJS) have ample TEMP space (2-3 times the size of
   selection) to ensure objects are processed without failure.
3. JAVA Heap Size: Ensure that the LCM_CLI
   Script and the Adaptive Job Server
   (AJS) have the heap size configured appropriately when
   handling a large number of objects. Depending on the objects planned for your
   promotion, you may be required to increase the maximum heap size to 10 or 15 GB
   or more to ensure the job finishes and in time.
   - Heap size on LCM CLI tool
   can be configured by editing the script (bat or sh) and updating the parameter
   -Xmx10g.
   - The AJS Heap size can be
   configured on the CMC, AJS server properties and updating the command line
   argument -javaargs Xmx15g.
   Refer to the BI platform
   Administrators Guide for more details on preparing systems for bulk
   promotion.
4. Ensure that the TARGET server has ROLLBACK turned OFF. This
   is particularly important while IMPORTing or promoting a large data
   volume.
   If rollback is turned on, it will be making a copy of the content
   from the target system for backup purposes - thereby significantly affecting the
   performance of the job.
5. To improve the overall job performance, promote with security turned OFF.
6. for quicker execution, Ensure that you have turned off all the below
   options on the lcm_cli.properties
   file -
   exportDependencies=false
   includeSecurity=false
   stacktrace=false
   consolelog=true

Promotion Strategy

1. Promote Security with Users & user groups : First promote desired groups
   & users to the target BI4 system before any content is migrated.
2. Optionally Setup security at target group or user level so that security
   calculations may be avoided during the content promotion. This is because
   including security during a job runtime will bring back all users and groups
   associated with the folder or object thereby causing performance issues due to
   delay in these calculations.
3. Promote dependencies : Promote any pre-determined dependent objects such as
   universes & connections so that they are already present in target and may
   be avoided during the content promotion to improve performance associated with
   dependency calculations.
4. Promote folders and content: For bulk promotions, use the LCM CLI tool (with
   dependency calculation and security disabled for better performance) and promote
   folders and content by using designed cms queries to pull data from specific
   folders.
5. Multiple queries capability: Take advantage of the support for multiple
   queries within the same CLI.properties file.
6. Run LCM CLI on version BI4.1 SP02 or higher to take advantage of the
   performance improvements.   

Promoting FULL repository content using Promotion Management Tool
( LCM CLI 😞

The  BI
platform Administrators Guide describes the procedure  to promoting FULL CMS

repository (system database + filestore ) content from one system to another

using the promotion management tool (CLI).  This information is available since

BI 4.1 SP03 BI Platform Administrators guide and higher under the section titled

"Promoting full repository content using the promotion management
tool".

The procedure describes actions required for a successful promotion of

content from one BI4.x deployment to another using below steps.

1. Preparing source and target systems
2. Migration strategies and
3. Full system promotion steps including the exact Infostore
   queries required to accomplish these steps.
   
   
   1. promoting users and user groups
   2. promoting dependent objects
   3. promoting primary objects
   4. post-promotion

All this is available in the BI Platform Adminstrators guide. Here is a quick

link to the to BI Platform 4.1 SP04 Administrators guide for

your reference.

Samples:

• 
  Sample properties file while generating an lcmbiar file using a query.
  The same properties
  file can be used to promote (action=promote) to a live (Destination) system by
  proving the destination details (see attached properties file)
  Refer to the
  command-line tool parameters for a full list of
  parameters.
  
  CLI.properties
  
  action=export
  exportLocation=C:/export [ where "export" can be the
  Export location created for lcmbiar files. The
  files are created with a timestamp eg:
  Query_Administrator_Fri Mayxx xx-xx-xx.lcmbiar ]
  LCM_CMS=BIPW08R2:6400
  LCM_userName=Administrator
  LCM_password=Password1
  LCM_authentication=SecEnterprise
  
  Source_CMS=BIPW08R2:6400
  Source_userName=Administrator
  Source_password=Password1
  Source_authentication=SecEnterprise
  
  exportQueriesTotal=1
  exportQuery1=SELECT TOP 100000 static,
  relationships, SI_PARENT_FOLDER_CUID,
  SI_OWNER, SI_PATH FROM CI_INFOOBJECTS,CI_APPOBJECTS,CI_SYSTEMOBJECTS WHERE DESCENDANTS ("SI_NAME='Folder
  Hierarchy'",  "SI_ID in (18, 47, 48, 57 )")
  
  exportDependencies=false
  includeSecurity=false
  
  stacktrace=false
  consolelog=false
• Here are a few sample infostore queries: 
  
  TIP: For best results, test your queries for finite counts
  (eg TOP 10) or by choosing a small group or folder running them using the BI
  Platform query builder (AdminTool web application).
  
  NOTE: Query designing is outside the scope of SAP
  support.
  
  
  
  • To select all the content in the folder ABC and its subfolders:
    exportQuery1=SELECT TOP 10000 static,
    relationships, SI_PARENT_FOLDER_CUID,
    SI_OWNER, SI_PATH FROM CI_INFOOBJECTS,CI_APPOBJECTS,CI_SYSTEMOBJECTS WHERE (DESCENDENTS("si_name='Folder Hierarchy'","si_name='ABC' and si_kind='Folder'")) AND si_kind='Folder'
  • To bring back all folders and objects in a particular folder like the
    ‘Report Samples’ Folder and all its sub folders (all descendents)
    SELECT TOP 100000 static,
    relationships, SI_PARENT_FOLDER_CUID,
    SI_OWNER, SI_PATH FROM ci_infoobjects,ci_appobjects,ci_systemobjects where DESCENDENTS("si_name='Folder Hierarchy'","si_name='Report Samples' and si_kind='Folder'")
  • To retrieve only direct children of a particular folder like the 'Report
    Samples' Folder (this query will only bring the objects and folders which have
    this folder as their parent)
    SELECT TOP 100000 static, relationships,
    SI_PARENT_FOLDER_CUID, SI_OWNER, SI_PATH FROM
    ci_infoobjects,ci_appobjects,ci_systemobjects where CHILDREN("si_name='Folder Hierarchy'","si_name='Report Samples' and si_kind='Folder'")
  • To retrieve all the inboxes, favourites, personal categories, and custom
    roles:
    SELECT TOP 100000 static, relationships, SI_PARENT_FOLDER_CUID, SI_OWNER, SI_PATH FROM CI_INFOOBJECTS,CI_APPOBJECTS,CI_SYSTEMOBJECTS WHERE DESCENDANTS ("SI_NAME='Folder
    Hierarchy'",  "SI_ID in (18, 47, 48, 57 )")
  • To retrieve ALL users and sub-user groups that belong to a user
    group Eg: SAPtestGroup
    SELECT TOP 100000 static, relationships,
    SI_PARENT_FOLDER_CUID, SI_OWNER, SI_PATH FROM
    ci_infoobjects,ci_appobjects,ci_systemobjects WHERE 
    
    DESCENDENTS("SI_NAME='UserGroup-User'","SI_NAME='SAPTestGroup' AND
    SI_KIND='UserGroup'")
  • In above query, if you would like to include the parent group (SAPtestGroup) along with the all the descendent
    usergroups and users -
    SELECT TOP
    100000 static, relationships, SI_PARENT_FOLDER_CUID, SI_OWNER, SI_PATH FROM
    ci_infoobjects,ci_appobjects,ci_systemobjects WHERE SI_NAME='SAPTestGroup' or
    DESCENDENTS("SI_NAME='UserGroup-User'","SI_NAME='SAPTestGroup'")
  • If you only want to retrieve groups and descendent user groups (No users),
    then you may change the query slightly as below -
    SELECT TOP 100000
    static, relationships, SI_PARENT_FOLDER_CUID, SI_OWNER, SI_PATH FROM
    ci_infoobjects,ci_appobjects,ci_systemobjects WHERE SI_KIND='UserGroup' AND
    (SI_NAME='SAPTestGroup' or
    DESCENDENTS("SI_NAME='UserGroup-User'","SI_NAME='SAPTestGroup' AND
    SI_KIND='UserGroup'"))
    
    
  • NOTE: The BI platform Administrators Guide has several queries under the
    promotion management chapter.
    For more information on the infostore queries, refer to the BusinesssObjectsInfostore Queries series blog on the SCN

Regards,

Saloua

[Former Member · Accepted solution]

Hello, Saloua,

what will happen, if:

1. I migrate full content of the BOBI 4.1 by LCM CLI from CMO to FMO and then I migrate it again after few days without previous ones deleting?
2. I migrate full content of the BOBI 4.1 by LCM CLI from CMO to FMO and then I migrate it again after few days with previous ones deleting, but meanwhile some of deleted users created some new reports, jobs, etc.?

By another words: Will new import replace old data by fresh data from the same source? (And data stays consistent?) I can't find relevant SAP note for this.

Thank you.

[denis_konovalov]

Please search before posting , here is KBA that answers your question : https://service.sap.com/sap/support/notes/1969259