I really appreciate the quality of akvk's recent article "Scheduling BODS Jobs Sequentially and Conditionally". And the technical accuracy is high -- yes, you can accomplish what you are trying to do with the techniques discussed in the article. Love the visuals, too.
However.
I cannot really recommend this kind of solution. Data Services is not an enterprise scheduling or orchestration tool. This approach suffers a bit from Maslow's law of the instrument: "if the only tool you have is a hammer...treat everything as if it were a nail." Yes, I love Data Services and Data Services is capable of doing all of these things. Is it the best tool for this job?
Not exactly. And this question is answered in the first paragraph that mentions chaining workflows. Data Services already gives you the capability to encapsulate, chain together, and provide conditional execution of workflows. If jobs only contain one dataflow each, why are you calling them jobs and why do you want to execute these jobs together as a unit? Data Services is a programming language like other programming languages, and some discretion needs to be taken for encapsulation and reusability.
I do really like the use of web services for batch job launching. It is a fantastic feature that is underutilized by DS customers. Instead, I see so many folks struggling to maintain tens and sometimes hundreds of batch scripts. This is great for providing plenty of billable work for the administration team, but it isn't very good for simplifying the DS landscape. The web services approach here will work and seems elegant, but the section about "sequencing using web services" does not sequence the jobs at all. It just sequences the launching. Batch jobs launched as web services are asynchronous... you call the SOAP function to launch the job, and the web service provider replies back with whether the job was launched successfully. This does not provide any indication of whether the job has completed yet. You must keep a copy of the job's runID (provided to you as a reply when you launch the job successfully) and use the runID to check back with the DS web service function Get_BatchJob_Status (see section 3.3.3.3 in the DS 4.1 Integrator's Guide). [Note: scheduling and orchestration tools are great for programming this kind of logic.]
Notice how it would be very hard to get true dependent web services scheduling in DS since you would have to implement this kind of design inside of a batch job:
This convoluted design is functionally IDENTICAL to the following and does not rely on web services:
I'm also hesitant to recommend a highly customized DS job launching solution because of supportability. When you encapsulate your ETL job launching and orchestration in an ETL job, it's not very supportable by the consultants and administrators who will inherit this highly custom solution. This is why you invest in a tool like Tidal, Control-M, Maestro, Tivoli, Redwood, etc., so that the scheduling tool encapsulates your scheduling and monitoring and notification logic. Put the job execution logic into your batch jobs, and keep the two domains separate (and separately documentable). If you come to me with a scheduling/launching problem with your DS-based highly customized job launching solution, I'm going to tell you to reproduce the problem without the customized job launching solution. If you can't reproduce the problem in a normal fashion with out-of-the-box DS scheduling and launching, you own responsibility for investigating the problem yourself. And this increases the cost to you of owning and operating DS.
If you really want to get fancy with conditional execution of workflows inside of a job, that is pretty easy to do.
Let me know if this makes sense. If you see weird errors, search the KBase or file a SAP Support Message to component EIM-DS.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
38 | |
19 | |
13 | |
13 | |
10 | |
10 | |
10 | |
10 | |
8 | |
8 |