Im trying to create a "Master" job that controls a batch process, in my case it would be a very big plus to be able to start a non specific job using a component like tRunJob.. Standard its possible to select a job that needs to be started (hard coded) in the tRunJob component but what i want to do is to decide at Run time by passing this component a parameter (eg the name of the job)
if this is not possible i was wondering if it could be done by creating a new routine that can be used (called) with a tJava component, whats your opinion about this? and have u got some suggestion? code examples? all help is welcome!
Last edited by Albert (2008-01-03 02:36:34)
Standard its possible to select a job that needs to be started (hard coded) in the tRunJob component but what i want to do is to decide at Run time by passing this component a parameter (eg the name of the job)
Yes, you can pass a parameter Dynamically to the child job via tRunjob component.
I have created a simple scenario:
child job: define a context:name, there is only a tJava component.
father job: use tRunjob component to run child job and pass a parameter danamically to child job.
Please see the screenshot.
Feel free to post your questions!
Thanks for the quick reply and good example! Actually i meant something different, i was wondering if i could tell the tRunJob component
what job to run by passing it a parameter (like the job name) at run time, but standard the tRunJob component can run only one specific job (determined when u choose from the drop down list inside the components properties) not at run time and not dynamically (the job to run is chosen during the design and hard coded) so for example lets say that i have the following jobs:
-MainJob -> decides what job to start.
-Job1 -> small job that does something simple
-Job60 -> ---
It would be possible to make a MainJob with 60 tRunJob components that all start a different Jobx jobs but what i want is some
tRunJob component that i can tell (by passing some parameter) what job to run (at runtime). So a tRunJob component that can start a job just by telling the tRunJob component what job to run.
for example lets say that u have a loop and that u want to start job1 ... job60 with 1 tRunJob component just by telling tRunJob what job to run so something like this:
tRunJob(start job1) ..... tRunJob(start job60)
It seems to me that this is not possible with the current tRunJob component so i was wondering if it is possible to create some routine that can do the same as tRunJob but do this for any job at runtime.. something like this maybe:
StartJob(String jobName, String jobParameters) this can than be started from a tJava component?
So the Mainjob can use the tJava component to call some routine with the appropriate parameters (jobname and job parameters) and start the job with these parameters.
so basically the same functionality as the tRunJob component but with the option to chose a job at runtime, whats your opinion about this? is this possible?
Thanks for the help shong!
Your issue is intereting for me!! I will discuss it with other members in Talend or report a new feature on our bugtracker.
Thanks for your support!
Thanks for your help and feedback,
simone's description (in the bugtracker) is very recognizable
btw, the export part is a problem right? isn't it possible to just show a popup menu that allows the manual selection of jobs to include
in the export if a job is exported that contains a tDynamicRunJob (doesn't exist yet) component??
then it will be up to the user to decide/be responsible for the jobs that can be fired by tDynamicRunJob.
Last edited by Albert (2008-01-04 11:30:44)
I think we will work again on this feature, but in this case it will be planned for the 2.4 (too late for a 2.3 now).
As you've seen in the feature, the 2 main problems for this are on the export and when using in TOS.
For the export, as you said with a dialog with like a tree to select the jobs to include, it should be possible (or add like a dependency from one job to another one also)
After there is still the problem of execution in TOS as it will be really slow. If you execute the main job, it will look for the first job to execute, generate the code of the first job, then execute, then generate the code of the 2nd job, then execute etc...
(but if the user knows it's recommanded for "test only", it's ok maybe).
Could somebody confirm if v2.4.0M1 covers this feature request?
I confirm this feature is not present in 2.4.0M1. As you can currently see in the bugtracker, [Feature] 1859 is not fixed yet. It has even no "target version", ie it's not in the roadmap yet.
I have the same problem as Albert for Dynamic tRunJob. I have tRunjob master and may others tRunjob slave using the same parameter as the master one with promt. Can you tell me if the problem is resolved for Tos 2.2, I have the 2.2 version
Thanks in advance for your help.
Somewhere in the bug tracker for this, I remember seeing something about using a tSystem to do this?
Is that a possibility? I'm using Java and Unix once the job is deployed.
If we've exported the job scripts - couldn't we just shell out and run that job script?
I agree that a dynamic run ability would make things easier.
I'd love to know how simone ended up with 400 jobs.
Managing that has got to be a hard job.
Volker Brehm said:
I don't know anything about the actual state concerning this feature. But I think it would be easy to implement.
There is only one downside: Export of the job will be not able to export the called job. So you must export every possible subjob and ensure that it will be in the classpath. I do this too (not calling from a TOS job). But creating a custom component wouldn't be much challenging.
Additional I don't think that this will be much slower than the actual way the job is called.
Virtual Corner said:
To the experts - Shong, Plegall, et al.
I have a similar need to what Albert specified in this posting that originated in 2007. I do not see a way to dynamically specify the name of the job to run. The accompanying bug is in an open status. Could you clarify if there is another way of accomplishing the result?
Could you clarify if there is another way of accomplishing the result?
Yes, tSystem component can do a trick at the moment, export all the job script of jobs and pass dynamically the job name to tSystem.
Virtual Corner said:
I'll try it out. I need the the jobs (dynamically specified) to all be within the same transaction scope as the parent job. I take it this is not possible if the jobs are started via tSystem components.
Not sure if this is the exact issue, but here's what I have done.
I created a master job called job_dispatcher. It it, it calls a sub job called job_directory. In job_dispatcher, I choose multi thread option. and pass in a context called input_job_name (which is the job passed from the outside)/
In job_directory, I bring in all jobs that I would want to call at run time. In each job will linked from a tJava (so one tJava per job) with the following code:
boolean match = context.input_job_name.equals("job_orphan1");
where context.input_job_name is passed from job_dispatcher and "job_orphan1" is the name of the name of the job. so in another tJava, you will have "job_orphan2" or something like that.
Then connect tjava to the job with a Run If condition: match
To test, Export job_dispatcher.
Run job_dispatcher and pass in job_orphan2 as the name of the job you want to run.
It should execute that job only all others will failed at the condition. The multi thread allows to fire the test of the condition all at once.
has there been any advancement on this feature.
i have to load a KPI database with hundreds of KPIs as each KPI has its own job
i would like to use this design of Master-Children.
could someone post a description on how to use tSystem to achieve this?
I'm very interrested in this feature, the status of the case http://talendforge.org/bugs/view.php?id=1859 seems to be still open in the bugtracker.
Is there any news about this case ?
Did anybody already try to design a tRunDynamicJob based on the regular tRunJob ? Wich can handle a variable as __PROCESS__ ?