(OO) Support tip: Multi Instance step execution time increased
When executing parallel lanes ( multi-instance or parallel step), a delay has been observed for starting the lanes in batches.
Take for instance a Multi Instance step of 200 iterations ( 200 lanes to execute) with a throtte of 25 (setup from Studio authoring on the step).
The lanes will be executed in batches of 25 concurrently with this setting. But in time, for starting next batch of lanes, a delay is observed (next lanes will wait more and more senconds to start the next batch).
This is a known effect of the usage of multiple lanes step in OO 10.x.
The delay that appears between the end of one chunk to the other comes from the fact that at the end of each chunk of 25 executed lanes, there is an "invisible" split-Join step that gets executed on one of the RASes .
This cumulates in time, as more and larger contexts need to be re-united. This data is written to DB and at the end, the contexts of all lanes are re-united to one big context and written to DB in different tables.
That is why large iteration parallel execution might take more time to execute.
We recommend spliting the data to be processed in smaller batches and distribute them to more Multi-Instance Steps, to parallelize at Worker level.
Also, this large context data is loaded also to the memory of the executing RAS, so you might also have Out-of-Memory problem if contexts get really big. This is another reason why data should be processed in smaller chunks.