Prevent Azure data factory from running multiple instances of a single pipeline at same time

Today we will learn on how to prevent Azure data factory from starting another instance of the pipeline when previous one is still running
Scenario: You have a pipeline that has been set to run every 5 minutes. suddenly, you notice that the pipeline has started to take more than 5 minutes and the next instance of the pipeline exucutes after 5 minutes. this is giving you problems. Follow the steps below:

Steps:
  • Go to your pipeline
    • Click the grey area
  • Go to General > Concurrency
    • Set to 1
    • This will set the number of simultaneous runs of that pipeline to 1 
    • Now only 1 pipeline instance can be run at one time. all the other pipeline runs will be put on waiting stage 

1 comment:

  1. Good One. We had similar issue in our case and this came as a saviour

    ReplyDelete