Centralize logs with Azure Monitor and Storage account (ADLS)

Today we will learn on how to Send the logs of each resource within Azure to Azure Monitor as well as Azure Storage (ADLS)

• If you want to store the logs for longer duration, you can use Azure Data Lake (ADLS) for logs storage
• If you want to view the logs of each resource/activity within Azure on the fly, you can send each resource's logs to Azure Monitor (a centralized hub to viewing logs)

Today we will store the logs of Azure data factory in ADLS as well as view them in Azure Monitor.

Steps to be performed at Data factory side and Azure Monitor:

• Go to your data factory
• click Diagnostic settings

• Configure the diagnostic settings
• Select "Send to Log Analytics"
• This will send the logs to Azure Monitor
• Specify the subsctiption and log analytics workspace
• If you haven't created a log analytics workspace, create one.
• Select "Archive to a storage account"
• This will send the logs to ADLS
• Specify the subscription and Storage account where you want the logs to be stored
• If you specify this option, then you also have to specify the retention (in days) for logs
• Select what type of logs you want to be stored
• Tick mark all the logs.
• Now go to Azure Monitor to view the logs
• Type "Monitor" in the search bar
• Select a scope
• Select what resource's logs you want to view

• Select the Metric you want to view
• Select the Aggregation

• You are done. 
• The specific metric will be shown

Steps to see the logs in the Azure Storage (ADLS):

• Install Azure Storage Explorer in your computer.
• This utility will let you view your ADLS accounts and the data inside
• Configure the storage to your storage account
• Navigate to your Storage account
• Here, you will see different containers created automatically for data factory logs
• Pipeline runs
• Trigger runs
• activity runs
• You will also see a $logs container created that will store all the logs for all the resources day wise in a folder