Setting up Conditional Access alerts

We all know a strong Conditional Access design is one of the most important security measures to implement to protect your users and data.

One feature that is currently lacking in Conditional Access is change tracking and alerting. If an admin makes a mistake or a global admin account is breached and Conditional Access is changed, how will you know?

Setting up alerting on Conditional Access is quite straight forward and requires forwarding Azure AD logs to a Log Analytics Workspace. Forwarding these logs also has the added benefit of improving log retention past the default 30 days that is granted with an Azure AD Premium P1/P2 license.

Today we will walk through setting up log forwarding and setting up two types of alerts. The first for catching changes to Conditional Access, and the second for catching changes to Conditional Access exempt groups.


Forwarding Azure AD logs to Log Analytics

First, we create a new Log Analytics workspace. In the Azure portal search for Log Analytics and hit Create.

With the deployment of Log Analytics complete, browse to Azure AD in the Azure portal, select Monitoring, Audit Logs, and then Export Data Settings.

Select “AuditLogs” and “SignInLogs” and “Send to Log Analytics”.

Now we can set up the alerting. Azure Monitor allows us to trigger alerts based on KUSTO queries. In our examples we will be setting up email alerting, but you could also trigger SMS, a webhook or an Azure Automation runbook.

Setting up the alerts

Open the Log Analytics workspace in the Azure portal and scroll down to “Alerts”, listed under the Monitoring category. Click “New Alert Rule”. Click “Select Condition” and then “Custom log search”.

Under the search query field, enter the following KUSTO query:

AuditLogs | where Category == "Policy" | project ActivityDateTime, ActivityDisplayName , TargetResources[0].displayName, InitiatedBy.user.userPrincipalName

Change the Alert logic to read as:

Next you will need to create an action group to dictate what type of alert is generated and where it is sent to. The alert will show you the policy that has changed and the user responsible for making the change.

Note: This query will also alert on changes to App Proxy as they fall under the category “Policy” – if you use App Proxy in your organisation consider changing the name and content of the alert to reflect this.

Finally, we save the new alert rule. It should look similar to the screenshot below:

When a change is detected, an email is received. Here is an example alert:

Alerting on a CA exempt group change

For the second of our examples, we will alert on any user additions to a Conditional Access exempt group. In our environment, we have some devices such as meeting room phones that require Legacy Auth, but we wouldn’t want a genuine user added to this group. To begin we will need to find the group in Azure AD and make a note of its object ID. This will be used in our alerting query.

Next we set up the alerting as above, but this time the query should be:

AuditLogs | where TargetResources[1].id in (OBJECTID) and ActivityDisplayName == “Add member to group” | project ActivityDateTime, ActivityDisplayName , TargetResources[0].userPrincipalName, InitiatedBy.user.userPrincipalName

Make sure you substitute the OBJECTID for the Object ID of your group.

Mark’s final thoughts

With our alerting set up and working we can all rest a little bit easier knowing that changes cannot happen without us being aware. Microsoft are working on the many customer requests to have this type of alerting built into Conditional Access, but in the meantime – and for such an important component of an organisation’s security – this type of alerting must be considered essential.

Next steps

If you’d like help securing your identities and improving your cyber security posture, Chorus can help. We provide leading end-to-end Cyber Security Services, helping forward-thinking organisations transition to a modern Zero Trust Security Model. To discuss your cyber security challenges with one of our cyber security consultants, please get in touch today.