Use Maester to Monitor Microsoft 365 Security Settings

Living in an era where cyberattacks happen every second, admins should configure Microsoft 365 tenants with crucial security settings. But here’s the thing: merely configuring the security settings in Microsoft 365 is not enough; it’s about how you effectively monitor the security configurations.

Mistakes may happen and sometimes users/admins inadvertently make changes to the security settings, which might cause a big loophole in Microsoft 365 security. So, how do you stay on top of these critical security settings?

Don’t worry! Merill Fernando and his dedicated team answered the call with a game-changing solution – the Maester, a Microsoft security test automation framework. It’s a free tool born out of extensive research, raised to refurnish the way we monitor crucial security settings. Let’s dive into it in detail!

Maester is an open-source PowerShell-based test automation framework designed to help admins monitor their tenant’s security configurations effectively. It achieves this by running a set of tests to ensure that configurations comply with baseline security policies.

Maester includes over 40+ out-of-the-box tests from Entra ID Security Config Analyzer (EIDSCA) and 20+ Conditional Access tests. Additionally, each test result provides guidance and remediation steps to address any issues identified.

Note: Currently, the Maester tool focuses on Entra ID settings such as monitoring Conditional Access policies, authentication methods, Entra recommendations, and more. However, future updates aim to extend its capabilities to monitor security configurations across all Microsoft 365 products.

The Maester tool depends on Pester and Microsoft Graph PowerShell to work. By running the below cmdlets, both dependencies will automatically get deployed.

Once done, run the below cmdlet to connect to your Microsoft 365 tenant.

After installation, you can start using the Maester tool to run the tests & monitor your Microsoft 365 tenant’s security configurations.

To run pre-built security tests, execute the cmdlet below.

Once the tests have been executed, Maester will generate a comprehensive HTML report, offering detailed insights into each test. The Maester test report comprises two main sections:

Test Summary: This section provides an overview of the test results, including successful, failed, and untested cases. Presented in visually engaging representations, it offers a quick glimpse into the overall test outcome.

Test Details: Here’s where the real value lies. In this section, you’ll find a breakdown of all the tests, along with their status (Pass/Fail). Additionally, each test includes an “Info” column. Clicking on it reveals further details about the test, allowing for deeper analysis and troubleshooting.

You may ask, do we need to execute the above cmdlets every time we want to get the test results? But that’s not needed! Let’s see how!

Maester can seamlessly integrate with DevOps services to automate the monitoring of security configurations. By setting up Maester in Azure DevOps pipelines, GitHub, or Azure Automation, you can ensure that tests run automatically at regular intervals. This Maester automation guarantees that your tenant configuration remains compliant with your policies.

You have the flexibility to schedule tests to run daily or monthly and view the comprehensive test results within the respective DevOps service interfaces. Additionally, you can configure Maester to send an email summary at the end of each monitoring cycle using the “Send-MtMail” cmdlet in your GitHub/Azure DevOps daily monitoring workflow.

You might be wondering if the pre-built test cases cover all the scenarios you need. They may not, but don’t worry! With Maester, you can create custom tests tailored to your specific requirements and execute them seamlessly. Let’s dig it!

Maester is built on top of the Pester module, a PowerShell-based test framework (Security as Code – SaC) used for writing and running tests. Pester employs an English-like, easily understandable format for writing tests.

So, you need to use the Pester code and test it with Maester. To do it, follow the steps below.

• Create a Custom Test File in the Maester Folder.
• Add Security Tests to the File.
• Run the Custom Test Using Maester.

After installing the Maester module, a folder named “Custom” is automatically created in the same location as the module. This is where you need to create your custom test files. To create a custom test file there, follow the steps below.

1. Open Windows PowerShell ISE as you normally would.
2. Create an empty file named “<CompanyName>Entra.Test.ps1”.
3. Save the file in the “Custom” folder of the Maester module.

Note: Ensure that you use the exact filename format with the “Tests.ps1” suffix so that the file can be automatically discovered and run when you invoke the Maester module.

Now, you need to add the security tests inside the created file. Here, I have given an example to check whether a specific Conditional Access policy exists in the tenant or not.

Once done, you need to run the below cmdlet to execute the custom-created test.

Upon successful testing, Maester will provide you with the output of your tests. Furthermore, you can add more test cases or new test files based on your requirements and run with the above cmdlet.

Recently, Microsoft introduced a preview tool – “Conditional Access What If” in Microsoft Entra ID. This will help you understand the enforcement of your CA policies easily.

Typically, after configuring a CA policy with various location conditions and device-based conditions, manual sign-ins are performed to verify proper enforcement. However, this manual process consumes both time and effort.

But, with this tool, you can automatically simulate user sign-ins and check whether the CA policies are enforced on the sign-ins. Needed-one, right?

So, Maester allows you to define tests that can run against the CA policies using What If API. This capability allows you to identify potential loopholes in your conditional access policies before they are applied.

Maester offers over 20 pre-configured What If Conditional Access policy tests, including:

• Verification of at least one Conditional Access policy requiring MFA for admins.
• Validation of at least one Conditional Access policy configured with All Apps.
• Confirmation of at least one Conditional Access policy configured to block legacy authentication.
• Verification of at least one Conditional Access policy requiring MFA for all users.

Additionally, you have the flexibility to create your own What-If Conditional Access tests using the “Test-MtConditionalAccessWhatIf” cmdlet.

That’s all! We have covered all the essential functionalities of Maester to monitor Microsoft 365 security settings! Kudos to Merill and his team for giving us the Maester security monitoring tool! By the way, this module is constantly updating with new tests, you can get that using the “Update-MaesterTests”. Also, note that the current version of the module shows some duplications in skipped tests and also tests showing failed even though the exact configurations exist in the tenant. If you have any questions, don’t hesitate to reach out to us in the comment section below. We’re here to help!