Recently, I encountered an issue while working on a project that involved onboarding on-premises Syslog data sources to Azure Sentinel through a Log Analytics Workspace. The process was designed to use an on-premises data collector server via a private endpoint and Azure Monitor Private Link Service (AMPLS). Despite having proper planning and configuration, ran into a roadblock after implementing a Data Collection Rule (DCR). Here’s a detailed breakdown of the situation, the error, and how it was resolved.
The Scenario
The goal was straightforward: onboard Syslog data from an on-premises environment to Azure Sentinel. Here’s the step-by-step breakdown of what i did:
- Syslog Collector Server Setup: Created a Syslog collector server and installed Azure Arc to ensure that the server was available in the Azure portal for management.
- AMA Data Connector Installation: In Azure Sentinel, we installed the Azure Monitor Agent (AMA) data connector to facilitate log collection.
- DCR Rule Creation: Then created a Data Collection Rule (DCR) to define the logs wanted to collect from the Syslog server.
Solution Architect as simple as Below. Suggested by MS
However, even after completing these steps, I could not get the logs to appear in Azure Sentinel as expected.
Initial Diagnostics
To identify the issue, I went through multiple checkpoints, including:
- Checking the status of the Azure Monitor Agent (AMA)
- Verifying the Azure Arc connectivity
- Ensuring proper firewall connectivity between the source (Syslog server) and destination (Azure)
Everything appeared to be correctly configured, yet the issue persisted.
The Error
When i dove deeper into the logs, specifically under /var/opt/microsoft/azuremonitoringagent/log/mdsd.err, encountered the following error message:
Error –
[/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshGigToken.cpp:384,Getagentconfiguration] Response code: 403; Response: {"Error":"Invalid Access", "Message": "Data collection endpoint must be used to access configuration over private link"}. Error code: -2146172665
Error Breakdown
The error pointed to an issue with accessing the data collection endpoint over a private link, which explained why the configuration was not working as expected.
Error Code: -2146172665
- This error indicates a problem with the connection or access to the data collection endpoint, typically due to misconfigured access via the private link.
The Root Cause
Upon further investigation, I discovered that there was a missing DNS configuration. Although traffic was flowing to Azure via AMPLS, the Syslog server failed to resolve the necessary Azure endpoints properly. This was preventing the Data Collection Rule (DCR) from successfully sending logs to Azure Sentinel.
The Solution
To resolve the issue, I tested a direct DNS entry on the host file of the Syslog server. Here’s what worked:
- Edit the Hosts File: On the Syslog server, I manually added a direct DNS entry in the
/etc/hostsfile. This allowed the server to resolve the private endpoint’s DNS. - Restart AMA Agent: After adding the DNS entry, I restarted the Azure Monitor Agent (AMA) service to ensure the changes took effect.
- Logs Started Flowing: Once this was done, the Syslog data started flowing into Azure Sentinel as expected.
Key Takeaways
- Manual DNS Entry: In cases where DNS configurations are missing, a simple manual entry in the
/etc/hostsfile can resolve the issue. However, this is a temporary solution, and proper DNS configuration should be applied for long-term stability. (For a Temporary Fix ) - For Permanent Fix Ensure DNS Resolution: When using AMPLS and private endpoints, always verify that the necessary DNS resolution is configured for your data collector server. Even if traffic is flowing through AMPLS, missing DNS entries can prevent access to critical Azure endpoints.
Detailed example with AMPLS and network isolation.
Reference link – https://mortenknudsen.net/?p=1442
Here is a detailed illustration of the network and usage of DCE in a network isolation scenario.
