ExpertIncident 2023-06-20 - Duplicate Azure devices
Yesterday’s Azure device duplication issue (see https://status.logicmonitor.com/) is still active (details below).
This has had far-reaching implications for a few of our (Panoramic Data Limited)’s customers, including an alert storm.
At the current time, LogicMonitor have not posted a solution for de-duplicating the Azure VMs or silencing the alerts.
We have developed a script to do so, and we are happy to help you to (carefully!) bulk delete your duplicate Azure devices.
If you would like to hold a call to remedy this on your LM portal, please contact me direct via Linked In (https://www.linkedin.com/in/davidbond/).
However… IF you have a tame .NET dev, feed them some caffeine and instruct them as follows…
At a PowerShell prompt:
dotnet new install LogicMonitor.Cli
dotnet new logicmonitor --name "My.AzureDeduplicator"
code .
Edit and update appsettings.json
- Provide account, tokenId and tokenKey
- Set “Setting1” to the string version of the ResourceGroup ID that you want to deduplicate from
- Modify Application.cs as follows:
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
try
{
// Use _logger for logging
_logger.LogInformation("Application start. DeviceGroupId is set to {setting1}", _config.Setting1);
using var logicMonitorClient = new LogicMonitorClient(_config.LogicMonitorClientOptions);
var allDevices = await logicMonitorClient
.GetAllAsync<Device>(
cancellationToken
)
.ConfigureAwait(false);
var deviceGroupIdString = _config.Setting1;
var devices = allDevices
.Where(d => d.DeviceGroupIdsString.Split(',').Any(dgs => dgs == deviceGroupIdString))
.ToList();
_logger.LogInformation("Found {deviceCount} devices", devices.Count);
// Find a list of those with a similar name, but ending in #1
var devicesToDedupe = devices
.Where(d => d.DisplayName.EndsWith("#1") && devices.Any(d2 => d2.DisplayName == d.DisplayName.Replace("#1", "")))
.ToList();
foreach(var device in devicesToDedupe)
{
if(cancellationToken.IsCancellationRequested)
{
return;
}
_logger.LogInformation("Deduplicate {deviceName}?", device.DisplayName);
// Read a keypress from the user and check that it's a Y
if(Console.ReadKey().KeyChar == 'Y')
{
// Delete the duplicate
_logger.LogInformation("Deleting {deviceName}", device.DisplayName);
await logicMonitorClient
.DeleteAsync(device, cancellationToken)
.ConfigureAwait(false);
}
else
{
_logger.LogInformation("Skipping {deviceName}", device.DisplayName);
}
}
}
finally
{
_lifetime.StopApplication();
}
}
- READ AND UNDERSTAND THE CODE - BE SURE THAT THIS IS WHAT YOU WANT TO DO
dotnet run
This will prompt you for each device.
Usual caveats:
THE OWNER MAKES NO WARRANTIES, EXPRESS OR IMPLIED, WITH RESPECT TO THE CONFIDENTIAL INFORMATION AND HEREBY EXPRESSLY DISCLAIMS ANY AND ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
LogicMonitor’s Status page text:
LogicMonitor is currently investigating impacts to our LM Cloud Component(s) for MS AzureSubscribe
Update - We have rolled back the component responsible for adding duplicate Azure devices. We are continuing to investigate the best approach for removing the incorrectly added duplicate devices. The next update will be around 10am PT on Wednesday June 21st.
Jun 20, 2023 - 19:37 CDT
Identified - LogicMonitor has identified an issue with our LM Cloud application that may be creating duplicate Azure devices for some customer's accounts. We are currently working towards remediating the issue with our Engineering backend team. We will provide further updates in approximately 2 hours, or if more information is uncovered.
Jun 20, 2023 - 17:32 CDT
Investigating - LogicMonitor is currently investigating technical abnormalities, which may be impacting customer accounts. We will update once we have further information on the full scope of impact.
Jun 20, 2023 - 17:27 CDT
