Incident 2023-06-20 - Duplicate Azure devices
Yesterday’s Azure device duplication issue (seehttps://status.logicmonitor.com/) is still active (details below). This has had far-reaching implications for a few of our (Panoramic Data Limited)’s customers, including an alert storm. At the current time, LogicMonitor have not posted a solution for de-duplicating the Azure VMs or silencing the alerts. We have developed a script to do so, and we are happy to help you to (carefully!) bulk delete your duplicate Azure devices. If you would like to hold a call to remedy this on your LM portal, please contact me direct via Linked In (https://www.linkedin.com/in/davidbond/). However… IF you have a tame .NET dev, feed them some caffeine and instruct them as follows… At a PowerShell prompt: dotnet new install LogicMonitor.Cli dotnet new logicmonitor --name "My.AzureDeduplicator" code . Edit and update appsettings.json Provide account, tokenId and tokenKey Set “Setting1” to the string version of the ResourceGroup ID that you want to deduplicate from Modify Application.cs as follows: protected override async Task ExecuteAsync(CancellationToken cancellationToken) { try { // Use _logger for logging _logger.LogInformation("Application start. DeviceGroupId is set to {setting1}", _config.Setting1); using var logicMonitorClient = new LogicMonitorClient(_config.LogicMonitorClientOptions); var allDevices = await logicMonitorClient .GetAllAsync<Device>( cancellationToken ) .ConfigureAwait(false); var deviceGroupIdString = _config.Setting1; var devices = allDevices .Where(d => d.DeviceGroupIdsString.Split(',').Any(dgs => dgs == deviceGroupIdString)) .ToList(); _logger.LogInformation("Found {deviceCount} devices", devices.Count); // Find a list of those with a similar name, but ending in #1 var devicesToDedupe = devices .Where(d => d.DisplayName.EndsWith("#1") && devices.Any(d2 => d2.DisplayName == d.DisplayName.Replace("#1", ""))) .ToList(); foreach(var device in devicesToDedupe) { if(cancellationToken.IsCancellationRequested) { return; } _logger.LogInformation("Deduplicate {deviceName}?", device.DisplayName); // Read a keypress from the user and check that it's a Y if(Console.ReadKey().KeyChar == 'Y') { // Delete the duplicate _logger.LogInformation("Deleting {deviceName}", device.DisplayName); await logicMonitorClient .DeleteAsync(device, cancellationToken) .ConfigureAwait(false); } else { _logger.LogInformation("Skipping {deviceName}", device.DisplayName); } } } finally { _lifetime.StopApplication(); } } READ AND UNDERSTAND THE CODE - BE SURE THAT THIS IS WHAT YOU WANT TO DO dotnet run This will prompt you for each device. Usual caveats: THE OWNER MAKES NO WARRANTIES, EXPRESS OR IMPLIED, WITH RESPECT TO THE CONFIDENTIAL INFORMATION AND HEREBY EXPRESSLY DISCLAIMS ANY AND ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. LogicMonitor’s Status page text: LogicMonitor is currently investigating impacts to our LM Cloud Component(s) for MS AzureSubscribe Update-We have rolled back the component responsible for adding duplicate Azure devices. We are continuing to investigate the best approach for removing the incorrectly added duplicate devices. The next update will be around 10am PT on Wednesday June 21st. Jun20,2023-19:37CDT Identified-LogicMonitor has identified an issue with our LM Cloud application that may be creating duplicate Azure devices for some customer's accounts. We are currently working towards remediating the issue with our Engineering backend team. We will provide further updates in approximately 2 hours, or if more information is uncovered. Jun20,2023-17:32CDT Investigating-LogicMonitor is currently investigating technical abnormalities, which may be impacting customer accounts. We will update once we have further information on the full scope of impact. Jun20,2023-17:27CDT