Locked
Discussion interactions are currently paused.
Add device to Collector with least devices in rest api
I hope someone here can help,
Using powershell and the rest API I would like to do a lookup of selected collectors find which has the least amount of devices on it then add the a new device .
I have the script for adding the device and that works great just need the collector lookup, has anyone done something similar.
Thank you in advanced for all help provided
Paul
Paul_Carroll
Posted 7 years ago·Last reply 6 years ago
49 comments
Cole McDonald
·6 years agoNo such luck... could you verify that this is the correct URL structure for this @Forrest Evans - LM?
Even being really pedantic about it still isn't taking it:
same error:
Cole McDonald
·6 years agoThank you Forrest. As always, a font of great information. I assume from the reading that the default is "refresh"
From the link above (to save folks some clicking):
I'll be putting this in place to day and I'll report back... hopefully with a new $data block for the script and calling it done!
Forrest_Evans_-
·6 years agoHi Cole,
predef.* attributes are used by our Topology Service and can't be updated through the API, only through LogicModules. They aren't specific to Azure as they may be applied to standard devices as wel
To update a device with Properties and not affect others you'll want to update the device using PATCH command documented here: https://www.logicmonitor.com/support/rest-api-developers-guide/v1/devices/update-a-device/#PATCH. Pay particular attention to the customProperties opType option.
Note: I haven't looked through the code. I'm just offering this based on the last two posts.
~Forrest
Cole McDonald
·6 years agoNo updates from LM on this last point. It's the last piece I've got on this. I'm going to put it in place anyway as it works on the other ABCGs in our environment and doesn't do anything to the other when it fails due to the azure resources being REST API incompatible due to naming convention issues with their properties. I suspect it's just a miss and will magically start working at some point. (Again, this is where I point out that we're not liable if you implement this and it doesn't work in your environment).
hopefully they get this sorted out soon.
Cole McDonald
·7 years agoI've found an issue that I'm working with LM to address. If the device is an auto-discovered Azure resource, some of the custom properties applied during that process seem to not allow the REST API to alter the device at all. So if you have azure based resources in a collector group, they won't move. The specific property names mentioned in the error message start with "predef." ( predef.externalResourceID, predef.externalResourceType ).
So currently, my code won't work for azure resources. Works like a champ for us on Hyper-V hosted VMs and physicals.
Cole McDonald
·7 years agohere's the updated calculate-threshold and call that uses the settings from the config (which did work for me, so the <##> can be removed from that portion of the code):
Cole McDonald
·7 years agoOK... here's the fix for that miss:
This replaces the same chunk of code in the previous post and includes an if statement to exclude any device with a custom property "collector"
Cole McDonald
·7 years agoFound a miss... it shifts the collector VMs as well... I have to catch for that yet.
Cole McDonald
·7 years ago!!! Successful first run against our environment! The balance in the group I tested against went from (devices:instances) 39:7112 / 175:6803 to 107:6880 / 107:7036
Doesn't seem to be any uptick in alerts following the transition. CPU spike on one of the collectors... but I have reconfigured one of them in the group to fail faster than the other (as a remediation for network socket resource exhaustion on the collector causing a backlog of tasks). I'll need to transfer those settings across to the other one (more threads, shorter timeouts in key areas that were getting bogged down) to get a real read on the changes. So far, so good.
Cole McDonald
·7 years agoJust to show where we're at right now... The only part not working is the threshold PATCH at the end of the group loop (same disclaimer as always; neither I nor Beyond Impact 2.0, LLC warranty this software. It's aggressive, use with caution):
Cole McDonald
·7 years agoLast piece of the puzzle... I need to set the autobalance threshold of the ABCG. Assuming that everything else works codewise (it's the same function everything else goes through successfully)... what am I missing here?
Cole McDonald
·7 years agoNever mind... I've successfully tested moving 1 server. Have to look at the impact on the collectors.
Cole McDonald
·7 years ago@Forrest Evans - LM I've got all of the pieces in place... except for the precollectorid change... I'm still stuck on that a bit. Let's assume I'm looking to move all of the devices in an ABCG into arbitrary collectors, then allow the ABCG to adjust as necessary once everything has been set. From a logic standpoint should that be a 2 step process?
Since I'm looking to make this as portable as possible, I have to assume that the memory sizes of the collectors are different. In terms of calculating that threshold, let's assume I have a 2Gb and a 4Gb collector in a group. If the group overall has 3000 incidents we want A to handle half as much as the other... so how does that apply to that threshold? If we set it for the # for A, it'll never trigger for B ... if we set it for B, it'll never trigger on A which will be constantly overwhelmed. I would assume we have to set it to the smallest possible threshold and hope that the other/s don't end up balanced wrong and choke out? If we add a C at 4 Gb, what if B and C lose balance? If we set it for A, will they constantly try to balance? So in these cases, the calculator I've got does no good as it assumes evenly provisioned collectors in the group and we should set it to:
(instances / total ram) * lowest ram setting? (rounded up a touch to the nearest 50 to account for any slight disparities in instances/device counts)
All that said, I'm initially implementing with the calculator as it is, since our collectors are evenly provisioned within each group.
Cole McDonald
·7 years agoInitial testing results of my algorithm looks good. Once I've added the payload to change the preferred collector on the devices, I'll post the code. Device listing are "Id InstanceCount" so you can see how they are sorting:
Cole McDonald
·7 years agoThank you for that.
If I have a medium and a large in the same group, I'd need to determine threshold for each, then add the results or would a min/max/avg be a better option?
Here's the calculator I'm using currently. It works amazingly well for groups with the same size collectors in them:
/topic/2392-autobalanced-collector-group-threshold-calculator/Forrest_Evans_-
·7 years agoYes. ABCG can properly address collectors of different sizes. The JVM memory value is used to determine how many instances should be assigned to each collector. The configured threshold is used for Medium collectors and the value is scaled up/down.
Cole McDonald
·7 years agoI'm nearly done with my re-balancing 'Source. I've got it gathering all of the data and it's ready to start reassigning resources to different collectors. I have one collector that is showing up in the collector's interface, but doesn't come up in the return from the REST API call for /settings/collectors/collector. So the code won't consider that as a place to send devices.
Due to imbalanced instance counts per type of device in our environments, I'm ending up with all of the same type of device auto balancing to a single collector rather than evenly spread across both. It's an interesting sorting behavior caused by the balancing algorithm over time. As a result, when a batch script fires, it opens TCP ports for each of the instances being collected... and does so on a single collector in the group due to the strange sorting behavior (they only consider instance count). More resource exhaustion. I've ended up doubling threads and halving timeouts on script/batchscript/powershell/WMI on most of my collectors to get them to fail faster to keep the systems from log jamming. The collectors are all "Large."
Hopefully, I'll have this done soon and can post it here to help out others encountering problems with the autobalance rebalancing algorithms. I've also got it gathering the number of instances and using the threshold calculator I wrote (posted around here somewhere) to set the threshold at the end of the re-balance to make sure the balance is set correctly for the collector sizes and the number of instances.
@Forrest Evans - LM, can collectors of different sizes be added into an ABCG? I'd have to figure out how to account for that in the threshold calculations. I'm also working on building it out for groups with > 2 collectors to make sure it's handing them out to all of them equally.
Forrest_Evans_-
·7 years agoIn my tests I had to wrap the preferredCollectorId value of zero in double quotes:
{"name":"10.1.1.22","displayName":"My Device","autoBalancedCollectorGroupId": 4,"preferredCollectorId":"0"}tduong
·7 years agoHi All,
I'm trying to create a new device with auto balance group with the below code. But I'm getting response error message "Collector (id=0) does not exist" status 1404. Not sure what went wrong... please advise what need to be done if you've come across this before.
Thank you in advance
Cole McDonald
·7 years agoThat's where I was trying to do it. It's just not cooperating... so I'll have to play with the Regex a bit to get it to cooperate, assuming the property name "name" I'm referencing is correct.
Mike Moniz
·7 years agoTry looking at https://www.logicmonitor.com/support/datasources/active-discovery/datasource-discovery-filters. If I understand the issue correctly, too many DataSource Instances and you want to ignore some, you can fix that without any REST API work and making changes within the system instead.
Cole McDonald
·7 years agohttp://woshub.com/user-profile-disks-in-windows-server-2012-r2-rds/
I'm trying to have it pass them by as they all have a very discernable and easily regex-able naming convention. I have to figure out how to address them in the filters in the LM interface... now that I see the REST docs, I assume that the fieldnames are going to be the same in the interface as in the REST object... I may try again (unsuccessful last time - Drinking from the firehose is fun).
Mike Moniz
·7 years agoWow, that is an interesting situation. If I understand the situation, LM sees that your Terminal server as suddenly gaining some 2000 new volumes in-and-out that it wants to monitor. It then sends out 2000 WMI requests to query each disk for free space and the like. And with 10 servers, that comes out to 20,000 individual remote WMI calls every 3 minutes or so? I can see that killing a collector.
If I understand that correctly, perhaps it's best to instead write a custom replacement Volume DataSource to either ignore all these symlink volumes, if you don't need to monitor them, or have it use Batch Script to convert 2000 WMI calls into just one or two calls that collects all the volume details all at once.
Cole McDonald
·7 years agoI do have a meeting that is being scheduled with a dev. I can ask this at that meeting if you'd like.
Forrest_Evans_-
·7 years agoWhile you pass 0 to set it to AutoBalanced the actual collector that is currently doing the monitoring will be represented in the return value for 'preferredCollectorId'
I'd have to try and test your specific scenario and potentially confirm with the developers to see how it would respond.
Cole McDonald
·7 years agoInteresting, preferredCollectorId points to an individual collector on every device in our environment currently. Should they all be set to 0? When we setup, it was before autobalancing was available, we switched the collector groups that we had to autobalancing afterwards. What would the impact be of setting the group to 0, preferred collector to the new collector we want it to point to, once it moves, set the group back to the original? Will it tend to shift back or will it be re-evaluated by the group and potentially shifted back even after I move it manually?
Cole McDonald
·7 years agoWe use User Profile Disks in a couple of our environments. When a user logs in, it creates a symlink to a mounted VHD volume. As this reads as the system adding a device, it registers as an instance under the volume dataSources. When 2000 users log into a load balanced RDP Terminal Server set, that's a ton of instances that come online all at once. One of the drawbacks is that the symlink doesn't disappear when the user logs out. So every user's symlink is potentially on every server in the LBSet. So 2K users across 10 Terminal servers = 20K potential devices that LM can't differentiate between a live link and a dead link. So it ties up the resources for 18K instances while they wait to time out. It also seems as though (when looking at !tlist c-wmi h=<IP for 1 TS>) that it's holding onto them even after it's timed out... so the socket pool never empties on the collector and a ping query with no sockets becomes a "host down" alert in LM for each host that isn't able to respond to the ping that is never happening.
We auto generate tickets based on those host down alerts, so we were bombarded with alerts the other day that would have been hundreds of false tickets that we'd need to pursue (our management uses those tickets as metrics as a basis for our performance as employees). Luckily, I hadn't turned on the ticketing yet or we would have had a deluge like we haven't seen since the great AX-AOS alert storms of '17
Forrest_Evans_-
·7 years agoIf you are changing the collectorId within Auto Balanced Collectors there are some other "rules" to follow. If you want to switch a device from being Auto Balanced to directly assigned you have to set `autoBalancedCollectorGroupId` to 0 and `preferredCollectorId` to the collector ID you want it directly assigned to. To set it back to AutoBalanced from directly assigned do the reverse: `autoBalancedCollectorGroupId` sets to id of the group and `preferredCollectorId` to 0
With Version 2 of the API you don't need the $queryparam part. You can just patch with the payload.
Mike Moniz
·7 years agoWant to post the REST call part of the script? Are you sending it as a PATCH http vert (vs PUT or GET)?
Cole McDonald
·7 years agoAs a test, I'm doing this:
It's returning with no errors, but the value doesn't seem to be changing.
Forrest_Evans_-
·7 years agoThanks for the feedback. I'm confused why user logins would have such an impact on a collector, but I look forward to seeing your datasource.
Cole McDonald
·7 years agoThe issue we're running into isn't with the memory... it's with TCP/IP Socket exhaustion. The number of Devices * instances was hitting the limit and we were getting 'host unreachable' messages constantly. Moving them to an autobalanced group was supposed to alleviate it by balancing them... which it never did as it's too conservative. I would venture a guess that your software keeps track of outstanding requests per instance/device. If a balanced needs to occur, I would set a flag to have it movePending, as soon as the threads had all returned, then the collector move happens.
I'm looking for real balance, not just above the individual collector's threshold to get it back under the threshold. When 2000 users log into an environment when their shift starts and all of the traffic hits an unbalanced load, one of the collectors stops delivering data, the other is fine. That's a problem with the balancer that should be easy to fix. I know that this kind of load is possible as we had the exact same environment being monitored by SCOM (technically, it still is - and I'm having to go back to it whenever LM fails to get telemetry).
I have guidance about how to get the collector moves to work and as soon as I'm done with the dataSource I'm building to run twice a day, I'll post that code here.
Mike Moniz
·7 years agoAh thanks! I haven't played with Auto Balancing yet but it would be great to get those numbers for non-balanced collectors also.
Forrest_Evans_-
·7 years agoWhen you enable Auto Balanced Collectors on a Collector Group a new column is added with the instance count.
I do have an enhancement on the list to always show this count rather than only on Auto Balanced Groups.
Mike Moniz
·7 years agoHow can you determine the number of instances a collector has? Assuming this is not the same as the number of devices a collector has.
Forrest_Evans_-
·7 years agoFew clarifications around the Auto Balanced Collector solution and when it chooses to balance devices:
The AutoBalance is intentionally conservative because when devices are moved between collectors there is a risk of them missing polls. The balancing activity will not trigger a rebalance (even if the button is pushed) if the collector is not over it's own unique threshold limit. In the current iteration, all instance types are treated equally.
Since collectors of different sizes can handle different loads we set the default threshold to 10,000 as the number of instances threshold for 2GB of JVM memory, which corresponds to a medium sized collector. The math to determine how many actual instances a collector can handles is the following:
(Target_Collector_Mem/Medium_Mem)^1/2 * Medium_Threshold
For example, if user sets (2G) collector's threshold to 10,000, for a large(4G) collector, the threshold will be scaled to: (4/2)^1/2 * 10000 = 14140
I checked out your current collectors and none of them are over their corresponding thresholds. The biggest one only had 12,209 instances on a large collector.
As for adding devices if you want to add a new device to an AutoBalance Collector Group without choosing the collector the payload below would be a very minimal example to do that. The key parts being the autoBalancedCollectorGroupID is provided as an integer and 'preferredCollectorId' is set to 0 for the device to be AutoBalanced. This should add the device to the collector with the lowest instance count.
{"name":"10.1.1.22","displayName":"My Device","autoBalancedCollectorGroupId": 4,"preferredCollectorId":"0"}~ForrestMike Moniz
·7 years agoI was able to update the collector on a device by using PATCH:
https://www.logicmonitor.com/support/rest-api-developers-guide/v1/devices/update-a-device/
Forrest_Evans_-
·7 years agoWe just finished with LevelUp. I'll connect with Support to review the issues you were running into and get back to you tomorrow (Friday)
Cole McDonald
·7 years agoI'm at the point in my script where I'm ready to deliver the payload... and I'm informed that we don't have programmatic access to change the collectors for a device. So that was apparently 20 hours of wasted development time - <grumble>.
@Forrest Evans - LM Is there any progress on fixing the autobalancers? I don't see updates in my ticket to that end.
Cole McDonald
·7 years agoWorking, thank you. I was not using the query params, so I was accidentally succeeding before with my other calls... corrected now.
Mike Moniz
·7 years agoI was being more general in my example. When using the LM API, you need to separate the endpoint part (/device/devices) from the query part (part after the "?") because of how LM does API authentication. Which is likely why you are getting auth failed error.
So using the example Paul provided above, you would use something like this:
(You would also need to change the ending part of his script as devices have different properties than collectors.
Cole McDonald
·7 years agoI'm still not getting any response from the API using the URL above. Here's the response I get... the function to retrieve data works for other queries:
I get a good response with device/devices raw. I get the above error with just the ?size=1000 as well as all of the following pieces in that query you'd posted. I've gone so far as to separate all of the individual pieces and test them alone and no dice. Am I using the wrong question mark?
Mike Moniz
·7 years agoCalling the REST API with the below should get you a list of all the devices assigned to that collector:
Replace COLLECTOR_ID_HERE with the collector ID number which you can get from the Collector Settings page and/or via the API. If there are more than 1000 devices on a collector then you need to call this multiple times with an index as the API maxes at 1000 devices per call. I believe there are examples in the API docs.
Cole McDonald
·7 years agoI did... but I need the solution before you'll be able to fix it. I ended up on a call with Casey and we walked through troubleshooting it. The metrics being used to balance aren't evaluating correctly for our environment, so the load is unbalanced. I am using a far simpler method of determining load to perform a balance which will reduce our VM spend (as opposed to adding more collectors to handle the load) to be able to monitor our environments.
I'm at the point in the script where I just need to grab the list of devices reporting to the Collector with the most devices associated with it, and then I'll make a random selection to cover the sum difference to redistribute and change them to the other Collector/s in the group. We currently don't have more that 2 collectors / group, but I'm writing my script as if we did to make it more portable and save time in the future having to re-read what I'd done in the first place to refashion it for a broader use case.
Ticket # 155345 or your reference. If you're interested to see how I'm doing what I'm doing, I'm making a DataSource (specifically so I can schedule its run) that I will applyTo our primary Collector so it has somewhere to run. It then uses the REST API to gather data and make changes to the environment. I'd love you guys to add a "ScriptSource" that would allow us to trigger a script based on an alert of a specific type, or on a schedule. This would allow me to write auto-remediation scripts for common simple alerts so we could save the staffing costs to manually address them. Right now, we have SCOM performing quite a bit of small maintenance tasks, but we'll be turning it off (replaced by LM) and it's going to get really busy with menial tasks for our technicians when we do.
Forrest_Evans_-
·7 years agoHey Cole,
Are you having trouble with the newly released Auto Balanced Collectors feature? If so please open a ticket so we can look into it and help.
~Forrest
Cole McDonald
·7 years agoHave you found an efficient way to get a list of devices reporting to a specific Collector? I'm having autobalancer issues and am writing my own as a datasource so I can fire it off once a day. I've got all of the logic solved and am just at the point where I have to get the list of devices reporting to the Collector with the most of them, then randomly select x number of them and assign them to the other collector(s) in the collector group.
Paul_Carroll
OP7 years agoThanks Joe.
Really appreciated
I have added the below which filters by selected collectors and selects the one with the least number of machines
Joe Williams
·7 years agoHere is what I use to "report" on collectors and their versions.
The numberOfHosts would help you out in this case.
Forrest_Evans_-
·7 years agoHi Paul,
We are actively working on an enhancement to LogicMonitor to support Load Balanced Collectors. One of the key features is that when adding a device you can pick a Collector Group instead of an individual collector. More to come in the next month or two!
~Forrest