Collector Service Account Locked Out
We have 5 different AD domains that we want to monitor. We created dynamic groups for each domain and added the associated credentials for that domain.
For some reason after we added more than one domain, the original collector account started getting locked out. We reset the password at every spot we can find and even changed it on the service on each collector, and still the account gets locked out.
We put in a ticket but support says it must be set wrong somewhere, which we'd agree with but where? Our logs show it is getting locked on the collectors, sometimes the same one sometimes other. As soon as we unlock the account we can restart the services.
mpunderw
Posted 1 year ago·Last reply 1 year ago
12 comments
Cole McDonald
·1 year agoWith a list of the 4625 events in the Windows security log, check the amount of time between each 4625. See if you can workout the pattern there. Then you can check any 'Sources in the Modules section that are set to collect at that specific frequency... in LM for that collector, open a debug console and do a !tlist. Check for anything that seems to be failing there. I also recall a Collector Debug/Status report in the Settings > Collectors page that had helped me ID failing modules at some point.
Cole McDonald
·1 year agoOn either the DC or the LM collector, in the event log you're looking for Security:4625 and Security:4740 events.
These will contain a Status and substatus that will tell you what specifically is happening. The 4625 for the service account that occurs just before the locked is the reason it's locking out.
mpunderw
OP1 year agoIt's getting locked out because bad username or password, and it's coming ONLY from one of the collectors. Once we unlock the account it will work fine for hours, days, or weeks. Then out of the blue, it just starts failing with bad password until the account is locked. The only way we've been able to keep LM running is to setup a cron job that checks if the account is locked out every 5 seconds and unlocks it if it is, and it has worked for several weeks now.
Cole McDonald
·1 year agoI'm assuming the creds are stored as wmi.user and wmi.pass. Sounds like some dataSource isn't respecting those are set and using the account the collector is set to. Tracking down that culprit takes some effort going through DataSource code and evaluating how each is utilizing the credentials or not.
My first stab at that would be to pull all of the 'Sources from the RestAPI using Powershell into an array. Then Filtering for any script that doesn't -match the string "wmi.user"
Alternatively, if you can match it to a specific PID (usually needs conversion from Hex in the event logs), get-wmiobject win32_process | where processid -eq <PID> might get you at the commandline property to see which script is being called to be able to more positively match it, but that's got a short lifespan.
Tying a lockout cadence to it could allow you to sort the dataSources by run frequency to help narrow it down as well.
mpunderw
OP1 year agothat's sounds awesome, how do you do any of that?
mpunderw
OP1 year agoAnd the service account the collector is running with is the same account and password used for wmi.user/pass and winrm.user/pass. Even if it ignores what properties we set, it should still be good since we're using the same account for everything.
mpunderw
OP1 year agoLM support recommended to use a separate account for wmi and winrm. We setup two new accounts, updated the attributes, and restarted the collectors. In under a minute both accounts were locked out. All coming from one of the log collectors.
We asked about removing it and rebuilding it, and we were told that all configuration data is stored in LM so we would just run into the same problem.
Dave Lee
·1 year agoI don't know if this applies in your situation, but we chased our tails for a while over constant account lockouts. It turns out that some special chars (actually, currency symbols) in the password we were using didn't play nice with some of our PowerShell based modules. Once we changed the password to remove the currency symbols, our problems went away. MS apparently doesn't consider currency symbols to be "special" for the purposes of satisfying password complexity policies anyway.
mpunderw
OP1 year agoWinRM and the password works fine outside of LogicMonitor, in the application is the only place that it fails.
mpunderw
OP1 year agoAnd to add, once we unlock the account, there are a large number of devices that don't start monitoring again and the only way to get them working is to remove them and re-add them.
Shack
·1 year agoDo you need your collector to be running with the service accounts? I'd suggest switching them to run as system and then just control wmi.user and wmi.pass on each dynamic group.
I'd create a report that includes all of the Windows devices and add properties to the report for wmi.user and wmi.pass. Also include the collector properties - system.prefcollectorid, system.prefcollectordesc. This will at least let you see where the creds are set.
mpunderw
OP1 year agoThat's how the collectors installed themselves, that wasn't a choice we made. What else is that runas account used for and what would stop working if we changed it to System?
I don't think we knew you could create reports. How does one go about that?