Forum Discussion

mpunderw's avatar
mpunderw
Icon for Neophyte rankNeophyte
3 months ago

Collector Service Account Locked Out

We have 5 different AD domains that we want to monitor. We created dynamic groups for each domain and added the associated credentials for that domain.

For some reason after we added more than one domain, the original collector account started getting locked out. We reset the password at every spot we can find and even changed it on the service on each collector, and still the account gets locked out.

We put in a ticket but support says it must be set wrong somewhere, which we'd agree with but where? Our logs show it is getting locked on the collectors, sometimes the same one sometimes other. As soon as we unlock the account we can restart the services. 

12 Replies

  • mpunderw's avatar
    mpunderw
    Icon for Neophyte rankNeophyte
    3 months ago

    And to add, once we unlock the account, there are a large number of devices that don't start monitoring again and the only way to get them working is to remove them and re-add them. 

    • Shack's avatar
      Shack
      Icon for Advisor rankAdvisor
      3 months ago

      Do you need your collector to be running with the service accounts?  I'd suggest switching them to run as system and then just control wmi.user and wmi.pass on each dynamic group.   

      I'd create a report that includes all of the Windows devices and add properties to the report for wmi.user and wmi.pass.  Also include the collector properties - system.prefcollectorid, system.prefcollectordesc.  This will at least let you see where the creds are set.  

      • mpunderw's avatar
        mpunderw
        Icon for Neophyte rankNeophyte
        3 months ago

        That's how the collectors installed themselves, that wasn't a choice we made. What else is that runas account used for and what would stop working if we changed it to System?

        I don't think we knew you could create reports. How does one go about that?

  • Dave_Lee's avatar
    Dave_Lee
    Icon for Advisor rankAdvisor
    3 months ago

    I don't know if this applies in your situation, but we chased our tails for a while over constant account lockouts.  It turns out that some special chars (actually, currency symbols) in the password we were using didn't play nice with some of our PowerShell based modules.  Once we changed the password to remove the currency symbols, our problems went away.  MS apparently doesn't consider currency symbols to be "special" for the purposes of satisfying password complexity policies anyway.

    • mpunderw's avatar
      mpunderw
      Icon for Neophyte rankNeophyte
      3 months ago

      WinRM and the password works fine outside of LogicMonitor, in the application is the only place that it fails. 

  • Cole_McDonald's avatar
    Cole_McDonald
    Icon for Professor rankProfessor
    3 months ago

    On either the DC or the LM collector, in the event log you're looking for Security:4625 and Security:4740 events.

    These will contain a Status and substatus that will tell you what specifically is happening.  The 4625 for the service account that occurs just before the locked is the reason it's locking out.

    4624 - Auth Success
    4625 - Auth Failure
    4740 - Lockout
    4767 - Unlock
    4624 - Logon
    4634 - Logoff
     
    # Failed Logon Event Codes - 4625 - https://system32.eventsentry.com/codes/field/Netlogon%20Error%20Codes
    0xC000005E There are currently no logon servers available to service the logon request.
    0xC0000064 user name does not exist
    0xC000006A user name is correct but the password is wrong
    0xC0000234 user is currently locked out
    0xC0000072 account is currently disabled
    0xC000006F user tried to logon outside his day of week or time of day restrictions
    0xC0000070 workstation restriction, or Authentication Policy Silo violation (look for event ID 4820 on domain controller)
    0xC0000193 account expiration
    0xC0000071 expired password
    0xC0000133 clocks between DC and other computer too far out of sync
    0xC0000224 user is required to change password at next logon
    0xC0000225 evidently a bug in Windows and not a risk
    0xc000015B The user has not been granted the requested logon type (aka logon right) at this machine
    0xC00000DC  Indicates the Sam Server was in the wrong state to perform the desired operation.
    0xC000018D STATUS_TRUSTED_RELATIONSHIP_FAILURE

     

    • mpunderw's avatar
      mpunderw
      Icon for Neophyte rankNeophyte
      3 months ago

      It's getting locked out because bad username or password, and it's coming ONLY from one of the collectors. Once we unlock the account it will work fine for hours, days, or weeks. Then out of the blue, it just starts failing with bad password until the account is locked. The only way we've been able to keep LM running is to setup a cron job that checks if the account is locked out every 5 seconds and unlocks it if it is, and it has worked for several weeks now. 

      • Cole_McDonald's avatar
        Cole_McDonald
        Icon for Professor rankProfessor
        3 months ago

        I'm assuming the creds are stored as wmi.user and wmi.pass.  Sounds like some dataSource isn't respecting those are set and using the account the collector is set to.  Tracking down that culprit takes some effort going through DataSource code and evaluating how each is utilizing the credentials or not.

        My first stab at that would be to pull all of the 'Sources from the RestAPI using Powershell into an array.  Then Filtering for any script that doesn't -match the string "wmi.user"

        Alternatively, if you can match it to a specific PID (usually needs conversion from Hex in the event logs), get-wmiobject win32_process | where processid -eq <PID> might get you at the commandline property to see which script is being called to be able to more positively match it, but that's got a short lifespan.

        Tying a lockout cadence to it could allow you to sort the dataSources by run frequency to help narrow it down as well.

  • Cole_McDonald's avatar
    Cole_McDonald
    Icon for Professor rankProfessor
    2 months ago

    With a list of the 4625 events in the Windows security log, check the amount of time between each 4625.  See if you can workout the pattern there.  Then you can check any 'Sources in the Modules section that are set to collect at that specific frequency... in LM for that collector, open a debug console and do a !tlist.  Check for anything that seems to be failing there.  I also recall a Collector Debug/Status report in the Settings > Collectors page that had helped me ID failing modules at some point.