Forum Discussion

Cole_McDonald's avatar
Cole_McDonald
Icon for Professor rankProfessor
30 days ago
Solved

Deep Dive troubleshooting question

I'm having difficulties tracking down the source of a Service Account Lockout occuring in a fairly complex domain structure.

I've found it necessary to set WMI creds at the top of the heirarchy, with sub-groups having different WMI creds, as well as individual devices with their own.

On the domain controller that would be targeted with a sub-group WMI cred set, I see most 'Sources collecting correctly, but a few times a day, the SA is locking out.  I find the 4740 on the DC, but can't quite track down the specific event (should be a 4625) causing the lockout.  The only 4625 I'm finding show the Main group WMI creds... which are in a different domain.

Seems as though some 'Sources are using the wrong WMI creds (or defaults based on the Collector's SA).

The question: Is there a way to get a !TLIST showing the WMI.USER being leveraged in the debug console for each job?

  • We struggled with something similar recently... In our case, we had a single WMI account across all hosts in a group and we knew it was correct, but we would see accounts logouts happening almost constantly.

    We thought the same as you - that there is a credential cached somewhere, or a credential set wrong on a host or a credential hard coded somewhere.  We didn't come across a way of troubleshooting it, one of my colleagues did find the root cause though.

    In our case, the issue was due to currency chars in the password - specifically the $ sign.   This was causing some of our PowerShell based property/data sources to do handle parts of the password as if we wanted it to do string interpolation.  Of course, the variable it was replacing portions of our password with didn't exist, so it had the effect of removing a portion of the password at runtime.  We were using double quotes rather than single quotes in a PowerShell script.

    Dave

6 Replies

  • I'm not sure about !TLIST and such, but I believe most, if not all, WMI requests are actually run from PowerShell (or atleast it use to) by the Collector. Perhaps it's worth enabling scriptblock logging in PowerShell on the Collector(s) so you can audit the commands that PS are running. Then check if you can see the WMI requests commands themselves from there.

  • We struggled with something similar recently... In our case, we had a single WMI account across all hosts in a group and we knew it was correct, but we would see accounts logouts happening almost constantly.

    We thought the same as you - that there is a credential cached somewhere, or a credential set wrong on a host or a credential hard coded somewhere.  We didn't come across a way of troubleshooting it, one of my colleagues did find the root cause though.

    In our case, the issue was due to currency chars in the password - specifically the $ sign.   This was causing some of our PowerShell based property/data sources to do handle parts of the password as if we wanted it to do string interpolation.  Of course, the variable it was replacing portions of our password with didn't exist, so it had the effect of removing a portion of the password at runtime.  We were using double quotes rather than single quotes in a PowerShell script.

    Dave

    • Cole_McDonald's avatar
      Cole_McDonald
      Icon for Professor rankProfessor

      There was a $... I changed all my built-in var grabs to String Literal for the 'Sources I've written.  Hopefully, this is the beast.  Thank you!

      • Dave_Lee's avatar
        Dave_Lee
        Icon for Advisor rankAdvisor

        fun?  At least three of us here had a go at trying to figure it out and failed to spot it.  I dread to think about the number of hours we wasted trying to track down a potential stale password.  At one point I thought it might be specific to a PowerShell module - so I was close!  I didn't spot that we had a module that wasn't using a literal for the password, nor did I spot the password had a $.

        Anyway, you live and learn - and can share the experience so perhaps others don't have to suffer quite as much! 🤣