Forum Discussion

Antony_Hawkins's avatar
8 years ago

DataSources_List PropertySource

(Edit, October 2020 - see later comment below for a version that does all this without needing an API call)

Q: What does it do?

A: Calls the LogicMonitor API and finds, for each applicable device, a list of DataSource Names that exist on the Device, i.e. have at least one instance. This also includes ConfigSources active on the Device. These are put together as a semicolon-seperated list in the property auto.activedatasources:

c79ef7414884f314cf54c8317eb8bd77.png

Q: Why? I can just look at the Device in the Device tree.

A: You can, but... This PropertySource means you can create Dynamic Groups and Device Inventory Reports based on the presence or absence of *any* combination of DataSources on Devices.

Q: Can you give me a real-world example of this being useful?

A: Yes. On a Microsoft SQL server, LogicMonitor will find the SQL Services via WMI, but the actual SQL Server metrics are found via Perfmon or JDBC methods. If we can't access Perfmon or JDBC for any reason (required services not running, lack of credentials, etc) we'll be missing the SQL stats but quite probably getting all the other Windows metrics (via WMI). This can be very difficult to spot if you have one affected SQL server out of 1,000.

With this PropertySource running, you could create a Dynamic Group with a custom query of:

auto.ActiveDataSources =~ "WinSQLServices-" && auto.ActiveDataSources !~ "WinSQLServer-" && auto.ActiveDataSources !~ "WinSQLServer_JDBC-"

This will list all the devices on which we've found SQL Services but have failed to find any SQL metrics.

Similarly Exchange Services are discovered via WMI, whilst most Exchange metrics come from Perfmon.

Or, maybe you want to create a Group to contain all your Linux servers that are running Apache web server and MySQL:

isLinux() && auto.ActiveDataSources =~ "Apache-" && auto.ActiveDataSources =~ "Mysql-"

You will need:

A REST API Access ID and Access Key for a user with permissions to at least view all Devices within the account (or all Devices in whichever Groups you wish to consider). These should be set as properties 'apiaccessid.key' and 'apiaccesskey.key' within the Devices tree (generally the root level, or a Group level if appropriate).

The PropertySource will use these to find the list of DataSources present on the Device, and create the auto.activedatasources property for all devices.

auto.activedatasources will have one of two error messages if the required API details are not present or do not give sufficient access to query the Device; this could of course also be used to create Dynamic Groups and in Device Inventory Reports.

Note that DataSources are listed by their (unique) 'Name' rather than their (not necessarily unique) 'Displayed As' string seen in the Device tree.

v1.0.0: ZLTDWM

Current limitations:

The auto.property fields are limited to 2,000 characters, so if the list of DataSource Names exceeds this size, the data will be truncated. This is likely to be an edge case only seen on test Devices with multiple custom DataSources applied.

Auto Properties are updated daily or when Active Discovery is run for a Device, so there may (will!) be a time lag between DataSources being applied to a Device and those DataSources appearing in this auto.activedatasources property list.

  • On 8/26/2021 at 11:56 AM, Antony Hawkins said:

    @Brandon Sellers um... no idea on the cause, in terms of a first guess at least. I'm more than happy to take a look, calendar gaps permitting (pretty hectic/full currently, but will see what I can do), if you can point me at a working and a non-working example in your portal (PM me the details).

    Based on an email exchange with support today it sounds like this won't work in the new version if executed from within SSE, it must be executed by Agent in this new non-API flavor.  Perhaps his Medium collector wasn't using SSE?

  • @Antony Hawkins i've installed your property source 2.1.0 yesterday. Although i can get it to run using the Test Script button within the PropertySource itself, and it returns data, it doesn't write any values to resources\devices. I've run an active discovery from a device, several times, and run a Force device match but nothing. Any ideas? is there anything i need to do other than install it? I've left the default Applies To - system.collectorid != "-2", does this need changing?

  • Anonymous's avatar
    Anonymous

    When you click the "test applies to" button, are all your resources matched? They should be. 

  • 13 hours ago, Stuart Weenig said:

    When you click the "test applies to" button, are all your resources matched? They should be. 

    Yes

  • 2 hours ago, Slim said:

    Yes

    Ok, sorted it out. Had to change the collector config from SSE to Agent based. Another question, What's the implication of changing from SSE to Agent?

  • Anonymous's avatar
    Anonymous

    Try here. I don't have much practical experience with it myself.

  • I also can't give a definitive answer on the implications, I'm afraid.

    I suspect, if all the scripts being run by the collector are nice and clean and elegant and graceful in their failures, very little. I think the SSE was implemented to protect the collector operation in the event of a module script being... less than all of those things.

  • From my past conversations with support, the only real difference was that it provides a dedicated execution space (JVM) for script execution with dedicated memory.   So if you uses Agent your script/batchscript/etc run inside Agent memory space and have the possibility to exhaust Agent memory or terminate Agent if your scripts do not exit properly;  with SSE you can only impact the SSE process itself.   That being said, SSE didn't exist for many years and all scripts ran in Agent, so keeping it that way isn't a big deal really, especially if you don't do a lot of LogicModule dev.  SSE only kicks on if there is an available memory of 50% of the RAM allocated to Agent anyway (meaning if you configure a collector that uses 8GB for Agent, it would want 4GB for SSE) and if that's not available it will suspend usage of SSE anyhow.  Also, because of that memory allocation logic, you'll find the docs recommend not using SSE with many virtual-machine collectors due to how they report RAM to the underlying guest OS (see the docs).  We have a hodge-podge of SSE and non-SSE collectors at various clients and there's been no discernible difference operationally.  We have this community DS that's the subject of this thread running on a dozen non-SSE linux VM collectors and it's had no issues.

  • ...and indeed, having dug into the logs, Dev agree this is SSE:

    Quote

     

    The instance of of CollectorDb when accessed through the Agent and SSE is different. When accessed through the SSE, the instance of CollectorDb has less data access - which is by design, considering the memory footprint. This has nothing to do we collector size.

    The workaround for this to change the groovy.script.runner to agent from sse in the agent.conf. But this needs to be done with a bit caution. If the load on the collector is high and then we change from sse to agent - then we might face memory bloating hampering agent's working.