Forum Discussion

Dave_Lee's avatar
Dave_Lee
Icon for Advisor rankAdvisor
6 months ago

Extracting datapoints via API

Hi all

I wonder if anyone else has found a good way to pull metric data out of LM into other systems?  I've been having a play and have something working.  My code is doing this:

  • Query the API for any devices where auto.wmi.operational=true
  • loop through the devices and for each:
    • make an API call to get the datasource id for "WinVolumeUsage" under this device
    • make an API call to get the instances under this datasource
    • loop through all the instances found
      • make an API call to get the datapoints for each instance for the last few hours
      • write that data out to an object so it can be later written to another database

It does work, but it's fairly slow (a few seconds per device) as it's having to make multiple API calls for every server it finds.

I had a look at using a report to do this, but it seems reports have a 500 row limit and the dataset I'm looking for is more like tens of thousands.  I'm also not sure if I could pull report data through the API.

Anyway doing this kind of thing successfully?

thanks

Dave

  • Dave_Lee's avatar
    Dave_Lee
    5 months ago

    Using a report worked for me.  I create 3 different Resource Metric Trend reports for CPU, RAM, disk utilisation.  Scheduled them to run daily to generate a CSV report.  I've got about 600 servers in there at the moment, so was a few thousand rows in the Volumes report, which seems to run fine as a scheduled report and only seems to take a minute or two to run.

    I have my script call the /report/links/{reportId} endpoint, then download the most recent CSV and process that data.  This is far quicker than iterating through all the API calls needed to gather the data.

    The most granular you can go is 2 hours of data, so I'm able to get the average for each data point over the 2 hours before the report was run.

    Report frequency is 1 day at the most, but I found that I could reschedule the report from the API as well.  So, if I have the script run just after the report, it could get the average for the last 2 hours then the script can reschedule the report to run again in 2 hours.

  • Anonymous's avatar
    Anonymous
    5 months ago

    You could cache the device/instance list and only refresh it every hour or day since it's not likely to change that much minute to minute. That would save you a little, not much, time.

    You could make a dashboard widget and pull the data from there. That would give you high resolution data, based on the most recent poll by LM. /dashboard/widgets/{id}/data.

    You could schedule a report then fetch the data from that report via API. That might be the fastest. You'd have to know the report ID, which wouldn't change. Then you'd have to call an undocumented API endpoint "/report/links/{reportId}" which will give you the list of links pointing to the data. Figure out which one is the most recent and pull it using the link. It wouldn't be high resolution data.

    • Dave_Lee's avatar
      Dave_Lee
      Icon for Advisor rankAdvisor
      5 months ago

      Using a report worked for me.  I create 3 different Resource Metric Trend reports for CPU, RAM, disk utilisation.  Scheduled them to run daily to generate a CSV report.  I've got about 600 servers in there at the moment, so was a few thousand rows in the Volumes report, which seems to run fine as a scheduled report and only seems to take a minute or two to run.

      I have my script call the /report/links/{reportId} endpoint, then download the most recent CSV and process that data.  This is far quicker than iterating through all the API calls needed to gather the data.

      The most granular you can go is 2 hours of data, so I'm able to get the average for each data point over the 2 hours before the report was run.

      Report frequency is 1 day at the most, but I found that I could reschedule the report from the API as well.  So, if I have the script run just after the report, it could get the average for the last 2 hours then the script can reschedule the report to run again in 2 hours.

  • Dave_Lee's avatar
    Dave_Lee
    Icon for Advisor rankAdvisor
    5 months ago

    I had a play with using a report to pull data, but LM refused to run it as there would be >3200 instances returned and there seems to be a maximum of 500 instances in a report.

    I could look at a dashboard to display the most recent data, but don't those have a maximum of 500 rows as well?

    • Anonymous's avatar
      Anonymous
      5 months ago

      Check with support to make sure that limitation exists in the report. I didn't know that was there. That limit might only be in adhoc (run now) reports, so scheduling the report may be a way to get around it. 

    • Shack's avatar
      Shack
      Icon for Advisor rankAdvisor
      5 months ago

      I think the limitations vary by widget.