Forum Discussion

dorian_fuchs's avatar
dorian_fuchs
Icon for Neophyte rankNeophyte
12 days ago

REST API 503 errors

Hello everyone

We have been using various PowerShell scripts for years and have never had any problems with them. Now, since last week Monday, we are getting random 503 “server is busy” HTTP errors on some GET queries such as Devices, Device DataSources, Dashboards, etc.
There is no recognizable pattern to the errors.
Does this problem also occur for others using script automation via the REST API (occurs in V2 and V3)?

I have submitted a support request to LM, but progress is slow...

Have a nice day everyone
Dorian

rest api

15 Replies

Sort By
  • Joe_Williams's avatar
    Joe_Williams
    Icon for Expert rankExpert
    11 days ago

    I guess 1 question is, does your automation/scripts look at the headers to see your api limits? I don't know off the top of my head if when you hit your call limit, if it returns a 503, but you might be querying a little to fast and hitting the api limit.

    • Stuart_Weenig's avatar
      Stuart_Weenig
      Icon for Legend rankLegend
      11 days ago

      I think that would be a 429, not a 503. 400-499 means you did something wrong. 500-599 means the server did something wrong.

      • Joe_Williams's avatar
        Joe_Williams
        Icon for Expert rankExpert
        11 days ago

        Yeah, I wasn't sure what the status code would be as I have always coded around it. But this is LogicMonitor, I love it, use it, praise it, but it is LogicMonitor and logic doesn't always play a role ;)

  • thedietz's avatar
    thedietz
    Icon for Neophyte rankNeophyte
    12 days ago

    i have several powershell scripts that run daily and no errors for me, the main ones backing up all my logicmodules

  • thedietz's avatar
    thedietz
    Icon for Neophyte rankNeophyte
    11 days ago

    I don' t think that would give a 503.   If my API has more than the limit (1000 results) it simply returns just the first 1000.   I get around this w/ a DO While loop if i need all results

    • Joe_Williams's avatar
      Joe_Williams
      Icon for Expert rankExpert
      11 days ago

      That is not the limit I was referring to. Each api endpoint has a defined number of calls allowed in a defined period of time. 

      Here is an example of my how my python function evaluates the api limits.

      returned_headers = response.headers
api_limit = int(returned_headers['x-rate-limit-limit'])
api_left = int(returned_headers['x-rate-limit-remaining'])
api_threshold = api_limit - api_left
if api_threshold > 5:
	time.sleep(int(returned_headers['x-rate-limit-window']))

       

  • dorian_fuchs's avatar
    dorian_fuchs
    Icon for Neophyte rankNeophyte
    3 days ago

    Hello everyone

    I have now received feedback from LM that on March 26, 2024 auto throttling for external API calls was activated and therefore the 503 HTTP errors are coming (nowhere mentioned in the releases note, etc.).
    The feedback from LM on my question how we can handle such errors in the future was simply "script a catch and try again after a certain time".

    What do you think about this? Is this our problem now if the API has resource bottlenecks? Is LM leaving us to our fate here? Should LM increase the resources of the API?
    In my opinion, a 429 HTTP error would be the right "procedure".
    1. for 429 errors, the person who makes a request can do something about it AND also do something against it.
    2. in the case of 503 errors, the person making the request can usually do nothing about it if there is a high load on the API AND also do nothing against it.

    Have a nice weekend soon
    Dorian