REST API 503 errors
Hello everyone
We have been using various PowerShell scripts for years and have never had any problems with them. Now, since last week Monday, we are getting random 503 “server is busy” HTTP errors on some GET queries such as Devices, Device DataSources, Dashboards, etc.
There is no recognizable pattern to the errors.
Does this problem also occur for others using script automation via the REST API (occurs in V2 and V3)?
I have submitted a support request to LM, but progress is slow...
Have a nice day everyone
Dorian
Dorian Fuchs
Posted 2 years ago·Last reply 2 years ago
14 comments
Dorian Fuchs
OP2 years agoHello everyone
I have now received feedback from LM that on March 26, 2024 auto throttling for external API calls was activated and therefore the 503 HTTP errors are coming (nowhere mentioned in the releases note, etc.).
The feedback from LM on my question how we can handle such errors in the future was simply "script a catch and try again after a certain time".
What do you think about this? Is this our problem now if the API has resource bottlenecks? Is LM leaving us to our fate here? Should LM increase the resources of the API?
In my opinion, a 429 HTTP error would be the right "procedure".
1. for 429 errors, the person who makes a request can do something about it AND also do something against it.
2. in the case of 503 errors, the person making the request can usually do nothing about it if there is a high load on the API AND also do nothing against it.
Have a nice weekend soon
Dorian
LM User
·2 years agoAuto throttling?
What Joe_Williams posted earlier is what they mean by this.
A 429 would be the right thing to do as it explicitly means you've hit the rate limit. Sending back a 503 indicates that the server is having issues, not that there's anything wrong with our request.
Mike Moniz
·2 years agoLM has always had rate limiting and it's documented at https://www.logicmonitor.com/support/rest-api-developers-guide/overview/rest-api-rate-limit. Is this something different or is it no longer doing 429 anymore?
Justin Dietz
·2 years agoI don' t think that would give a 503. If my API has more than the limit (1000 results) it simply returns just the first 1000. I get around this w/ a DO While loop if i need all results
Joe Williams
·2 years agoThat is not the limit I was referring to. Each api endpoint has a defined number of calls allowed in a defined period of time.
Here is an example of my how my python function evaluates the api limits.
LM User
·2 years agoHow long until they build pagination into the SDK?
Joe Williams
·2 years agoHonestly, I don't use the SDK as I always had odd results on items returning differently sometimes. So I wrote my own Python function that handles pagination, api limitations, etc and returns it always in the same manner.
LM User
·2 years agoYes, i have one of those too. But the SDK should include it.
The problems you've had have been around filtering and returning specific columns, right? I've also had weird situations where a call shouldn't require a value but it does. Like if you're patching the display name, you also have to include a couple other fields. The code under the hood of the SDK doesn't handle it on some methods.
LM, please open source your SDK and put it into github so we can submit pull requests and "make LM friendly to the developers." (your words not mine).
Joe Williams
·2 years agoYeah, filtering was a huge pain. And whatever version I was using awhile ago, would change the returned columns between snake case and something else randomly I believe.
Joe Williams
·2 years agoI guess 1 question is, does your automation/scripts look at the headers to see your api limits? I don't know off the top of my head if when you hit your call limit, if it returns a 503, but you might be querying a little to fast and hitting the api limit.
LM User
·2 years agoI think that would be a 429, not a 503. 400-499 means you did something wrong. 500-599 means the server did something wrong.
Joe Williams
·2 years agoYeah, I wasn't sure what the status code would be as I have always coded around it. But this is LogicMonitor, I love it, use it, praise it, but it is LogicMonitor and logic doesn't always play a role ;)
Justin Dietz
·2 years agoi have several powershell scripts that run daily and no errors for me, the main ones backing up all my logicmodules
LM User
·2 years agoI run a lot of automation and haven't had any errors crop up.