Dell ECS Flux API DataSource

I am currently working on replacing the DataSource elements for CPU and Memory for the Dell ECS S3 Platform we use, but have hit a problem in building the JSON I can get this to work in Python, but I am really struggling in Groovy where I get 400 Bad request where I see a Syntax as the problem.

Here is the JSON I am sending as I would write it in python and working:

sendData = {"query":"from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \""+hostname+"\")"}

Here is it in Groovy format which has been through a few iterations.

dataString = ["query":'from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \"'+hostname+'\")']

The actual code is here it is a copy of one of the original ECS Data Sources then adapting the getApi element to use the flux API as opposed to the dashboard API elements.

import groovy.json.JsonSlurper
import groovy.json.JsonOutput
import groovy.json.JsonBuilder
import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit
import com.santaba.agent.groovyapi.http.*;
import com.santaba.agent.util.Settings

hostname = [Removed]
user = [Removed]
pass = [Removed]
collectorplatform = "linux"

// Temp lines
println "Successfully read params"
def nodeid = [Removed]
// Temp lines end
if (nodeid == null) {
    println nodeid
    println "auto.emc.ecs.node.nodeid is not set! Please see Technical Notes."
    return 1
}

debug = true

def success = false

def token = login()
//Temp Lines
println "out of login"
println token
// End Templines
if (token) {
    def response = getApi(token)
    println "Response below"
    println response
    } else if (debug) { println "Bad API response: ${response}"}

return success ? 0 : 1

def login() {
    println "in login"
    File tokenCacheFile
    if (collectorplatform == 'windows') tokenCacheFile = new File("emc_ecs_tokens" + '\\' +  hostname + "-emc_ecs_tokens.txt") 
    if (collectorplatform == 'linux') tokenCacheFile = new File("emc_ecs_tokens" + '/' + hostname + "-emc_ecs_tokens.txt") 
    if (debug) println "Token cache filename is: ${tokenCacheFile}"

    // If we have a non-empty readable token cache file, then extract and return the token
    if (tokenCacheFile.exists() && tokenCacheFile.canRead() && tokenCacheFile.readLines().size() == 1 && !tokenCacheFile.readLines()[0].contains("null")) {
        if (debug) println "Token cache file exists and is non-empty"
        def cachedToken = tokenCacheFile.readLines()[0]
        if (cachedToken) {
            if (debug) println "Extracted token from cache file: ${cachedToken}"
            return cachedToken
        }
    } else if (!tokenCacheFile.exists()) { // token cache file does not exist, create it
        if (debug) println "Token cache file does not exist, creating..."
        new File("emc_ecs_tokens").mkdir()
        tokenCacheFile.createNewFile()
    } else if (tokenCacheFile.text != '') { // malformed token cache file
        println "Bad token file: ${tokenCacheFile.readLines()}\nClearing..."
        tokenCacheFile.text = ''
    } else if (debug && tokenCacheFile.text == '') { // token cache file has been cleared, proceed and rebuild
        println "Session token file is cleared. Rebuilding..."
    }

    // Fetch new token using Basic authentication, set in cache file and return
    if (debug) println "Checking provided ${user} creds at /login.json..."
    def userCredentials = "${user}:${pass}"
    def basicAuthStringEnc = new String(Base64.getEncoder().encode(userCredentials.getBytes()))
    def loginUrl = "https://${hostname}:4443/login.json".toURL()
    def loginConnection = loginUrl.openConnection()
    loginConnection.setRequestProperty("Authorization", "Basic " + basicAuthStringEnc)
    def loginResponseBody = loginConnection.getInputStream()?.text
    def loginResponseCode = loginConnection.getResponseCode()
    def loginResponseToken = loginConnection.getHeaderField("X-SDS-AUTH-TOKEN")
    println loginResponseCode
    if (loginResponseCode == 200 && loginResponseToken) {
        if (debug) println "Retrieved token: ${loginResponseToken}"
        tokenCacheFile << loginResponseToken
        if (debug) println "Set token in cache file"
        return loginResponseToken
    } else {
        println "STATUS CODE:\n${loginResponseCode}\n\nRESPONSE:\n${loginResponseBody}"
        println "Unable to fetch token with ${user} creds at /login.json"
    }
    println "Something unknown went wrong when logging in"
}

def getApi(token, alreadyFailed=false) {
    def dataUrl = "https://"+hostname+":4443/flux/api/external/v2/query";
    
    if (debug) println "Trying to fetch data from ${dataUrl}..."

    //def GetStatus = JsonOutput.toJson([query:'from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \"${hostname}\")'])
    dataString = ["query":'from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \"'+hostname+'\")']
    def GetStatus = JsonOutput.toJson(dataString)

    println GetStatus
    println "GetStatus Class"
    println GetStatus.getClass()
    def dataHeader = ['X-SDS-AUTH-TOKEN':token,'Content-Type':'application/json','accept':'application/json']
    // Now we can retrieve the data.
    httpClient = Client.open (hostname,443)

    def dataData = httpClient.post(dataUrl,GetStatus,dataHeader);

    if ( !(httpClient.getStatusCode() =~ /200/))
    {
        println "Failed to retrieve data "+httpClient.getStatusCode
        return(1)
    }
    String dataContent = httpClient.getResponseBody()
    println httpClient.getStatusCode
    println dataContent
    return new JsonSlurper().parseText(dataContent)   
}

Thanks in advance for any help.

SteveBamford

Posted 9 months ago·Last reply 8 months ago

4 comments

SteveBamford

OP8 months ago

Ok after weeks of thinking this was the JSON that was the problem it would appear it is the Token being saved to the file which was in the original Dell ECS Dashboard Queries, this was the only thing I wasn't doing in Python, I have stripped it out and I now have some working code that works for the CPU, I will build out the specific real data sources and post them in the community for people to use if they wish. I don't understand why it wasn't returning a HTTP 401 error code as opposed to a 400.

My head hurts now 😄

SteveBamford

OP8 months ago

Just an Update, I changed collector and got a 400 response which still returned the data I have a case open with Dell as this definitely seems to be device related and I have also double checked my logic against other similar data sources and don't have this issue.

I am yet to hear anything from Dell

Michael Rodrigues

·9 months ago

SteveBamford I think it's just an escaping issue. Try this:

def flux = "from(bucket: \"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \"${hostname}\")"
def jsonBody = groovy.json.JsonOutput.toJson([query: flux])

SteveBamford

OP8 months ago

Hi Mike_Rodrigues so I implemented as you suggested and it is still failing I did need to make a slight amendment for the hostname to be picked up but that was me altering the code before posting here

def flux = "from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \""+hostname+"\")"

def jsonBody = JsonOutput.toJson([query: flux])

When I look at the output I can see it is being formatted the same as the python working query in JSON Body and yet I still get bad syntax.

Output:
Json Body {"query":"from(bucket:\"monitoring_op\") |> range(start: -5m) |> filter(fn: (r) => r._measurement == \"cpu\" and r.cpu == \"cpu-total\" and r._field == \"usage_idle\" and r.host == \[REMOVED]\")"} Type class java.lang.String

Response
"Failed to retrieve data HTTP/1.1 400 Bad Request"
"he request cannot be fulfilled due to bad syntax."

I even validated against the JSON Example in the documentation as well, I am going to raise a request with the vendor and request the logs from the device incase they tell me anything. There is little difference in the Python code to the groovy code, both are string type as well, the method I have used on other API calls for other vendors which I know isn't a guarantee in its self, but should be for what is quite a simple request.

Any other suggestions gratefully appreciated.