MTR path monitoring
I have not published this given the security implications running something like this... Would love some help to try reduce the risks
Provided the collector is running Linux (CentOS has only been tested) and the host has mtr installed this will monitor the path out to your LogicMonitor account along with the Latency, the path is stored as a Instance Level property on the device look at the updateState code... Every time this runs it compares the path and the previous path from the instance level property. If the path changes it updates another instance property named alert_message with the old vs new path so you can put it in your favourite tool to see the diff, if you set the datapoints to alert you will get both old and new path's in the alert messages you receive.
You can also map the alerts to spikes in latency ect.
If anyone has a nicer way of managing all this please do!!!
<span class="pun"><?</span><span class="pln">xml version</span><span class="pun">=</span><span class="str">"1.0"</span><span class="pln"> encoding</span><span class="pun">=</span><span class="str">"UTF-8"</span><span class="pln"> </span><span class="pun">?></span><span class="pln">
</span><span class="tag"><feed</span><span class="pln"> </span><span class="atn">version</span><span class="pun">=</span><span class="atv">"1.0"</span><span class="pln"> </span><span class="atn">hasPendingRequests</span><span class="pun">=</span><span class="atv">"false"</span><span class="pln"> </span><span class="tag">></span><span class="pln">
</span><span class="tag"><company></company></span><span class="pln">
</span><span class="tag"><status></span><span class="pln">200</span><span class="tag"></status></span><span class="pln">
</span><span class="tag"><errmsg></span><span class="pln">OK</span><span class="tag"></errmsg></span><span class="pln">
</span><span class="tag"><interval></span><span class="pln">0</span><span class="tag"></interval></span><span class="pln">
</span><span class="tag"><entry</span><span class="pln"> </span><span class="atn">type</span><span class="pun">=</span><span class="atv">"predatasource"</span><span class="tag">></span><span class="pln">
</span><span class="tag"><version></span><span class="pln">1631754840</span><span class="tag"></version></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">MTR to LogicMonitor</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><displayedas></span><span class="pln">MTR to LogicMonitor</span><span class="tag"></displayedas></span><span class="pln">
</span><span class="tag"><description></span><span class="pln">This Datasource will only work on Linux currently tested with CentOS 8 only, if you want to add this to a collecotr you will need to add a property to the instance mtr.monitoring</span><span class="tag"></description></span><span class="pln">
</span><span class="tag"><collector></span><span class="pln">script</span><span class="tag"></collector></span><span class="pln">
</span><span class="tag"><hasMultiInstances></span><span class="pln">false</span><span class="tag"></hasMultiInstances></span><span class="pln">
</span><span class="tag"><schedule></span><span class="pln">180</span><span class="tag"></schedule></span><span class="pln">
</span><span class="tag"><appliesTo></span><span class="pln">isCollectorDevice() && getPropValue("mtr.monitoring")</span><span class="tag"></appliesTo></span><span class="pln">
</span><span class="tag"><wildcardauto></span><span class="pln">false</span><span class="tag"></wildcardauto></span><span class="pln">
</span><span class="tag"><wildcardpersist></span><span class="pln">false</span><span class="tag"></wildcardpersist></span><span class="pln">
</span><span class="tag"><wildcardlinuxscript></wildcardlinuxscript></span><span class="pln">
</span><span class="tag"><wildcardlinuxcmdline></wildcardlinuxcmdline></span><span class="pln">
</span><span class="tag"><wildcardwinscript></wildcardwinscript></span><span class="pln">
</span><span class="tag"><wildcardwincmdline></wildcardwincmdline></span><span class="pln">
</span><span class="tag"><wildcardgroovyscript></wildcardgroovyscript></span><span class="pln">
</span><span class="tag"><wildcardschedule></span><span class="pln">1440</span><span class="tag"></wildcardschedule></span><span class="pln">
</span><span class="tag"><wildcarddisable></span><span class="pln">false</span><span class="tag"></wildcarddisable></span><span class="pln">
</span><span class="tag"><wildcarddeleteinactive></span><span class="pln">false</span><span class="tag"></wildcarddeleteinactive></span><span class="pln">
</span><span class="tag"><agdmethod></span><span class="pln">none</span><span class="tag"></agdmethod></span><span class="pln">
</span><span class="tag"><agdparams></agdparams></span><span class="pln">
</span><span class="tag"><group></span><span class="pln">Custom Monitoring</span><span class="tag"></group></span><span class="pln">
</span><span class="tag"><tags></tags></span><span class="pln">
</span><span class="tag"><technology></technology></span><span class="pln">
</span><span class="tag"><adlist></span><span class="pln"><![CDATA[{"agdmethod":"none","agdparams":"","id":0,"filters":[],"params":{}}]]></span><span class="tag"></adlist></span><span class="pln">
</span><span class="tag"><schemaVersion></span><span class="pln">2</span><span class="tag"></schemaVersion></span><span class="pln">
</span><span class="tag"><dataSourceType></span><span class="pln">1</span><span class="tag"></dataSourceType></span><span class="pln">
</span><span class="tag"><attributes></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">scripttype</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></span><span class="pln">embed</span><span class="tag"></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">scriptgroovy</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></span><span class="pln">import com.santaba.agent.groovyapi.expect.Expect;
import com.santaba.agent.groovyapi.snmp.Snmp;
import com.santaba.agent.groovyapi.http.*;
import com.santaba.agent.groovyapi.jmx.*;
import org.xbill.DNS.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import groovy.json.*
import org.apache.commons.codec.binary.Hex
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import com.santaba.agent.util.Settings
import java.security.MessageDigest
import com.santaba.agent.live.LiveHostSet
import org.apache.http.client.methods.*
import org.apache.http.entity.ContentType
import org.apache.http.entity.StringEntity
import org.apache.http.impl.client.CloseableHttpClient
import org.apache.http.impl.client.HttpClients
import org.apache.http.client.config.RequestConfig
import org.apache.http.HttpHost
import org.apache.http.util.EntityUtils
import java.net.URLEncoder
proxy_host = hostProps.get("proxy.host")?hostProps.get("proxy.host"):null
proxy_port = hostProps.get("proxy.port")?hostProps.get("proxy.port"):null
String apiId = hostProps.get("lmaccess.id")?:hostProps.get("apiaccessid.key")
String apiKey = hostProps.get("lmaccess.key")?:hostProps.get("apiaccesskey.key")
def portalName = hostProps.get("lmaccount")?:Settings.getSetting(Settings.AGENT_COMPANY)
if (proxy_host) {
System.getProperties().put("proxySet", "true");
System.getProperties().put("proxyHost", proxy_host);
System.getProperties().put("proxyPort", proxy_port);
}
def currentPath = instanceProps.get('mtr.path')
def address = "${portalName}.logicmonitor.com";
proc = ('mtr --csv ' + address).execute().text;
ArrayList<String> path = new ArrayList<String>();
if(proc)
{
i = 0
proc.eachLine
{
try
{
hops = it.split(',')
if(i != 0) {
path.add(hops[5]);
i += 1;
}
else {
i += 1;
}
}
catch(e)
{
hopSteps += 'Error: ' + e + '\n';
}
}
fullPath = path.join(',')
if(fullPath != currentPath) {
updateState(fullPath, instanceProps.get("system.instanceid"), hostProps.get("system.deviceId"), instanceProps.get("instance"), portalName, apiId, apiKey, 'mtr.path')
updateState("Original Path:${currentPath} New Path: ${fullPath}", instanceProps.get("system.instanceid"), hostProps.get("system.deviceId"), instanceProps.get("instance"), portalName, apiId, apiKey, 'alert_message')
println "path_changed=1"
} else {
println "path_changed=0"
}
icmp_time = ("ping ${address} -c1").execute().text =~ /time=(\d+)/;
println "icmp_time=${icmp_time[0][1]}"
}
else
{
println 'UNKNOWN issue';
}
return 0;
// internal stuff
def updateState(message, instanceId, deviceId, dataSourceName, portalName, apiId, apiKey, field) {
datasourceId = getDataSourceId(deviceId, dataSourceName, portalName, apiId, apiKey)
instance_data = "{\"customProperties\":[{\"name\":\"mtr.path\",\"value\":\"${message}\"}]}"
old_data = apiGetV2(portalName, apiId, apiKey, "/device/devices/${deviceId}/devicedatasources/${datasourceId['ds_id']}/instances/${datasourceId['instance_id']}")
old_data['description'] = message
old_data['customProperties'] << [name: field, value: message]
json_data = groovy.json.JsonOutput.toJson(old_data)
response = rawPutV2("PUT",apiId,apiKey,portalName,"/device/devices/${deviceId}/devicedatasources/${datasourceId['ds_id']}/instances/${datasourceId['instance_id']}","",json_data)
}
def getDataSourceId(deviceId, dataSourceName, portalName,apiId,apiKey) {
//pp(do_request('GET', '/setting/datasources?filter=name:"Crestone custom test"'))
def args = [filter:"dataSourceName:\"${dataSourceName}\""]
data = apiGetV2(portalName, apiId, apiKey, "/device/devices/${deviceId}/devicedatasources", args)
instance_data = apiGetV2(portalName, apiId, apiKey, "/device/devices/${deviceId}/devicedatasources/${data['items'][0]['id']}/instances")
mapping_data = [instance_id: instance_data['items'][0]['id'], ds_id: instance_data['items'][0]['deviceDataSourceId']]
return mapping_data
}
def apiGetV2(portalName, apiId, apiKey, endPoint, Map args=[:]) {
def request = rawGetV2(portalName, apiId, apiKey, endPoint, args)
if (request.getResponseCode() == 200) {
def payload = new JsonSlurper().parseText(request.content.text)
return payload
}
else {
throw new Exception("Server return HTTP code ${request.getResponseCode()}")
}
}
def rawGetV2(portalName, apiId, apiKey, endPoint, Map args=[:]) {
def auth = generateAuth('GET', apiId, apiKey, endPoint)
def headers = ["Authorization": auth, "Content-Type": "application/json", "X-Version":"2", "External-User":"true"]
def url = "https://${portalName}.logicmonitor.com/santaba/rest${endPoint}"
if (args) {
def encodedArgs = []
args.each{ k,v ->
encodedArgs << "${k}=${java.net.URLEncoder.encode(v.toString(), "UTF-8")}"
}
url += "?${encodedArgs.join('&')}"
}
def request = url.toURL().openConnection()
headers.each{ k,v ->
request.addRequestProperty(k, v)
}
return request
}
def rawPutV2(_verb, _accessId, _accessKey, _account, _resourcePath, _queryParameters, _data){
responseDict = [:]
url = 'https://' + _account + '.logicmonitor.com' + '/santaba/rest' + _resourcePath + _queryParameters
StringEntity entity = new StringEntity(_data)
epoch = System.currentTimeMillis()
requestVars = _verb + epoch + _data + _resourcePath
hmac = Mac.getInstance('HmacSHA256')
secret = new SecretKeySpec(_accessKey.getBytes(), 'HmacSHA256')
hmac.init(secret)
hmac_signed = Hex.encodeHexString(hmac.doFinal(requestVars.getBytes()))
signature = hmac_signed.bytes.encodeBase64()
CloseableHttpClient httpclient = HttpClients.createDefault()
http_request = new HttpPut(url)
if (proxy_host) {
RequestConfig requestConfig = RequestConfig.custom()
.setProxy(new HttpHost(proxy_host, new Integer(proxy_port)))
.build();
http_request.setConfig(requestConfig);
}
http_request.addHeader("Authorization", "LMv1 " + _accessId + ":" + signature + ":" + epoch)
http_request.setHeader("X-Version", "2")
http_request.setHeader("Accept", "application/json")
http_request.setHeader("Content-type", "application/json")
http_request.setEntity(entity)
response = httpclient.execute(http_request)
responseBody = EntityUtils.toString(response.getEntity())
code = response.getStatusLine().getStatusCode()
responseDict['code'] = code ?: null
responseDict['body'] = responseBody ?: null
return responseDict
}
static String generateAuth(method,id, key, path) {
Long epoch_time = System.currentTimeMillis()
Mac hmac = Mac.getInstance("HmacSHA256")
hmac.init(new SecretKeySpec(key.getBytes(), "HmacSHA256"))
def signature = Hex.encodeHexString(hmac.doFinal("${method}${epoch_time}${path}".getBytes())).bytes.encodeBase64()
return "LMv1 ${id}:${signature}:${epoch_time}"
}</span><span class="tag"></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">windowsscript</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">linuxscript</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">windowscmdline</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">linuxcmdline</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"><attribute></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">properties</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><value></value></span><span class="pln">
</span><span class="tag"><comment></comment></span><span class="pln">
</span><span class="tag"></attribute></span><span class="pln">
</span><span class="tag"></attributes></span><span class="pln">
</span><span class="tag"><datapoints></span><span class="pln">
</span><span class="tag"><datapoint></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">path_changed</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><dataType></span><span class="pln">7</span><span class="tag"></dataType></span><span class="pln">
</span><span class="tag"><type></span><span class="pln">2</span><span class="tag"></type></span><span class="pln">
</span><span class="tag"><postprocessormethod></span><span class="pln">namevalue</span><span class="tag"></postprocessormethod></span><span class="pln">
</span><span class="tag"><postprocessorparam></span><span class="pln">path_changed</span><span class="tag"></postprocessorparam></span><span class="pln">
</span><span class="tag"><usevalue></span><span class="pln">output</span><span class="tag"></usevalue></span><span class="pln">
</span><span class="tag"><alertexpr></alertexpr></span><span class="pln">
</span><span class="tag"><alertmissing></span><span class="pln">1</span><span class="tag"></alertmissing></span><span class="pln">
</span><span class="tag"><alertsubject></span><span class="pln">MTR path changes to your LogicMonitor account</span><span class="tag"></alertsubject></span><span class="pln">
</span><span class="tag"><alertbody></span><span class="pln">MTR path has changed:
##alert_message##</span><span class="tag"></alertbody></span><span class="pln">
</span><span class="tag"><enableanomalyalertsuppression></enableanomalyalertsuppression></span><span class="pln">
</span><span class="tag"><adadvsettingenabled></span><span class="pln">false</span><span class="tag"></adadvsettingenabled></span><span class="pln">
</span><span class="tag"><warnadadvsetting></warnadadvsetting></span><span class="pln">
</span><span class="tag"><erroradadvsetting></erroradadvsetting></span><span class="pln">
</span><span class="tag"><criticaladadvsetting></criticaladadvsetting></span><span class="pln">
</span><span class="tag"><description></description></span><span class="pln">
</span><span class="tag"><maxvalue></maxvalue></span><span class="pln">
</span><span class="tag"><minvalue></minvalue></span><span class="pln">
</span><span class="tag"><userparam1></userparam1></span><span class="pln">
</span><span class="tag"><userparam2></userparam2></span><span class="pln">
</span><span class="tag"><userparam3></userparam3></span><span class="pln">
</span><span class="tag"><iscomposite></span><span class="pln">false</span><span class="tag"></iscomposite></span><span class="pln">
</span><span class="tag"><rpn></rpn></span><span class="pln">
</span><span class="tag"><alertTransitionIval></span><span class="pln">0</span><span class="tag"></alertTransitionIval></span><span class="pln">
</span><span class="tag"><alertClearTransitionIval></span><span class="pln">0</span><span class="tag"></alertClearTransitionIval></span><span class="pln">
</span><span class="tag"></datapoint></span><span class="pln">
</span><span class="tag"><datapoint></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">icmp_time</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><dataType></span><span class="pln">7</span><span class="tag"></dataType></span><span class="pln">
</span><span class="tag"><type></span><span class="pln">2</span><span class="tag"></type></span><span class="pln">
</span><span class="tag"><postprocessormethod></span><span class="pln">namevalue</span><span class="tag"></postprocessormethod></span><span class="pln">
</span><span class="tag"><postprocessorparam></span><span class="pln">icmp_time</span><span class="tag"></postprocessorparam></span><span class="pln">
</span><span class="tag"><usevalue></span><span class="pln">output</span><span class="tag"></usevalue></span><span class="pln">
</span><span class="tag"><alertexpr></alertexpr></span><span class="pln">
</span><span class="tag"><alertmissing></span><span class="pln">1</span><span class="tag"></alertmissing></span><span class="pln">
</span><span class="tag"><alertsubject></alertsubject></span><span class="pln">
</span><span class="tag"><alertbody></alertbody></span><span class="pln">
</span><span class="tag"><enableanomalyalertsuppression></enableanomalyalertsuppression></span><span class="pln">
</span><span class="tag"><adadvsettingenabled></span><span class="pln">false</span><span class="tag"></adadvsettingenabled></span><span class="pln">
</span><span class="tag"><warnadadvsetting></warnadadvsetting></span><span class="pln">
</span><span class="tag"><erroradadvsetting></erroradadvsetting></span><span class="pln">
</span><span class="tag"><criticaladadvsetting></criticaladadvsetting></span><span class="pln">
</span><span class="tag"><description></description></span><span class="pln">
</span><span class="tag"><maxvalue></maxvalue></span><span class="pln">
</span><span class="tag"><minvalue></minvalue></span><span class="pln">
</span><span class="tag"><userparam1></userparam1></span><span class="pln">
</span><span class="tag"><userparam2></userparam2></span><span class="pln">
</span><span class="tag"><userparam3></userparam3></span><span class="pln">
</span><span class="tag"><iscomposite></span><span class="pln">false</span><span class="tag"></iscomposite></span><span class="pln">
</span><span class="tag"><rpn></rpn></span><span class="pln">
</span><span class="tag"><alertTransitionIval></span><span class="pln">0</span><span class="tag"></alertTransitionIval></span><span class="pln">
</span><span class="tag"><alertClearTransitionIval></span><span class="pln">0</span><span class="tag"></alertClearTransitionIval></span><span class="pln">
</span><span class="tag"></datapoint></span><span class="pln">
</span><span class="tag"></datapoints></span><span class="pln">
</span><span class="tag"><graphs></span><span class="pln">
</span><span class="tag"><graph></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">Latency to LogicMonitor</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><title></span><span class="pln">Latency to LogicMonitor</span><span class="tag"></title></span><span class="pln">
</span><span class="tag"><verticallabel></span><span class="pln">ms</span><span class="tag"></verticallabel></span><span class="pln">
</span><span class="tag"><rigid></span><span class="pln">false</span><span class="tag"></rigid></span><span class="pln">
</span><span class="tag"><maxvalue></span><span class="pln">NaN</span><span class="tag"></maxvalue></span><span class="pln">
</span><span class="tag"><minvalue></span><span class="pln">NaN</span><span class="tag"></minvalue></span><span class="pln">
</span><span class="tag"><displayprio></span><span class="pln">1</span><span class="tag"></displayprio></span><span class="pln">
</span><span class="tag"><timescale></span><span class="pln">1day</span><span class="tag"></timescale></span><span class="pln">
</span><span class="tag"><base1024></span><span class="pln">false</span><span class="tag"></base1024></span><span class="pln">
</span><span class="tag"><graphdatapoints></span><span class="pln">
</span><span class="tag"><graphdatapoint></span><span class="pln">
</span><span class="tag"><name></span><span class="pln">ICMPms</span><span class="tag"></name></span><span class="pln">
</span><span class="tag"><datapointname></span><span class="pln">icmp_time</span><span class="tag"></datapointname></span><span class="pln">
</span><span class="tag"><cf></span><span class="pln">1</span><span class="tag"></cf></span><span class="pln">
</span><span class="tag"></graphdatapoint></span><span class="pln">
</span><span class="tag"></graphdatapoints></span><span class="pln">
</span><span class="tag"><graphvirtualdatapoints></span><span class="pln">
</span><span class="tag"></graphvirtualdatapoints></span><span class="pln">
</span><span class="tag"><graphdatas></span><span class="pln">
</span><span class="tag"><graphdata></span><span class="pln">
</span><span class="tag"><type></span><span class="pln">1</span><span class="tag"></type></span><span class="pln">
</span><span class="tag"><legend></span><span class="pln">ICMPms</span><span class="tag"></legend></span><span class="pln">
</span><span class="tag"><color></span><span class="pln">silver</span><span class="tag"></color></span><span class="pln">
</span><span class="tag"><datapointname></span><span class="pln">ICMPms</span><span class="tag"></datapointname></span><span class="pln">
</span><span class="tag"><isvirtualdatapoint></span><span class="pln">false</span><span class="tag"></isvirtualdatapoint></span><span class="pln">
</span><span class="tag"></graphdata></span><span class="pln">
</span><span class="tag"></graphdatas></span><span class="pln">
</span><span class="tag"></graph></span><span class="pln">
</span><span class="tag"></graphs></span><span class="pln">
</span><span class="tag"><overviewgraphs></span><span class="pln">
</span><span class="tag"></overviewgraphs></span><span class="pln">
</span><span class="tag"><scripts></span><span class="pln">
</span><span class="tag"></scripts></span><span class="pln">
</span><span class="tag"></entry></span><span class="pln">
</span><span class="tag"></feed></span>