Forum Discussion

Pooja's avatar
Pooja
4 years ago

How to monitor opensource elasticsearch in Logic Monitor

Hello All,

I have installed opensource(Not AWS) elasticsearch on my servers. Now I want to monitor the elasticsearch parameters in Logic Monitor. How can i do that? Are there any plugins available to monitor elasticsearch in LM. I found an article for monitoring AWS Elasticsearch through Cloudwatch , but nothing other than that. Any help would be greatly appreciated.

Thanks in advance!!

3 Replies

Sort By
  • Jeff8682's avatar
    Jeff8682
    4 years ago

     

    We have 3 datasources that we use for our internal Elk clusters.  Maybe they can help you or at least point you in the right direction for writing your own.  This is the one for shard health. 

    <?xml version="1.0" encoding="UTF-8" ?>
<feed  version="1.0" hasPendingRequests="false" >
  <company></company>
  <status>200</status>
  <errmsg>OK</errmsg>
  <interval>0</interval>
    <entry type="predatasource">
        <version>1573572263</version>
        <name>Elasticsearch_Cluster_Shards_veri</name>
        <displayedas>Cluster Shards_veri</displayedas>
        <description>Monitors Elasticsearch Cluster shard health.</description>
        <collector>webpage</collector>
        <hasMultiInstances>false</hasMultiInstances>
        <schedule>60</schedule>
        <appliesTo>hasCategory(&#34;elasticsearch&#34;) or system.deviceId == &#34;6986&#34;</appliesTo>
        <wildcardauto>false</wildcardauto>
        <wildcardpersist>false</wildcardpersist>
        <wildcardlinuxscript></wildcardlinuxscript>
        <wildcardlinuxcmdline></wildcardlinuxcmdline>
        <wildcardwinscript></wildcardwinscript>
        <wildcardwincmdline></wildcardwincmdline>
        <wildcardgroovyscript></wildcardgroovyscript>
        <wildcardschedule>1440</wildcardschedule>
        <wildcarddisable>false</wildcarddisable>
        <wildcarddeleteinactive>false</wildcarddeleteinactive>
        <agdmethod>none</agdmethod>
        <agdparams></agdparams>
        <group></group>
        <tags>cluster,shards,elasticsearch,health</tags>
        <technology></technology>
        <adlist><![CDATA[{"agdmethod":"none","agdparams":"","id":0,"filters":[],"params":{}}]]></adlist>
        <schemaVersion>2</schemaVersion>
        <dataSourceType>1</dataSourceType>
        <attributes>
        <attribute>
            <name>ip</name>
            <value></value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>port</name>
            <value>9200</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>followRedirect</name>
            <value>false</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>connectTimeout</name>
            <value>1000</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>readTimeout</name>
            <value>1000</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>request</name>
            <value>GET /_cluster/health?pretty&#38;human HTTP/1.0</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>ssl</name>
            <value>N</value>
            <comment></comment>
        </attribute>
        </attributes>
        <datapoints>
        <datapoint>
            <name>active_primary_shards</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>json</postprocessormethod>
            <postprocessorparam>active_primary_shards</postprocessorparam>
            <usevalue>body</usevalue>
            <alertexpr></alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody></alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>Number of active primary shards.</description>
            <maxvalue></maxvalue>
            <minvalue>0</minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>0</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        <datapoint>
            <name>active_shards</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>json</postprocessormethod>
            <postprocessorparam>active_shards</postprocessorparam>
            <usevalue>body</usevalue>
            <alertexpr></alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody></alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>Number of active shards.</description>
            <maxvalue></maxvalue>
            <minvalue>0</minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>0</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        <datapoint>
            <name>initializing_shards</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>json</postprocessormethod>
            <postprocessorparam>initializing_shards</postprocessorparam>
            <usevalue>body</usevalue>
            <alertexpr></alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody></alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>Number of initializing shards.</description>
            <maxvalue></maxvalue>
            <minvalue>0</minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>0</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        <datapoint>
            <name>relocating_shards</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>json</postprocessormethod>
            <postprocessorparam>relocating_shards</postprocessorparam>
            <usevalue>body</usevalue>
            <alertexpr></alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody></alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>Number of relocating shards.</description>
            <maxvalue></maxvalue>
            <minvalue>0</minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>0</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        <datapoint>
            <name>unassigned_shards</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>json</postprocessormethod>
            <postprocessorparam>unassigned_shards</postprocessorparam>
            <usevalue>body</usevalue>
            <alertexpr></alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody></alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>Number of unassigned shards.</description>
            <maxvalue></maxvalue>
            <minvalue></minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>0</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        </datapoints>
        <graphs>
        <graph>
            <name>Shards</name>
            <title>Shards</title>
            <verticallabel>shards</verticallabel>
            <rigid>false</rigid>
            <maxvalue>NaN</maxvalue>
            <minvalue>0.0</minvalue>
            <displayprio>1</displayprio>
            <timescale>1day</timescale>
            <base1024>false</base1024>
            <graphdatapoints>
        <graphdatapoint>
            <name>active_primary_shards</name>
            <datapointname>active_primary_shards</datapointname>
            <cf>2</cf>
        </graphdatapoint>
        <graphdatapoint>
            <name>active_shards</name>
            <datapointname>active_shards</datapointname>
            <cf>2</cf>
        </graphdatapoint>
        <graphdatapoint>
            <name>initializing_shards</name>
            <datapointname>initializing_shards</datapointname>
            <cf>2</cf>
        </graphdatapoint>
        <graphdatapoint>
            <name>relocating_shards</name>
            <datapointname>relocating_shards</datapointname>
            <cf>2</cf>
        </graphdatapoint>
        <graphdatapoint>
            <name>unassigned_shards</name>
            <datapointname>unassigned_shards</datapointname>
            <cf>2</cf>
        </graphdatapoint>
            </graphdatapoints>
            <graphvirtualdatapoints>
            </graphvirtualdatapoints>
            <graphdatas>
            <graphdata>
                <type>1</type>
                <legend>Active Primary</legend>
                <color>red</color>
                <datapointname>active_primary_shards</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            <graphdata>
                <type>1</type>
                <legend>Active</legend>
                <color>orange</color>
                <datapointname>active_shards</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            <graphdata>
                <type>1</type>
                <legend>Initializing</legend>
                <color>yellow</color>
                <datapointname>initializing_shards</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            <graphdata>
                <type>1</type>
                <legend>Relocating</legend>
                <color>green</color>
                <datapointname>relocating_shards</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            <graphdata>
                <type>1</type>
                <legend>Unassigned</legend>
                <color>blue</color>
                <datapointname>unassigned_shards</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            </graphdatas>
        </graph>
        </graphs>
        <overviewgraphs>
        </overviewgraphs>
        <scripts>
        </scripts>
    </entry>
</feed>

     

  • Jeff8682's avatar
    Jeff8682
    4 years ago

    Last one for cluster health status. 

    <?xml version="1.0" encoding="UTF-8" ?>
<feed  version="1.0" hasPendingRequests="false" >
  <company></company>
  <status>200</status>
  <errmsg>OK</errmsg>
  <interval>0</interval>
    <entry type="predatasource">
        <version>1573572264</version>
        <name>Elasticsearch_Cluster_Health_Status_veri</name>
        <displayedas>Cluster Health Status_veri</displayedas>
        <description>Elasticsearch Cluster status: 2 == green, 1 == yellow, 0 == red.</description>
        <collector>script</collector>
        <hasMultiInstances>false</hasMultiInstances>
        <schedule>60</schedule>
        <appliesTo>hasCategory(&#34;elasticsearch&#34;) or system.deviceId == &#34;6986&#34;</appliesTo>
        <wildcardauto>false</wildcardauto>
        <wildcardpersist>false</wildcardpersist>
        <wildcardlinuxscript></wildcardlinuxscript>
        <wildcardlinuxcmdline></wildcardlinuxcmdline>
        <wildcardwinscript></wildcardwinscript>
        <wildcardwincmdline></wildcardwincmdline>
        <wildcardgroovyscript></wildcardgroovyscript>
        <wildcardschedule>1440</wildcardschedule>
        <wildcarddisable>false</wildcarddisable>
        <wildcarddeleteinactive>false</wildcarddeleteinactive>
        <agdmethod>none</agdmethod>
        <agdparams></agdparams>
        <group></group>
        <tags>cluster,elasticsearch,health,status</tags>
        <technology></technology>
        <adlist><![CDATA[{"agdmethod":"none","agdparams":"","id":0,"filters":[],"params":{}}]]></adlist>
        <schemaVersion>2</schemaVersion>
        <dataSourceType>1</dataSourceType>
        <attributes>
        <attribute>
            <name>scripttype</name>
            <value>embed</value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>scriptgroovy</name>
            <value>import groovy.json.JsonSlurper
import org.apache.http.client.methods.HttpGet
import org.apache.http.conn.ssl.SSLConnectionSocketFactory
import org.apache.http.conn.ssl.SSLContextBuilder
import org.apache.http.conn.ssl.TrustSelfSignedStrategy
import org.apache.http.impl.client.HttpClients
import org.apache.http.util.EntityUtils

def hostname = hostProps.get(&#34;system.hostname&#34;)
def slurper = new JsonSlurper()

def cluster_status_map = [&#34;green&#34; : 2,
                          &#34;yellow&#34;: 1,
                          &#34;red&#34;   : 0]

def request_endpoint = &#34;http://${hostname}:9200/_cluster/health&#34;
def request_headers = [&#34;User-Agent&#34;  : &#34;LogicMonitor Polling Agent&#34;,
                       &#34;Content-Type&#34;: &#34;application/json&#34;];

/*
 * http_get - do an HTTP GET with apache http client
 *  @param string request_endpoint
 *  @param map request_headers
 *  @return map [code,status,body]
 */

def http_get(request_endpoint, request_headers)
{
    def http_get = new HttpGet(request_endpoint);
    // what type of request?
    if (request_endpoint.startsWith(&#39;https:&#39;))
    {
        // https -- create an ssl-safe client
        ssl_builder = new SSLContextBuilder();
        ssl_builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
        ssl_sf = new SSLConnectionSocketFactory(ssl_builder.build(), SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
        client = HttpClients.custom().setSSLSocketFactory(ssl_sf).build();
    }
    else
    {
        // vanilla http -- use the default client
        client = HttpClients.createDefault();
    }
    // iterate over request headers
    request_headers.each()
    { key, value -&#62;
        http_get.setHeader(key, value);
    }
    def http_response = client.execute(http_get);
    def response_message = http_response.getEntity();
    def response_status = http_response.getStatusLine();
    def response_code = response_status.getStatusCode();
    def response_body = EntityUtils.toString(response_message);
    http_response.close();
    return 
    ;
}

// Make HTTP GET call
def get_response = http_get(request_endpoint, request_headers);

// Parse result to JSON
def result = slurper.parseText(get_response[&#39;body&#39;])

// Print the results
println cluster_status_map[result.status]

return 0;
    </value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>windowsscript</name>
            <value></value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>linuxscript</name>
            <value></value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>windowscmdline</name>
            <value></value>
            <comment></comment>
        </attribute>
        <attribute>
            <name>linuxcmdline</name>
            <value></value>
            <comment></comment>
        </attribute>
        </attributes>
        <datapoints>
        <datapoint>
            <name>cluster_status</name>
            <dataType>7</dataType>
            <type>2</type>
            <postprocessormethod>none</postprocessormethod>
            <postprocessorparam></postprocessorparam>
            <usevalue>output</usevalue>
            <alertexpr>= 1 0</alertexpr>
            <alertmissing>1</alertmissing>
            <alertsubject></alertsubject>
            <alertbody>The ElasticSearch Cluster at ##HOST## is reporting a state of ##VALUE## putting it in a ##LEVEL## state.

The cluster has been in this state since ##START## or for ##DURATION##.</alertbody>
            <enableanomalyalertsuppression></enableanomalyalertsuppression>
            <adadvsettingenabled>false</adadvsettingenabled>
            <warnadadvsetting></warnadadvsetting>
            <erroradadvsetting></erroradadvsetting>
            <criticaladadvsetting></criticaladadvsetting>
            <description>ElasticSearch Cluster status:

2 = green,
1 = yellow,
0 = red.</description>
            <maxvalue>2</maxvalue>
            <minvalue>0</minvalue>
            <userparam1></userparam1>
            <userparam2></userparam2>
            <userparam3></userparam3>
            <iscomposite>false</iscomposite>
            <rpn></rpn>
            <alertTransitionIval>1</alertTransitionIval>
            <alertClearTransitionIval>0</alertClearTransitionIval>
        </datapoint>
        </datapoints>
        <graphs>
        <graph>
            <name>Elasticsearch Cluster Health Status</name>
            <title>Elasticsearch Cluster Health Status</title>
            <verticallabel>status</verticallabel>
            <rigid>false</rigid>
            <maxvalue>2.0</maxvalue>
            <minvalue>0.0</minvalue>
            <displayprio>1</displayprio>
            <timescale>1day</timescale>
            <base1024>false</base1024>
            <graphdatapoints>
        <graphdatapoint>
            <name>cluster_status</name>
            <datapointname>cluster_status</datapointname>
            <cf>1</cf>
        </graphdatapoint>
            </graphdatapoints>
            <graphvirtualdatapoints>
            </graphvirtualdatapoints>
            <graphdatas>
            <graphdata>
                <type>1</type>
                <legend>Status</legend>
                <color>blue</color>
                <datapointname>cluster_status</datapointname>
                <isvirtualdatapoint>false</isvirtualdatapoint>
            </graphdata>
            </graphdatas>
        </graph>
        </graphs>
        <overviewgraphs>
        </overviewgraphs>
        <scripts>
        </scripts>
    </entry>
</feed>

     

  • Anonymous's avatar
    Anonymous
    4 years ago

    @Jeff8682, I suggest you submit these to the LM Exchange for easier distribution. This has the added benefit of getting a security review of the code.