Email Ingestion

I haven't looked at the script to decipher the text output, but does it look like this?

Rereading this post and it occurs to me that you would probably do better with a datasource rather than an eventsource. Are you looking for pure unread count per mailbox? Or are you trying to use the subject line matching in the code to only count the number of unreads matching your given subject? If you're just looking for unread emails that match the default subject line in the code, you could replace this line:

debugOutput(debug, "foundCount: ${foundCount}")

with this line:

println("foundCount: ${foundCount}")

And comment out this line:

println(new JsonBuilder(events).toPrettyString())

Make this your datasource script and make a datapoint called foundCount using multi-line key/pair with the key of "foundCount". 

If the script is currently marking items as read, it should continue to do that, but the datapoint should reflect the number of items it marked as read.

Like this:

import jakarta.mail.*
import jakarta.mail.internet.*
import jakarta.mail.search.*
import groovy.json.JsonBuilder
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
def debug = false
def deleteProcessedEmails = false
def imapHost = hostProps.get("apc.imap.host") // Required
def imapType = hostProps.get("apc.imap.type") // Optional
def emailUser = hostProps.get("apc.email.user") // Required
def emailPass = hostProps.get("apc.email.pass") // Required
def msgSubject = hostProps.get("apc.email.subject") // Optional, can include a propertyname (example: "##system.hostname##") for dynamic replacement
def deleteProcessedEmailProp = hostProps.get("email.deleteProcessed") // Optional
def emailInbox = hostProps.get("apc.email.folder") ?: "Inbox" // Example: "Inbox/Errors"
def hostName = hostProps.get("system.displayname")
// Default our email subject if one wasn't specified via property...
if (!msgSubject) {msgSubject = "Incident Notification"} 
// See if a property was specified in our subject line...
def dynamicProp = msgSubject =~ ".*#{2}(.+)#{2}.*"
if (dynamicProp.size() > 0) {
    def dynamicPropName = dynamicProp[0][1]
    def dynamicPropValue = ""
    // Allow simply specifying ##hostname## to replace with system.hostname...
    if (dynamicPropName.toLowerCase() == "hostname") {
        dynamicPropValue = hostName
    } else {// Otherwise, grab the property named (example: "Alert Test ##aws.instanceid##")...
        dynamicPropValue = hostProps.get(dynamicPropName)
    }
    if (dynamicPropValue != "") {
        msgSubject = msgSubject.replaceAll("##" + dynamicPropName + "##", dynamicPropValue)
    }
}
// Default is to NOT delete processed emails (will just mark them as read)...
if (deleteProcessedEmailProp && deleteProcessedEmailProp.toLowerCase() == "true") {
    deleteProcessedEmails = true
}
def protocolDebug = true
if (debug) {
    println "imapHost=${imapHost}"
    println "imapType=${imapType}"
    println "emailUser=${emailUser}"
    println "emailFolder=${emailInbox}"
    println "msgSubject=${msgSubject}"
    println "deleteProcessedEmails=${deleteProcessedEmails}"
    // println "emailAddr=${emailAddr}"
}
def eventList = []
// Do we have all required params?...
if (!imapHost || !emailUser || !emailPass){
    debugOutput(debug, "errorCode=-1")
    return 0
}
protoDebugFile = new PrintStream('../logs/' + hostName + '-APCEmailAlert-protocol.log')
scriptDebugFile = new File('../logs/' + hostName + '-APCEmailAlert-debug.log')
scriptDebugFile.delete()
/*
 * IMAP Setup
 */
imapProps = System.getProperties()
if (imapType == "SSL"){
    store_type = "imaps"
    imapProps.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory")
    imapProps.setProperty("mail.imap.socketFactory.fallback", "false")
    imapProps.setProperty("mail.imap.socketFactory.port", "993")
    imapProps.setProperty("mail.imaps.ssl.trust", "*")
    imapProps.setProperty("mail.imaps.ssl.checkserveridentity", "false")
} else if (imapType == "TLS"){
    store_type = "imap"
    imapProps.setProperty("mail.imap.starttls.enable", "true")
    imapProps.setProperty("mail.imap.socketFactory.port", "143")
} else {
    store_type = "imap"
    imapProps.setProperty("mail.imap.socketFactory.port", "143")
    imapProps.setProperty("mail.imap.socketFactory.class", "javax.net.SocketFactory")
}
imapProps.setProperty("mail.store.protocol", store_type)
imapSession = Session.getInstance(imapProps, null)
imapSession.setDebug(protocolDebug)
imapSession.setDebugOut(protoDebugFile)
imap = imapSession.getStore(store_type)
/*
 * Connect to IMAP
 */
def tries = 5
while (tries > 0){
    tries--
    try{
        // scriptDebugFile.append("Attempting IMAP Connection\n")
        debugOutput(debug, "Attempting IMAP Connection")
        imap.connect(imapHost, emailUser, emailPass)
        tries = -1
    }catch (Exception e){
        error_string = "IMAP Connection Error: " + e.message + "\n"
        // scriptDebugFile.append(error_string)
        debugOutput(debug, error_string)
        println(error_string)
        sleep(1000)
    }
}
// Were we able to connect to imap server?...
if (tries == -1){
    // Yes. log some debugging...
    // scriptDebugFile.append("IMAP Connection Successful\n")
    debugOutput(debug, "IMAP Connection Successful")
} else {
    // No. Exit...
    debugOutput(debug, "errorCode=3")
    protoDebugFile.close()
    return 0
}
/*
 * Search inbox for the message we created earlier
 */
// inbox = imap.getFolder("Inbox")
inbox = imap.getFolder(emailInbox)
// Search for unread messages containing our subject (Note: our search term is converted to lowercase above)...
subjectTerm = new SubjectTerm(msgSubject)
unreadTerm = new FlagTerm(new Flags(Flags.Flag.SEEN), false)
search_term = new AndTerm(subjectTerm, unreadTerm)
foundMessage = false
try {
    // Open the inbox do the search...
    inbox.open(Folder.READ_WRITE)
    // scriptDebugFile.append("Initiating new inbox search on subject \"${search_term}\"\n")
    debugOutput(debug, "Initiating new inbox search on subject \"${search_term}\"")
    searchResults = inbox.search(search_term)
    //  Message messages[] = inbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), false))
    foundCount = searchResults.size()
    debugOutput(debug, "foundCount: ${foundCount}")
    println("foundCount: ${foundCount}")
    // Did we find any messages?...
    if (foundCount == 0){
        // No -- sleep for a bit...
        // scriptDebugFile.append("  - search found no messages -- sleeping\n")
        debugOutput(debug, "  - search found no messages -- sleeping")
        sleep(1000)
    } else {
        //  Yes. Iterate through the search results array...
        // scriptDebugFile.append("  - search found ${foundCount} messages\n")
        debugOutput(debug, "  - search found ${foundCount} messages")
        // Iterate through results...
        searchResults.each { imap_message ->
            foundSubject = imap_message.getSubject()
            foundSubject = foundSubject.replaceAll(",","")
            foundSubject = foundSubject.replaceAll("\\(","")
            foundSubject = foundSubject.replaceAll("\\)","")
            // Does the subject of this message match the subject of the message we sent earlier?...
            if (foundSubject =~ msgSubject) {
                // Yes -- exit the loop...
                tries = -1
                foundMessage = true
                // scriptDebugFile.append("  - located target message \"${foundSubject}\"\n")
                debugOutput(debug, "  - located target message \"${foundSubject}\"")
                emailBody = getText(imap_message).trim()
                emailBody = emailBody.replaceAll("\\r|\\n", "")
                emailBody = emailBody.replaceAll("\\<.*?\\>", "")
                emailBody = emailBody.replaceAll("font-family:[^;']*(;)", "")
                emailBody = emailBody.replaceAll("font-size:[^;']*(;)", "")
                emailBody = emailBody.trim().replaceAll("\\s+", " ")
                // From epoch to human readable...
                String date = new java.text.SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz ").format(new Date())
                eventList.push([
                    "happenedOn": date.toString(),
                    "severity"  : "Critical",
                    "message"   : foundSubject,
                    "source"    : hostName
                ])
            }
            // Flag any messages that matched this search for deletion...
            if (deleteProcessedEmails) {
                imap_message.setFlag(Flags.Flag.DELETED, true)
                // scriptDebugFile.append("  - delete flag set on \"${foundSubject}\"\n")
                debugOutput(debug, "  - delete flag set on \"${foundSubject}\"")
            }
            imap_message.setFlag(Flags.Flag.SEEN, true)
        }
        // Expunge the deleted messages...
        if (deleteProcessedEmails) {
            // scriptDebugFile.append("  - expunging deleted message(s)\n")
            debugOutput(debug, "  - expunging deleted message(s)")
            inbox.expunge()
        }
    }
    // Close the mailbox...
    inbox.close(true)
    // scriptDebugFile.append("  - inbox closed\n")
    debugOutput(debug, "  - inbox closed")
} catch (Exception e){
    // Something blew up...
    error_string = "IMAP Inbox Search Error: " + e.message
    // scriptDebugFile.append(error_string + "\n")
    debugOutput(debug, error_string)
    println(error_string)
    // Because something blew up, lets make sure our inbox is closed...
    inbox.close(true)
    sleep(1000)
}
// Did we find the message?...
if (!foundMessage) {
    // No. Close and exit...
    debugOutput(debug, "errorCode=4")
    protoDebugFile.close()
    return 0
} else {
    def events = [events: eventList]
    // println(new JsonBuilder(events).toPrettyString())
    // def jsonOut = JsonOutput.toJson(eventsMap)
    // println JsonOutput.prettyPrint(jsonOut)
}
// Close the IMAP connection...
// scriptDebugFile.append("Closing IMAP Connection\n")
debugOutput(debug, "Closing IMAP Connection")
imap.close()
// Everything went well...
debugOutput(debug, "errorCode=0")
return 0

// Capture/log debug output if flagged to do so...
def debugOutput(Boolean debug, String message) {
    if (debug) {
    println "IMAP Connection Successful"
    scriptDebugFile.append("${message}\n")
    }
}

// Credit to Mike Suding for the following function...
def getText(Part p) {
    if (p.isMimeType("text/*")) {
        String s = (String)p.getContent()
        textIsHtml = p.isMimeType("text/html")
        return s
    }
    if (p.isMimeType("multipart/alternative")) {
        // Prefer html over plain text...
        Multipart mp = (Multipart)p.getContent()
        String text = null
        for (int i = 0; i < mp.getCount(); i++) {
            Part bp = mp.getBodyPart(i)
            if (bp.isMimeType("text/plain")) {
                if (text == null)
                    text = getText(bp)
                    continue
            } else if (bp.isMimeType("text/html")) {
                String s = getText(bp)
                if (s != null)
                return s
            } else {
                return getText(bp)
            }
        }
        return text
    } else if (p.isMimeType("multipart/*")) {
        Multipart mp = (Multipart)p.getContent()
        for (int i = 0; i < mp.getCount(); i++) {
            String s = getText(mp.getBodyPart(i))
            if (s != null)
            return s
        }
    }
    return EMPTY
}

Hello Stuart,

Our aim is to monitor any unread emails in the mailbox. Once detected, the LM has to generate an alert if the Display name matches.

Ok, then i think you would do better with a datasource using the code above.  i didn't test the code above, but it's minor changes from your working code. 

If we use Event Source, messages are visible, but can we also view messages when utilizing a DataSource? I believe we can observe the count of unread messages, correct?

Yes, I reached out to support again. Here's the response I received:

I was never able to get this script working. This is why my suggestions are the following;

1. Can we add this device as a resource within Logicmonitor
2. Can we get this data from this device in a different way;  i.e. snmp syslogs or traps, etc?
3. Does this device write a log file that we could potentially monitor for this data?
4. Another option would be to get in touch with your CSM, who can put you in touch with our professional services team to build out a datasource/eventsource for you.

 
Another potential option may be to add a collector where that resource is and have it just poll this device like for real monitoring versus just getting email.

But we can't monitor by doing SNMP Trap, Because Devices are hosted in Cloud. It is not capable of sending the SNMP traps

Hello Stuart,

Apologies for the delayed response; I wasn't available last week.

After running the script yesterday, I encountered a JSON error that wasn't present before. Could you please advise why this error is occurring and provide guidance on how and where to rectify it? I've included a snapshot of the error for reference.

I utilized the script as an event source since it allows us to view the content of the email. Using a data source would only display the email count.

While running the script in debug mode, it executed successfully and returned with code 0. However, encountering the mentioned error occurred when running the script with the event source.

Could you setup a 30-minute call to discuss this further? It would help me explain the issue more effectively.

Hey Stuart,

The JSON Format isn't the problem. We've identified the cause of the error. Earlier today, we updated the email password and successfully tested the script, yielding the expected output. However, the password is inexplicably becoming locked within moments. I'm uncertain as to why this is occurring, but it's resulting in a JSON error. If you have any insights into why the password might be getting locked, I'd appreciate your input.