Container

This might be my last article before leaving my beloved LogicMonitor. It has been a privilege to be part of the Support team but life happens to everyone and I know all things work together for good....

 

Now the Docker monitoring. We know the standard monitoring described and explained here: https://www.logicmonitor.com/support/monitoring/applications-databases/docker/, which takes advantage of Google cAdvisor, a tool, if I may define it, by Google to monitor Docker containers, elaborated nicely in greater details here: https://github.com/axibase/cadvisor-atsd. This tool is equipped with API which LogicMonitor uses to collect some data about the running Docker. There is a readily available datasource for such. 

To make life more exciting as a support team member, about a year ago I received a request from one among ALL our important Clients, by the name of RS (a.k.a Robert), whose request is the following:

We are using cadvisor to provide docker metrics to Logic Monitor. Is there a way to snag a history of the size of each container

that is definitely beyond Support and usually it will go to our excellent Monitoring Engineering team whose responsibility is to develop LogicModules, but I embarked on a challenging journey and within a month, a datasource for that purpose was ready.

This is beside the matter, I am in the midst of gaining knowledge about containers and came across an orchestration system for containers which is boasted to have a 'self-healing' mechanism, Kubernetes. Now I recognize that in the modern cloud infrastructure, self-healing is not uncommon. The indisputable remarkable cloud providers of this generation already have the same kind of feature or products in their offerings, auto/self-scaling, auto/self-replicating, auto-scheduling, etc. This is in-parallel (or even deploying the idea) of the fast-progressing development of Artificial Intelligence in cloud and system infrastructure, a system that has the intelligence to heal itself. 

Taking the benefit of Kubernetes that I have installed, researching container metrics in regards with the size began and the answer lies within this statement: 

Quote

"You want a system to offer and push their own health status to the monitoring layer, because who knows better about status than the system itself?"

 

 

 

Exactly the same goes with cAdvisor for Docker which has API to monitor itself and that is the very method that LogicModules "Docker_Containers_cAdvisor20" is using to collect data from Docker containers. Everything we need for monitoring is already available and provided by the source...the cAdvisor API.

Some sample metrics presented by cAdvisor API in a web-based display:

We just need to find out how to process those outputs and put the numeric data into datasource.

 

Fast forward, this is the final product, the monitoring of virtual size of Docker containers (orchestrated by Kubernetes) in my test and please take note of the instance names and wildvalues:

Note: it is also seen (in the 2nd screenshot, a datapoint by the name "virtual_size_sum" and "virtual_size_mbyte".

 

Now the next question is: how the data numbers come to be?

Firstly, let us take an example of one container highlighted above as an example for the calculation. The instance (container) has a name of "k8s_carts_carts-794f6cc876-z9d6p_sock-shop_7655a496-d347-11e7-90ef-000c29412e95_0"and has wildvalue of:

8afc5e907b809b9a15884b518fdff1678cad54d2562d6d0b63bd3fb5ce77d5a3

If we trace back, that wildvalue can be found in the data collected from cAdvisor API as well, in a text-based result as below, which are the output of running Docker processes:

Note: there are 2 (two) processes for the containers, hence the SUM of the virtual size datapoint ("virtual_size_sum").

When we zoom into only those 2 processes with the same wildvalue of one container, we will see clearer:

It makes the total virtual size as follows:

virtual size sum = 1795608576 + 1560576 = 1797169152 B (= 1797169152/1024/1024 = 1713.9140625 MB)

(note: virtual size PID 5686 = 1560576 , virtual size PID 5810 = 1795608576)

 

The number is precisely what cAdvisor web-based data earlier shows:

upsize screenshots:

 

 

If you see the total virtual size number, that is what is presented by LogicModule collected from the cAdvisor:

 

The datasource had actually been submitted, but it is not released yet in the community and is still undergoing a review process by the official team, so it may take a while. (Locator Code:JAGW6D). Therefore there is no way to download it from LM Exchange yet.

 

If you wish to have it, you may contact the excellent Support Team and they will search my past case to get hold of the datasource. Alternatively, you may drop an email in my personal address: purnadi.k@gmail.com as I will sadly be leaving soon.

 

Life is too short, so live your lives to the fullest and may we meet again, one way or another. Have a fun day! Cheers.