Data Source Consolidation and Transition Strategy

Ashish Sikarwar · 2024-09-04T12:18:36.841000Z

Hello,Current Situation:We have multiple data sources across different data centers for monitoring metrics such as CPU, Memory, Disk, Network, and others. Currently, these are managed separately, leading to a fragmented view of the data.Objective:Our goal is to consolidate and standardize these data sources so that eventually, we will have a single data source for each metric (e.g., CPU, Memory) across all data centers. This consolidation will allow us to bring all relevant data under one dashboard without needing to create multiple tables for each metric (since each table can only have one data source).Challenges:One of the main challenges is the potential data loss when moving devices from one data source to another. If the data source is changed, the historical data may be deleted, which is a significant concern during the transition.Proposed Solution:To mitigate this risk, I am considering using Kafka to send a duplicate stream of data. This would create a redundant data source that could be used until the transition to the consolidated data source is fully complete. This approach would ensure that no data is lost during the transition and provide a safety net as we standardize our monitoring setup.Request for Guidance:I am seeking advice on the best way to proceed with this consolidation and transition process. Specifically, I would like to know:The most efficient way to standardize data sources across all data centers.How to effectively use Kafka or another method to maintain data integrity during the transition period.Thanks,

Tech Forums

Product Discussions

Hello,

Current Situation:

We have multiple data sources across different data centers for monitoring metrics such as CPU, Memory, Disk, Network, and others. Currently, these are managed separately, leading to a fragmented view of the data.

Objective:

Our goal is to consolidate and standardize these data sources so that eventually, we will have a single data source for each metric (e.g., CPU, Memory) across all data centers. This consolidation will allow us to bring all relevant data under one dashboard without needing to create multiple tables for each metric (since each table can only have one data source).

Challenges:

One of the main challenges is the potential data loss when moving devices from one data source to another. If the data source is changed, the historical data may be deleted, which is a significant concern during the transition.

Proposed Solution:

To mitigate this risk, I am considering using Kafka to send a duplicate stream of data. This would create a redundant data source that could be used until the transition to the consolidated data source is fully complete. This approach would ensure that no data is lost during the transition and provide a safety net as we standardize our monitoring setup.

Request for Guidance:

I am seeking advice on the best way to proceed with this consolidation and transition process. Specifically, I would like to know:
1. The most efficient way to standardize data sources across all data centers.
2. How to effectively use Kafka or another method to maintain data integrity during the transition period.

Thanks,

Ashish Sikarwar

Posted 1 year ago·Last reply 1 year ago

6 comments

Ashish Sikarwar

OP1 year ago

Sure, Thanks a lot!

LM User

·1 year ago

The collector has its own set of datasources that monitor its health. Unfortunately, there's not a health index that sums up the health of the collector, but you should be able to look at the various individual collector datasources to make sure you're not overloading the collector.

Ashish Sikarwar

OP1 year ago

Got the storage point clear.
For Impact on collector - Yes I meant more CPU resources etc. yes I got it we can consider reducing the polling.

Thank you!

LM User

·1 year ago

This is why metric families are needed.

I've not been in this situation myself because it would take a lot of effort to get into a situation where datasources are resource/location specific. It's just not how the OOTB datasources are built.

Why not just expand/build the one module you want for CPU and have it work across all datacenters, keeping the old ones in place until you have enough history on the desired datasource? I don't understand why you wouldn't just have one datasource apply to all your datacenters.