Forum Discussion

asikarwar's avatar
asikarwar
Icon for Neophyte rankNeophyte
13 days ago

Data Source Consolidation and Transition Strategy

Hello,

Current Situation:

  • We have multiple data sources across different data centers for monitoring metrics such as CPU, Memory, Disk, Network, and others. Currently, these are managed separately, leading to a fragmented view of the data.

Objective:

  • Our goal is to consolidate and standardize these data sources so that eventually, we will have a single data source for each metric (e.g., CPU, Memory) across all data centers. This consolidation will allow us to bring all relevant data under one dashboard without needing to create multiple tables for each metric (since each table can only have one data source).

Challenges:

  • One of the main challenges is the potential data loss when moving devices from one data source to another. If the data source is changed, the historical data may be deleted, which is a significant concern during the transition.

Proposed Solution:

  • To mitigate this risk, I am considering using Kafka to send a duplicate stream of data. This would create a redundant data source that could be used until the transition to the consolidated data source is fully complete. This approach would ensure that no data is lost during the transition and provide a safety net as we standardize our monitoring setup.

Request for Guidance:

  • I am seeking advice on the best way to proceed with this consolidation and transition process. Specifically, I would like to know:
    1. The most efficient way to standardize data sources across all data centers.
    2. How to effectively use Kafka or another method to maintain data integrity during the transition period.

Thanks,

  • Stuart_Weenig's avatar
    Stuart_Weenig
    Icon for Mastermind rankMastermind
    13 days ago

    This is why metric families are needed. 

    I've not been in this situation myself because it would take a lot of effort to get into a situation where datasources are resource/location specific. It's just not how the OOTB datasources are built.

    Why not just expand/build the one module you want for CPU and have it work across all datacenters, keeping the old ones in place until you have enough history on the desired datasource? I don't understand why you wouldn't just have one datasource apply to all your datacenters.

  • Stuart_Weenig's avatar
    Stuart_Weenig
    Icon for Mastermind rankMastermind
    13 days ago

    The collector has its own set of datasources that monitor its health. Unfortunately, there's not a health index that sums up the health of the collector, but you should be able to look at the various individual collector datasources to make sure you're not overloading the collector.

  • asikarwar's avatar
    asikarwar
    Icon for Neophyte rankNeophyte
    13 days ago

    Why not just expand/build the one module you want for CPU and have it work across all datacenters, keeping the old ones in place until you have enough history on the desired data source?

    Will it cost significantly more in terms of storage? Will it have an impact on the collectors where since we will have data sources for each metric type?

    This was done in the past and not properly planned. I know we must have one data source.

    It is what it is but not is the time to make the change. Thank you for your response.

    • Stuart_Weenig's avatar
      Stuart_Weenig
      Icon for Mastermind rankMastermind
      13 days ago

      You don't pay for storage and it's not stored on the collector. It's stored on the LM platform. Your license allows you to have unlimited number of datasources per resource. So having a new one active while the old one is active doesn't impact anything for you. Yes, there will be more tasks to execute so think about that, but you can manage that by decreasing the poll rate of the old DS in a phased approach.

  • asikarwar's avatar
    asikarwar
    Icon for Neophyte rankNeophyte
    13 days ago

    Got the storage point clear.
    For Impact on collector - Yes I meant more CPU resources etc. yes I got it we can consider reducing the polling.

    Thank you!