Forum Discussion

Lewis_Beard's avatar
Lewis_Beard
Icon for Expert rankExpert
23 days ago

Config Source Cached?

I have a deprecated config source from LM that collects from FortiGate Firewalls, collects a config, and it keeps having a difference check on lines with #conf_file_ver=. We have kept this deprecated config source because it has some customizations. But I figured out, or so I thought, why that regex wasnt working to ignore those lines, because the LM UI when viewing the config isnt showing some white space characters.

So made a clone, applied it to just a couple of servers, changed the regex. Tested it in regex101 and did it a couple different ways, cant get it working but I know the regex is right. But anyway, waiting for real changes in that line is super tedious.

So I changed my clone to just hard code spit out a "config" of 5 lines of the top of the config, with the groovy making a random change to one character on the line I'm testing. So that I could at least check hourly (the shortest config running time).

When I test my config script in the Config Source editor, it does exactly what I want, changing 1 character on the line I want to ignore. But when I save and let it run on my actual device, the first time it runs, it generates a random number in the slot I wanted. But ALL repeats generate that same number. Whereas in the edit/test of the config source its different every time.

SO I'm just wondering why the heck my config script is making the value I did on the first time it ran but never changing it, even though the Last Checked time is changing every hour, implying the config source is running.

Is there some weird caching that happens when the device isnt really being queried and the script is just generating sample data? I dont get it.

But prior to that when I was working with a clone that really pulled the config, I could get neither of these to work, even though they both match perfectly on regex101:
        Ignore lines with regex "(^|\s+|\t+)#conf_file_ver.*"
        Ignore lines with regex "^\s*#conf_file_ver.*"

But thats beside the point. I cant figure out what weird LM quirk is messing up my regex, both of which work on regex101 and one of which matches the ones LM uses in that datasource (the first one).

My real issue is wondering if a script config that generates data from a script without reaching out to the device, that is hourly, why is LM delivering me the same config from that script every time, even though with I edit the configsource and test my own script, the random changes the character correctly?

Driving me wild over here. :)

You can see in the image that after a few hours where its clearly been running a bunch of times, LM is giving me the 8:04 am run on both sides of the DIFF even though if you look, you can see that its checked it .... 4 more times now, at 9, 10, 11, and 12.

GRR. I'm sure its a ME thing. But wondering if there is something I'm missing.

Config-Diff
  • Lewis_Beard's avatar
    Lewis_Beard
    Icon for Expert rankExpert
    23 days ago

    Oh and I meant to clarify, my regex101 regular expression is working on the real config and on all my fake ones, with both of the regex examples.

    And I alsm meant to clarify or highlight in the image, the last digit on line 2 of the config changes with my randomize code CORRECTLY WHEN TESTING THE CONFIG SOURCE IN EDIT MODE. But once I save it, it runs once, gives that last digit as random 0-9, and then just never changes every time LM invokes the collection. At least so far.

    • Stuart_Weenig's avatar
      Stuart_Weenig
      Icon for Mastermind rankMastermind
      23 days ago

      I totally get your frustration. It's what i've been dealing with trying to use the common configsources where the logic is spread over a dozen property sources too.

      Lots to unpack here so let me restate to understand the problem: you have a configsource. The target data changes often. You have a config check alerting on diff and you have some exclusions in that check that use regex. That regex matches the line that changes, so you shouldn't see alerts. All of this is happening as expected. However, you are expecting to see a new version registered in LM, but you're not.

      Is that it?

      • Lewis_Beard's avatar
        Lewis_Beard
        Icon for Expert rankExpert
        23 days ago

        One extra step at the end.

        I made a clone of the config check that, instead of reaching out to the host to grab the config, it prints out the top few lines of a config I grabbed once, hard coded, but with ONE character on the line in question altered, at random 0-9, on the line of concern. Because I dont want to wait on the REAL config source to happen change once every 2 weeks to test it.

        And while in test mode for the config source, I see the line changing that one character every time I hit play to test, once I apply this config source to my ONE applies-to target, it runs the random thing and give me my "fake config" but it NEVER EVER re-runs the script, seemingly, such that my line of concern changes.

        Its always giving me the original run.

        The code is 3 lines:

        def myNum = Math.abs(new Random().nextInt() % 10).toString();

        print("#config-version=FGT61F-7.0.14-FW-build0601-240206:opmode=1:vdom=0:user=lm\n#conf_file_ver=542255083128888" + myNum + "\n#buildno=0601\n#global_vdom=1\nconfig system global");

        return 0

         

  • Lewis_Beard's avatar
    Lewis_Beard
    Icon for Expert rankExpert
    23 days ago

    So basically yes you are right, but because the line doesnt change often, I'm hard-coding a hastening of the line change. It should have a 90% chance to change each hour when the config gets pulled again. Supposedly. But it keeps delivering me the original (it runs hourly, supposedly).

  • Lewis_Beard's avatar
    Lewis_Beard
    Icon for Expert rankExpert
    21 days ago

    I am starting to think this is a feature, that maybe LM doesnt have a later version of the Config that wasnt considered a change.

    I abandoned my script clone that printed out fake (but changing) data, and went back to the first clone that really reads from the device, and I made the diff check for that item to just ignore changes to any line that has conf_file_ver= on the line. I changed the apply to false() and back to my 3 target test devices, just to clear out any old configs or alerts.

    It ran on all 3 devices around 8:33 am today, and collected every hour. At some point I saw 13:33 or whatever as the most recent check, but the config was just showing the most recent update as being from 8:33 still. So I thought it hadnt changed. But on a whim, I did a manual pull of the config and it HAS changed.

    So I waited until 14:33 and after LM checked again (and I reloaded the page) and it STILL shows the data from 8:33 am and thats the only existing config. So I'm guessing LM doesnt actually store a newer version of the config, if the items that flag it as a change, wouldnt otherwise make the config be considered a change.

    At least, it seems that way to me. If thats the case then my difference test is now working and I'll just see what changes happen naturally over the weekend, and if they have all changed on that one line and LM is still only providing the one from 8:33 am today but has still been checking, then I'll feel sure about it, at which point I'll edit the original.

    Cheers!

     

    • Stuart_Weenig's avatar
      Stuart_Weenig
      Icon for Mastermind rankMastermind
      21 days ago
      Lewis_Beard wrote:

      LM doesnt actually store a newer version of the config, if the items that flag it as a change, wouldnt otherwise make the config be considered a change.

      That's what i was poorly explaining before. When LM pulls the config, unless an "important" part of the config has changed, it doesn't register a new version of the config.

      • Lewis_Beard's avatar
        Lewis_Beard
        Icon for Expert rankExpert
        21 days ago

        Well, except now I'm more confused than ever. 2 of the configs across the 3 servers in my applies-to conditions have have changed, specifically that line. LM has polled them both since they changed, and the one I was referring to above still hasnt thrown a difference error, so I thought my difference check was finally working.

        But that second one that changed, the same line changed, and it shows it as a diff. So its back to the drawing board. Good grief.

        I dont see why this difference check wouldn't work, and even less certain why LM decided to cache one of them and give no alert, but change the other and show an alert, even though both have changed and both I can see the last check was after the change.

        If I could get this cleared up I'd knock 900 warnings off my portal. I dont get it. I almost wonder if LM is oracle database under the hood and its treating those underscores, which I'm assuming should be literals, as some kind of wildcard like oracle does with underscores. Surely not. Makes no darned sense to me.

        And this isnt my first rodeo. I've successfully done difference tests on custom configs several times and tested then and they worked. So I got no idea why editing an existing one is being weird.

        I suppose I may as well try regex looking for "conf.file.ver= or something, with the . matching any 1 character in an attempt to avoid the underscores.

        Grr. I know I'm not doing something stupid. But I think I'm doing something stupid. :)