Config Source Cached?
I have a deprecated config source from LM that collects from FortiGate Firewalls, collects a config, and it keeps having a difference check on lines with #conf_file_ver=. We have kept this deprecated config source because it has some customizations. But I figured out, or so I thought, why that regex wasnt working to ignore those lines, because the LM UI when viewing the config isnt showing some white space characters.
So made a clone, applied it to just a couple of servers, changed the regex. Tested it in regex101 and did it a couple different ways, cant get it working but I know the regex is right. But anyway, waiting for real changes in that line is super tedious.
So I changed my clone to just hard code spit out a "config" of 5 lines of the top of the config, with the groovy making a random change to one character on the line I'm testing. So that I could at least check hourly (the shortest config running time).
When I test my config script in the Config Source editor, it does exactly what I want, changing 1 character on the line I want to ignore. But when I save and let it run on my actual device, the first time it runs, it generates a random number in the slot I wanted. But ALL repeats generate that same number. Whereas in the edit/test of the config source its different every time.
SO I'm just wondering why the heck my config script is making the value I did on the first time it ran but never changing it, even though the Last Checked time is changing every hour, implying the config source is running.
Is there some weird caching that happens when the device isnt really being queried and the script is just generating sample data? I dont get it.
But prior to that when I was working with a clone that really pulled the config, I could get neither of these to work, even though they both match perfectly on regex101:
Ignore lines with regex "(^|\s+|\t+)#conf_file_ver.*"
Ignore lines with regex "^\s*#conf_file_ver.*"
But thats beside the point. I cant figure out what weird LM quirk is messing up my regex, both of which work on regex101 and one of which matches the ones LM uses in that datasource (the first one).
My real issue is wondering if a script config that generates data from a script without reaching out to the device, that is hourly, why is LM delivering me the same config from that script every time, even though with I edit the configsource and test my own script, the random changes the character correctly?
Driving me wild over here. :)
You can see in the image that after a few hours where its clearly been running a bunch of times, LM is giving me the 8:04 am run on both sides of the DIFF even though if you look, you can see that its checked it .... 4 more times now, at 9, 10, 11, and 12.
GRR. I'm sure its a ME thing. But wondering if there is something I'm missing.
Lewis_Beard
OP1 year agoSo basically yes you are right, but because the line doesnt change often, I'm hard-coding a hastening of the line change. It should have a 90% chance to change each hour when the config gets pulled again. Supposedly. But it keeps delivering me the original (it runs hourly, supposedly).
Lewis_Beard
OP1 year agoOh and I meant to clarify, my regex101 regular expression is working on the real config and on all my fake ones, with both of the regex examples.
And I alsm meant to clarify or highlight in the image, the last digit on line 2 of the config changes with my randomize code CORRECTLY WHEN TESTING THE CONFIG SOURCE IN EDIT MODE. But once I save it, it runs once, gives that last digit as random 0-9, and then just never changes every time LM invokes the collection. At least so far.
LM User
·1 year agoI totally get your frustration. It's what i've been dealing with trying to use the common configsources where the logic is spread over a dozen property sources too.
Lots to unpack here so let me restate to understand the problem: you have a configsource. The target data changes often. You have a config check alerting on diff and you have some exclusions in that check that use regex. That regex matches the line that changes, so you shouldn't see alerts. All of this is happening as expected. However, you are expecting to see a new version registered in LM, but you're not.
Is that it?
Lewis_Beard
OP1 year agoOne extra step at the end.
I made a clone of the config check that, instead of reaching out to the host to grab the config, it prints out the top few lines of a config I grabbed once, hard coded, but with ONE character on the line in question altered, at random 0-9, on the line of concern. Because I dont want to wait on the REAL config source to happen change once every 2 weeks to test it.
And while in test mode for the config source, I see the line changing that one character every time I hit play to test, once I apply this config source to my ONE applies-to target, it runs the random thing and give me my "fake config" but it NEVER EVER re-runs the script, seemingly, such that my line of concern changes.
Its always giving me the original run.
The code is 3 lines:
def myNum = Math.abs(new Random().nextInt() % 10).toString();
print("#config-version=FGT61F-7.0.14-FW-build0601-240206:opmode=1:vdom=0:user=lm\n#conf_file_ver=542255083128888" + myNum + "\n#buildno=0601\n#global_vdom=1\nconfig system global");
return 0
LM User
·1 year agoI think this may be a bug, but LM may say working as designed. Since the line is ignored, I'm not sure how LM is supposed to handle a change in an ignored line. Obviously it shouldn't alert, and it isn't. But should it register a new version? Maybe, maybe not. One reason i can think of to not register a new version is that if the ignored line contained the current time, it would change every time it was collected. If LM registered a new version for every change, including an ignored line with the time on it, you'd have a new version every poll cycle. For this reason, I think LM may consider this "working as designed".
The script may be running, not using a cache or anything else. Your "test script" executions have intimated as much. It comes down to LM's decision whether or not to consider the difference sufficient enough to merit registering a new version in the UI.
Clarify one more thing though: your screenshots of the diff view above show the same timestamp on both sides. Obviously that's a UI inconvenience that you can show the same thing on both sides, or that the same thing comes up on both sides by default. I'm not sure i completely understand: are you saying it's registering a new version when there truly is a change, but the new version has the same content as the old version?