Ad Widget

**catkins** · 29-12-2021, 07:23

I worked around the failure by making the Zabbix external script:

Create a temporary file
Run a bash script wrapper for the Selenium script via "at now", giving it the temporary file path
Wait for the "at" job to finish
Collect item names and values from the temporary file and send them to Zabbix

OK except the Zabbix external script now takes a little more than 30 seconds to finish (Selenium is heavy, it starts a full web browser)

**catkins** · 31-12-2021, 07:07

> I worked around the failure ...
The workaround did not work because the external script's stdout pipe (and stderr pipe), normally used to write data to Zabbix had disappeared after the "at" job finished.

> OK except the Zabbix external script now takes a little more than 30 seconds to finish
Was a bug in my script, a wait loop that did not break. After fixing, run time was ~7 sec

**catkins** · 09-01-2022, 08:47

Regards "the external script's stdout pipe (and stderr pipe), normally used to write data to Zabbix had disappeared after the "at" job finished", confirmed the pipe disappeared soon after the at job finished (fd10 was a copy of fd1 when called by Zabbix)

Code:

DEVEL: run_get_comap_data.sh_via_at.sh function run_get_comap_data_sh before at job finished. For fd 10
buf: l-wx------ 1 zabbix zabbix 64 Dec 31 14:08 /proc/1303/fd/10 -> pipe:[3575077]
pipe_n: 3575077
sh 1302 zabbix 1w FIFO 0,12 0t0 3575077 pipe 1302,sh,2w 1303,run_get_c,10w 1589,grep,10w 1590,grep,10w
sh 1302 zabbix 2w FIFO 0,12 0t0 3575077 pipe 1302,sh,1w 1303,run_get_c,10w 1589,grep,10w 1590,grep,10w
run_get_c 1303 zabbix 10w FIFO 0,12 0t0 3575077 pipe 1302,sh,1w 1302,sh,2w
grep 1589 zabbix 10w FIFO 0,12 0t0 3575077 pipe 1302,sh,1w 1302,sh,2w
grep 1590 zabbix 10w FIFO 0,12 0t0 3575077 pipe 1302,sh,1w 1302,sh,2w
DEVEL: run_get_comap_data.sh_via_at.sh function run_get_comap_data_sh after at job finished. For fd 10
buf: l-wx------ 1 zabbix zabbix 64 Dec 31 14:08 /proc/1303/fd/10 -> pipe:[3575077]
pipe_n: 3575077
run_get_c 1303 zabbix 10w FIFO 0,12 0t0 3575077 pipe
grep 1603 zabbix 10w FIFO 0,12 0t0 3575077 pipe
grep 1604

Examined the Zabbix zbx_execute source code. Did not see anything which would explain the behaviour

Worked around by creating a script, independent of Zabbix, scheduled to run repeatedly to get the values and write them to a cache file and changed the Zabbix external script to get item names and values from the cache file. An ugly asynchronous solution but needs must when the devil drives.

**tim.mooney** · 12-01-2022, 00:39

This is a nice bit of debugging and thank you for posting it.

Does the web interface you're scraping require JavaScript capability in the "browser" you're using? I was just thinking you might have an easier time scripting it with a less-complicated web scraper, but that's only an option if something like wget, curl, or the web modules for a scripting language have enough features. Typically the simpler scripting options have to be abandoned when JavaScript is required in the browser. I suppose you considered them and were forced to go the Selenium route?

Note that sometime after Zabbix 6.0 releases, there's more advanced web scraping features planned for the product. They mention plans for Selenium-like capabilities in the roadmap, I believe.

**catkins** · 12-01-2022, 09:26

> Does the web interface you're scraping require JavaScript

Yes, it has

Code:

<script type="text/javascript" src="md5.js"></script>

<script language="JavaScript">
function DoHash()
{
var psw = document.getElementById('psw_id');
var hpsw = document.getElementById('hpsw_id');
var nonce = hpsw.value;
hpsw.value = MD5(nonce.concat(psw.value));
psw.value = '';
return true;
}
</script>

That is run on clicking OK after entering an authentication string

> I was just thinking you might have an easier time scripting it with a less-complicated web scraper

Thanks for the suggestion, Tim

wget was tried but the second call got "Too many other clients connected. Try your request later"
lynx and elynx were rejected as incomplete and not currently maintained
Python's beautifulsoup
- Did not have Xpath support (all the Xpaths were known from the Selenium development)
- Was extended with lxml's etree for Xpath support
- Considered extending beautifulsoup with
  - dryscape "might work but probably doesn't"
  - https://www.npmjs.com/package/jssoup "might work but is not widely used"
- Was found not to be able to click web elements
- Considered adding click functionality by extending with "requests" but I did not find how to make that work
- Rejected because it cannot click
Python's scrapy was considered. It was rejected because "https://docs.scrapy.org/en/latest/to...hitecture.html shows a much more elaborate scheme than we need but we can probably use just the parts we need but a quick reading of https://docs.scrapy.org/en/latest/topics/commands.html did not show how to do that".
Python's mechanicalsoup was tried but I did not find how to make it click
PhantomJS was considered but is not maintained
https://stackoverflow.com/questions/...-with-requests. Has an example of writing a JavaScript or JQuery function to enter data and click then running the function. Was not pusued because it was the only reference to using the technique and had not got a "that worked for me" comment

> Note that sometime after Zabbix 6.0 releases, there's more advanced web scraping features planned

I started with Zabbix web scraping and soon found it was too basic for the task

BTW, for anyone considering Selenium, it is essential to try-except all errors which may stop the script before running driver.quit() because

Unclosed browser processes accumulate and seem to interfere with Selenium. When killed they become zombies (rare on current Linuxes)
A error crashed Selenium script process sends SIGTERM to its parent process

Ad Widget

Script fails when run as external script. OK by cron, at and interactive

Script fails when run as external script. OK by cron, at and interactive

Comment

Comment

Comment

Comment

Comment