AS/400 Monitoring solutions

[Kos]

I've uploaded the updated version (v0.7.1) at share.zabbix.com site.

Gerald, try, please, this version to check the auto-reconnection mechanism. Unfortunately, I have a limited possibility to test this feature seriously, so your experience is valueable for me.

I hope that this version should, finally, work with acceptable stability in your configuration
In any case - thank you in advance for your feedback!

Update: I've updated the template also to include additions that appeared in v0.7.0 (as400.disk.state and vfs.fs.state metrics) as well as macro with context.

[KevC]

Hello KevC,
I'm glad that my work is useful for anybody :-)

Regarding your questions:

Full Message id:
You can filter by the full EventID field using a regular expression if you use it in the item's key. In this case the filtering is performed on the agen's side, where is has a full EventID (contrasting to filtering in trigger on the server side, where only digital part of EventID is available). By the way, documentation has an example of this trick.

Users:
Unfortunately, there is no such functionality at the moment. In theory, API provides the methods getUser() (returns the sender job's user) and getCurrentUser() (returns the current user name). However, in general case, both these methods could return an empty string (""). Additionally, it's impossible to transfer these values onto Zabbix server (the only way that I see is to concatenate this field to some ather existing field like text of message). Finally, it will require to extend the Item's key to accept an additional parameter. So, at the moment I'm not sure if it is really needed. Maybe, filtering by the Job name could be enough in your case?

Library:
As I unferstand, the library name is part of IFS path name of the message queue (again, see example in documentation). You could consult with your AS/400 admin for a details, but I'm supposing that you could use something like that as the queue name: "/QUSRSYS.LIB/MIMIX.MSGQ".

Hi Kos,
Seems we can do most everything we needs except filter based on user name. Our Developers do testing on PROD machines and we need to filter messages generated by the developers. Seems there are no other options for them to indicate this is a test message. Would it be possible to append to the beginning or end of the string and if blank, either don't tack it on or put something like "username_blank". I will be OOO next week, I will check when I return, maybe you will have a new version by then.

[Kos]

Zabbix Agent emulator v0.7.2

Hi Kos,
Seems we can do most everything we needs except filter based on user name. Our Developers do testing on PROD machines and we need to filter messages generated by the developers. Seems there are no other options for them to indicate this is a test message. Would it be possible to append to the beginning or end of the string and if blank, either don't tack it on or put something like "username_blank". I will be OOO next week, I will check when I return, maybe you will have a new version by then.

OK, the version V0.7.2 is available.

The only difference is a new feature: possibility to filter messages in message queue by the "current user" metadata. An additional parameter to key eventlog[...] is a case-insensitive regular expression describing needed users, for example:

eventlog[QSYSOPR,,40,,,100,,zabbix|QsYs]

(all messages from the QSYSOPR message queue, with severity 40 and above, written by "ZABBIX" or "QSYS" users).

An additional parameter to config file is also added: as400UserAsMessagePrefix (with default value = zero). If it set to 1, then the name of current user is added to the message text as prefix (to have possibility use it in a triggers).

[RohrbaGe]

Kos,

unfortunateley still the same.
I installed in the morning your new version.

At 15:00 it stopped like extected to work,
but in the evening when the AS400 was available again no values came.

Attached the log. As I had problems to upload a .zip so I renamed to .sql.
So you need to rename to zip back.

Do you think the behaviour will be different, if I run the agent on the AS400 diretly or do you sthink it will be still the same?

[Kos]

Gerald,
thank you for a feedback. I could repeat that it is very valuable for me as it helps me to find and fix a defects.

Analyzing this log, I can see several different problems:

1) there is no fixed timeout for connection to AS/400 system. For example, thread #16 asked to the AS/400 at 15:07:46.297 but received negative result (timeout) at 15:23:17.245 only - after 15 and a half minutes! Another thread, #17, started request at 15:23:12.638 but received timeout at 15:38:43.452 (the same 15 and half minutes). As there was no specific timeout set in my program, it seems that this duration is operating system-dependent.

2) during this timeout a Zabbix-server has time to send a lot of new requests. They are queued by agent for processing, but all threads are busy this moment; so it's similar to DOS-attack to agent. Most of these requests should be timed out from the Zabbix-server side (probably, with the messages in the zabbix_server.log).

3) it seems that my program has a bug that reveal itself in the high-load condition like described above. As result, the listener thread goes to the dead-lock state and stops to responce.

I believe that I'll fix, at least, the last mentioned problem soon.

The agent running on the AS/400 system directly, evidently, has slightly different behaviour: it does not connect to this system via network; so, communications to AS/400 always should be OK. However, if you stop AS/400 or its networking subsystem, then you will have no connectvity to it from Zabbix-server.
Another thing you can try is the usage of active mode. It uses a single thread only for all metrics, and a bit different communication mechanism to Zabbix-server. I think that it is a bit more reliable for your situation.

At the same time, I'll be grateful if you could take a once more test with an updated version (I hope to publish it before a weekend).

[Kos]

Zabbix Agent emulator v0.7.3

Version 0.7.3 uploaded.

Welcome to try it.

[RohrbaGe]

Kos,

just installed the new version.
Let`s see the difference.


Regards

Gerald

[RohrbaGe]

Kos,
unfortunately the same behaviour.
When the AS400 is online again, still no values are coming.



Attached the log file, I renamed the zip to xml because of the size limitations.


Regards

Gerald

[Kos]

Kos,
unfortunately the same behaviour.
When the AS400 is online again, still no values are coming.

Attached the log file, I renamed the zip to xml because of the size limitations.

Gerald,
it seems that this log file was created by the agent v0.7.0.
Could you like, please, to repeat this test with version 0.7.3?
Thanks in advance!

[Kos]

By the way, if someone like to meet with me personally, it could be happened at Zabbix Conference 2017 in September, Riga. The official agenda contains my speech about this agent simulator for AS/400

[sancho]

Hi Kos,
Previously, the agent disconnected from the zabbixserver every 2 days + -, after this last update (0.7.3) has not happened again.
I do not know if this was the same problem that Gerald had.

Thank you again for the great work you are doing.

[RohrbaGe]

Gerald,
it seems that this log file was created by the agent v0.7.0.
Could you like, please, to repeat this test with version 0.7.3?
Thanks in advance!

Kos, you are right, somehow I mixed the versions.
I installed now the actual one.
After my vacation I found the time to check and correct.
Next weekend we will see if it works.
Sanchos comment makes some hope.

[Kos]

Thanks all for a replies.
Just a note: I will be out of office next 2 weeks (till the end of month); therefore I could not reply on forum quickly.
But, anyway, I'll wait for results of your testing.

[RohrbaGe]

Kos,

I found some time for testing.
I installed your client, agend files and the .jar files on the as400.

One problem I noticed is the name of the json-simple-1.1.1.jar.
Running before the agent from a remote ubuntu, I provided with
classpath a different file name.
On the as400 this seems not to work.
On the links in your documentation I did not found the right file,
just json_simple-1.1.jar I found. Maybe my mistake somewhere.

However, with my low AS400 knowledge I got it running.
We have V7R1 running, default is Java 1.6. I do not want to change the default, because we are using Java apps in our ERP system.
So I finally found a way to make it working.
Probably not perfect, but maybe it gives some others with old Java as default an idea.
As written above, I doubt somehow that the Classpath works.
But I do not want to store the .jar files in the standard AS400 folders.

But defintely the right Java version is started with my CL.

I wrote my first CL....
strpdm

File QCLSRC
Library QGPL

F6 ( Create new file)
Source file ZABBIXJAVA
Type CLP
Description Zabbix Java Agent


0001.00 PGM
0002.00 RMVENVVAR ENVVAR(JAVA_HOME)
0003.00 MONMSG MSGID(CPF0000)
0004.00 RMVENVVAR ENVVAR(CLASSPATH)
0005.00 MONMSG MSGID(CPF0000)
0006.00
0007.00 ADDENVVAR ENVVAR(JAVA_HOME) +
0008.00 VALUE('/QOPENSYS/QIBM/ProdData/JavaVM/jdk80/32bit')
0009.00 ADDENVVAR ENVVAR(CLASSPATH) +
0010.00 VALUE('/home/ZABBIX/jt400.jar:/home/ZABBIX/+
0011.00 json-simple-1.1.1.jar')
0012.00 JAVA CLASS('/home/ZABBIX/ZabbixAgent.jar') +
0031.00
0032.00 ENDPGM

Save the file and mark with 14. (Compile)

Login as user ( I created user ZABBIX before )

CALL ZABBBIXJAVA

The Java shell is opened, that´s something I need to check.
But at the end I will run the agent in an own subystem like you wrote in your documentation.
I 'm using your template also and added some of our special items.

So for the next weeks I will probably run the remote agent and the agent on the as400 parallel. I want to check the behaviour, when the AS400 is doing a full backup on sundays and is not available by IP.

Thanks for your help and good luck for your presentation.

I do see your client as a very good basic for monitoring the AS400.
Still a lot`s of companies are using this good old technology.
With your agent a monitoring of resources needed for ERP
processes is very simple and helps the admin's a lot.


Gerald

[Kos]

Hello Gerald,
thank you for sharing of your experience, it is interesting and useful.

On the links in your documentation I did not found the right file,
just json_simple-1.1.jar I found. Maybe my mistake somewhere.

I've checked once more. The link in documentations is this, it's the main page of this project. The section "Getting Started" up to now contains this link where I obtained this JAR-file before. Unfortunately, this link seems to be broken now :-(
Probably, more correct page could be the following: it contains a direct links to some files, including json-simple-1.1.1.jar. The only note: my browser by some reason tries to save this file with ".zip" extension instead of ".jar", it should be renamed correctly.

At the same time, there are a lot of other sites where this JAR could be downloaded from. For example, this page (the first diskette icon has a correct link) or this page (the "Download (BUNDLE)" link).

Probably, I need to update a documentation with the more "fresh" links; thank you for a pointing me to this problem.

It is still very interesting for me your testing with the "remote" mode of this agent :-)