AS/400 Monitoring solutions

[MNfromH]

I've seen zabbix agents running on OpenWRT so I'm certain that any zabbix agent, with a little modification, will run on the i as well.

I don't think that a linked libperfstat will do since nothing libperfstat does works in pase.

An aproach to fix this could be a modified version of the agent with no reference to libperstat at all. On the i you wouldn't notice any differecne since it doesn't return any sane results anyway.

I've acces to an AIX 7.1 box with gcc. I guess I'll give it a try later.

[MNfromH]

Oh, just to make sure. You are now running the agent with a user that has GRPPRF(*NONE) ?

[FabrizioA]

Hi MNfromH,

sorry, but I re-checked now and my ZBX user belong to ZABBIX group profile.

Previously, the agent can't start when I login with the QSECOFR user... then I created the ZBX user (copied from QSECOFR) and I have assigned ZABBIX group profile to him.

[RohrbaGe]

MNfromH,

would you share your AS400 scripts?
Especially the one for the message queue I'm very interested in.

Also I would like to get the zabbix agent on he AS400 installed.
Unfortunately Iḿ not too much expert on OS400, so it will take a while.....





Regards

Gerald

[FabrizioA]

Hi RohrbaGe,

basically I have used the Aix's binary agent on the AS400.

[RohrbaGe]

FabrizioA,

on the screenshot from MNformH I saw the scipts he is using.
Main interest for us is to get a message, when a msgw is
stopping executing other jobs.

|-- local.ibmi.msgw.sh

Are you using any script to get aware for qsysopr messages?


Regards

Gerald



ser@idev:/etc/zabbix # tree /etc/zabbix/
/etc/zabbix/
|-- scripts
| |-- local.ibmi.asp.total.gb.sh
| |-- local.ibmi.asp.used.gb.sh
| |-- local.ibmi.asp.used.percent.sh
| |-- local.ibmi.cpu.load.percent.sh
| |-- local.ibmi.hostname.sh
| |-- local.ibmi.job.status.sh
| |-- local.ibmi.msgw.sh
| `-- userparameters.conf
`-- zabbix_agentd.conf

1 directory, 10 files

[MNfromH]

Hi all,

sorry, somehow I didn't get notified about any new messages here.

@RohrbaGe
The zabbix 1.4.6 AIX binaries do work well on the i using QSH/PASE and creating all config files in a unixish way in the IFS. See my first few posts for more details.

The content of local.ibmi.msgw.sh:

content=$(system 'wrkactjob' | grep MSGW | awk '{printf "%s ( %s/%s/%s )\n",$1,$3,$2,$1 }' )
if [ "$content" == "" ] ; then
  stored_content="$(cat /tmp/zbxtmpmsgw.txt)"
  if [ "$stored_content" == "-" ] ; then
    echo "-"
  else
    echo "OK: $stored_content"
    echo "-" > /tmp/zbxtmpmsgw.txt
  fi
else
  echo "$content" | tee /tmp/zbxtmpmsgw.txt
fi
exit 0

It will do the following:

First, it checks for any job in the system with the status MSGW and formats them to:

"JOBNAME ( JOBID/USER/JOBNAME )" <-- This is the text that will be used in the notification mail. You can copy paste this to use the text in the brackets to use with WRKJOB .

For example:

somejob ( 544743/QPGMR/somejob )

It will also store the jobs names (it works with multiple jobs at the same time) in /tmp/zbxtmpmsgw.txt .

On the next run of the script, the script will check the list of all jobs in MSGW again, compare it with the j obs in /tmp/zbxtmpmsgw.txt and will return

OK: somejob ( 544743/QPGMR/somejob )

in case the job no longer resides in MSGW.

If no job is found, it return just:

-

In Zabbix I created the following trigger:

Name: MSGW: {ITEM.VALUE}
Expression:

{ibmi:local.ibmi.job.msgw.str(OK: )}#1  | {ibmi:local.ibmi.job.msgw.str(-)}#1

So every returned value with "OK: " or "-" is fine and the trigger will be activated when some thing else gets returned.


There is a downside to this though: The while script needs 1s on average to be processed.

The PASE and QSHELL environemnt is incredibly slow on processing scripts like this because it wastes loads of time starting each command as a separate process. Compared to a real unix/linux system that is.

[MNfromH]

Are you using any script to get aware for qsysopr messages?

Nope. We log most of the critical messages, interesting messages to IFS logfiles.

You could do something with

system 'DSPMSG MSGQ(QSYSOPR)'

but depending on size of the msgq this could take ages to process.

[Kos]

Hello, guys!

I have some ideas about alternative approach to this topic.

First of all, there is a Java interface to AS400: IBM Toolbox for Java. This interface exists a long time already, so it is mature enough. It provides access to some important AS400 internals, including message queues, output queues, subsystems and jobs. What is even more valuable, some classes (such as ProgramCall and ProgramCallDocument) provide an interface to the native AS400 API; so, in theory - it's possible to access any existing AS400 system API.

On the other hand, there is a protocol describing communications between Zabbix-server and Zabbix-agent (link1, link2). Moreover, there are open sources of entire Zabbix product, so it's possible to become familiar with the "reference implementation" of this protocol.

So, my idea is: to write a Java-program emulating the behaviour of Zabbix-agent. In theory, such program could run either on the AS400 system directly or on some other host and connect to AS400 via network (both modes have their pros and contras). So, from the Zabbix-server's perspective, such program looks like a usual Zabbix-agent. At the same time, this program can use Java API and AS400 API to obtain needed information. Ideally if this program could support the most of standard Zabbix-agent's item keys (as far as they have sense for this architecture).

I tried to write such a program on my own. I had some experience in Java programming, but the last 15 years my primary work is more the system administration; so I'm not sure that my result is a perfect. Nevertheless, it is good "proof of concept": it really works.

Of course, it is not a full-fledged Zabbix-agent and it has a lot of limitations. Some of them:

• lack of IPv6 (Ipv4 only supported);
• no encryption (sorry);
• only limited subset of standard metrics supported, and even in this case: some of them have a bit different semantics of parameters;
• some config file's parameters are recognized, but really ignored (PidFile, EnableRemoteCommand, Alias, AllowRoot, Include, UserParameter and LoadModule);
• global regular expressions in key parameters for the active mode are not supported (meanwhile);
• minimum value for the StartAgents parameter is 1 (i.e. active-only mode is not supported).

At the same time, some supported features are:

• the config and log files of the same format as the original Zabbix-agent;
• both passive and active modes (according to configured parameters);
• multi-threading: for active mode - separate thread for each Zabbix server, for the passive mode - configured number of threads;
• the following metrics (important for us):
  • proc.num[<name>,<user>,<state>,<subsystem>] – number of jobs with the given name, user and state in the given subsystem. All parameters are optional (default: all). <subsystem> is regular expression. <state> is a case-insensitive standard abbreviation of job state (like "sigw", "LCKW" or "OUTQ", default: all running);
  • vfs.fs.discovery, vfs.fs.size[<ASP>,<mode>] – discovering and size of ASP's (at least, should be ASP1, i.e. system ASP);
  • eventlog[<name>,<regexp>,<severity>,<source>,<eventid>,<max lines>,<mode>] – messages from the message queue <name>. The first parameter is mandatory (all others are optional), it could be either fully qualified IFS path name or the name only (in the last case the IFS path name is formed as "/QSYS.LIB/" + name + ".MSGQ"). Unlike to standard metric, the <severity> parameter is not a regular expression but a number restricting the minimal level. <regexp>, <source> and <eventid> are regular expressions, <source> is the name of job generated the message.
  • as400.subsystem[<subsystem>,<library>] – string describing the state of subsystem (usually, "*ACTIVE" or "*INACTIVE");
  • as400.queue.size[<outputQueue>,<library>] – number of spool-files in the specified output queue;
  • as400.disk.discovery, as400.disk.size[<disk>,<mode>] and as400.disk.state[<disk>] – discovering, size and state of the disks (or LUN's for SAN-attached devices).

So, at the moment I'd like to ask if such program could be interesting to someone else. In this case, probably, it could be published as the open source somewhere like https://share.zabbix.com after some time (at least, to prepare documentation). Probably, the separate topic could be created on forum (with a screenshots, examples, etc.).

[Voxie]

So, at the moment I'd like to ask if such program could be interesting to someone else. In this case, probably, it could be published as the open source somewhere like https://share.zabbix.com after some time (at least, to prepare documentation). Probably, the separate topic could be created on forum (with a screenshots, examples, etc.).

We are interested in an agent for OS400.
Can you share the source code or the working "proof-of-concept" so we can test it.

[MNfromH]

This sounds very interesting indeed!

If you need any testing, I'd be more then happy to test.
Is there a way to add UserParameters?

Max

[Kos]

OK, thank you for interest. 2 replies during the first hour - it is a good signal
Well, I'll prepare my project for a publishing (as I wrote, at least some documentation is needed).

Is there a way to add UserParameters?

At the moment - no
Unfortunately, AS400 architecture is specific enough. For example, unlike to Unix/Linux, is has no concept "standard output (stdout)". So, it' difficult for me to understand what exactly should perform this "Zabbix agent emulator" processing the UserParameters. Maybe, someone has a more clear ideas?

[Kos]

I had registered myself on https://share.zabbix.com and I've prepared some description (at least, how to start, and the list of supported metrics). At the moment, I'm waiting for share.zabbix.com mainteners to accept my yesterday's query to add "AS/400" item into "Operating Systems" list.

[Kos]

My project published on share.zabbix.com.
I've prepared a detailed description (metrics supported, requirements, starting) and some screenshots as example.

Welcome to testing and commenting

Update:
Version updated: metrics system.localtime, system.users.num and system.cpu.num added.

[Kos]

Version updated (0.5.0).
Fixed bug: large replies from server (>10KB) was not processed correctly (for example, a big list of active checks).
Metrics proc.cpu.util and proc.cpu.util.discovery added and documented.

Update:
Version updated to 0.5.1 (small bugs fixed).
Example of template published also.