AS/400 Monitoring solutions

[Kos]

Gerald,
thank you for your feedback, it is valueable for me.

It is a interesting effect about negative values for CPU utilization. I didn't see that; but I'll check this moment. What kind of metric do you use? I mean, is it a specific job (identified by JobNumber/JobName/JobUser triplet) or some kind of summary statistic (filtered by subsystem, by jobname, by username, or not filtered at all)?

Monitoring of disk I/O is not implemented yet.
I need to investigate how difficult to add this fuctionality.

[RohrbaGe]

Kos,

that´s the item:
proc.cpu.util[,,,,avg1,]

So it`s a summary.

It went down to minus 400% e.g.
It was before a connection drop to the I5 or the java process was restartet

But there where some other negative values before also.
Attached a screenshot, so you got an idea.
It does not happen very often.

Regards

Gerald

[RohrbaGe]

Kos,

during the weekend we are doing a full backup of the as400.
That meean also TCPIP is shutdown, the as400 is not reachable by IP.

It´s clear, that during this time I get no values.
But when the machine is reachable by IP again, the Java client running from a remote ( the zabbix server) is not delivering any value.
I need to stop and start the java programm to get it working again.

Probably you have a different situation and your machine is never offline.
Unfortunatley we have on the As400 the old Java version only, in some month we get a new machine, I hope we can upgrade to a newer OS version.

Maybe it´s easy to fix. I can run a scheduler, buts this is no good solution.

Regards

Gerald

[Kos]

Gerald,

Thank you for a feedback.

1) negative values for proc.cpu.util metric. I'll check that, seems like a bug.

2) autoreconnect for described configuration was implemented in v0.6.0. The current version is 0.6.1 (bug with releasing some resources was fixed). What version do you use in your tests? Are there some messages in the log (there should be something like "communication error, trying to reconnect...")?

[RohrbaGe]

Kos,

I´m using 6.01 since the beginning of my tests.

It seems that the log file is missing entries during this time.


Reagrds

Gerald

12:20170217:160457.742 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 143632/SOFTM/UCS08 nicht gefunden.
12:20170219:060024.752 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 144293/SOFTMOPR/SFIREUPD nicht gefunden.
12:20170219:150525.418 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 144465/QSECOFR/QP0ZSPWP nicht gefunden.
12:20170219:150534.691 Procstat.updateJobinfoList() communication to AS/400 error: java.net.SocketException: Connection reset
16:20170219:152316.468 ZabbixAgent.process() 'system.uname' error: java.net.SocketException: Keine Route zum Zielrechner (Read failed)
12:20170219:174248.811 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 144621/QCPMGTDIR/QP0ZSPWP nicht gefunden.
12:20170219:174251.270 Procstat.updateJobinfoList() communication to AS/400 is working again
12:20170219:174257.231 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 144838/QCPMGTDIR/QP0ZSPWP nicht gefunden.
12:20170219:174730.197 Procstat.updateJobinfoList() error: com.ibm.as400.access.AS400Exception: CPF3C53 Job 144968/QCPMGTDIR/QP0ZSPWP nicht gefunden.

[Kos]

Thank you for your feedback.
These messages:

12:20170219:150534.691 Procstat.updateJobinfoList() communication to AS/400 error: java.net.SocketException: Connection reset
[...]
12:20170219:174251.270 Procstat.updateJobinfoList() communication to AS/400 is working again

are normal for auto-reconnection mechanism. First of them is result of Java exception (some subclass of java.io.IOException), the second - result of reconnection. It is normal if there are several pair of similar messages (each Java thread has its own flag of communitation state).

However, this message has no the appropriate recovering message:

16:20170219:152316.468 ZabbixAgent.process() 'system.uname' error: java.net.SocketException: Keine Route zum Zielrechner (Read failed)

java.net.SocketException is also subclass of IOException, so it should be catched and processed.
What kind of check do you use for 'system.uname' metric - active or passive?

I can run a scheduler [...]

Just for the case if you decide to use some kind of scheduler: this agent emulator could be stopped gracefully just sending him request for 'agent.exit' metric (for example, using 'zabix_sender' utility). Of course, the config file must allow the appropriate sender's host (in the 'Server=' config. file parameter).

[Kos]

I found some strange negative values for CPU utilization, but in general it seems to be ok. I will try to find where the negative values, (very seldom)
are coming from.

Gerald, thank you for pointing me to this problem. I was able to reproduce it.

It seems like a bug (or, at least, some undocumented "feature") in API used. We get the total CPU used (in milliseconds) for each job, and compare it with the previous value for the same job to obtain the CPU usage for the last "tick". In theory, the total CPU usage for the job should be increasing only. In reality, sometimes we suddenly receive 0 (zero) for several jobs that before already had non-zero total CPU usage. As result, the negative values are appearing.

I've implemented a workaround for this case, so I'll upload an updated version in several days.

[Kos]

I've implemented a workaround for this case, so I'll upload an updated version in several days.

Version 0.6.3 was published here.

Change list:

• some additional debugging info added;
• initialization improved. For example, if the JSON library could not be found, the program will stop gracefully immediately (instead of some strange semi-live state);
• negative values of CPU Usage should be avoided now.

[sancho]

Hi Kos,
Just to write to thank you for the fantastic work you have done, until now we had not been able to include our AS400 in monitoring zabbix except by SMNP.

Congratulations

Sorry for my bad English

[RohrbaGe]

Kos,

I missed that you updated....

I have upgraded just now my agent.

I read you fixed the isssue with negative values.

Did you fixed also the issue with the none reachable AS400?
At the weekend our AS400 is not reachable by IP, at Sunday 18:00 it´s again available. I have to restart the agent, to get it working again.
In case of other issues or network problems we want get the actual status.
With a trigger we can make this visible, but this is not the best way.


Regards

Gerald

[Kos]

Did you fixed also the issue with the none reachable AS400?

There was no changes regarding this point.
Gerald, did you read my remark here?
What type of checks do you use - active or passive?

[RohrbaGe]

Kos,

I thought I answered this, but probably I was interrupted and forgot about.


I use zabbig agent only, no active.


In some weeks we will get a new i5, I hope then we can get the new Java version to run the agent direct on the AS400 and not like now on the zabbix server.


Regards
Gerald

[grantd]

Hi Kos,
Having a little issue getting it to start on my Linux system to connect to the AS/400 without a X11 setup

Below is the error message, the directory listing, Zabbix log, and Zabbix conf files

Thanks,
Grant



[root]# java -jar ZabbixAgent.jar /opt/zabbix_as400/zabbix_agentd.conf
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(Graphic sEnvironment.java:204)
at java.awt.Window.<init>(Window.java:536)
at java.awt.Frame.<init>(Frame.java:420)
at java.awt.Frame.<init>(Frame.java:385)
at com.ibm.as400.access.ToolboxSignonHandler.setupPas swordDialog(ToolboxSignonHandler.java:573)
at com.ibm.as400.access.ToolboxSignonHandler.handleSi gnon(ToolboxSignonHandler.java:496)
at com.ibm.as400.access.ToolboxSignonHandler.connecti onInitiated(ToolboxSignonHandler.java:50)
at com.ibm.as400.access.AS400.promptSignon(AS400.java :2929)
at com.ibm.as400.access.AS400.signon(AS400.java:4246)
at com.ibm.as400.access.AS400.getVRM(AS400.java:2451)
at com.ibm.as400.access.ProgramCall.setParameterList( ProgramCall.java:855)
at com.ibm.as400.access.ProgramCall.setProgram(Progra mCall.java:896)
at com.ibm.as400.access.ProgramCall.<init>(ProgramCal l.java:317)
at com.ibm.as400.access.SystemStatus.loadInformation( SystemStatus.java:1013)
at com.ibm.as400.access.SystemStatus.loadInformation( SystemStatus.java:970)
at com.ibm.as400.access.SystemStatus.getSystemName(Sy stemStatus.java:789)
at as400.metric.As400Metrics$1.process(As400Metrics.j ava:30)
at as400.thread.ZabbixAgent.process(ZabbixAgent.java: 25)
at as400.Config.setDefaultsAndValidate(Config.java:29 7)
at as400.thread.ZabbixAgent.run(ZabbixAgent.java:106)
Exception in thread "ZabbixAgent config"


[root]# ls -la
total 4804
drwxr-xr-x 2 root root 121 Apr 4 13:10 .
drwxr-xr-x 3 root root 22 Apr 4 13:07 ..
-rw-rw-r-- 1 root root 23931 Mar 31 09:28 json-simple-1.1.1.jar
-rw-rw-r-- 1 root root 4779571 Apr 4 13:03 jt400.jar
-rw-rw-r-- 1 root root 8056 Apr 4 13:09 zabbix_agentd.conf
-rw-rw-rw- 1 root root 2185 Apr 4 13:10 zabbix_agentd.log
-rw-rw-r-- 1 root root 102241 Mar 2 09:34 ZabbixAgent.jar


[root]# cat zabbix_agentd.log
8:20170404:131034.675 Starting Zabbix Agent v0.6.3
8:20170404:131034.677 using configuration file: /opt/zabbix_as400/zabbix_agentd.conf
8:20170404:131034.678 Oracle Corporation Java version "1.8.0_121"
8:20170404:131034.678 OpenJDK Runtime Environment (build 1.8.0_121-b13)
8:20170404:131034.679 OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
8:20170404:131034.679 Open Source Software, JTOpen 9.1, codebase 5770-SS1 V7R3M0.00 built=20160705 @RF
11:20170404:131034.911 agent #1 started [collector]
13:20170404:131034.919 agent #3 started[listener #2]
12:20170404:131034.920 agent #2 started[listener #1]
11:20170404:131034.928 Exception in Procstat.updateJobinfoList():
java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(Graphic sEnvironment.java:204)
at java.awt.Window.<init>(Window.java:536)
at java.awt.Frame.<init>(Frame.java:420)
at java.awt.Frame.<init>(Frame.java:385)
at com.ibm.as400.access.ToolboxSignonHandler.setupPas swordDialog(ToolboxSignonHandler.java:573)
at com.ibm.as400.access.ToolboxSignonHandler.handleSi gnon(ToolboxSignonHandler.java:496)
at com.ibm.as400.access.ToolboxSignonHandler.connecti onInitiated(ToolboxSignonHandler.java:50)
at com.ibm.as400.access.AS400.promptSignon(AS400.java :2929)
at com.ibm.as400.access.AS400.signon(AS400.java:4246)
at com.ibm.as400.access.ProgramCall.chooseImpl(Progra mCall.java:411)
at com.ibm.as400.access.ProgramCall.run(ProgramCall.j ava:771)
at com.ibm.as400.access.JobList.load(JobList.java:154 0)
at com.ibm.as400.access.JobList.getJobs(JobList.java: 991)
at as400.perfstat.Procstat.updateJobinfoList(Procstat .java:209)
at as400.thread.CollectorThread.run(CollectorThread.j ava:30)
11:20170404:131034.929 agent #1 stopped [collector]
14:20170404:131034.931 agent #4 started[listener #3]
14:20170404:131034.931 agent #4 stopped[listener #3]
12:20170404:131036.925 agent #2 stopped[listener #1]
13:20170404:131036.927 agent #3 stopped[listener #2]
1:20170404:131036.928 Zabbix Agent stopped. v0.6.3


[root]# cat zabbix_agentd.conf
# This is a config file for the Zabbix agent daemon (i5/OS)
# To get more information about Zabbix, visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: PidFile
# Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_agentd.pid

### Option: LogFile
# Name of log file.
#
# Mandatory: no
# Default:
# LogFile=/tmp/zabbix_agentd.log

LogFile=/opt/zabbix_as400/zabbix_agentd.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1

### Option: DebugLevel
# Specifies debug level
# 0 - no debug
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default:
# DebugLevel=3

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: EnableRemoteCommands
# Whether remote commands from Zabbix server are allowed.
# 0 - not allowed
# 1 - allowed
#
# Mandatory: no
# Default:
# EnableRemoteCommands=0

### Option: LogRemoteCommands
# Enable logging of executed shell commands as warnings.
# 0 - disabled
# 1 - enabled
#
# Mandatory: no
# Default:
# LogRemoteCommands=0

##### Passive checks related

### Option: Server
# List of comma delimited IP addresses (or hostnames) of Zabbix servers.
# Incoming connections will be accepted only from the hosts listed here.
#
# Mandatory: no
# Default:
# Server=

Server=MYZABBIXSERVER

### Option: ListenPort
# Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10055

### Option: ListenIP
# List of comma delimited IP addresses that the agent should listen on.
# First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

### Option: StartAgents
# Number of pre-forked instances of zabbix_agentd that process passive checks.
# (If set to 0, disables passive checks and the agent will not listen on any TCP port.)
# Currently must be minimum 1.
#
# Mandatory: no
# Range: 1-100
# Default:
# StartAgents=3

##### Active checks related

### Option: ServerActive
# List of comma delimited IPort (or hostnameort) pairs of Zabbix servers for active checks.
# If port is not specified, default port is used.
# If port is not specified, square brackets for IPv6 addresses are optional.
# If this parameter is not specified, active checks are disabled.
# Example: ServerActive=127.0.0.1:20051,zabbix.domain
#
# Mandatory: no
# Default:
# ServerActive=

#ServerActive=zabbix.yourcompany.com

### Option: Hostname
# Unique, case sensitive hostname.
# Required for active checks and must match hostname as configured on the server.
# Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

### Option: HostnameItem
# Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
# Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
# HostnameItem=system.hostname

### Option: HostMetadata
# Optional parameter that defines host metadata.
# Host metadata is used at host auto-registration process.
# An agent will issue an error and not start if the value is over limit of 255 characters.
# If not defined, value will be acquired from HostMetadataItem.
#
# Mandatory: no
# Range: 0-255 characters
# Default:
# HostMetadata=

### Option: HostMetadataItem
# Optional parameter that defines an item used for getting host metadata.
# Host metadata is used at host auto-registration process.
# During an auto-registration request an agent will log a warning message if
# the value returned by specified item is over limit of 255 characters.
# This option is only used when HostMetadata is not defined.
#
# Mandatory: no
# Default:
# HostMetadataItem=

### Option: RefreshActiveChecks
# How often list of active checks is refreshed, in seconds.
#
# Mandatory: no
# Range: 60-3600
# Default:
# RefreshActiveChecks=120

### Option: BufferSend
# Do not keep data longer than N seconds in buffer.
#
# Mandatory: no
# Range: 1-3600
# Default:
# BufferSend=5

### Option: BufferSize
# Maximum number of values in a memory buffer. The agent will send
# all collected data to Zabbix Server or Proxy if the buffer is full.
#
# Mandatory: no
# Range: 2-65535
# Default:
# BufferSize=100

### Option: MaxLinesPerSecond
# Maximum number of new lines the agent will send per second to Zabbix Server
# or Proxy processing 'log' and 'logrt' active checks.
# The provided value will be overridden by the parameter 'maxlines',
# provided in 'log' or 'logrt' item keys.
#
# Mandatory: no
# Range: 1-1000
# Default:
# MaxLinesPerSecond=100

############ ADVANCED PARAMETERS #################

### Option: Alias
# Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.
# Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.
# Different Alias keys may reference the same item key.
# For example, to retrieve the ID of user 'zabbix':
# Alias=zabbix.userid:vfs.file.regexp[/etc/passwd,^zabbix:.[0-9]+),,,,\1]
# Now shorthand key zabbix.userid may be used to retrieve data.
# Aliases can be used in HostMetadataItem but not in HostnameItem parameters.
#
# Mandatory: no
# Range:
# Default:

### Option: Timeout
# Spend no more than Timeout seconds on processing
#
# Mandatory: no
# Range: 1-30
# Default:
# Timeout=3

### Option: AllowRoot
# Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent
# will try to switch to user 'zabbix' instead. Has no effect if started under a regular user.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Default:
# AllowRoot=0

### Option: Include
# You may include individual files or all files in a directory in the configuration file.
# Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.
#
# Mandatory: no
# Default:
# Include=

# Include=/usr/local/etc/zabbix_agentd.userparams.conf
# Include=/usr/local/etc/zabbix_agentd.conf.d/

####### USER-DEFINED MONITORED PARAMETERS #######

### Option: UnsafeUserParameters
# Allow all characters to be passed in arguments to user-defined parameters.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
# UnsafeUserParameters=0

### Option: UserParameter
# User-defined parameter to monitor. There can be several user-defined parameters.
# Format: UserParameter=<key>,<shell command>
# See 'zabbix_agentd' directory for examples.
#
# Mandatory: no
# Default:
# UserParameter=

####### LOADABLE MODULES #######

### Option: LoadModulePath
# Full path to location of agent modules.
# Default depends on compilation options.
#
# Mandatory: no
# Default:
# LoadModulePath=${libdir}/modules

### Option: LoadModule
# Module to load at agent startup. Modules are used to extend functionality of the agent.
# Format: LoadModule=<module.so>
# The modules must be located in directory specified by LoadModulePath.
# It is allowed to include multiple LoadModule parameters.
#
# Mandatory: no
# Default:
# LoadModule=

####### AS/400-specific parameters #######

### Option: as400ServerHost
# DNS name or IP address of AS/400 host to connect to.
# Default is localhost (acceptable if the agent is running on the AS/400 host directly).
#
# Mandatory: no
# Default:
as400ServerHost=SOMEHOST

### Option: as400Password
# Password for a user on AS/400 (username can be set via "User=" parameter).
# Default is acceptable if the agent is running on the AS/400 host directly).
# Mandatory: no
# Default:
User=SOMEUSER
as400Password=SOMEPASSWORD

### Option: as400EventIdAsMessagePrefix #######
# If Zabbix Agent should add the Event ID to the message text as a prefix
#
# Mandatory: no
# Range: 0-1
# Default:
# as400EventIdAsMessagePrefix=1

[wheresmyhammer]

AS400 Agent Not Working for me

Kos

I followed instructions for running agent on AS/400, but am encountering the following error

Exception in thread "main" java.lang.UnsupportedClassVersionError: JVMCFRE003 bad major version; class=as400/thread/ZabbixAgent, off
at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java :286)
at java.security.SecureClassLoader.defineClass(Secure ClassLoader.java:74)
at java.net.URLClassLoader.defineClass(URLClassLoader .java:538)
at java.net.URLClassLoader.defineClass(URLClassLoader .java:449)
at java.net.URLClassLoader.access$300(URLClassLoader. java:77)
at java.net.URLClassLoader$ClassFinder.run(URLClassLo ader.java:1041)
at java.security.AccessController.doPrivileged(Access Controller.java:448)
at java.net.URLClassLoader.findClass(URLClassLoader.j ava:427)
at java.lang.ClassLoader.loadClass(ClassLoader.java:6 76)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:358)
at java.lang.ClassLoader.loadClass(ClassLoader.java:6 42)
Java program completed with exit code 1

I am running i5/OS v7.1 and have Java 7 and 8 installed already.

• Created a home directory (e.g. /home/ZABBIX/agentd)
• Added objects; json-simple-1.1.1.jar, jt400.jar, zabbix_agentd.conf, and ZabbixAgent.jar
• When I submit the job - SBMJOB CMD(JAVA CLASS('/home/ZABBIX/agentd/ZabbixAgent.jar')
  PARM('/home/ZABBIX/agentd/zabbix_agentd.conf')) JOB(ZBXSVC) JOBQ(QSYS/QSYSNOMAX)
  USER(ZABBIX), I receive the error message listed above. There are no logs created as the job doesn't even appear to submit.

Any help/feedback is appreciated.


Thanks

[Kos]

Hello,
Thank you for a feedbacks!

Grant,
the Java Toolbox could try to start X11 if it unable to connect to AS/400 system with the provided credentials. Check, please, that you can connect to this system with the same username and password (config file parameters User/as400ServerHost/as400Password) - for example, just via "green screen".

wheresmyhammer,
you wrote:

I am running i5/OS v7.1 and have Java 7 and 8 installed already.

According to the error message, it seems that by default java still starts with JRE v1.6. Can you check the environment variable JAVA_HOME value, please?

Gerald,
I found a mistake with my re-connection implementation. I'll try to fix it.