SYSSTAT Module

The SYSSTAT module is used to obtain run-time information about the remote operating system running on a node. The module communicates with a remote agent daemon called vacm_sys_statd to obtain the remote information.

Module Features

The SYSSTAT module provides the end user or client with the following information:

The module also provides the ability to obtain information via the vacm_sys_stat_proxy in the event that the remote system is behind a firewall. Traffic between the module and the agent daemon can be encrypted using OpenSSL is VACM was compiled with OpenSSL support.

Installing and Setting Up the SYSSTAT Agent Daemon

The agent daemon is part of the vacm_node_exports package and must be installed on any system that is to be monitored via SYSSTAT. If secure communication between the module and the agent is desired, VACM should be compiled with OpenSSL support, and OpenSSL should be installed on each remote system to be monitored. (See 'Encryption and Security Considerations for VACM'.) The SSL certificate 'pem' file should be installed as /etc/vacm_sys_statd.pem.

To install the agent, simply transfer and install the vacm_node_exports package on every remote system you wish to monitor. Then add the agent daemon into your system startup scripts. Once installed, a Nexxus authentication password must be assigned to the daemon to ensure that only authorized Node Controllers may connect to the daemon. To initialize the Nexxus authentication password, simply run the daemon manually. You will be prompted for the password which will be used to authenticate remote connections. This password is stored in /etc/vacm_sys_statd.passwd and may be copied to the remaining nodes you wish to monitor if you wish to have the same Nexxus authentication password for every node.

Once the password is set, you can restart the daemon to begin normal operation.

Configuring the vacm_sys_stat_proxy for a node

If a remote node is behind a firewall or non-routable network, the module may not be able to communicate with the remote agent daemon. In this case the vacm_sys_stat_proxy is used to proxy connections from the module to the agent. The proxy server is located in the vacm_node_exports package. If secure communication is being used, then OpenSSL must be installed on the firewall or proxy machine. The SSL certificate 'pem' file should be installed as /etc/vacm_sys_stat_proxy.pem. Add the proxy server into your startup scripts, and run it to set the Nexxus proxy authentication password. Once set, simply re-run the proxy server for normal operation.

Configuring the SYSSTAT Module To Manage a Node

Before a node can be monitored with SYSSTAT, the module needs to know the Nexxus authentication password for a node. This password is used to ensure that only authorized Nexxus connections are accepted by a remote agent daemon. The address of the remote machine is pulled out from the 'IP_ADDRESS' node global variable. The maximum password length is 40 characters and must contain only printable characters.
FORMAT:
SYSSTAT:CONFIGURATION:<NODE_ID>:<PASSWORD>
RESPONSES:
JOB_STARTED
JOB_ERROR:PASSWORD_TOO_LONG   -- Password exceeds 40 characters
JOB_ERROR:BAD_PASSWORD        -- Password contains non printable 
                                 characters
JOB_COMPLETED

Configuring a node to be monitored via SYSTAT proxy

A remote proxy server is specified for a node with the 'SET_PROXY' command. The proxy is unset by specifying 'NONE' for <PROXY_ADDR>.
FORMAT:
SYSSTAT:SET_PROXY:<NODE_ID>:<PROXY_ADDR>:<PROXY_PASSWORD>
RESPONSES:
JOB_STARTED
JOB_ERROR:PASSWORD_TOO_LONG   -- Password exceeds 40 characters
JOB_ERROR:BAD_PASSWORD        -- Password contains non printable
                                 characters
JOB_ERROR:BAD_ADDRESS         -- Invalid address specified
JOB_COMPLETED

Obtaining Node Memory Statistics

To return realtime memory statistics on a remote node, use the 'MEM' command:
FORMAT:
SYSSTAT:MEM:<NODE_ID>
RESPONSES:
JOB_STARTED
MEM:<T_MEM>:<F_MEM>:<S_MEM>:<BUF>:<CACHE>:<T_SWAP>:<F_SWAP>
JOB_COMPLETED
FIELDS:
<T_MEM>  - Total System Memory
<F_MEM>  - Free System Memory
<S_MEM>  - Shared System Memory
<BUF>    - Buffers
<CACHE>  - Cache
<T_SWAP> - Total Swap
<F_SWAP> - Free Swap

Obtaining Node CPU Load

Current CPU load statistics are obtained with the 'CPU_LOAD' command:
FORMAT:
SYSSTAT:CPU_LOAD:<NODE_ID>
RESPONSES:
JOB_STARTED
CPU_LOAD:<1_MIN_AVG>:<5_MIN_AVG>:<15_MIN_AVG>
JOB_COMPLETED

Obtaining Node Uptime

The 'UPTIME' command returns the number of seconds the remote system has been up:
FORMAT:
SYSSTAT:UPTIME:<NODE_ID>
RESPONSES:
JOB_STARTED
UPTIME:<SECONDS>
JOB_COMPLETED

Obtaining Node Mounted Filesystem Information

The 'FS' command returns a list of mounted filesystems on a remote node and their disk space usage:
FORMAT:
SYSSTAT:FS:<NODE_ID>
RESPONSES:
JOB_STARTED
FS:<DEVICE_NAME>:<TOTAL_BLK>:<USED_BLK>:<FREE_BLK>:<PERCENT_USED>:<MNT_PT>
JOB_COMPLETED

Obtaining a List of Users Online a Node

The 'WHO' command returns a list of users currently logged into a node:
FORMAT:
SYSSTAT:WHO:<NODE_ID>
RESPONSES:
JOB_STARTED
FS:<USER_NAME>:<TTY>
JOB_COMPLETED

Obtaining a List of Processes Running On a Node

Process listings are retrieved from a remote node with the 'PS' command:
FORMAT:
SYSSTAT:PS:<NODE_ID>
RESPONSES:
JOB_STARTED
PS:<PID>:<USERNAME>:<CPU %>:<MEM %>:<RSS>:<STATE>:<TTY>:<COMMAND>
JOB_COMPLETED
FIELDS:
<PID> - Process ID
<USERNAME> - Username running process
<CPU %>    - Percentage of CPU currently in use
<MEM %>    - Percentage of memory currently in use
<RSS>      - Resident set size of process
<STATE>    - Current state of process
<TTY>      - TTY associated with process
<COMMAND>  - Command name

Obtaining a Nodes Kernel Version

The current kernel version of a remote node is obtained with the 'VERSION' command:
FORMAT:
SYSSTAT:VERSION:<NODE_ID>
RESPONSES:
JOB_STARTED
VERSION:<KERNEL_VERSION>
JOB_COMPLETED

Obtaining a Nodes APM Status

The current advanced power management status of a node is retrieved with the 'APM' command.
FORMAT:
SYSSTAT:APM:<NODE_ID>
RESPONSES:
JOB_STARTED
APM:<LINE_STATUS>:<BATT_STATUS>:<PERC_CAPACITY>:<UNITS_REMAINING>:
    <MEASURE>
JOB_COMPLETED
FIELDS:
<LINE_STATUS>     - AC line status
<BATT_STATUS>     - Battery status
<PERC_CAPACITY>   - Percentage battery remaining
<UNITS_REMAINING> - Number of 'units' of battery 
                    capacity remaining
<MEASURE>         - Measure of units'

Obtaining an Interrupt Allocation List for a Node

A hardware interrupt map of a remote node is obtained with the 'INTERRUPTS' command:
FORMAT:
SYSSTAT:INTERRUPTS:<NODE_ID>
RESPONSES:
JOB_STARTED
INT:<INT_NO>:<INT_SRC>:<INT_NAME>:<CNT_CPU0>:<CNT_CPU1>:<CNT_CPU2>:
    <CNT_CPU3>
JOB_COMPLETED
FIELDS:
<INT_NO>   - Interrupt number
<INT_SRC>  - Interrupt source
<INT_NAME> - Interrupt name
<CNT_CPU0> - Interrupt count for CPU 0
<CNT_CPU1> - Interrupt count for CPU 1 (may be 'NONE')
<CNT_CPU2> - Interrupt count for CPU 2 (may be 'NONE')
<CNT_CPU3> - Interrupt count for CPU 3 (may be 'NONE')

Obtaining an I/O Port Allocation List for a Node

An I/O port map for a node is retrieved with the 'IOPORTS' command:
FORMAT:
SYSSTAT:IOPORTS:<NODE_ID>
RESPONSES:
JOB_STARTED
IO:<ADDRESS_RANGE>:<FUNCTION>
JOB_COMPLETED

Obtaining a DMA Channel Allocation List for a Node

The DMA channel allocation list is retrieved with the 'DMA' command:
FORMAT:
SYSSTAT:DMA:<NODE_ID>
RESPONSES:
JOB_STARTED
DMA:<CHANNEL>:<FUNCTION>
JOB_COMPLETED

Obtaining Swapfile Statistics for a Node

Detailed swapfile statistics for a node are obtained withthe 'SWAP' command:
FORMAT:
SYSSTAT:SWAP:<NODE_ID>
RESPONSES:
JOB_STARTED
SWAP:<SWAP_FILE>:<SWAP_TYPE>:<SWAP_SIZE>:<SWAP_USED>:<SWAP_PRI>
JOB_COMPLETED
FIELDS:
<SWAP_FILE> - Swap filename
<SWAP_TYPE> - Swap type (either 'partition' or 'file')
<SWAP_SIZE> - Swap file/partition size
<SWAP_USED> - Swap used
<SWAP_PRI>  - Swap priority

SYSSTAT Module Node Global Variable Requirements

The SYSSTAT module requires that the IP_ADDRESS variable be set to the internet address of a node. This address is used to connect to the remote vacm_sys_statd.