EMP Module
The Emergency Management Port module (EMP) is designed to communicate with an IPMI compliant Baseboard Management Controller (BMC) over either the serial port, or a serial port connected to a network terminal server. On Intel(tm) based motherboards, the EMP port is hardwired to COM2 (ttyS1 in Linux). Through the BIOS, this port can be enabled and configured. In this section, we will go through the EMP module features, how to setup your management topology for using EMP to manage multiple machines, how to setup EMP on an Intel based machine, and how to perform EMP operations using the VACM EMP module.
Module Features
The EMP module provides the end user or client with the ability to perform many low level hardware monitoring and management functions. Many of these functions are available 'Out of Band', meaning that they can be performed regardless of hardware power state or condition. Some of these functions include:
Chassis power on, power off, hard reset, power cycle
Retrieval of hardware inventory information such as Board Manufacturer, Product Name, Serial Numbers
Ability to set an asset tag (or other string) that is maintained in the hardware NVRAM and available to the local operating system
Board level hardware failure and warning log management
Realtime chassis hardware status queries
Realtime sensor data retrieval supporting onboard temperature, fan speed, and voltage sensors
Front panel power light flash control
Sensor value threshold retrieval
Setting Up a Management Topology for EMP
For best results, spend some time planning how the nodes connect to the Nexxus Node Controller. EMP is a peer to peer implimentation; you need one serial port for every node you wish to connect to. If you are using console mode, it is recommended you use COM1 for console redirection, requiring two serial ports for every node you wish to monitor and manage. There are many ways such a setup can be accomplished.Multiport serial cards such as the Comtrol Rocketport(tm) work well since they have low latency, however, each card takes up a PCI slot on your Node Controller. Rocketports are currently available at a density of 32 ports per card, giving you the capability of servicing 16 nodes per card if you are using console mode on the recommended second serial port. High end terminal servers such as Cisco(tm) chassis with ASYNC cards work extremely well over a network, the only requirement being that the physical network that carries the EMP data between the terminal server and the Node Controller should be isolated as to ensure that only EMP traffic is travelling on the wire. If you don't isolate EMP traffic, network traffic spikes could cause latency problems with the EMP hardware. Cisco hardware is also generally a lot more expensive than conventional multiport serial cards. Shown below are two diagrams illustrating the aforementioned EMP wiring strategies: [INSERT ROCKETPORT PICTURE HERE] [INSERT CISCO PICTURE HERE]
Configuring EMP On a Remote System
The EMP hardware can be configured on most systems through the system BIOS. Currently supported platforms are the Intel Nightshade, Nightlight, and Lancewood motherboards. To enable and configure the EMP properly for VACM use with these platforms, enter the BIOS configuration area and go to the 'Server Management' section. You should configure the EMP parameters to look like this:
Figure 5-1. Bios Setup for EMP
If you are going to enable console redirection mode so you can remotely enter the BIOS and effect changes, enter the 'Console Redirection' level and set the parameters as:
Figure 5-2. Bios Setup for Serial Console
This configuration shows the console to be connected to COM1. This is the *recommended* behaviour. You *can* setup the console on COM2, but if your system has a problem booting or otherwise locks up while console mode is engaged, then you will *not* be able to reset it remotely via EMP. You should also enable the boot time diagnostics screen. If the diagnostics screen is disabled, console mode will not function properly.
The EMP module has a number of extended features that are not normally available via EMP but are related to the BMC. These extended features are accessed via a TCP/IP network, communicating with the EMP_EXTENTD daemon that must be running on all target machines. Since a daemon is used on the remote machine, these features are only available 'in-band'; when the operating system is up and running. The emp_extentd daemon allows the EMP module to perform realtime sensor queries, and blink the front panel power indicator. The extensions daemon also provides support for the system hardware watchdog built into the BMC. The daemon can be configured to automatically power cycle the system, if the operating system locks up due to software or hardware failure. The extensions daemon communicates with the BMC via the IPMI KCS interface. This interface is a local ISA bus interface that the local system uses to perform low level BMC operations. The IPMI_KCS driver must be installed in the node's running kernel for the extensions daemon to communicate with the BMC. The IPMI_KCS driver patches are available in the nexxus/nexxus_modules/emp/support_utilities directory, and apply to the 2.2.14 kernel (the patch should apply to newer 2.2 kernels just fine). To install the patch, log into the remote node and transfer the patch over. Then apply the patch to your running kernel. The following example assumes the patch has been transferred to the /usr/src/linux directory/.
[root@H101 /]# cd /usr/src/linux [root@H101 linux]# patch -p 1 ../ipmi_kcs_version.linux-2.2.14.patch patching file `arch/i386/kernel/traps.c' patching file `drivers/char/Config.in' patching file `drivers/char/Config.in.orig' patching file `drivers/char/Makefile' patching file `drivers/char/Makefile.orig' patching file `drivers/char/ipmi_kcs.c' patching file `drivers/char/ipmi_kcs.h' patching file `drivers/char/misc.c' patching file `include/linux/ipmi_kcs_ioctls.h' patching file `include/linux/miscdevice.h' [root@H101 linux]# |
The file ipmi_kcs.c, ipmi_kcs.h, ipmi_kcs_ioctls.h, miscdevice.h, and misc.c are the actual source files for the driver. The traps.c file contains a modified NMI handler which will cause the front panel power indicator to begin flashing should the daemon be configured in "non-power-cycle- mode" and the watchdog times out.
Once the patch is installed, it is necessary to reconfigure the kernel to use the driver. Using the kernel configuration tool of your choice, enter the 'character devices' menu and enable the 'IPMI KCS Interface'. Once enabled, save your kernel configure, rebuild your kernel and install.
The extensions daemon itself takes two command line arguments. The first argument determines whether or not the daemon opens a network socket and waits for sensor requests from the EMP Module. This is for situations where a user may wish to take advantage of the watchdog functions of the daemon, but not the sensor functions. This argument is either 'ON' or 'OFF'. The second argument is the watchdog firing action which can either be set to 'REBOOT' or 'NOREBOOT'. In REBOOT mode, the hardware watchdog will power-cycle the machine if the watchdog fires. In NOREBOOT mode, the hardware generates an NMI (non maskable interrupt) that causes the kernel to *attempt* to blink the front panel power indicator (depending on the cause of the software or hardware failure, the kernel may not be in a state to execute code).
Here is an example of running the daemon in the foreground with sensor extensions enabled, in reboot mode:
[root@H101 /root]# ./emp_extentd on reboot VACM EMP Extensions Daemon (c) 1999,2000 San Mehat (nettwerk@valinux.com) --Sensor Extensions enabled. --Watchdog Action is REBOOT. |
The following example shows running the daemon in the foreground with sensor extensions enabled, and in non-reboot mode:
[root@H101 /root]# ./emp_extentd on noreboot VACM EMP Extensions Daemon (c) 1999,2000 San Mehat (nettwerk@valinux.com) --Sensor Extensions enabled. --Watchdog Action is NMI ONLY. |
The IPMI_KCS driver provides a /proc/ipmi interface that is used to see watchdog status, and view hardware inventory records stored in the NVRAM. The interface is shown below:
Driver Version : 1.1 BMC Version : 1.14 IPMI Version : 0.9 WD Timer Status : STARTED WD Timeout Action : NONE WD Pre-Timeout IRQ : NMI WD Last Pet : 964752365 (28 seconds ago) WD countdown : 27 Board Area Records: Board Manuf. : Intel Board Prod Name : L440GX+ Board Serial : IMLW94304081 Board Part : 721242-008 Product Area Records: Product Manuf. : VA Linux Product Name : FullOn 2x2 Product Part : VAR101679 Product Version : Not Available Product Serial : Not Available Product Asset : San's FullON(26 byte maximum size) |
The first section displays the driver version, BMC firmware version, and IPMI version compliance numbers. The second section displays information on the watchdog. WD Timer Status indicates weather the watchdog is currently running or not. WD Timeout Action indicates what action is taken when the watchdog timer expires. WD Pre-Timeout IRQ indicates what action is taken just before the watchdog timer expires. The WD Last Pet field gives the system tick timer timestamp of the last watchdog ping. WD countdown indicates the current hardware countdown value of the watchdog in 1/10's of a second. The third section contains information about to the motherboard manufacturer, name, serial number, and part number. The third field contains information about the overall Product manufacturer name, part, version and serial number. The last field contains the currently configured asset tag. The asset tag can be programmed via the EMP module remotely and can be used for inventory control, or storing small bits of configuration data such as an IP address, or ethernet hardware address.
Configuring the EMP Module To Manage a Node
Before a node can be managed by the EMP module, the module needs to be told what serial port device or what TCP/IP address and port to connect to. Also, if the target node's BIOS has been configured to require an EMP password, the module will need to be notified. The CONFIGURATION command is used to either configure the device and password parameters for a node, or to disable VACM EMP management for a node altogether. If no password is configured, the PASSWORD field should be 'NONE'. To disable VACM EMP management for a node, the DEVICE/ADDRESS field should be 'NONE'
FORMAT:
EMP:CONFIGURATION:<NODE_ID>:<DEVICE/ADDRESS>:<PASSWORD>
RESPONSES:
JOB_STARTED
STATUS:<STATUS>
JOB_ERROR:(errno string)
JOB_COMPLETED
FIELDS:
<DEVICE/ADDRESS> - either a /dev/device entry, or a network
address and port to connect to. Network
addresses should be specified as
xxx.xxx.xxx-yyy, where x is the network
address portion, and y is the target
port portion. If <DEVICE/ADDRESS> is
set to 'NONE', then EMP management
and monitoring is disabled for the
specified node.
<STATUS> - As the EMP module negotiates with the target
node hardware, the
following may be returned to
signify the negotiation progress:
'ENGAGING' - Thread is engaging to port
'ENGAGE_FAILED' - Unable to engage to port
'PROTOCOL_DETECTED' - EMP protocol detected
'PROTOCOL_UNAVAILABLE' - EMP protocol not detected on port
'CONNECTION_ACCEPTED' - Connection accepted by hardware
'CONNECTION_DENIED' - Bad password during connection
authentication with h/w
'INTERFACE_TEMP_LOCKED'- Too many bad passwords; interface
locked for 30 seconds
'DOWNLOADING_FRU' - FRU download in progress
'DOWNLOADING_SDR' - SDR download in progress
'DOWNLOADING_SEL' - SEL download in progress
NOTES:
If you receive a PROTOCOL_UNAVAILABLE message, the node *may*
be in console mode (which is part of the reason you shouldn't
do console on COM2). If the node goes into console mode at any
point during the negotiation, a
'Protocol driver not attached' JOB_ERROR message will be sent. |
Retrieving a Remote Node's BMC Information
The current version of BMC firmware and IPMI version compliance is obtained with the 'BMC_INFO' command:
FORMAT:
EMP:BMC_INFO:<NODE_ID>
RESPONSES:
JOB_STARTED
BMCINFO:<BMC FIRMWARE VERSION>:<BMC MANUFACTURER>:
<IPMI_VERSION>:<HARDWARE_REVISION>
JOB_ERROR:(errno string)
JOB_COMPLETED |
Retrieving a Configured Node's Module Configuration
Retrieve a node's configuration from the EMP module with the 'INQUIRY' command:
FORMAT:
EMP:INQUIRY:<NODE_ID>
RESPONSES:
JOB_STARTED
INQUIRY:<NODE_ID>:<SERIAL NUMBER>:
<DEVICE ADDRESS>:<STATE>:<PASSWORD>
JOB_ERROR:(errno string)
JOB_COMPLETED
NOTES:
IP address information can be retrieved with the nexxus 'GET_VAR' command. |
Retrieving a Node's Current Connection Status
Determine the protocol state of the node with the 'NODE_STATUS' command:
FORMAT: EMP:NODE_STATUS:<NODE_ID> RESPONSES: JOB_STARTED NODESTATUS:<DEVICE ADDRESS>:<STATE> JOB_ERROR:(errno string) JOB_COMPLETED FIELDS: <STATE> can be one of: 'NEGOTIATING' - Negotiation in progress 'CONSOLE' - Node is in console mode 'DEAD' - Pending internal removal 'INITIALIZING'- Thread is starting 'DETECTED' - Protocol detected (Everything OK) 'NOT_DETECTED'- Protocol not detected. (Waiting for protocol) |
Retrieving a Node's Inventory Information
Obtain inventory information from the FRU with the 'NODE_INV_INFO' cmd. This information is static and kept in the target nodes NVRAM
FORMAT:
EMP:NODE_INV_INFO:<NODE_ID>
RESPONSES:
JOB_STARTED
NODEINVINFO:<B_MAN>:<B_NAME>:<B_PART>:<B_SER>:
<B_ETH>:<P_MAN>:<P_NAME>:<P_PART>:
<P_VER>:<P_SER>:<ASSET>
JOB_ERROR:(errno string)
JOB_COMPLETED
FIELDS:
<B_MAN> - Board Manufacturer
<B_NAME>- Board Name
<B_PAR> - Board Part Number
<B_SER> - Board Serial Number
<B_ETH> - Board Ethernet Address (NOT USED)
<P_MAN> - Product Manufacturer
<P_NAME>- Product Name
<P_PART>- Product Part Number
<P_VER> - Product Version
<P_SER> - Product Serial Number
<ASSET> - Asset Tag |
Setting the Asset Tag on a Node
The asset tag for a node is stored in the hardware NVRAM. This asset tag can be used to store any type of string data from inventory control information to configuration data. The asset tag can be read remotely through the module using the NODE_INV_INFO command, or locally using the IPMI_KCS driver's /proc/ipmi interface. The size of the asset tag is restricted to the field space allocated in the FRU. By default on some systems this may be 0. To ensure enough space, reload your FRU and SDRs using your vendor supplied FRU and SDR updater and specify an asset tag with as many placeholder bytes as you wish to program. VA Linux customers may have the field length already expanded. If not, contact technical support and request the BMC/SDR/FRU updater floppy images for your particular server. If you attempt to write a tag larger than the served space in the FRU, you will receive an error. To set the asset tag, use the SET_ASSET_TAG command.
FORMAT: EMP_SET_ASSET_TAG:<NODE_ID>:<TAG> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Retrieving a Node's Current Chassis Status
The current chassis status of a node's chassis can be obtained with the CHASSIS_STATUS command. This command also returns the last power event (if any).
FORMAT:
EMP:CHASSIS_STATUS:<NODE_ID>
RESPONSES:
JOB_STARTED
CSTATUS:<PS_PWRON>:<PS_POVER>:<PS_INTLCK>:<PS_CTLFLT>:
<PS_PWRRSTPOL>:<LPE_ACFAIL>:<LPE_POVER>:
<LPE_INTLCK>:<LPE_PFLT>:<LPE_CMD>:<M_INT>:
<M_SEC>:<M_DFLT>:<M_FAN>
JOB_ERROR:(errno string)
JOB_COMPLETED
FIELDS:
<PS_PWRON> - Power State - Power On?
<PS_POWER> - Power State - Power Overload?
<PS_INTLCK> - Power State - Interlock Active?
<PS_PFLT> - Power State - Power Fault?
<PS_CTLFLT> - Power State - Power Control Fault?
<PS_PWRRSTPOL> - Power State - Power Restore Policy?
<LPE_ACFAIL> - Last Power Event - AC Failed?
<LPE_POVER> - Last Power Event - Power Overload?
<LPE_INTLCK> - Last Power Event - Interlock Power Down?
<LPE_PFLT> - Last Power Event - Power Fault?
<LPE_CMD> - Last Power Event - Commanded ON/OFF?
<M_INT> - Misc Chassis State - Intrusion Active?
<M_SEC> - Misc Chassis State - Secure Mode Active?
<M_DFLT> - Misc Chassis State - Drive Fault?
<M_FAN> - Misc Chassis State - Cooling Fan Fault? |
Retrieving a List of Chassis Capabilities
A brief list of chassis capabilities can be retrieved with this command. These chassis capabilities can be used to determine if some of the response fields of a CHASSIS_STATUS are valid. Chassis capabilities are retrieved with the CHASSIS_CAPABILITIES cmd.
FORMAT:
EMP:CHASSIS_CAPABILITIES:<NODE_ID>
RESPONSES:
JOB_STARTED
CCAPABLE:<PROVIDES_INTRUSION?>:<PROVIDES_SECUREMODE?>:
<PROVIDES_FPNMI?>:<PROVIDES_POWERINTERLOCK?>
JOB_ERROR:(errno string)
JOB_COMPLETED |
Powering Down the Chassis
The node can be powered down with the POWER_DOWN command. Depending on the state of the remote hardware, there may be a short delay before the hardware actually powers down. This command is a restricted command. If restricted mode is enabled in the remote nodes BIOS, an error will be returned.
FORMAT: EMP:POWER_OFF:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Powering Up the Chassis
The node can be powered up with the POWER_ON command. This command is a restricted command. If restricted mode is enabled in the remote nodes BIOS, an error will be returned.
FORMAT: EMP:POWER_ON:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Hard Resetting the Chassis
The node can be hard reset with the HARD_RESET command. This command is a restricted command. If restricted mode is enabled in the remote nodes BIOS, an error will be returned.
FORMAT: EMP:HARD_RESET:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Power Cycling the Chassis
The node can be power cycled with the POWER_CYCLE command. When issued, the target node firmware will power off the node, wait a few seconds, then power it up again. This command is a restricted command. If restricted mode is enabled in the remote nodes BIOS, an error will be returned.
FORMAT: EMP:POWER_CYCLE:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Sending a Chassis Front Panel NMI
If the target node supports Front Panel NMI (See CHASSIS_CAPABILITIES cmd), a front panel NMI can be strobed with the PULSE_FP_NMI command. The function of this command is vendor specific. This command is a restricted command. If restricted mode is enabled in the remote nodes BIOS, an error will be returned.
FORMAT: EMP:PULSE_FP_NMI:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Downloading the System Event Log
The EMP hardware system event log can be retrieved with the 'DOWNLOAD_LOG' command. The log is downloaded from the module's disk based node SEL file. The module queries the hardware every 5 seconds, and downloads any new entries. New entries are then deleted from the hardwarei's event buffer to make room for new events. Since the module attempts to modify the remote SEL buffer whenever it queries it, restricted mode must be disabled for SEL functions to work.
FORMAT:
EMP:DOWNLOAD_LOG:<NODE_ID>
RESPONSES:
JOB_STARTED
SEL:<NODE_SERIAL>:<RECORD_ID>:<RECORD_TYPE>:<RECORD_TIMESTAMP>:
<RECORD_GENERATOR>:<RECORD_FORMAT>:<SENSOR_TYPE>:<SENSOR_NAME>:
<DESCRIPTION>
JOB_ERROR:(errno string)
JOB_COMPLETED |
Clearing the System Event Log
The CLEAR_LOG command is used to remove all entries from the disk based system event log that the module keeps for a specific node.
FORMAT: EMP:CLEAR_LOG:<NODE_ID> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Receiving System Event Logs Asynchronously
Sometimes a client may want to receive system event log entries as they come in, and not have to constantly poll for them. The LOG_TAIL command is used to enable and disable asynchronous event log entry notification.
FORMAT: EMP:LOG_TAIL:<NODE_ID>:<ON/OFF> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Retrieving a List of Sensors on a Node
A list of available sensors on each node and their types can be retrieved with the GET_SENSOR_LIST command.
FORMAT: EMP:GET_SENSOR_LIST:<NODE_ID RESPONSES: JOB_STARTED SENSOR:SENSOR_NUMBER:<NAME>:<TYPE> JOB_ERROR:(errno string) JOB_COMPLETED FIELDS: <TYPE> - Either 'ANALOG', 'DIGITAL', or 'UNSUPPORTED'. |
Retrieving Sensor Thresholds for a Sensor
Threshold values for a sensor can be retrieved with the GET_SENSOR_THRESHOLDS command.
FORMAT:
EMP:GET_SENSOR_THRESHOLDS:<NODE_ID>:<SENSOR_NUMBER>
RESPONSES:
JOB_STARTED
THRESHOLD:<SENSOR_NUMBER>:<MIN_VAL>:<LWR_NONCRIT>:<LWR_NONREC>:
<LWR_CRIT>:<UPR_NONCRIT>:<UPR_NONREC>:<UPR_CRIT>:<MAX_VAL>
JOB_ERROR:(errno string)
JOB_COMPLETED
FIELDS:
<MIN_VAL> - Minimum Sensor Value
<LWR_NONCRIT> - Lower Non-Critical Threshold
<LWR_NONREC> - Lower Non-Recoverable Threshold
<LWR_CRIT> - Lower Critical Threshold
<UPR_NONCRIT> - Upper Non-Critical Threshold
<UPR_NONREC> - Upper Non-Recoverable Threshold
<UPR_CRIT> - Upper Critical Threshold
<MAX_VAL> - Maximum Sensor Value
NOTES:
Fields which have a return value of 0.0 indicate no threshold set. |
Reading a Sensor
A realtime sensor reading can be obtained from a node with the READ_SENSOR command. This command will function only if the EMP Extensions Daemon is running on the remote node.
FORMAT:
EMP:READ_SENSOR:<NODE_ID>:<SENSOR_NUMBER>
RESPONSES:
JOB_STARTED
SENSOR:<SENSOR_NUMBER>:<VAL>:<UNIT>
SENSOR:<SENSOR_NUMBER>:<CONDITION>
JOB_ERROR:(errno string)
JOB_COMPLETED
FIELDS:
<VAL> - Numerical value (Returned for ANALOG sensors only)
<UNIT> - String unit name which VAL refers to
(Returned for ANALOG sensors only)
<CONDITION> - Condition string for the sensor
(Returned for DIGITAL sensors only) |
Setting the Flash State of a Node's Front Panel Power Indicator
The IDENTIFY command is used to change the state of the front panel power indicator on a node. When identify is turned ON, the front panel power indicator will flash. When identify is turned OFF, the front panel power indicator will return to it's normal state. This command wil function only if the EMP Extensions Daemon is running on the remote node, and is currently only available on Intel LW440GX (Lancewood) class motherboards.
FORMAT: EMP:IDENTIFY:<NODE_ID>:<ON/OFF> RESPONSES: JOB_STARTED JOB_ERROR:(errno string) JOB_COMPLETED |
Resetting the EMP Module for a Node
The REFRESH command is used to reset the module thread for a node. The thread will disconnect from the EMP port, reconnect and renegotiate. See the CONFIGURATION command for a description of response messages.
FORMAT: EMP:REFRESH:<NODE_ID> RESPONSES: (see section 4.2.6) |
Unsolicited Messages
The following unsolicited messages may be sent by the EMP module at any time: SEL:<NODE_SERIAL>:<RECORD_ID>:<RECORD_TYPE>:<RECORD_TIMESTAMP>: <RECORD_GENERATOR>:<RECORD_FORMAT>:<SENSOR_TYPE>:<SENSOR_NAME>: <DESCRIPTION>
EMP Module Node Global Variable Requirements
The EMP module requires that the IP_ADDRESS variable be set to the internet address of a node only if sensor requests are to be performed.
EMP Module Supported Hardware List
The following hardware has been qualified for remote management with the EMP module:
Intel Nightshade / Nightlite Server Boards
Intel Lancewood Server Boards rev A -> G with BMC firmware 1.14
Intel Koa/D'Iberville Server Boards
Comtrol Rocketport cards & Rocketport breakout boxes
Cisco Systems 36xx chassis with 16/32 port async cards
Standard 16550 internal serial ports
Linux 2.2.13 and greater (SMP/UP)