IPMI
101
There are two modes.
- "ipmitool cmd" on the machine itself, which doesn't go thru the network, and always work.
- "ipmitool -H bmc.ip.addr cmd" to use the "lanplus" network interface, which will require working IP stack, and require for example for the bmc ip to be pingable and reachable via the IPMI protocol. (port ??)
If ipmitool doesn't work, returning with "Error in open session response message : no matching cipher suite"
try
ipmitool -C 3 ...
to specify cipher to use.
102
Dell DRAC and HP iLo are implementation of IPMI.
Could use embeded NIC (shared with OS)?
Dedicated NIC provides more functionality.
when enabled, can ssh in (eg into the iDRAC port).
The standard iDRAC supports IPMI and basic web interface to look at temperature, fru status, reboot machine, etc.
The Enterprise iDRAC adds remote (GUI) console, AD-based user auth.
IPMI sol (serial over lan), isol, etc is avail from iDRAC, presumably even in standard version.
For Dell, IPMI don't seems to have way to change hyperthreading settings, not even in the delloem commands.
Disabling hyperthreading seems to req interactive BIOS changes.
Linux kernel param noht doesn't work (or was it only supposed to work at install time?)
~~
sky4-5 has decent config, bio2 don't.
There is likely an IPMI-only w/o iDRAC on the machine as well, eg the low end 1U box
CMD
ipmitool lan print 1 # iDRAC interface (could be shared if set so). 1 refer to channel 1, which typically only channel.
ipmitool lan set 1 ipaddr 10.10.58.140 # set iDRAC interface IP (should be, need to verify it works)
ipmitool lan set 1 defgw ipaddr 10.10.58.1 # set default gateway
ipmitool lan set 1 vlan id 20 # if in network require vlan id tagging, set to use the correct id (eg: 20)
ipmitool sdr # print environment statistics
ipmitool fru
ipmitool sol info # supposed to provide serial over lan info
ipmitool shell # get to ipmi> shell so that don't have to invoke the tool each time. it is NOT gettingo inside iDRAC shell.
ipmitool delloem lan # dell specific command. lan section set whether to use dedicated NIC or shared.
# but seems like iDRAC is configured yet this info can still be not set.
# so iDRAC can probably speak IPMI and interfaces with it, but it is not an instance of IPMI.
# if work, just tell whether dedicated or shared nic
ipmitool delloem lan set dedicated # disable use of shared IPMI. set once iDRAC enterprise module is installed.
ipmitool delloem powermonitor # display power usage info
ipmi tag to vlan 20 (default is untagged):
ipmitool lan set 1 vlan id 20
ipmitool user list 1 # list users configured to use ipmi
ipmitool chassis identify # blink location becon (typically blue) to help find machine
ipmitool chassis bootdev pxe options=persistent # change to use pxe boot by default on every boot.
ipmitool -I lan -U root -P calvin -H 10.8.1.76 chassis bootdev pxe options=persistent
# issue the ipmi command remotely to a remote host using the remote iDRAC interface
ipmitool fru # list all fru (eg 4 node in 2U chassis would list service tag and board serial of each node)
ipmitool dcmi set_asset_tag # set asset tag, presumably for the whole chassis ...
ipmitool dcmi asset_tag # retrieve asset tag info
ipmitool dcmi get_temp_reading # cpu temperature
ipmitool sensor # many sensor info, may include inlet temperature
ipmitool sel # system event log, overview
ipmitool sel list # get list of events and dates. eg, PSU AC lost. additional example output:
3 | 08/12/2021 | 16:49:18 | Unknown #0x2e | | Asserted
4 | 08/12/2021 | 16:49:18 | Memory #0x45 | Redundancy Degraded from Fully Redundant | Asserted
1 | 10/22/2020 | 00:44:54 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
2 | 01/25/2021 | 19:00:24 | Power Supply #0x18 | Fully Redundant | Asserted
3 | 09/11/2021 | 22:23:05 | Memory #0x1b | Monitor | Asserted
4 | 10/27/2021 | 17:32:09 | Memory #0x1b | Monitor | Asserted
ipmitool sensor | grep Current # on Dell C6400 see output like this, and 1 PSU likely dont have AC power:
Current 1 | 0.000 | Amps | ok | na | na | na | na | na | na
Current 2 | 2.400 | Amps | ok | na | na | na | na | na | na
# if IPMI isn't working, in RHEL 6, it is a kernel module that need to be started
# for machines with module:
module load shared
module load ipmitool
service ipmi start
Finding Dell Chassis ServiceTag from a node using hexcode
For dell C6320 and the like, the chassis has its own service tag.
hex code via ipmi "raw" command can retrieve them.
conversion would be needed. eg
xxd -r reverse hex conversion to ascii, avail from vim-common rpm
# Dell C6320 vintage 2017
ipmitool raw 0x30 0xc8 0x01 0x00 0x0b 0x00 0x00 0x00 | xxd -r
# additional ipmi raw command to try to get chassis code
# The C6220II should use the same commands as the C6320, but if the system is too old the original version of this unit won’t work.
# The C6100 and C6220 do not have a unique chassis service tag to query.
# To query the Set FCB value run ALL of the following commands in this order
# (It generate and execute some sort of reservation, each command should produce some output,
# till the last one should yield a serial number of the chassis)
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00 0x00
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00 0x01
ipmitool raw 0x30 0xC8 0x01 0x00 0x0c 0x00 0x02 0x00 0x00 0x00 | xxd -r
Programming Dell Chassis from a node using IPMI RAW commands
While some chassis have svc tag programmed out of factory, others are not.
For inventory purpose, if one wish to program in the service tag so that they can be queried.
Here is an example of steps to program svc tag of "bjhkqd2" to the chassis via one of its node.
bjhkqd2 in hex is 62 6a 68 6b 71 64 32
b j ...
vvvv vvvv
ipmitool raw 0x30 0xC8 0x00 0x00 0x0B 0x00 0x00 0x00 0x0B 0x00 0x11 0x0A 0x62 0x6a 0x68 0x6b 0x71 0x64 0x32
## commit
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00
IPMI RAW
Dell "Using IPMITOOL RAW commands for remote management".
It is pdf from dell. includes IPMI, BIOS config for SOL, escape seq keys, etc.
# raw ipmi command can potentially set what temp server shutdown when overheat
# raw cuz it is hex code being fed to the service processor.
ipmitool -I open sel list
# list raw events... maybe plaform specific.
# Over temperature raw 0x04 0x02 0x04 0x01 0x01 0x01 0x09 00 00
# Over voltage raw 0x04 0x02 0x04 0x02 0x11 0x01 0x09 00 00
Obscure IPMI cmd for Asus
May only be relevant to ESC4000 G4
ipmitool raw 0x32 0x71 0x00 0x01 0x01 # set ipmi/bmc to use shared nic
ipmitool lan set 8 cipher_privs XaaaaaaaaaaaXXX # reset cipher suite allowed for channel 8
Obscure IPMI cmd for Penguin Relion 2940 w/ Gigabyte mboard
ipmitool raw 0x2e 0xcc 0x5e 0x2b 0 0x0C 0x01 0x02 # set ipmi/bmc to use share nic
# there maybe a bios settings to put the NIC in shared mode, but some notes has it that it wasn't very reliable,
# so support provided the ipmitool raw command.
# this was for the 4 nodes in 2U w/ intel E5-2640 v4 CPU
Dell ipmitool
Dell maintain their own version of ipmi that handles complex situation that RHEL7 ipmitool didn't.
eg:
eg, this command is needed by some new Dell eg PE R7525 :
/local/tin/dell_tool/iDRACTools/ipmitool/usr/bin/ipmitool delloem lan set shared with lom1
GNU freeipmi / ipmi-oem
Ref:
gnu freeipmi aka ipmi-oem.
In CentOS 7 has freeipmi* rpm
may provide oem commands to set ipmi
eg appro, which is/uses Intel S2600JF
oem get-bmc-services
eg with Dell
/usr/sbin/ipmi-sensors
ipmi-oem Dell get-power-capacity
ipmi-oem Dell get-active-lom-status
/usr/sbin/bmc-config ??
Serial over IPMI SoL
This doesn't quite redirect grub login to serial yet.
But don't really want to redirect VGA to serial.
Only want to spawn a serial login session so that ipmi sol
can gain a login.
inittab and getty no longer works in RHEL7, below config is for systemd.
## /etc/systemd/system/mgetty.service ::
[Unit]
Description=Smart Modem Getty(mgetty)
Documentation=man:mgetty(8)
Requires=systemd-udev-settle.service
After=systemd-udev-settle.service
[Service]
Type=simple
ExecStart=/sbin/mgetty -r -s 115200 /dev/ttyS1
# -r is for direct connection (ie null modem)
Restart=always
PIDFile=/var/run/mgetty.pid.ttyS1
[Install]
WantedBy=basic.target multi-user.target
ls -Z /usr/sbin/mgetty
ls -Z /var/log/mgetty.*
chcon --type getty_exec_t /var/log/mgetty.*.log
### still need more tweaking, but don't really prevent mgetty from creating a login session.
systemctl start mgetty.service
systemctl enable mgetty.service
# enable will create sym links in /etc/systemd/system/basic.target.wants and multi-user.target.wants
# create ipmi user #3 to have access
ipmitool user list
ipmitool user set password 3 NewIpmiPwForUserNum3 # if need to change password
ipmitool user enable 3
ipmitool channel setaccess 1 3 ipmi=on privilege=4
ipmitool user priv 3 4 1
Generic usage:
# using serial over lan (sol, ipmi 2.0)
ipmitool -H hostname -U username -P pw list # see list of user
ipmitool -H hostname -U username -P pw sol activate # initial sol connection
ipmitool -I lan -U $username -P $pass -H $nodename sol activate # activte serial over lan
# should drop into console. escape seq to control IPMI session:
# ~. = exit
# ~ ^Z = suspend
# ~B = send break
# ~? = help
# Esc R Esc r Esc R = ctrl-alt-del
# Esc 1 = F1
ipmitool -I lan -U $username -P $pass -H $nodename sol deactivate # disconnect prev sol, clean up measure
Specific usage example:
csh
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol info # 115.2 should be default speed
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol activate instance=1 # maps to /dev/ttyS1
# ~? = help (but note that ssh also use ~ as escape, so may need ~~?
# ~. = terminate session
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol deactivate # if some old session was left running
exit
rm ~/.history
# most ipmitool over lan uses 623.
# dell idrac says use UDP 623
# but on SMC, can use telnet to port 623 and get connection established, so it is over TCP??!!
# test connectivity eg:
telnet 172.10.0.128 623
possible test for connectivity:
traceroute --port=623 172.10.0.128
traceroute --port=33434 is the default, it is Destination UDP port that this pokes at.
ref:
http://snarkybrill.blogspot.com/2015/08/mgetty-in-systemd-for-modem-dial-in.html
# http://thenetworksherpa.com/using-ipmi-serial-lan-consoles/
# ref: (for windows, but mostly applicable to linux)
# https://www.symantec.com/connect/articles/how-do-sol-serial-over-lan-aka-console-redirection-dell-servers
https://topics-cdn.dell.com/pdf/idrac9-lifecycle-controller-v3212121_users-guide_en-us.pdf
page 31,32 discuss firewall req for iDRAC. states UDP 623.
Remote console need VNC on port TCP/5900 and 5901
Remote Power Cycle using IPMI
linux machine with ipmitool (provides ipmi cli) installed, so can also do:
ipmitool -H hostname -U username -P pass power status
credentials are IDRAC credentials
*should* be blade specific when hostname is iDRAC of blade even when description say chassis. Have not been able to get it to work.
ipmitool -I lan -U $username -P $pass -H $nodename mc info # ipmi ver info
ipmitool -I lan -U $username -P $pass -H $nodename chassis power status
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power status
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power down
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power on
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power cycle
# nodename is the DNS name of the ipmi address of a host, eg ipmi-HOSTNAME
# most of time use lanplus interface
# lan may work for some basic commands, but more sophisticated command req it to be run over lanplus
# dell def username pass is root calvin
ipmitool mc reset cold # reset the service processor on the node itself, cold? booting it?
# dell C6320 need the following to configure IPMI:
ipmitool delloem lan set shared with lom1
ipmitool -H
sol payload enable 1 2
# these supposed to work, but haven't been able to get them to work :(
# may need to configure grub level serial redirect
# but realistically haven't been able to connect to serial, not just a blank screen...
Verify status
ipmitool lan print 1
ipmitool lan print | grep 'IP Address'
Finding power usage
# Dell C6320, 32 skylake cores in use. drawing 336W
# Merger reads busy server drawing 3.7A x 2 feed x 202V / 4 nodes per chassis = 373W.
# ipmitool sensor | grep 'Pwr Consumption'
Pwr Consumption | 336.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# on, but idle skylake server, 96 GB RAM
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# on, but idle skylake server, 192 GB RAM
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# when machine is ipmi off:
Pwr Consumption | 0.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Power on power consumption remotely monitored by ipmitool :
Pwr Consumption | 0.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 56.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 112.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 126.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 140.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 252.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 210.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 182.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 182.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Ref
Solaris#IPMI
hoti1
bofh1
