SOL - No More KVM

See http://buttersideup.com/docs/howto/IPMI_on_Debian.html
See also PXE boot and IPMI

IPMI allows you to configure the BMC on Dell machines (probably others) to talk to the computer via its serial port and relay the input, and output to another machine, over the LAN interface. E.g. Serial over LAN support. SOL requires IPMI version 2 which is only available on a few Dells at this time (see here for details. This is quite some ways from a replacement for a non-graphical console (e.g. KVM), but it's close enough to get excited about it. If it works.

This interface requires that everything related to the console that you are interested in seeing and responding to must use the serial console. That is:

  • BIOS of your machine
  • BIOS of peripherals
  • Syslinux boot menu
  • Boot loader (grub)
  • Kernel

All of these must be configured to write to the serial port, rather than (or in addition to) the normal console. Sun machines have been doing this for ages. This sounds like an really long complex list, but fortunately, Dell and probably some other manufacturers offer an option to forward some console traffic over the serial port. This means that you don't need to configure all of the above, just a couple. Read on for my experience with Dells.

My goal is just to get through a basic boot - in particular, boot the machine, be able to invoke BIOS or much more interestingly, invoke PXE boot.

This sequence is a necessity if you have dozens or more machines at a remote site and need to install a new system (e.g. move from i386 to amd64). If just one of the steps requires you to press enter, you're screwed - you need a KVM (keyboard and VGA monitor, mouse not needed). Many places have invested in a serial console switch like those at http://www.kvms.com/ to name just one place. Yikes, they are expensive!

BIOS SETUP

For this you need a console because SOL is not enabled yet. I'd hoped this would work on all my PowerEdge machines (SC1425, PE1850, PE1950 and PE2950). SOL is new enough that older machines don't support it at all, or enough. For this to have a chance to work, your machine's BIOS must support IPMI 2.0. In my world that means only 1950s and 2950s work. Boot into Dell's BIOS and choose the following:

  Move to  'Serial Communication'
  Set
    Serial Communication .......  On with Console Redirection via COM2
    External Serial Connector .. COM2
    Failsafe Baud Rate ......... 57600
    Remote Terminal Type ....... VT100/VT220
    Redirection After Boot ..... Enabled

  Save and reboot

Note - enabling 'Redirection After Boot' works wonders. You do not need to configure Grub or Syslinux to use the serial console, since Dell's BIOS is handling that for you. In fact, setting up serial console for GRUB (as described in the URL above) actually messes things up. Don't configure Syslinux either... Dell's redirection is all the magic you need for that part.

KERNEL SETUP

You must tell the kernel to generate messages to the serial port as well as the console. Almost certainly your kernel was compiled with serial console support. Mine was, so I don't have any notes on what to enable, sorry. Enabling serial console consists of simply passing some parameters to the kernel at boot time.

   vi /boot/grub/menu.lst
   
   #  Add 'console=tty1 console=ttyS1,57600n8' after 'ro'
   kernel    /vmlinuz-2.6.24-mos root=/dev/sda3 ro console=tty1 console=ttyS1,57600n8
   
   #  In addition add this to kopt= keyword too so it is propagated
   #  when grub-update is done (probably not wanted)
   #  Note kopt is a COMMENT in your grub menu file.

   # kopt=root=/dev/sda3 ro console=tty1 console=ttyS1,57600n8

The URL above has notes about changing getty. I did not need to do anything with this, Debian seemed to have it all set up. Lucky me. As a final step you need to reboot this machine so the kernel gets these parameters.

USING SERIAL CONSOLE

At this point you should have a machine (I'll call it HOSTNAME) which has the BIOS changes made and has been booted with the serial console parameters. On another machine do:

  ipmitool -I lanplus -H HOSTNAME -U ROOTUSER sol activate
  Password:
  [SOL Session operational.  Use ~? for help]

This just invokes ipmitool to communicate with HOSTNAME using IPMI. The 'lanplus' option means to use IPMI v2.0. This is where you'll find out if your HOSTNAME actually supports the protocol you need. If you get the SOL Session prompt, you're set. Now reboot HOSTNAME (ipmitool can powercylce the machine), you'll see something reassuringly familiar - those messages from BIOS.

                                                                   F2 = Setup
  Phoenix ROM BIOS PLUS Version 1.10 1.0.1                 F10 = Utility Mode
  Copyright 1985-1988 Phoenix Technologies Ltd.               F11 = Boot Menu
  Copyright 1990-2006 Dell Inc.                                F12 = PXE Boot
  All Rights Reserved

  Dell System PowerEdge 1950
  www.dell.com
  Testing memory.  Please wait.
  
  etc

Realize this SOL is a pretty ugly terminal interface. There will likely be lots of funny looking characters etc, but the good news is that it can be used. You just need to know how to enter some special characters - press the escape key once and then one more character. ESC + 1 sends F1 to HOSTNAME. ESC + 2 sends F2. You can guess F1 to F9 easily enough. ESC+0 is F10. ESC+! is F11 and ESC+@ is F12 (for PXE boot). The arrow keys work in Setup as you'd hope. The last thing you need to know is how to get out of this:

  ~.             # Yes, tilde + period
  
  >              # Now back at your normal shell prompt

  #   Now deactivate the interface
  ipmitool -I lanplus -H HOSTNAME -U ROOTUSER sol deactivate

All it all, SOL is pretty easy to get going, your BIOS willing. I'm not interested in doing much with the kernel after it gets booted, so I've not played with that at all. I just wanted access to those things before the kernel - so I can control what gets booted (or installed).

Terminal Sessions with getty

Once SOL is working for console support, you might want to set up a serial TTY so you can get in when SSH or the network is not working. Once the machine is booted, the serial port can be used for a conventional login. Here are some notes on what to do for Debian systems. Other systems will be similar.

Modify /etc/inittab to enable the serial getty by adding this line:

T1:23:respawn:/sbin/getty -L ttyS1 57600 vt100

The speed setting may need to be changed for your hardware. Most recent systems can handle 57600, but some now default to 115200 (although I bet you can set this in the BIOS). The device name (ttyS1) could differ on your system too. Anyway, after changing inittab, do this to get the getting working:

kill -HUP 1

On Ubuntu systems you'll find the file /etc/event.d/ttyS1 and you should modify the line

exec /sbin/getty 57600 ttyS1

with the correct speed etc. Then issue the command start ttyS1 to get it going. Now you're ready to open the serial console connection with:

ipmitool -I lanplus -H IPMIAddress -U IPMIUser sol activate

You may need to hit 'enter' to get a login prompt. If there is already and active connection elsewhere deactivate the serial connection as mentioned above.

Pretty slick. This set of tricks has made us decide to NOT purchase a serial KVM, avoiding all that cabling. A clear win for us.