Recover from a failed Linux boot

Most Linux computers use the Grand Unified Bootloader (GRUB) — more specifically, GRUB 2 — to control the handoff from the computer's firmware to the kernel. GRUB 2 provides sophisticated boot-time user-interaction features that give you control over the boot process. You probably won't use these features every day, but they can be important in handling problem situations — such as a failure of the computer to boot after a kernel upgrade, disk swap, or other system change. A few tips and GRUB commands can help you boot the computer in such situations and save valuable time. You can also use emergency boot disks to boot your normal Linux installation even if a problem occurs with the regular initial stages of boot loader activation.
Broadly speaking, GRUB recovery addresses two types of problems: problems that you can solve by using GRUB's built-in shell and those that require an external tool. I cover both types of problems in this article, with the Super GRUB2 Disk as an example of an external tool that you can use when necessary. (This article emphasizes GRUB 2, but some of the information presented here applies to GRUB Legacy too. Read about the differences between GRUB Legacy and GRUB 2 in "Migrate to GRUB 2.")

Understanding GRUB 2's control structures

Before delving into recovery details, you should understand some basics of how GRUB works. A computer's boot process is complex, and knowing something about the boot path can help you solve problems if the boot process veers from that path. I'll begin by describing where boot code and GRUB files reside on a computer so that you can find them — or identify what might be missing if a problem is caused by a missing file. A complete description of the GRUB configuration file format is beyond this article's scope, but I do cover the configuration basics, which can help you correct simple errors (such as an incorrectly specified root file system).

Finding GRUB files

The boot process on most computers that use the Basic Input/Output System (BIOS) involves code that's stored in various locations on the disk. These locations include the Master Boot Record (MBR), officially unallocated disk sectors, and the partition's boot sector (also known as the Partition Boot Record [PBR]). These records can be overwritten by other boot loaders, overwritten by a virus or a low-level disk utility with needs that conflict with GRUB's, or damaged by misuse of a low-level utility such as dd. When such damage occurs, GRUB is unlikely to start at all, and you might need to use a tool such as Super GRUB2 Disk.
Many newer computers use the Extensible Firmware Interface (EFI) rather than BIOS. On such systems, GRUB code isn't stored in the MBR, PBR, or officially unallocated disk sectors. Instead, it is in an EFI boot loader file with an .efi extension on the EFI System Partition (ESP) — a partition with a File Allocation Table (FAT) format that typically appears at the start of the disk. EFI systems aren't susceptible to the same types of low-level boot loader damage as BIOS systems, but they can malfunction because of changes to the computer's non-volatile RAM (NVRAM) settings. In such cases, you might need to perform an emergency boot and then use the efibootmgr utility to restore GRUB as the default boot loader.
In addition to the low-level BIOS or EFI boot files, GRUB 2 relies on conventional files in /boot/grub. These include file system drivers, video drivers, fonts, and the GRUB configuration file (grub.cfg). Because these files reside in a normal Linux file system, the earlier boot stages must include at least one rudimentary Linux file system driver. If these files are damaged, GRUB might launch normally but be unable to start your operating system; or GRUB might start up and present nothing but a grub> prompt.

Editing the configuration file

On most systems, the GRUB 2 configuration file is /boot/grub/grub.cfg or /boot/grub2/grub.cfg. However, some EFI-based installations place it in a directory on the ESP. This might be /boot/efi/EFI/grub/grub.cfg or some other similar location. But in most cases, the grub.cfg file isn't meant to be edited directly; instead, it's pieced together by scripts. You can find some of the component pieces in the /etc/grub.d directory. If you need to make changes to your standard GRUB 2 configuration, edit those files.
However it's built, grub.cfg consists of both global options and OS- or kernel-specific boot stanzas. Linux distributions set up their global GRUB 2 configurations properly for most computers. If you have an unusual configuration, you might need to study your global GRUB 2 configuration to find the cause of a problem.
GRUB 2's boot stanzas define individual OSs or kernels. The boot stanzas typically appear in the last half of the grub.cfg file. Listing 1 shows an example:

Listing 1. Example GRUB 2 stanza to boot Linux

1 2 3 4 5 6 7 8 9 10 11

menuentry 'Ubuntu, with Linux 3.2.0-24-generic-pae' {    recordfail    gfxmode $linux_gfx_mode    insmod gzio    insmod part_gpt    insmod reiserfs    set root='(hd0,gpt6)'    search --no-floppy --fs-uuid --set=root 313324f5-a9ed-4e80-b541-dc9e5eeb89fc    linux   /vmlinuz-3.2.0-23-generic-pae root=/dev/sda7 ro quiet splash $vt_handoff    initrd  /initrd.img-3.2.0-23-generic-pae }

Some key points about the entry in Listing 1 include:

• The insmod command loads the driver modules.
• The set root line identifies the partition from which the kernel and initial RAM disk are read, but the search line then overrides this value and locates the partition by the Universally Unique Identifier (UUID) number of the file system that it contains.
• The linux line identifies a Linux kernel and sets the options that are passed to it.
• The initrd line identifies an initial RAM disk file that's passed to the kernel.

Interacting with GRUB 2 at boot time

To be able to fix problems, you should first understand the normal boot process and the ways in which you can adjust it. Frequently, you can fix minor problems by using GRUB 2's built-in editor to tweak your boot options. You can sometimes recover from more-serious problems by using GRUB 2's built-in shell.

Understanding the normal boot process

Traditionally, GRUB displays a text-mode menu, similar to the one shown in Figure 1, that displays your boot options. (On many installations today, GRUB hides this menu unless you press a key.)

Figure 1. The GRUB menu

In a normal boot, you use the up- and down-arrow keys to navigate through the menu and then select your desired entry by pressing Enter. On Linux systems, GRUB then loads the kernel and initial RAM disk and passes control of the computer to the kernel.

Changing your boot options

If you get to a GRUB menu similar to the one shown in Figure 1 but your selection fails to start, there might be a problem with the boot entry. GRUB includes a simple text editor so that you can make temporary changes your boot stanzas at runtime. To change your boot options, select the entry in the GRUB menu that's closest to what you want to achieve and then press the e key. The result resembles Figure 2:

Figure 2. The GRUB text editor

The lines in Figure 2 are the same as those in the boot stanza in Listing 1. You can edit these entries much as if you were using a text-mode text editor in Linux. Any changes that you make in this editor are temporary. (To learn how make your changes permanent, see Making permanent repairs, later in this article.)
One common reason to edit a boot stanza in the text editor is to make a one-boot change. For instance, suppose that you want to boot into single-user mode to perform low-level maintenance, but there's no single-user entry in GRUB. You can achieve your goal by editing the boot stanza and adding single to the end of the linux line. When you're done, press Ctrl-x or F10 to boot, as the prompt at the bottom of the screen reminds you.
If you create a new GRUB entry and it fails to start, you might be able to discover the problem by examining the boot entry. Perhaps the entry contains a typo, such as linu instead of linux. Maybe you omitted the initrd line. Perhaps you specified the wrong root file system. You might be able to correct such problems by using your knowledge of your system and of GRUB 2 configuration generally. In other cases, though, you might lack critical knowledge. For instance, you might need to learn what your root file system's identifier is. In such cases, or when problems are more severe, you can use the GRUB 2 shell.

Using the GRUB 2 shell

GRUB includes its own built-in shell where you can type commands similar to those you can type in Bash or other Linux text-mode shells. The GRUB shell is simple by Linux standards, but it's adequate for many emergency maintenance tasks. To enter the shell from the GRUB main menu (Figure 1), press c. To enter the shell from the GRUB editor (Figure 2), press Ctrl-c or F2. The result resembles Figure 3:

Figure 3. The GRUB shell

The GRUB 2 shell supports a range of commands, many of which are similar or identical to the commands used in grub.cfg to control the menu-driven boot process. If you're familiar enough with the GRUB 2 configuration file format, you can boot your computer by typing commands at the shell. In practice, you're more likely to use the shell for recovery operations. Table 1 presents some of the commands that are most likely to be useful in this respect. GRUB 2 supports many additional commands, which you can read about in the GRUB documentation.

Table 1. Common GRUB 2 command-line commands

Command	Options	Explanation
boot	None	Boot a kernel or chain loader that's already been defined.
cat	filename	Display the contents of the specified file.
configfile	filename	Load the specified configuration file.
help	[command]	Display help—either a list of commands or help on the specified command.
initrd	filename	Load the specified initial RAM disk file.
insmod	module_name	Load the specified module (driver).
linux	filename	Load the specified Linux kernel.
ls	[arg]	List devices or files on a device.
reboot	None	Reboot the computer.
set	envvar=value	Set an environment variable equal to a value.

One problem that can motivate use of GRUB's command line is GRUB's inability to locate its own configuration file. Reinstalling GRUB, as I describe in this article's Making permanent repairs section, is the long-term solution. In the meantime, though, you can issue a few commands to bring up your regular GRUB menu and boot Linux. To begin, you must identify the partition on which GRUB is installed. You can do this with the help of the ls command. Used without any options, ls displays the disks and partitions that GRUB can detect. You can then look inside particular partitions by specifying the device's name with a trailing slash (/), as shown in Listing 2:

Listing 2. Using ls to view devices or the contents of file systems

1 2 3 4 5 6

grub> ls (hd0) (hd0,gpt5) (hd0,gpt4) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) grub> ls (hd0,gpt5)/ abi-3.2.0-22-generic grub/ initrd.img-3.2.0-22-generic memtest86+bin System.map-3.2.0-22-generic vmcoreinfo-3.2.0-22-generic vmlinuz-3.2.0-22-generic

The example in Listing 2 shows a computer with a single disk (hd0) that holds five globally unique identifier (GUID) Partition Table (GPT) partitions. The contents of (hd0,gpt5) appear to be a Linux /boot partition, including a GRUB configuration directory (grub/). You might need to peek inside other partitions before finding your Linux /boot partition. If your system doesn't use a separate /boot partition, you must look for your Linux root (/) partition instead.
With the home of the GRUB configuration file identified, you can tell GRUB where to find it by setting the prefix and root environment variables. These variable identify, respectively, the directory in which grub.cfg lives and the partition on which it resides:

1 2	grub> set prefix=(hd0,gpt5)/grub grub> set root=(hd0,gpt5)

From here, you can load the normal module and launch it to bring up the GRUB menu:

1 2	grub> insmod normal grub> normal

Using Super GRUB2 Disk

A BIOS-only tool

Super GRUB2 Disk is a BIOS-only tool. If you normally boot in EFI mode, Super GRUB2 Disk isn't helpful. EFI-mode booting doesn't rely on code that's stored in the MBR or a disk's boot sector, which are code areas that — when damaged — make Super GRUB2 Disk most useful. In case of EFI boot problems, your firmware's boot manager or an ancillary program, such as rEFInd, might help you recover from problems caused by corrupted NVRAM settings. I also recommend backing up the files in an EFI computer's ESP in case that partition becomes damaged. Because EFI doesn't rely on hidden code, you can restore the ESP with a simple file-copy operation. (Full disclosure: I maintain rEFInd.)

In some cases, GRUB won't even give you a grub> prompt, or you might have trouble resolving a problem even with the prompt. In these cases, the Super GRUB2 Disk rescue tool can help.

Preparing for disaster

Even if you can boot successfully now, I recommend that you have copy of Super GRUB2 Disk on hand for immediate use when you need it. The Super GRUB2 DISK download is a hybrid image file with a .iso extension. You can use dd to copy this file to a floppy disk, a CompactFlash (CF) disk, a Universal Serial Bus (USB) flash drive, or a similar type of disk. Alternatively, you can use cdrecord or a GUI optical disk tool to copy the file to a CD-R.
After you create your boot medium, I recommend that you test it — ideally, on multiple computers — to familiarize yourself with the tool and to verify that it works on the hardware you use.

Booting with Super GRUB2 Disk

After you prepare a Super GRUB2 Disk, you can boot it like any other bootable disk. In some cases, you might need to change your boot order by pressing a key during your boot process. F2, F10, and F12 are common choices, but you should consult your computer's manual for details. When Super GRUB2 Disk boots, you're greeted by a display, similar to the one shown in Figure 4, that includes options to detect OSs or enable various types of support:

Figure 4. GRUB menu that's displayed when you boot Super GRUB2 Disk

If your computer uses redundant array of independent disks (RAID) or Logical Volume Management (LVM) — or relies on older Parallel ATA (PATA) disks or external USB disks — you might need to activate those features by selecting them and pressing Enter. When that's done, you can try the detection options. I find that the Detect any GRUB2 configuration file (grub.cfg) and Detect any GRUB2 installation (even if the MBR is overwritten) options generally work best for recovering a damaged GRUB installation. But the Detect any Operating System option might also be worth trying.
If the detection is successful, you should see a new GRUB menu of options. On a single-OS installation, this menu probably contains just one entry that identifies the GRUB configuration file by its path, as in (hd0,gpt5)/grub/grub.cfg. When you select this option, your installation's normal GRUB screen should appear. (Fonts and colors might be different, but the menu options should work normally.)

Making permanent repairs

Repairs such as those that I've described so far are impermanent. You might boot Linux successfully, but as soon as you reboot you end up with the original GRUB screen. To make your changes permanent, you need to take additional steps.
The simplest of these steps is to adjust your GRUB configuration file. Although you can edit grub.cfg directly to alter your settings, this approach is inadvisable because automated scripts are likely to reconstruct the file from other files whenever you upgrade your distribution-provided kernel. Instead, edit files in /etc/grub.d and default global settings in /etc/default/grub. You can then generate a new grub.cfg file from the Linux command prompt by using grub-mkconfig:

1	grub-mkconfig -o /boot/grub/grub.cfg

If your problem is that GRUB brought up only a grub> prompt or didn't start at all, you must reinstall GRUB to your hard disk:

1	grub-install /dev/sda

In some cases, you might need to install to a device other than /dev/sda, such as /dev/sdb. Installing GRUB 2 to a partition is generally inadvisable. If you're installing GRUB to a GPT disk on a BIOS-based computer, ensure that the computer includes a BIOS Boot Partition. Without it, GRUB might refuse to install or might be unreliable. If you're installing GRUB to an EFI-based computer, omit the device specification and ensure that your ESP is mounted at /boot/efi. grub-install copies the necessary files to this directory (and hence to the ESP) automatically. If GRUB doesn't start on an EFI-based computer because of improper NVRAM settings, you might be able to fix those within the firmware itself, but details vary greatly among implementations. Alternatively, if you can boot an emergency system in EFI mode, you can use efibootmgr to restore your boot loader:

1	efibootmgr -c -l \\EFI\\loaderdir\\loadername.efi -L MenuName

Conclusion

GRUB 2 is a flexible tool for directly booting Linux (and several other) OS kernels. But because of vulnerabilities in the boot process and GRUB's own complexity, problems can occur that can render a system unbootable. In such situations, knowing how to edit individual GRUB stanzas, use the GRUB command line, and use Super GRUB2 Disk are invaluable skills. By using these techniques, you can recover from various boot problems and boot into your normal installation. At that point, you can edit your GRUB 2 configuration file or reinstall the boot loader to make your repairs permanent.