18 December, 2013

Optimizing Fedora Startup Performance

Disclaimer: This isn't a full course meal on the topic - just the main course. I'm concentrating on the low-hanging fruits here & ignoring stuffs which require excessive amount of warnings without significant impact.

In case you're interested about the ignored bits - you may wanna check this blog that covers some of those parts: http://www.harald-hoyer.de/2013/11/13/fedora-boot-optimization/

Update 2: Reviewing the Fedora 20 change-sets, I have a feeling that although following is a generic way of optimizing Fedora startup - there are so many radically awesome changes inbound with Heisenbug, that you should definitely consider a fresh install to get rid of many ghosts of the past (if you can afford to).

To start with

I recently (yesterday) performed a fresh install of Fedora 20 on my test-rig & then onward, rebooting my other system appeared to be considerably slower (compared to it).

This is a system on which Fedora 17 was initially installed & then innumerable amount of packages were installed over the time along with next 3 release upgrades (with FedUp). So you can imagine, a cleanup was long overdue.

How bad?

Well, "yewww!" would be the best adjective, so to say:
[root@debs ~]# systemd-analyze
Startup finished in 2.526s (kernel) + 10.188s (initrd) + 2min 43.299s (userspace) = 2min 56.015s
Damn... Which spawn of devil is screwing up my userspace-startup!
[root@debs ~]# systemd-analyze blame
     1min 1.856s sm-client.service
     1min 1.271s sendmail.service
         15.865s firewalld.service
         13.805s plymouth-quit-wait.service
         11.665s rsyslog.service
          9.543s accounts-daemon.service
          9.293s dnf-makecache.service
          7.995s systemd-cryptsetup@luks<trimmed-volume-guid>.service
          7.024s NetworkManager.service
          6.392s httpd.service
          4.696s rtkit-daemon.service
          4.671s avahi-daemon.service
          4.670s systemd-logind.service
          4.417s gdm.service
          2.839s NetworkManager-wait-online.service
          2.617s polkit.service
          2.573s systemd-cryptsetup@luks<trimmed-volume-guid>.service
          2.361s chronyd.service
          2.003s wpa_supplicant.service
          1.885s systemd-udev-settle.service
          1.353s plymouth-start.service
          1.064s systemd-readahead-replay.service
          1.021s jexec.service
          0.XXXs [trimmed]
John "F" Kennedy!

For me the immediate reaction is: "Why do I even have sendmail... must be the BugZilla test-setup" — but the particulars may vary person-to-person system-to-system.

Update 1: As Sankarshan points out, sendmail has _been_ the default MTA for quite a while — not an after-setup.

What now?

Go through the list before & figure out which nutjobs you wanna kill with fire, which ones simply don't make sense to have around anymore — and remove them.
[root@debs ~]# yum remove xxx yyy zzz
Won't chew the food for you here... if you can't figure out the package-names to remove, you should try different ways to optimize the system (how about a fresh-install, eh! :P).

There are some stuff which you may need, but don't need during startup (NetworkManager, iSCSI, RAID). So removing them isn't a good idea - but you can safely disable them:
[root@debs ~]# systemctl disable xxx yyy zzz
I'm going with systemctl disable instead of systemctl mask because, it's more revertible, in case things go wrong. For brave souls, go ahead with whichever you see fit.

Each time you optimize some certain parts - reboot & repeat.

Not only it gives you the idea how the optimization is going, in case things go wrong, you have less troubleshooting to perform to pinpoint which caused it exactly.

Close Enough!

Once you feel comfortable with the startup speed &  systemd-analyze blame output, then it's time to wrap up.
[root@debs ~]# yum autoremove
This will uninstall the packages which were installed as dependency of other packages, but that main package has been removed (generally, rendering these useless)... completely safe to remove.

Also, the startup depends on the HDD performance. It's better not to use LVM partitioning (at least on the startup disk). Also, make sure to check for file-system errors, bad-sectors & volume usage-percentage. Reclaim some HDD space if required (start with cleaning /var/log/*). Anyway, out of the scope of this post.

I've got my system down to 40sec for a complete startup (of which 50% is decrypting LUKS filesystem — can't do shit about it) — that works for me!

Extra mile...

If you're still not happy, you can go on ahead further optimizing these parts:
  • Remove the leftover scripts from Live Media
  • Disable/remove Plymouth (i.e. GUI Bootscreen)
  • Disable avahi, bluetooth et al. services at startup
  • Disable Firewall, SELinux etc. (bet you know what you're doing)
  • Find the failing processes/services with systemctl --failed and fix 'em
  • ...and the list goes on! (What all did I miss?)

Statutory warning

Always have a fail-proof/alternate kernel to boot to (especially if you're going to mingle with dracut/initramfs-rebuilds). Have a Live USB ready, in case you need to fix /boot (btw, keep a backup of /boot), or need to chroot the installation in case it doesn't boot up.

If your installation doesn't have too much customization, then considering a fresh install is a really good idea (n00b tip: having /home on a different mount point makes it much easy to format the system, even with different distro but still have the user configurations retained).