When doing kernel development, doing it in a virtual machine can be very convenient, if there’s no need for actual hardware devices or features. This is especially true for network or client/server development where multiple physical machines would otherwise be needed. Plus, VMs reboot much faster than actual hardware!

The #1 tip: a shared development directory

My preferred setup is to use KVM via virt-manager. I use my editor and the compiler on the host, and then mount my development directory on the guest, and then install the compiled modules there. This lets development on the host remain undisturbed by unstable kernel versions and new target distro versions. In fact, my host is still on RHEL 6, although I’m working on features for much more current kernels and distro releases.

I use NFS to export a mount point, and then mount it in an identical location in each guest. Then, edit and build the kernel on the host, using the ‘O’ kernel make option to keep .config and build files separate from the kernel’s git tree, although both the source and build dirs are under the mount point accessible to the guest. Finally on the guest, “make O=/path/to/buildfiles modules_install install” and everything’s ready to test.

On a more recent host, and with guests that support it, an easier way to set up a shared directory would be using VirtFS. NFS is a little fiddly to set up for the first time, and virtfs looks pretty easy, and a little faster and secure, even.

Other tips

1. Guest debug output onto the host. Set up a virtual serial port and point it at a file on the host. Then, add “console=ttyS0,115200 console=tty0” to the guest kernel command line. This will output everything to the file as well as keep outputting to the guest console. Then ‘tail -f‘ the file, and you can be assured any kernel oopses or other messages will be captured. The file will be truncated every time the guest is restarted, BTW.
2. Turn on all relevant debugging options (under “kernel hacking”) when compiling your kernel. If you start with the distribution’s .config file, many won’t be set. I’d recommend turning everything on. Also turn on frame pointers, and configure out drivers and subsystems the guest won’t need. ‘make localmodconfig‘ (run from the guest) might help here.
3. If you are in an edit/compile loop, use the ‘M=’ make option to just build where you’re hacking, and save make from scanning the whole tree. Just use ‘make modules‘ (on the host) and ‘make modules_install‘ (on the guest) to save time (posttest??? arrgh) and not reinstall a kernel if it hasn’t actually changed. The build will increment the kernel version, which is printed during build and in ‘uname -v‘ output, only if it actually built a new kernel image.
4. Get an SSD. Pays for itself in saved time almost instantly.
5. Use ccache. If a source file has already been built with identical headers, ccache keeps the object around and saves the build from doing repeated work. This is as simple as installing ccache on the host, and then setting ‘export CC="ccache gcc"‘ in your .bash_profile.
6. Make sure your guest has two or more virtual CPUs, in order to properly expose yourself to races when testing.
7. Set guests to auto-login and turn off screen savers. Use Ctrl-R (reverse history search) in bash aggressively when repeating test steps on the guest. Give different guests different-colored backgrounds to tell them apart more easily.
8. Make sure lockdep checking is enabled, and hasn’t fired earlier in the boot process, because then it turns itself off.
9. Familiarize yourself with magic-sysrq feature in Documentation/sysrq.txt, and poking it via /proc/sysrq-trigger.

I hope some of this is helpful, and I’d love to know about more tips and techniques people have.

EDIT: Some more tips in G+ comments here!