My Video for Mayday.us: We all deserve a say!

Leave a reply

This is a video Jen and I made for the Mayday.us Maydayin30 contest. If you don’t mind giving them your email address, you can still vote for it here.

We went down to Pioneer square and just got some footage of random folks, trying not to be too obtrusive. I used my Blackmagic Pocket Cinema Camera, along with the Metabones Pocket Speedbooster and a Nikon 55-300 zoom. Unfortunately my monopod’s bottom foot needed tightening, so the video was a bit shaky. Shooting longer distances didn’t help, either.

Back at home, I cut the best parts together and wrote an appropriate voice-over text. I tried reading it myself first (sitting in the car in the driveway, to minimize reverb) but it wasn’t nearly as good as Jennifer’s readings, so that’s who you hear. Then I quickly added a piano based score using Ableton’s Piano sample set, and that was it.

It turned out very differently from other submissions. It’s a long shot but who knows? For me, I learn something new in each little project I complete, so just for that it was valuable to me.

A program calling command-line tools is the moral equivalent of web scraping.

4 Replies

I gave this talk at LPC 2012. It promotes the idea that programs layered on top of human-centric interfaces is a bad idea.

http://groveronline.com/wp-content/uploads/2014/09/all-plumbing-needs-an-api.pdf

The timing of this post with the announcement of the most recent bash vulnerability is not entirely coincidental.

Emacs and using multiple C code styles

4 Replies

I primarily work on Linux, so I put this in my Emacs config:

; Linux mode for C
(setq c-default-style
      '((c-mode . "linux") (other . "gnu")))

However, other projects like QEMU have their own style preferences. So here’s what I added to use a different style for that. First, I found the qemu C style defined here. Then, to only use this on some C code, we attach a hook that only overrides the default C style if the filename contains “qemu”, an imperfect but decent-enough test.

(defconst qemu-c-style
  '((indent-tabs-mode . nil)
    (c-basic-offset . 4)
    (tab-width . 8)
    (c-comment-only-line-offset . 0)
    (c-hanging-braces-alist . ((substatement-open before after)))
    (c-offsets-alist . ((statement-block-intro . +)
                        (substatement-open . 0)
                        (label . 0)
                        (statement-cont . +)
                        (innamespace . 0)
                        (inline-open . 0)
                        ))
    (c-hanging-braces-alist .
                            ((brace-list-open)
                             (brace-list-intro)
                             (brace-list-entry)
                             (brace-list-close)
                             (brace-entry-open)
                             (block-close . c-snug-do-while)
                             ;; structs have hanging braces on open
                             (class-open . (after))
                             ;; ditto if statements
                             (substatement-open . (after))
                             ;; and no auto newline at the end
                             (class-close)
                             ))
    )
  "QEMU C Programming Style")

(c-add-style "qemu" qemu-c-style)

(defun maybe-qemu-style ()
  (when (and buffer-file-name
       (string-match "qemu" buffer-file-name))
    (c-set-style "qemu")))

(add-hook 'c-mode-hook 'maybe-qemu-style)

Gnome 3: Setting the screen lock delay to more than 15 minutes

4 Replies

Gnome since 3.8 has restricted the Blank Screen time to between 1 and 15 minutes, or “Never”, to disable screen blanking/locking entirely. If this isn’t granular enough, you can set other values like so:
<del datetime="2014-03-25T22:25:42+00:00">dconf write /org/gnome/desktop/session/idle-delay 1800</del>
gsettings set org.gnome.desktop.session idle-delay 1800
The value is in seconds, so here we set the delay to 30 minutes (60*30=1800). It seems that once doing this, the UI will show “Never”, but the set value is still used correctly.
There is also a “Presentation Mode” shell extension that adds a button to inhibit screen lock, but for me, I still wanted to have it automatically lock, but just a little bit slower.

EDIT: dconf didn’t actually work! Apparently gsettings is the way to go.

Kernel Development with VMs: Some Tips

4 Replies

When doing kernel development, doing it in a virtual machine can be very convenient, if there’s no need for actual hardware devices or features. This is especially true for network or client/server development where multiple physical machines would otherwise be needed. Plus, VMs reboot much faster than actual hardware!

The #1 tip: a shared development directory

My preferred setup is to use KVM via virt-manager. I use my editor and the compiler on the host, and then mount my development directory on the guest, and then install the compiled modules there. This lets development on the host remain undisturbed by unstable kernel versions and new target distro versions. In fact, my host is still on RHEL 6, although I’m working on features for much more current kernels and distro releases.

I use NFS to export a mount point, and then mount it in an identical location in each guest. Then, edit and build the kernel on the host, using the ‘O’ kernel make option to keep .config and build files separate from the kernel’s git tree, although both the source and build dirs are under the mount point accessible to the guest. Finally on the guest, “make O=/path/to/buildfiles modules_install install” and everything’s ready to test.

On a more recent host, and with guests that support it, an easier way to set up a shared directory would be using VirtFS. NFS is a little fiddly to set up for the first time, and virtfs looks pretty easy, and a little faster and secure, even.

Other tips

  1. Guest debug output onto the host. Set up a virtual serial port and point it at a file on the host. Then, add “console=ttyS0,115200 console=tty0” to the guest kernel command line. This will output everything to the file as well as keep outputting to the guest console. Then ‘tail -f’ the file, and you can be assured any kernel oopses or other messages will be captured. The file will be truncated every time the guest is restarted, BTW.
  2. Turn on all relevant debugging options (under “kernel hacking”) when compiling your kernel. If you start with the distribution’s .config file, many won’t be set. I’d recommend turning everything on. Also turn on frame pointers, and configure out drivers and subsystems the guest won’t need. ‘make localmodconfig’ (run from the guest) might help here.
  3. If you are in an edit/compile loop, use the ‘M=’ make option to just build where you’re hacking, and save make from scanning the whole tree. Just use ‘make modules’ (on the host) and ‘make modules_install’ (on the guest) to save time (posttest??? arrgh) and not reinstall a kernel if it hasn’t actually changed. The build will increment the kernel version, which is printed during build and in ‘uname -v’ output, only if it actually built a new kernel image.
  4. Get an SSD. Pays for itself in saved time almost instantly.
  5. Use ccache. If a source file has already been built with identical headers, ccache keeps the object around and saves the build from doing repeated work. This is as simple as installing ccache on the host, and then setting ‘export CC="ccache gcc"’ in your .bash_profile.
  6. Make sure your guest has two or more virtual CPUs, in order to properly expose yourself to races when testing.
  7. Set guests to auto-login and turn off screen savers. Use Ctrl-R (reverse history search) in bash aggressively when repeating test steps on the guest. Give different guests different-colored backgrounds to tell them apart more easily.
  8. Make sure lockdep checking is enabled, and hasn’t fired earlier in the boot process, because then it turns itself off.
  9. Familiarize yourself with magic-sysrq feature in Documentation/sysrq.txt, and poking it via /proc/sysrq-trigger.

I hope some of this is helpful, and I’d love to know about more tips and techniques people have.

EDIT: Some more tips in G+ comments here!

O’Reilly Webcast: The Linux Way, and natural selection among distros

1 Reply

Before speaking at OSCON 2013, I gave a webcast version of my talk, and it’s now posted to Youtube (actual talk starting at 4m30s.)

  1. Tenets of the Unix Way
  2. History of Unix and Linux
  3. Modern Linux and how it may diverge from Unix
  4. The Linux Way — What is it?

The most significant change between this talk and the talk I gave six weeks later was that I figured out what the actual conclusion of the talk was:

Just like the Unix philosophies enabled the Unix command-line to develop and evolve more rapidly, the mix-and-match nature of Linux distros enables the Linux OS to also evolve more rapidly.

A distro is essentially its pool of packages, plus a handful of mutually-exclusive choices about how to run things. For example, init system, packaging system, and update frequency. These are intrinsic to the distro’s make-up, its ‘DNA’, and go beyond mutually-installable packages like apps and even desktop environments.

The cool thing is that the packages themselves are not part of a distro’s DNA. A new distro can rise up, change fundamental things about how the OS runs, and not have to fork those packages. Compared to forking more monolithic Unixes like *BSD or OpenSolaris, the barrier to entry for a new distro is relatively low, which is maybe why there are so many!

We now have a ‘gene pool’ of distros making different choices, and natural selection acts on this pool as users pick which distro they will use, and developers pick which to build on. This ensures that the Linux OS’s evolution is ultimately driven by its users. A popular distro can make some unpopular changes, but if it keeps doing so, eventually it will hurt its user base so much that other distros will take its place. This is a very good thing, and the conclusion I was trying to reach at the end of this webcast.

New screencast: 10 New Features in LIO and targetcli

1 Reply

I posted a new screencast that talks about ten new ease-of-use features that are new in Fedora 18.

10 New Features in LIO and targetcli

  1. Easier storage->ACL setup
  2. Name shows up as LUN model name
  3. Tags for initiator aliases and grouping
  4. ‘info’ command
  5. IPv6 portal support
  6. WWNs normalized
  7. Only show HW fabrics that are present
  8. 10 previous configs saved
  9. More info in summary
  10. iSER support
  11. Better sorting

Fedora for short-lifespan server instances

9 Replies

I read Máirín Duffy’s coverage of the Fedora Board’s userbase discussion. Really interesting. I wanted to add my take.

tl;dr: Puppet/Chef make Fedora’s short support period much less of an issue.

The OS is a building block

I’ve been watching a lot of videos on DevOps lately. Several close friends of mine are sysadmins and I’ve been learning a lot from them about the transformation that their profession is undergoing. From this year’s ChefConf, Adam Jacob’s keynote and the talk by Sascha Bates really impressed on me the big change in how admins should view machines — They’re not permanent, or even semi-permanent. They are ephemeral snowflakes that may live a year, or just an hour, so don’t get too attached.

Part of why admins like VMs is because the isolation they provide between different services. I used to run mail, DNS, and httpd from a single machine. Everything was mostly separate but not quite everything. They had separate userids but everyone’s config was in /etc, even touching the same files, sometimes. A full disk affected everybody. /var/log/messages didn’t split up their logging cleanly (by default, anyway.) It all just was built assuming there would be an admin at a command shell who could use their brain to resolve the conflicts to make everything play nice on a single OS image.

One service per instance

Admins adopted VMs for isolation, increased density, and better per-service resource allocation, but then ran into other problems. The setup that they did by hand once per new-hardware now was once per instance. (Editing /etc/sudoers for the fiftieth time gets old.) The tools then evolved further, until today one may keep no persistent state in an instance. Now, an instance is kickstarted into existence and configured automatically for the one job it will ever do. The sysadmin’s job isn’t to herd boxen any more, it’s to build ’em, run ’em, and then reap ’em.

All the OS mechanisms for co-existing server processes, they’re now either obsolete or vestigial to some degree. What is important is the malleability of the OS to assume all the tasks it may be asked to – rather like a stem cell needs to be able to become a nerve or muscle, but never needs to be both.

Never upgrade, just redeploy

Let’s come back to the odd fact that Fedora is both a precursor to RHEL, and yet almost never used in production as a server OS. I think this is going to change. In a world where instances are deployed constantly, instances are born and die but the herd lives on. Once everyone has their infrastructure encoded into a configuration management system, Fedora’s short release cycle becomes much less of a burden. If I have service foo deployed on a Fedora X instance, I will never be upgrading that instance. Instead I’ll be provisioning a new Fedora X+1 instance to run the foo service, start it, and throw the old instance in the proverbial bitbucket once the new one works.

Cheap and easy virt and config management gives admins what they’ve always wanted — stability when they want it (run a LT support distro image, or for the VM host) or the latest stuff for their fast-moving business-oriented instances, by running a fast-update or rolling-release distro.

What Fedora should do

We’re already working on some of these to some degree — I think we should try to do even more to ensure Fedora is useful for the fast-update instance role.

First, Fedora needs to be able to be small. Nobody’s going to read the manpages on a throwaway instance, nobody’s even going to run vi. Image size matters when multiplied for each instance. Can we get by without /usr/share/doc/* and its thousands of copies of the GPL text? Fedora seems pretty good but there must be more we can do.

Second, we need to ensure Fedora supports the packages people are really using these days. Latest Ruby. Latest OpenStack. Vagrant. Django. Chef. Puppet. All the weird JS stuff that’s popular now on GitHub. 🙂 Continue to improve packaging tools so it’s easier for new contributors to do their first package, as well as for long-time packagers to maintain more packages. And not just package for contribution to Fedora, but for admins to package for solely internal distribution. Like Sascha Bates stresses in her talk, packaging is a huge benefit to automation, but it does require effort. It can be easier.

Finally, I think we need to continue to look at how easy it is to configure and manage an instance of the OS, and tailor it more for automated configuration. I believe the key to this is adding programmatic interfaces where they are lacking. See my “All Plumbing needs an API” talk. Since we’re probably being configured by another piece of code rather than a person at the shell, we need clear, unambiguous programmatic interfaces with good error handling. Chef should not be calling cmdline tools and checking error codes, there should be a Ruby configuration library that natively controls the whatever-it-is directly! We want configuring Fedora to be fast, straightforward, and reliable.

Conclusion: Stable+fast-update is better than stable+self-built

Practically the whole history of Linux distros has been the conflict between stability and new features. With virtualization, one still must make this choice, but at a much finer granularity than before. If you’re going to re-instance within 6 months anyways, why manually build your latest-Ruby and whatnot to support your app on top of a stable distro image? Maybe just use Fedora for those.

Using qla2xxx with LIO on Fedora

1 Reply

In addition to turning your Fedora 18 box into an iSCSI target, LIO also supports other SCSI transport layers (‘fabrics’), such as Fibre Channel, with the qla2xxx fabric.

The most crucial bit is to verify that the qla2xxx driver has initiator mode disabled — it should be operating in target mode only. You can check this with:

cat /sys/module/qla2xxx/parameters/qlini_mode

It should say ‘disabled’. If it doesn’t, create a file called /usr/lib/modprobe.d/qla2xxx.conf and put:

options qla2xxx qlini_mode=disabled

in it. Then, run ‘dracut -f’ to rebuild your initrd, and reboot.

Some of you may be wondering: why /usr/lib/modprobe.d instead of /etc/modprobe.d ? This is because qla2xxx is likely loaded from the kernel’s initial ramdisk (initrd), and dracut, the initrd building tool, omits “host-specific” settings in /etc/modprobe.d. While you’re mucking around, also make sure the firmware package for your qla device, such as ql2200-firmware or similar, is also installed.

targetcli won’t let you create a qla2xxx fabric if qlini_mode is wrong. Once it lets you create the qla fabric, you can add luns to it and grant access permissions to acls exactly in the same manner as the other LIO fabrics.