Timely Intervention

I was replying to a post on the PC-BSD Forums about updating the ports tree, and it seemed to tie into a lot of things I've been looking on lately, so I decided to expand it a bit and explain how to do it.

The FreeBSD ports tree is an amazing piece of work. At the time of this writing, there are over 16,000 pieces of software in it, and it is just incredible how well most of them work. One of the most tedious things to do in a non-Windows environment is to install software from source (which is often how they are delivered). This is especially true in the development world, where support libraries and the like usually require a bunch of configuration, and careful building. When I was trying to get wxWidgets to install on Linux, it took forever to get it set up and installed right. Under FreeBSD, I simply did:

# cd /usr/ports/x11-toolkits/wxgtk28
# make all && make install && make clean 

I like to do the port build in three separate steps, rather than the more common # make install clean. It just seems to isolate problems better, and does a better job of building things without problems. Of course, these days I would just do # portmanager x11-toolkits/wxgtk28 -l and let portmanager do all the heavy lifting.

But in order for this process to work well, you have to be a stickler for keeping the ports tree up to date. It isn't surprising to realize that with over 16,000 ports, there are changes happening to it all the time. So you want to get a good system for keeping it up to date. I've written before about the much improved over cvsup way of doing it using portsnap, so that's what I'm going to talk about.

First, you need to get a crontab job set up to grab the updates. Many places out there still tell you to just go ahead and edit /etc/crontab as root. But that's just not the way to do it any more. Like virtually everything in /etc, the crontab file should be left alone, so that future updates can just overwrite it. About the only file that should get edited in there is rc.conf, and even for that I hear there's move afoot to create a custom version in /usr/local/etc which is where it belongs. So you don't want to just edit the system-wide /etc/crontab file, but rather you want to add a crontab job to root's personal crontab. You do it this way:

# setenv EDITOR ee
# crontab -e

Notice I set the EDITOR environment variable. In typical BSD fashion, the EDITOR variable tells crontab (and many system other config tools) what editor to use. The default editor is vi and, while I don't want to get into an editor war, I think I can safely say there isn't a more difficult editor for a newbie to use! I can use it enough to get some stuff done, and if you are going to be working in BSD land for long, I advise you to get minimally proficient as well. But if you set the EDITOR variable to your favorite editor, or at least 'ee' (easy editor), you'll find things much clearer. 'ee' is a very easy to use editor, and has lots of prompts. If your root shell isn't the default csh, you'll need to use the appropriate syntax to set the EDITOR variable (bash users would use # export EDITOR=ee, for instance).

After you type in the crontab -e command, you'll be dropped into your editor with a blank screen and a strange filename, like 'crontab.qNqprVSSlN'. Not much help really. What has happened is that it created a temporary crontab file and wants to you enter in, one line at a time, what to have it periodically do. The crontab(5) man page describes the format of this very picky file. There are 6 "fields" in the files, separated by white space. The first 5 fields tell it the date and time of when to run the 6th field, which is the command to run (the system /etc/crontab has a 'user' field before the command field, but because we're editing the user specific crontab, it isn't needed here). Read the man page for all the various ways to specify date and time, as it is incredibly flexible. We're going to use a shortcut, as here's what our crontab will look like:

@midnight /usr/sbin/portsnap cron

There's a bunch of special shorthand '@' specifiers you can use in the crontab which I only recently learned about. Here's the full list:

string	   meaning
------	   -------
@reboot	   Run once, at startup.
@yearly	   Run once a year, "0 0 1 1 *".
@annually	   (same as @yearly)
@monthly	   Run once a month, "0 0 1 * *".
@weekly	   Run once a week, "0 0 * * 0".
@daily	   Run once a day, "0 0 * * *".
@midnight	   (same as @daily)
@hourly	   Run once an hour, "0 * * * *".

So instead of the 5 date/time fields, you can use one of these special strings, which is a nice shorthand for a particular date/time set of fields (I can never remember the order of them!). Another thing to be careful with is the command you want to run (which is all the rest of the line after the last date/time specifier). The environment it runs in is pretty bare, so it is safer to specify the full path to command and assume almost no other environment variables are set (the crontab(5) man page tells more).

In this case, we are going to run portsnap's special 'cron' command. This tells portsnap to sleep for some random amount of time (up to 6 minutes) before actually doing the 'fetch' command and getting the updates. This is so all the lazy people like ourselves who insist on specifying a "simple" time like this don't bury the server every night precisely at midnight! At some point, you could probably change the time to be something like 13 3 * * *, to run it a 3:13am, a somewhat off time. But this will work for now.

So now, every night around midnight, portsnap will fetch all the /usr/port changes. But it won't actually update the port tree. It isn't really a good idea to run the 'update' command from a cron job, because terribly bad things would happen if you were in the middle of building a port when it suddenly got updated by a hidden cron job. I don't know what would happen; maybe the Earth would implode or the Yankees would win the World Series, but something too terrible to contemplate, so don't do it:-)

So before I build a port for the first time in a day, I do the update command:

# portsnap update
# portmanager security/sudo -l

It doesn't hurt to do the update too many times, but you only need to do it once after the 'fetch' command has been done. I'll bet I could even script something that could figure out if an update had been done since the fetch and run it automagically before doing the portmanager build. I'll have to look into that.

So there you have it. The only thing you have to remember to do is to run the portsnap update command before installing a new port, which isn't so hard. And you are left with a nicely updated /usr/ports tree.