Service Jails

Introduction

Warning: This page is very much WIP.

FreeBSD Jails allows a system administrator to partition a FreeBSD system, so if one jail/partition is compromised the rest of the system is not. For general information about Jails see the FreeBSD jail(2) manual page.

The major problem with using jail is normally how to keep each jail up-to-date. This normally isn't a problem if only using a few jails, since the manual proccess is fairly simple, but if managing many jail it can be quite timeconsuming and tedious proccess.

Service jails was designed to solve this problem by sharing as much as possible between jails, in a safe way, so it becomes more viable to create one jail per "service". Service in this context is a HTTP server, DNS server, SMTP server etc.

Thanks to "subdue" for trying out these instructions when they were even more incomplete than now, and asking a lot of good questions which have lead to improvements and new ideas.

Requirements

This was originally created for a new DMZ server I wanted to put every public service in a jail and preferably only one service. If possible, a model like this might also be used for the new www.FreeBSD.org setup. To do this I have set up the following requirements:

Design

The design heavily relies upon nullfs by having one master install which are then nullfs mounted to be used by each jail and one read-write file-system per jail. This means that the basic way of doing it requires a separate "device" for each jail - that device can be a physical disk, a partition on a disk, or a vnode backed md(4) device.

If it is not possible or desired to use a separate device, "service jails" can still be used by utilizing a read-write nullfs mounted "file system" instead. This of cause allows the jail to overflow that file-system. Using read-write nullfs for this has not yet been tested by the author and is therefor not documented here.

Note: Much here assume FreeBSD 6.0, but can probably be used with little modification for any recent FreeBSD release (5.3 or later).

Everything will be mounted under /j. The master root directory which will be the root of all jails will be /j/mroot. Inside each jail /s will be the read-write part of the system. The /s can be either a real device or e.g. a file backed memory device. The parts of the filesystem where read-write access is required will be symlinked in mroot into the per jail specific /s.

Create mroot with:

# Prepare skeleton dir - we need to move some dirs in there
mkdir /j/skel /j/skel/distfiles /j/skel/home /j/skel/tmp
mkdir /j/skel/usr-local /j/skel/usr-X11R6
# Now create the master root
mkdir /j/mroot
cd /j/mroot
# XXX, not really sure if the mtree commands are needed
mtree -U -d -f /etc/mtree/BSD.root.dist
mtree -U -d -f /etc/mtree/BSD.usr.dist -p usr
mtree -U -d -f /etc/mtree/BSD.include.dist -p usr/include
# Install our distribution bits
cat /mnt/cdrom/6.0-RELEASE/base/base.?? | tar --unlink --exclude boot -xpzf -
# Create the RW root used inside the jails
mkdir s
# Create a ports directory using portsnap
mkdir usr/ports
portsnap -p /j/mroot/usr/ports extract
# Remove dirs which we will change into symlinks
rmdir tmp usr/local
# Move directories created from our distribution bits into our
# skeleton directory since the jail will need these RW.
mv etc /j/skel
mv var /j/skel
# Now create symlinks from the RO root into the per jail RW root
ln -s s/etc etc
ln -s s/tmp tmp
ln -s s/var var
ln -s s/home home
ln -s ../s/usr-local usr/local
ln -s ../s/usr-X11R6 usr/X11R6
ln -s ../../s/distfiles usr/ports/distfiles
mtree -U -d -f /etc/mtree/BSD.local.dist -p /j/skel/usr-local
mtree -U -d -f /etc/mtree/BSD.x11.dist -p /j/skel/usr-X11R6
mtree -U -d -f /etc/mtree/BSD.var.dist -p /j/skel/var
# Set proper permissions on /tmp
chmod 1777 /j/skel/tmp
# We need to make sure we do not create jails without a root password
sed -E -i -e 's/root::/root:*:/' /j/skel/etc/master.passwd

This will create /root on the read-only file system, which might be a problem for some poeple. If a read-write /root is needed run the following (note, not tested yet):

mv /j/mroot/root /j/skel/root
ln -s s/root /j/mroot/root

Create /j/skel/make.conf:

WRKDIRPREFIX?=  /s/portbuild

Set up generic jail configuration in rc.conf:

# Jails
jail_enable="YES"
jail_set_hostname_allow="NO"
jail_list=""

Per jail creation:

This example has a jail named squid with the read-write part of the jail file-system on the device /dev/mirror/gm0s2b.

The following are placed in fstab per jail, where my jail rw directory is /dev/mirror/gm0s2b:

/j/mroot                /j/jails/squid  nullfs  ro              0 0
/dev/mirror/gm0s2b      /j/jails/squid/s ufs    rw              2 2

The jail read-write device (/dev/mirror/gm0s2b) need to be properly initialized with an UFS file-system, IE. normally simply by running newfs on like shown below. In general it is a good idea to enable soft-updates on the file system, so the -U paramter is used for that.

newfs -U /dev/mirror/gm0s2b

For rc.conf:

jail_list="... squid"
jail_squid_hostname="squid.nitro.dk"
jail_squid_ip="192.168.3.17"
jail_squid_rootdir="/j/jails/squid"
jail_squid_devfs_enable="YES"

And now create the needed directories as shown below. In this example cpdup, from the sysutils/cpdup port is used to mirror the skeleton directory into the jail. It can probably be substituted with rsync -a if rsync is prefered.

J=/j/jails/squid
mkdir $J
mount $J
# The new mount point "$J/s" will now "magically" appear since it is part
# of the shared master root.
mount $J/s
cpdup /j/skel $J/s

Upgrade jails

To upgrade the jails:

Note: This has not recieved much testing!

cd /j/mroot
find . -flags schg -print0 | xargs -0 chflags noschg
cat /mnt/cdrom/6.0-RELEASE/base/base.?? | tar --exclude boot --exclude etc --exclude var --exclude tmp --exclude usr/local --unlink -xpzf -

TODO: