I have been running an OpenBSD firewall cluster on my home network for some time. These two machines use CARP and pfsync to seamlessly fail over interfaces in case of a hardware or software failure. This has been quite convenient, and allows me to upgrade OpenBSD without losing internet connectivity. I only have one available public IP address, so that has to be on the CARP interface, ie each firewall can't have it's own public IP address. This meant that the backup machine had no connectivity to the outside world.

This weekend I discovered ifstated. This clever tool is really a finite state machine, that knows about the status of interfaces. You can also run your own shell commands as part of a state entry, transition, or exit. Now I have it configured to watch for transitions in the CARP interface, and if we are the backup, it changes the default route so that it points to the machine in the cluster that is the primary. Now the backup machine can get to the outside world.
Everyone always likes to put their own config files up when explaining this kind of thing, but I have a fairly complex configuration. I thought it might be more helpful to describe how to configure ifstated for the "Basic Example" found at http://www.countersiege.com/doc/pfsync-carp/.

The main difference between their configuration and mine is that they have a public IP address on their sis0 interfaces. I don't have that luxury. Without that IP address, the default route can only egress via the carp interface. If a particular node in the cluster is the backup node, the carp interface doesn't go anywhere, and you have no pipe to the outside world. In their example, they don't tell you what the next hop to the internet is, so we are going to pretend that it's 10.0.0.254. So, here's an /etc/ifstated.conf file for firewall A that will switch the default route when the status of the carp interface changes:

#
# ifstated.conf
#
# change the default route on the backup firewall so that it can
# get out to the rest of the world via the primary firewall
#
init-state master
carp_init = "carp0.link.unknown"
carp_master = "carp0.link.up"
state master {
init {
# change the default route to be out the router
run "route change default 10.0.0.254"
}
# probably just came up.  give things a change to sync
if $carp_init
run "sleep 5"
if ! $carp_master
set-state slave
}
state slave {
init {
# change the default route to go through the other firewall
run "route change default 192.168.0.253"
}
# if we become the master, set the state to master
if $inet_carp_master
set-state master
}

To make this config work on firewall B, s/192.168.0.253/192.168.0.254/.