Introducing gibson

Artwork by Mike Finch (CC BY 4.0)

 

As you may know, Emerald Onion systems run HardenedBSD. BSD systems in general, and HBSD in particular, provide numerous advantages to our team in operating secure and highly performant Tor relays. But BSD systems make up only a very small percentage of the Tor network. There are many similarities between BSD and Linux, with which many users may be more familiar, but the differences can be intimidating. We’re addressing this by launching gibson, a project to develop a suite of tools to address the needs of Tor service operators. The Tor network is more robust when it is diverse, and this is one way that we can encourage a more diverse Tor network and enhance our community.

It is important to say that while our initial focus is on BSD systems, our plan is to extend gibson to serve the Tor community regardless of platform. We’re starting with HBSD because it’s an obvious and natural choice for us; we believe in “dogfood“, and we want you to be assured that the code we share is used by our team in real deployments. In our mind it isn’t enough to make running Tor services easy, our tools must also help make services secure and reliable. At Emerald Onion, we do this by example.

What is gibson?

A generous description is that gibson is a no-dependency suite of cross-functional tools for creating and maintaining secure and robust Tor services. We currently support HardenedBSD systems, but plan to extend our support to FreeBSD, OpenBSD, and Linux in future releases.

We say that this is a generous description because we want our tools to take as light a touch as possible. To an experienced user, gibson may not appear to do much of anything at all. This is an intentional design decision. An experienced user might say, “Using gibson to do an update achieves the same outcome as just running these three commands I already know”. Truly, nothing would make us happier than this outcome. We want for everyone in the Tor community to be an expert of the tools that make the network operate, from Tor itself to the operating system, hardware, and everything in between. That said, we hope that experienced users will continue to use gibson, and that they will propose new solutions and new functionality.

The goal of gibson isn’t to make difficult Tor management tasks trivial. Rather, the goal of gibson is to make Tor management tasks consistent. We believe that secure systems are built around reproducible, auditable processes. Most of our maintenance is simple and mundane, but a small configuration error or a missed security patch could be catastrophic to the anonymity and security of Tor users. We want to eliminate possible sources of human error and to share our best practices with the community.

We also believe that users should be able to adopt gibson on existing systems without onerous effort, and should also be able to walk away from it whenever they wish without any lock-in. We want our tools to promote the adoption of correct processes and best practices. We also hope that they will be educational. Users new to BSD systems, new to Linux, and new to Tor should be able to look at our code and with minimal effort be able to understand what it does and why.

Finally, we say that gibson is cross-functional because our solution space is defined by the needs of a secure and robust Tor deployment. We do not seek to replace virtualization and jail tools (for instance, bhyve, virtualbox, ezjail, iocell, etc.). We do not seek to replace disk encryption tools (geli, LUKS, etc.). We’re not replacing any web servers, either (nginx, Apache, Caddy, etc.). What we are doing is providing tools to streamline the implementation of these other projects into a complete solution which addresses the needs of Tor administrators.

What does gibson do now?

Currently, gibson updates and controls Tor services running in HBSD jails. A simple and mundane task, but one that we want to make sure is done consistently during each of our maintenance windows. Our initial release of gibson is version 0.1, which is derived from a handful of scripts currently used in maintenance of Emerald Onion systems.

  • 0.0.1 Initial scripts used for EmeraldOnion system maintenance
  • 0.1.0 First version 0.0.1 scripts refactored into gibson: apply system and package updates in jails; start, stop, restart tor services in jails;

Where can I get gibson?

We’re working on getting gibson into the HardenedBSD ports repository, and it should land there shortly.  It will take a little more time for binary packages to become available for users who prefer to use pkg.

In the meantime, gibson is available at our GitHub, as is the files you need to sideload it as a port.  Detailed installation instructions are available there.

Roadmap (or: what will gibson do in the future?)

0.2 -> 0.5

  • Create and maintain template jail(s); clone templates into new jails and deploy tor services:
    • middle relays
    • exit relays
    • bridges
    • onion services (with nginix, initially)
  • Create and manage geli-encrypted ZFS pools
  • Initial creation of encrypted providers and pool members from specified devices
  • Replacement of devices due to failure or capacity expansion
  • Good ideas suggested (or implemented and submitted) by our community!

0.6 -> 0.9

  • Support for FreeBSD systems
  • Good ideas suggested (or implemented and submitted) by our community!

1.5+

  • Support for Linux systems
  • Support for non-geli encrypted filesystems
  • Support for non-ZFS storage pools
  • Good ideas suggested (or implemented and submitted) by our community!

House Style

gibson is always written in all-lowercase.

Logo

The gibson logo was generously donated by Mike Finch and is licensed by Emerald Onion as Creative Commons Attribution 4.0 International.

Tor on HardenedBSD

In this post, we’ll detail how we set up Tor on HardenedBSD. We’ll use HardenedBSD 11-STABLE, which ships with LibreSSL as the default crypto library in base and in ports. The vast majority of Tor infrastructure nodes run Linux and OpenSSL. Emerald Onion believes running HardenedBSD will help improve the diversity and resiliency of the Tor network. Additionally, running HardenedBSD gives us peace of mind due to its expertly crafted, robust, and scalable exploit mitigations. Together, Emerald Onion and HardenedBSD are working towards a safer and more secure Tor network.

This article should be considered a living document. We’ll keep it up-to-date as HardenedBSD and Emerald Onion evolve.

Initial Steps

Downloading and installing HardenedBSD 11-STABLE is simple. Navigate to the latest build and download the installation media that suits your needs. The memstick image is suited for USB flash drives. Boot the installation media.

Installing HardenedBSD is simple. Follow the prompts. Sample screenshots are provided below:

  1. Select Install:
  2. Select your keymap. If you use a standard US English keyboard, the default is fine:
  3. Choose a hostname:
  4. Select the distribution sets to install:
  5. Choose your filesystem. For purposes of this article, we’ll use ZFS for full-disk encryption:
  6. Selecting the Pool Type will allow you to configure your ZFS pool the way you want. We will just use a single disk in this article:
  7. Since we’re using a single disk, we’ll select the Stripe option:
  8. Select the disks to use in the pool. Only a single disk for us:
  9. After selecting the disks, you’ll go back to the original ZFS setup menu. We’ve made a few changes (Encrypt Disks, Swap Size, Encrypt Swap):
  10. Review the changes:
  11. Set the password on your encrypted ZFS pool:
  12. Validate the password:
  13. Encrypted ZFS will initialize itself:
  14. HardenedBSD will now install distribution sets:
  15. Set the root password:
  16. If you want to set up networking, select the network device to configure. In this article, we’ll set up a dynamic (DHCP) network configuration:
  17. We want to use IPv4:
  18. We want to use DHCP:
  19. It will try to acquire a DHCP lease:
  20. At Emerald Onion, we put IPv6 first. However, in this example article, we won’t use IPv6 as it’s not currently available. So we’ll choose no when prompted to set up IPv6:
  21. Ensure the DNS information is correct and make any changes if needed:
  22. It’s now time to choose the system timezone. Select the region:
  23. We chose America. We’ll choose United States for the country next:
  24. Finally we’ll chose the actual timezone:
  25. Confirm the timezone:
  26. Because we use NTP, we’ll skip setting the date:
  27. We’ll also skip setting the time:
  28. Select the services to start at boot:
  29. Select the system hardening options. HardenedBSD sets options one through five by default, so there’s no need to set them here.
  30. We will go ahead and add an unprivileged user. Make sure to add the user to the “wheel” group for access to use the su program:
  31. Set the user’s details:
  32. HardenedBSD is now installed! Exit the installer. The installer will do things in the background so there may be some delay between exiting and the next prompt:
  33. We don’t want to make further modifications to the installation prior to rebooting:
  34. Go ahead and reboot:

The installation is now complete!

Installing Tor

Installing Tor is simple, too. Once HardenedBSD is installed and you’ve logged in, run the following command:

# pkg install tor

The Tor package on HardenedBSD, and its upstream FreeBSD, currently does not ship with a modified Tor configuration file, which can be found at /usr/local/etc/tor/torrc. Tor isn’t set up to log outside of initial startup messages. You will need to edit the Tor configuration file to suit your needs. Take a look at the tor(1) manpage for all the available configuration options.

In our set up, Tor listens on TCP ports 80 and 443 as an unprivileged user. We need to tell HardenedBSD to allow non-root users to be able to bind to ports that traditionally require root privileges:

# echo 'net.inet.ip.portrange.reservedhigh=0' >> /etc/sysctl.conf
# service sysctl start

Multi-Instance Tor

At Emerald Onion, we run multiple instances of Tor on the same server. This allows us to scale Tor to our needs. The following instructions detail how to set up multi-instance Tor. The same instructions can be used for single-instance Tor.

We named our instances simple names: instance-01, instance-02, instance-03, and so on. Each instance has its own configuration file, located at /usr/local/etc/tor/torrc@${instance_name}. We first set up a template config file:

Nickname EmeraldOnion%%INSTANCE%%
Address tor01.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
Log notice file /var/log/tor/instance-%%INSTANCE%%/notices.log
OutboundBindAddressExit %%IP4ADDR%%
OutboundBindAddressOR %%IP4ADDR%%
DirPort %%IP4ADDR%%:80
ORPort %%IP4ADDR%%:443
ORPort %%IP6ADDR%%:443
RelayBandwidthRate 24 MBytes
RelayBandwidthBurst 125 MBytes
MyFamily %%FAMILY%%
IPv6Exit 1
ExitPolicy accept *:*
ExitPolicy accept6 *:*
SocksPort 0

The next script installs the appropriate config file based on the above template. Some things are sanitized. Shawn, who wrote the script, is a fan of zsh.

#!/usr/local/bin/zsh

ninstances=5

family=""

for ((i=1; i <= ${ninstances}; i++)); do
	instance=$(printf '%02d' ${i})

	family=""
	for ((k=1; k <= ${ninstances}; k++)); do
		[ ${k} -eq ${i} ] && continue
		[ ${#family} -gt 0 ] && family="${family},"
		family="${family}EmeraldOnion$(printf '%02d' ${k})"
	done

	sed -e "s/%%INSTANCE%%/${instance}/g" \
		-e "s/%%IP4ADDR%%/192.168.1.$((${i} + 10))/g" \
		-e "s/%%IP6ADDR%%/\[fe80::$((${i} + 10))\]/g" \
		-e "s/%%FAMILY%%/${family}/g" \
		tmpl.config > /usr/local/etc/tor/torrc@instance-${instance}
	mkdir -p /var/log/tor/instance-${instance}
	chown _tor:_tor /var/log/instance-${instance}
	chmod 700 /var/log/instance-${instance}
done

We then instructed the Tor rc script not to run the default instance of Tor:

# sysrc tor_disable_default_instance=YES

Then we tell the rc system which Tor instances we want to run and set Tor to start at boot:

# sysrc tor_instances="instance-01 instance-02 instance-03 instance-04 instance-05"
# sysrc tor_enable=YES

Then we start Tor. The first time the Tor rc script starts Tor, it will create the data and logging directories for you with the proper permissions.

# service tor start

Keeping HardenedBSD and Tor Up-To-Date

Updating HardenedBSD is simple with hbsd-update. We publish updates for base periodically. Feel free to use hbsd-update as often as you’d like to check for updates to the base operating system.

For example:

# hbsd-update
# shutdown -r now

To update your packages, including Tor, use:

# pkg upgrade

Tor Service Management Basics

The tor rc script uses SIGINT when shutting Tor down. This causes Tor to shutdown in an ungraceful manner, immediately halting connections from clients. Instead of using the traditional service tor stop command, directly issue SIGTERM to the instance you wish to stop.

# service tor status instance-01
tor is running as pid 70918.
# kill -SIGTERM 70918

If you’d like to stop all instances in a graceful way at the same time:

# killall -SIGTERM tor

In a multi-instance setup, you can tell the service command which instance you want to control by appending the instance name (the portion after the @ symbol of the torrc file) at the end of the command. For example, to reload the config file for instance-01, issue the following command:

# service tor reload instance-01

If you want to reload the config file for all instances, simply remove the instance name from the above command. The rc script will issue the reload command across all instances.

If you’d like to look at an instance’s log file, you can use the tail command:

# tail -f /var/log/tor/instance-01/notices.log

Future Work

In the future, we would like to further harden our Tor setup by having each instance deployed in its own HardenedBSD jail. Once that is complete, we will document and publish the steps we took.