We’re back up after a 3 week outage

On Wednesday (10/3/2018) @ around 5:45pm Pacific Time we came back up after being down for about 3 weeks starting on 9/11/2018.

The initial reason we went down was because the Seattle Internet eXchange (SIX) administratively shut BGP access to our port because of a violation. We were transmitting DNS requests from an IP Address that we do not own and so the SIX took steps to protect the SIX fabric and shutdown our BGP access. That was an acceptable response and is consistant with the SIX rules. The root cause as to why we were transmitting DNS requests from an IP Address we do not own is still unknown. Due to the fact we don’t log firewall actions or collect netflow statistics makes it technically impossible for us to establish an empirical RCA with facts and numbers.

The contributing factor to the length of our downtime is directly related to the fact that we are a volunteer led and run organization. Nobody gets paid. And so we have to rely on our volunteers’ free time and good will. Also, we do not yet have an Out of Band (OOB) solution in place, meaning that both service and remote management of the environment were lost at the same time. This meant physically traveling to the data center, gathering current configs for analysis. We analyzed our current config and found no reason for this to happen. Beyond that, any other statements would be a pure guess without logs.

In response to the fact we have low resources, we are now soliciting additional volunteer help to add another BSD Network Admin to the team so that we aren’t relying on or putting stress on just one individual.

We are happy to be back up and running. We are currently reviewing our infrastructure architecture to find ways to reduce outages both of this kind and duration.

We apologize to those who have donated for running exit relays and are going to extend the lifetime of the relays by this downtime and any further potential downtime.

Edge routing with HardenedBSD

This work would not have been possible without the help of Shawn (from HardenedBSD), the determination of Jordan, Christian, Yawnbox, and the patience of the Seattle Internet Exchange.

History

Emerald Onion set out to maximize the use of free/libre open source software for our compute environment, especially for the routing infrastructure (routing is actually all that we do). Before our partnership with HardenedBSD, we chose pfSense because we were most experienced with it. We had considered OPNSense but decided to move directly to HardenedBSD.

As part of our ongoing research and development for the enhancement of privacy infrastructure, we’re actively working on identifying and testing open source hardware.

Why did we move from pfSense to HardenedBSD?

  • Misplaced ARPs in pfSense were occurring, for unknown reasons, and could not be mitigated
  • Overall software minimization and attack surface reduction
  • A “common operating environment” — to have the same OS and application stack as our Tor router
  • Flexibility, like being able to use the most up-to-date software packages and latest features
  • Overall greater diversity and security for the Tor network

HardenedBSD – base operating system configuration

For our current configuration, we are using stock HBSD 11-stable.

rc.conf
ipv6_gateway_enable="YES"
gateway_enable="YES"
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="eo-pf-01.emeraldonion.org"
unbound_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
openbgpd_enable="YES"
powerd_enable="YES"
vnstat_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"

ipv6_activate_all_interfaces="YES"
ifconfig_ix0="inet 23.129.64.1/24"
ifconfig_ix0_ipv6="inet6 2620:18c::1 prefixlen 36"

pf_enable="YES"
pf_rules="/usr/local/emeraldonion/etc/pf.rules"
pflog_enable="NO"

# Unused
ifconfig_igb0="down"
ifconfig_igb1="down"

# SIX
ifconfig_ix1="inet 206.81.81.158/23 group eo_egress"
ifconfig_ix1_ipv6="inet6 2001:504:16::6:cdb prefixlen 64 accept_rtadv"
sysctl.conf
# $FreeBSD$
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
security.bsd.stack_guard_page=1

# SIX requirement
net.link.ether.inet.max_age=14400

# Allow router solicitations whilst being a router
net.inet6.ip6.rfc6204w3=1
loader.conf
aesni_load="YES"
geom_eli_load="YES"
geli_nvd0p5_keyfile0_load="YES"
geli_nvd0p5_keyfile0_type="nvd0p5:geli_keyfile0"
geli_nvd0p5_keyfile0_name="/boot/encryption.key"
vfs.root.mountfrom="zfs:zroot/ROOT/default"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vfs.zfs.min_auto_ashift=12
zpool_cache_load="YES"
zpool_cache_type="/boot/zfs/zpool.cache"
zpool_cache_name="/boot/zfs/zpool.cache"
geom_eli_passphrase_prompt="YES"
zfs_load="YES"

PF – routing and firewall configuration

#########
#
# Emerald Onion 
#
#########
#
###### Variables and data used throughout this document:
#
### Dynamic tables
table <hardened_updates> persist
table <hardened_updates6> persist
table <ssh_permitted_hosts> { } persist
table <tor_instances> const { 23.129.64.0/24, !23.129.64.1 } persist
table <tor_instances6> const { 2620:18c::0/36, !2620:18c::1 } persist

# SIX BGP sources
h4_six_bgp="{ 206.81.80.0/23 }"
h6_six_bgp="{ 2001:504:16::0/64 }"
h4_six_bgp_nonat="206.81.80.0/23"
h6_six_bgp_nonat="2001:504:16::0/64"

# Private addresses
h_tor_relay="23.129.64.10"

### Ports of note
p_ssh = "22"
p_dns = "53"
p_hbsd_update = "{ 80 443 }"
p_tor = "{ 80 443 }"
p_ntp = "123"
p_bgp = "179"
p_internal_permitted = "{" $p_ntp $p_dns "}"

### Interfaces - Outside
i_six = "ix1"
# functionally equivalent to eo_egress interface group - need to collapse rules using this alias
i_wan = "{" $i_six "}"
a_wan = "23.129.64.1" 
a6_wan = "2620:18c::1"

### Interfaces - Inside
i_lan = "{" ix0 "}"

### Permissible ICMP
icmp_types = "{ echoreq unreach }"
icmp6_types = "{ toobig echoreq neighbrsol neighbradv }"
i_wan_icmp6_types="{ echoreq listendone routeradv neighbrsol neighbradv redir }"

#
###### Options section
# Do not process loopback traffic
set skip on lo0
set limit { states 200000, frags 40000, src-nodes 50000 }

# Scrub (normalize) packets to ensure consistent rule assignment
scrub in all

#
###### NAT/RDR
#
### NAT Rules
# Anything sent out by the host to a BGP peer should use the native address. Everything else should NAT to the advertised address
nat on $i_six inet from ($i_six) to ! $h4_six_bgp_nonat -> $a_wan
nat on $i_six inet6 from ($i_six) to ! $h6_six_bgp_nonat -> $a6_wan

#
### Port Forwarding (RDR)
# Remember your filter rules below! These rules only translate incoming traffic, the filter rules decide what is allowed....
# These two rules should be collapsed to use eo_egress...
rdr on $i_six inet proto tcp from any to $a_wan port $p_ssh -> $h_tor_relay port $p_ssh
rdr on $i_six inet proto tcp from any to $a_wan port = 2222 -> $a_wan port $p_ssh

#
###### Filter
#
# Strict default policy
block all

# Block spurious emissions
block quick on $i_six inet6 from 2620:18c::1/128 to any

# Permit DNS requests from anything - interface group eo_egress includes $i_wow and $i_six
pass out quick on eo_egress inet proto { tcp udp } to any port = $p_dns keep state 
pass out quick on eo_egress inet6 proto { tcp udp } to any port = $p_dns keep state

# Permit openBGPd
pass out quick on $i_six inet proto tcp from ($i_six) to $h4_six_bgp port = $p_bgp keep state
pass out quick on $i_six inet6 proto tcp from ($i_six) to $h6_six_bgp port = $p_bgp keep state
pass in quick on $i_six inet proto tcp from $h4_six_bgp to ($i_six) port = $p_bgp keep state
pass in quick on $i_six inet6 proto tcp from $h6_six_bgp to ($i_six) port = $p_bgp keep state

# Permit SSH from authorized...
pass in quick on eo_egress inet proto tcp from <ssh_permitted_hosts> to $h_tor_relay port $p_ssh keep state
pass in quick on eo_egress inet proto tcp from <ssh_permitted_hosts> to $a_wan port $p_ssh keep state
pass out quick on ix0 inet proto tcp from <ssh_permitted_hosts> to $h_tor_relay port $p_ssh keep state
# ...and block from anyone else
block in quick on eo_egress inet proto tcp from any to any port $p_ssh 
block in quick on eo_egress inet6 proto tcp from any to any port $p_ssh

# Permit anything to the Tor servers
pass in quick on eo_egress inet from any to <tor_instances> 
pass out quick on ix0 inet from any to <tor_instances> 
pass in quick on ix0 inet from <tor_instances> to !(ix0) 
pass out quick on eo_egress inet from <tor_instances> to any 
pass in quick on eo_egress inet6 from any to <tor_instances6> 
pass out quick on ix0 inet6 from any to <tor_instances6> 
pass in quick on ix0 inet6 from <tor_instances6> to !(ix0) 
pass out quick on eo_egress inet6 from <tor_instances6> to any

# Allow Tor traffic to transit the router but do not allow it towards our management network
#pass in quick on ix0 from ix0:network to !(ix0) 
#pass out quick on eo_egress from ix0:network to any 
## IPv6 address hasn't been configured on this interface yet
#pass in quick on ix0 inet6 from ix0:network to !(ix0) 
#pass out quick on $i_wan inet6 from ix0:network to any

## Permitted egress traffic
# Hardened BSD Updates
# Script provided to update <hardened_updates> and <hardened_updates6> tables on-demand although the addresses should be relatively stable
# Any host is allowed to these servers on 80/443
pass out quick on eo_egress inet proto tcp from $a_wan to <hardened_updates> port $p_hbsd_update keep state
#pass out quick on eo_egress inet6 proto tcp from $a_wan to <hardened_updates6> port $p_hbsd_update keep state

# Permit ICMP facilities
pass out quick on eo_egress inet proto icmp all icmp-type $icmp_types keep state
pass out quick on eo_egress inet6 proto ipv6-icmp all icmp6-type $icmp6_types keep state
pass in quick on eo_egress inet proto icmp all icmp-type $icmp_types keep state
pass in quick on eo_egress inet6 proto ipv6-icmp icmp6-type $icmp6_types keep state
pass in quick on $i_six inet6 proto ipv6-icmp from any to { ($i_six) ff02::1/16 } icmp6-type $i_wan_icmp6_types keep state
# And for internal services...
pass quick on $i_lan inet proto icmp all icmp-type $icmp_types keep state
pass quick on $i_lan inet6 proto ipv6-icmp all icmp6-type $icmp6_types keep state

# Permit DNS and NTP requests to us from our internal interfaces
pass in quick on $i_lan inet proto { tcp udp } from ix0:network to (ix0) port $p_internal_permitted keep state

# Protect the router from internal users - block everything else (not DNS or NTP requests)
block in quick on ix0 from ix0:network to (ix0)

Unbound – DNS configuration

#
# See unbound.conf(5) man page, version 1.7.0.
#

# The server clause sets the main parameters.
server:
# whitespace is not necessary, but looks cleaner.

# verbosity number, 0 is least verbose. 1 is default.
verbosity: 1

# number of threads to create. 1 disables threading.
num-threads: 16

# specify the interfaces to answer queries from by ip-address.
# The default is to listen to localhost (127.0.0.1 and ::1).
# specify 0.0.0.0 and ::0 to bind to all available interfaces.
# specify every interface[@port] on a new 'interface:' labelled line.
# The listen interfaces are not changed on reload, only on restart.
interface: 0.0.0.0

# control which clients are allowed to make (recursive) queries
# to this server. Specify classless netblocks with /size and action.
# By default everything is refused, except for localhost.
access-control: 23.129.64.1/24 allow

# file to read root hints from.
# get one from https://www.internic.net/domain/named.cache
root-hints: "/usr/local/etc/unbound/root.hints"

# enable to not answer id.server and hostname.bind queries.
hide-identity: yes

# enable to not answer version.server and version.bind queries.
hide-version: yes

# Sent minimum amount of information to upstream servers to enhance
# privacy. Only sent minimum required labels of the QNAME and set QTYPE
# to A when possible.
qname-minimisation: yes

# Use 0x20-encoded random bits in the query to foil spoof attempts.
# This feature is an experimental implementation of draft dns-0x20.
use-caps-for-id: yes

# if yes, Unbound rotates RRSet order in response.
rrset-roundrobin: yes

# Python config section. To enable:
# o use --with-pythonmodule to configure before compiling.
# o list python in the module-config string (above) to enable.
# o and give a python-script to run.
python:

# Remote control config section.
remote-control:

OpenBGPD – BGP configuration

AS 396507

fib-update yes
holdtime 90

router-id 206.81.81.158

# IPv4 network
network 23.129.64.0/24
# IPv6 network
network 2620:18C::/36

#### IPv4 neighbors ####
group "AS-SIXRSv4" {
remote-as 33108
neighbor 206.81.80.2 {
descr "SIXRS_rs2v4"
announce self
local-address 206.81.81.158
enforce neighbor-as no
max-prefix 200000
}
neighbor 206.81.80.3 {
descr "SIXRS_rs3v4"
announce self
local-address 206.81.81.158
enforce neighbor-as no
max-prefix 200000
}
}
group "AS-HURRICANE-Transit-v4" {
remote-as 6939
neighbor 206.81.80.40 {
descr "HE_transit_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 100000000
}
}
group "AS-ALTOPIAv4" {
remote-as 6456
neighbor 206.81.80.10 {
descr "ALT_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 20 restart 30
}
neighbor 206.81.81.41 {
descr "ALT_rs2v4"
announce self
local-address 206.81.81.158
max-prefix 20 restart 30
}
}
group "AS-POCKETINETv4" {
remote-as 23265
neighbor 206.81.80.88 {
descr "POK_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 600
}
}
group "AS-DOOFv4" {
remote-as 395823
neighbor 206.81.81.125 {
descr "DOOF_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 5
}
}
group "AS-PCHv4" {
remote-as 3856
neighbor 206.81.80.81 {
descr "PCH_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 600
}
}
group "AS-PCHWNv4" {
remote-as 42
neighbor 206.81.80.80 {
descr "PCHWN_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 600
}
}
group "AS-WOBv4" {
remote-as 64241
neighbor 206.81.81.87 {
descr "WOB_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 5
}
}
group "AS-MISAKAv4" {
remote-as 57695
neighbor 206.81.81.161 {
descr "MISAKA_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 300
}
}
group "AS-RISUPv4" {
remote-as 16652
neighbor 206.81.81.74 {
descr "RISUP_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 20
}
}
group "AS-CLDFLRv4" {
remote-as 13335
neighbor 206.81.81.10 {
descr "CLDFLR_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 1000
}
}
group "AS-GITHUBv4" {
remote-as 36459
neighbor 206.81.81.89 {
descr "GITHUB_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 100
}
neighbor 206.81.81.90 {
descr "GITHUB_rs2v4"
announce self
local-address 206.81.81.158
max-prefix 100
}
}
group "AS-YAHOOv4" {
remote-as 10310
neighbor 206.81.80.98 {
descr "YAHOO_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 2000
}
neighbor 206.81.81.50 {
descr "YAHOO_rs2v4"
announce self
local-address 206.81.81.158
max-prefix 2000
}
}
group "AS-SYMTECv4" {
remote-as 27471
neighbor 206.81.81.169 {
descr "SYMTEC_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 40
}
neighbor 206.81.81.170 {
descr "SYMTEC_rs2v4"
announce self
local-address 206.81.81.158
max-prefix 40
}
}
group "AS-AKAMAIv4" {
remote-as 20940
neighbor 206.81.80.113 {
descr "AKAMAI_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 200
}
}
group "AS-WAVEv4" {
remote-as 11404
neighbor 206.81.80.56 {
descr "WAVE_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 3000
}
}


group "AS-GOOGLEv4" {
remote-as 15169
neighbor 206.81.80.17 {
descr "GOOGLE_rs1v4"
announce self
local-address 206.81.81.158
max-prefix 15000
}
}


#### IPv6 neighbors ####
group "AS-SIXRSv6" {
remote-as 33108
neighbor 2001:504:16::2 {
descr "SIXRS_rs2v6"
announce self
local-address 2001:504:16::6:cdb
enforce neighbor-as no
max-prefix 60000
}
neighbor 2001:504:16::3 {
descr "SIXRS_rs3v6"
announce self
local-address 2001:504:16::6:cdb
enforce neighbor-as no
max-prefix 60000
}
}
group "AS-HURRICANE-Transit-v6" {
remote-as 6939
neighbor 2001:504:16::1b1b {
descr "HE_transit_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 100000000
}
}
group "AS-ALTOPIAv6" {
remote-as 6456
neighbor 2001:504:16::1938 {
descr "ALT_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 20 restart 30
}
neighbor 2001:504:16::297:0:1938 {
descr "ALT_rs2v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 20 restart 30
}
}
group "AS-POCKETINETv6" {
remote-as 23265
neighbor 2001:504:16::5ae1 {
descr "POK_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 600
}
}
group "AS-DOOFv6" {
remote-as 395823
neighbor 2001:504:16::6:a2f {
descr "DOOF_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 5
}
}
group "AS-PCHv6" {
remote-as 3856
neighbor 2001:504:16::f10 {
descr "PCH_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 600
}
}
group "AS-PCHWNv6" {
remote-as 42
neighbor 2001:504:16::2a {
descr "PCHWN_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 600
}
}
group "AS-WOBv6" {
remote-as 64241
neighbor 2001:504:16::faf1 {
descr "WOB_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 5
}
}
group "AS-MISAKAv6" {
remote-as 57695
neighbor 2001:504:16::e15f {
descr "MISAKA_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 300
}
}
group "AS-RISUPv6" {
remote-as 16652
neighbor 2001:504:16::410c {
descr "RISUP_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 10
}
}
group "AS-CLDFLRv6" {
remote-as 13335
neighbor 2001:504:16::3417 {
descr "CLDFLR_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 200
}
}
group "AS-GITHUBv6" {
remote-as 36459
neighbor 2001:504:16::8e6b {
descr "GITHUB_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 20
}
neighbor 2001:504:16::346:0:8e6b {
descr "GITHUB_rs2v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 20
}
}
group "AS-YAHOOv6" {
remote-as 10310
neighbor 2001:504:16::2846 {
descr "YAHOO_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 200
}
neighbor 2001:504:16::306:0:2846 {
descr "YAHOO_rs2v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 200
}
}
group "AS-AKAMAIv6" {
remote-as 20940
neighbor 2001:504:16::51cc {
descr "AKAMAI_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 40
}
}
group "AS-WAVEv6" {
remote-as 11404
neighbor 2001:504:16::2c8c {
descr "WAVE_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 250
}
}


group "AS-GOOGLEv6" {
remote-as 15169
neighbor 2001:504:16::3b41 {
descr "GOOGLE_rs1v6"
announce self
local-address 2001:504:16::6:cdb
max-prefix 1000
}
}

#### Filtering Rules ####

deny from any
deny to any

# https://www.arin.net/announcements/2014/20140130.html
# This block will be subject to a minimum size allocation of /28 and a
# maximum size allocation of /24. ARIN should use sparse allocation when
# possible within that /10 block.
allow from any prefix 23.128.0.0/10 prefixlen 24 - 28 # ARIN IPv6 transition

## IPv4 ##
# SIXRS_rs2v4
allow from 206.81.80.2 inet prefixlen 8 - 24
allow to 206.81.80.2 inet prefixlen 8 - 24
# SIXRS_rs3v4
allow from 206.81.80.3 inet prefixlen 8 - 24
allow to 206.81.80.3 inet prefixlen 8 - 24
# HE_transit_rs1v4
allow from 206.81.80.40
allow to 206.81.80.40
# ALT_rs1v4
allow from 206.81.80.10 inet prefixlen 8 - 24
allow to 206.81.80.10 inet prefixlen 8 - 24
# ALT_rs2v4
allow from 206.81.81.41 inet prefixlen 8 - 24
allow to 206.81.81.41 inet prefixlen 8 - 24
# POK_rs1v4
allow from 206.81.80.88 inet prefixlen 8 - 24
allow to 206.81.80.88 inet prefixlen 8 - 24
# DOOF_rs1v4
allow from 206.81.81.125 inet prefixlen 8 - 24
allow to 206.81.81.125 inet prefixlen 8 - 24
# PCH_rs1v4
allow from 206.81.80.81 inet prefixlen 8 - 24
allow to 206.81.80.81 inet prefixlen 8 - 24
# PCHWN_rs1v4
allow from 206.81.80.80 inet prefixlen 8 - 24
allow to 206.81.80.80 inet prefixlen 8 - 24
# WOB_rs1v4
allow from 206.81.81.87 inet prefixlen 8 - 24
allow to 206.81.81.87 inet prefixlen 8 - 24
# MISAKA_rs1v4
allow from 206.81.81.161 inet prefixlen 8 - 24
allow to 206.81.81.161 inet prefixlen 8 - 24
# RISUP_rs1v4
allow from 206.81.81.74 inet prefixlen 8 - 24
allow to 206.81.81.74 inet prefixlen 8 - 24
# CLDFLR_rs1v4
allow from 206.81.81.10 inet prefixlen 8 - 24
allow to 206.81.81.10 inet prefixlen 8 - 24
# GITHUB_rs1v4
allow from 206.81.81.89 inet prefixlen 8 - 24
allow to 206.81.81.89 inet prefixlen 8 - 24
# GITHUB_rs2v4
allow from 206.81.81.90 inet prefixlen 8 - 24
allow to 206.81.81.90 inet prefixlen 8 - 24
# YAHOO_rs1v4
allow from 206.81.80.98 inet prefixlen 8 - 24
allow to 206.81.80.98 inet prefixlen 8 - 24
# YAHOO_rs2v4
allow from 206.81.81.50 inet prefixlen 8 - 24
allow to 206.81.81.50 inet prefixlen 8 - 24
# SYMTEC_rs1v4
allow from 206.81.81.169 inet prefixlen 8 - 24
allow to 206.81.81.169 inet prefixlen 8 - 24
# SYMTEC_rs2v4
allow from 206.81.81.170 inet prefixlen 8 - 24
allow to 206.81.81.170 inet prefixlen 8 - 24
# AKAMAI_rs1v4
allow from 206.81.80.113 inet prefixlen 8 - 24
allow to 206.81.80.113 inet prefixlen 8 - 24
# WAVE_rs1v4
allow from 206.81.80.56 inet prefixlen 8 - 24
allow to 206.81.80.56 inet prefixlen 8 - 24
# GOOG_rs1v4
allow from 206.81.80.17
allow to 206.81.80.17

## IPv6 ##
# SIXRS_rs2v6
allow from 2001:504:16::2 inet6 prefixlen 16 - 48
allow to 2001:504:16::2 inet6 prefixlen 16 - 48
# SIXRS_rs3v6
allow from 2001:504:16::3 inet6 prefixlen 16 - 48
allow to 2001:504:16::3 inet6 prefixlen 16 - 48
# HE_transit_rs1v6
allow from 2001:504:16::1b1b
allow to 2001:504:16::1b1b
# ALT_rs1v6
allow from 2001:504:16::1938 inet6 prefixlen 16 - 48
allow to 2001:504:16::1938 inet6 prefixlen 16 - 48
# ALT_rs2v6
allow from 2001:504:16::297:0:1938 inet6 prefixlen 16 - 48
allow to 2001:504:16::297:0:1938 inet6 prefixlen 16 - 48
# POK_rs1v6
allow from 2001:504:16::5ae1 inet6 prefixlen 16 - 48
allow to 2001:504:16::5ae1 inet6 prefixlen 16 - 48
# DOOF_rs1v6
allow from 2001:504:16::6:a2f inet6 prefixlen 16 - 48
allow to 2001:504:16::6:a2f inet6 prefixlen 16 - 48
# PCH_rs1v6
allow from 2001:504:16::f10 inet6 prefixlen 16 - 48
allow to 2001:504:16::f10 inet6 prefixlen 16 - 48
# PCHWN_rs1v6
allow from 2001:504:16::2a inet6 prefixlen 16 - 48
allow to 2001:504:16::2a inet6 prefixlen 16 - 48
# WOB_rs1v6
allow from 2001:504:16::faf1 inet6 prefixlen 16 - 48
allow to 2001:504:16::faf1 inet6 prefixlen 16 - 48
# MISAKA_rs1v6
allow from 2001:504:16::e15f inet6 prefixlen 16 - 48
allow to 2001:504:16::e15f inet6 prefixlen 16 - 48
# RISUP_rs1v6
allow from 2001:504:16::410c inet6 prefixlen 16 - 48
allow to 2001:504:16::410c inet6 prefixlen 16 - 48
# CLDFLR_rs1v6
allow from 2001:504:16::3417 inet6 prefixlen 16 - 48
allow to 2001:504:16::3417 inet6 prefixlen 16 - 48
# GITHUB_rs1v6
allow from 2001:504:16::8e6b inet6 prefixlen 16 - 48
allow to 2001:504:16::8e6b inet6 prefixlen 16 - 48
# GITHUB_rs2v6
allow from 2001:504:16::346:0:8e6b inet6 prefixlen 16 - 48
allow to 2001:504:16::346:0:8e6b inet6 prefixlen 16 - 48
# YAHOO_rs1v6
allow from 2001:504:16::2846 inet6 prefixlen 16 - 48
allow to 2001:504:16::2846 inet6 prefixlen 16 - 48
# YAHOO_rs2v6
allow from 2001:504:16::306:0:2846 inet6 prefixlen 16 - 48
allow to 2001:504:16::306:0:2846 inet6 prefixlen 16 - 48
# AKAMAI_rs1v6
allow from 2001:504:16::51cc inet6 prefixlen 16 - 48
allow to 2001:504:16::51cc inet6 prefixlen 16 - 48
# WAVE_rs1v6
allow from 2001:504:16::2c8c inet6 prefixlen 16 - 48
allow to 2001:504:16::2c8c inet6 prefixlen 16 - 48
# GOOG_rs1v6
allow from 2001:504:16::3b41
allow to 2001:504:16::3b41

# filter bogus networks according to RFC5735
deny from any prefix 0.0.0.0/8 prefixlen >= 8 # 'this' network [RFC1122]
deny from any prefix 10.0.0.0/8 prefixlen >= 8 # private space [RFC1918]
deny from any prefix 100.64.0.0/10 prefixlen >= 10 # CGN Shared [RFC6598]
deny from any prefix 127.0.0.0/8 prefixlen >= 8 # localhost [RFC1122]
deny from any prefix 169.254.0.0/16 prefixlen >= 16 # link local [RFC3927]
deny from any prefix 172.16.0.0/12 prefixlen >= 12 # private space [RFC1918]
deny from any prefix 192.0.2.0/24 prefixlen >= 24 # TEST-NET-1 [RFC5737]
deny from any prefix 192.168.0.0/16 prefixlen >= 16 # private space [RFC1918]
deny from any prefix 198.18.0.0/15 prefixlen >= 15 # benchmarking [RFC2544]
deny from any prefix 198.51.100.0/24 prefixlen >= 24 # TEST-NET-2 [RFC5737]
deny from any prefix 203.0.113.0/24 prefixlen >= 24 # TEST-NET-3 [RFC5737]
deny from any prefix 224.0.0.0/4 prefixlen >= 4 # multicast
deny from any prefix 240.0.0.0/4 prefixlen >= 4 # reserved

# filter bogus IPv6 networks according to IANA
deny from any prefix ::/8 prefixlen >= 8
deny from any prefix 0100::/64 prefixlen >= 64 # Discard-Only [RFC6666]
deny from any prefix 2001:2::/48 prefixlen >= 48 # BMWG [RFC5180]
deny from any prefix 2001:10::/28 prefixlen >= 28 # ORCHID [RFC4843]
deny from any prefix 2001:db8::/32 prefixlen >= 32 # docu range [RFC3849]
deny from any prefix 3ffe::/16 prefixlen >= 16 # old 6bone
deny from any prefix fc00::/7 prefixlen >= 7 # unique local unicast
deny from any prefix fe80::/10 prefixlen >= 10 # link local unicast
deny from any prefix fec0::/10 prefixlen >= 10 # old site local unicast
deny from any prefix ff00::/8 prefixlen >= 8 # multicast

vnStat – statistics gathering configuration

# vnStat 1.15 config file
##

# default interface
Interface "eth0"

# location of the database directory
DatabaseDir "/var/lib/vnstat"

# locale (LC_ALL) ("-" = use system locale)
Locale "-"

# on which day should months change
MonthRotate 1

# date output formats for -d, -m, -t and -w
# see 'man date' for control codes
DayFormat "%x"
MonthFormat "%b '%y"
TopFormat "%x"

# characters used for visuals
RXCharacter "%"
TXCharacter ":"
RXHourCharacter "r"
TXHourCharacter "t"

# how units are prefixed when traffic is shown
# 0 = IEC standard prefixes (KiB/MiB/GiB/TiB)
# 1 = old style binary prefixes (KB/MB/GB/TB)
UnitMode 0

# output style
# 0 = minimal & narrow, 1 = bar column visible
# 2 = same as 1 except rate in summary and weekly
# 3 = rate column visible
OutputStyle 3

# used rate unit (0 = bytes, 1 = bits)
RateUnit 1

# try to detect interface maximum bandwidth, 0 = disable feature
# MaxBandwidth will be used as fallback value when enabled
BandwidthDetection 1

# maximum bandwidth (Mbit) for all interfaces, 0 = disable feature
# (unless interface specific limit is given)
MaxBandwidth 1000

# interface specific limits
# example 8Mbit limit for 'ethnone':
MaxBWethnone 8

# how many seconds should sampling for -tr take by default
Sampletime 5

# default query mode
# 0 = normal, 1 = days, 2 = months, 3 = top10
# 4 = exportdb, 5 = short, 6 = weeks, 7 = hours
QueryMode 0

# filesystem disk space check (1 = enabled, 0 = disabled)
CheckDiskSpace 1

# database file locking (1 = enabled, 0 = disabled)
UseFileLocking 1

# how much the boot time can variate between updates (seconds)
BootVariation 15

# log days without traffic to daily list (1 = enabled, 0 = disabled)
TrafficlessDays 1


# vnstatd
##

# switch to given user when started as root (leave empty to disable)
DaemonUser ""

# switch to given user when started as root (leave empty to disable)
DaemonGroup ""

# how often (in seconds) interface data is updated
UpdateInterval 30

# how often (in seconds) interface status changes are checked
PollInterval 5

# how often (in minutes) data is saved to file
SaveInterval 5

# how often (in minutes) data is saved when all interface are offline
OfflineSaveInterval 30

# how often (in minutes) bandwidth detection is redone when
# BandwidthDetection is enabled (0 = disabled)
BandwidthDetectionInterval 5

# force data save when interface status changes (1 = enabled, 0 = disabled)
SaveOnStatusChange 1

# enable / disable logging (0 = disabled, 1 = logfile, 2 = syslog)
UseLogging 2

# create dirs if needed (1 = enabled, 0 = disabled)
CreateDirs 1

# update ownership of files if needed (1 = enabled, 0 = disabled)
UpdateFileOwner 1

# file used for logging if UseLogging is set to 1
LogFile "/var/log/vnstat/vnstat.log"

# file used as daemon pid / lock file
PidFile "/var/run/vnstat/vnstat.pid"


# vnstati
##

# title timestamp format
HeaderFormat "%x %H:%M"

# show hours with rate (1 = enabled, 0 = disabled)
HourlyRate 1

# show rate in summary (1 = enabled, 0 = disabled)
SummaryRate 1

# layout of summary (1 = with monthly, 0 = without monthly)
SummaryLayout 1

# transparent background (1 = enabled, 0 = disabled)
TransparentBg 0

# image colors
CBackground "FFFFFF"
CEdge "AEAEAE"
CHeader "606060"
CHeaderTitle "FFFFFF"
CHeaderDate "FFFFFF"
CText "000000"
CLine "B0B0B0"
CLineL "-"
CRx "92CF00"
CTx "606060"
CRxD "-"
CTxD "-"

OpenNTPD – network time protocol for system clock synchronization

# Upstream Servers
servers pool.ntp.org

 

Tor as infrastructure: challenges and opportunities

Infrastructure as a Service (IaaS) is commonly used within enterprise environments whereby organizations pay for someone else’s physical or virtual resources, including networking resources. Individuals also pay for IaaS when, for example, they need to deploy a VPS or VPN for personal use. From Gartner:

Infrastructure as a service (IaaS) is a standardized, highly automated offering, where compute resources, complemented by storage and networking capabilities are owned and hosted by a service provider and offered to customers on-demand.

 

Tor, the network of operators that make up the network, is IaaS. What is unusual about this terminology is that the Tor network is distributed, and utilization of this networking resource is free. Companies like Duck Duck Go, Facebook, New York Times, and every organization utilizing SecureDrop including the Associated Press, is using this IaaS in order to best support their users’ privacy rights.

A problem that Tor network operators have always faced is that setting up and maintaining the network is not free. Tor is free-to-use IaaS on purpose — people and services need to be able to use the network without attribution in order for Tor to provide specific guarantees of privacy.

If privacy infrastructure operators had better funding, we would be in a better position to support larger infrastructure needs. For example, if Mozilla or Brave ever wanted to collect browser telemetry over Tor onion services to best support the privacy of their users, the Tor network would likely need to pivot towards larger and more stable network operators. In this article, we will look at some of the challenges and opportunities for operators of privacy infrastructure.

Challenges

Emerald Onion was created, in part, to be able to demonstrate a successful Transit Internet Service Provider that only supports privacy IaaS. It was designed to best leverage existing laws in the United States in tandem with operational designs that require privacy-focused security properties. There have been and continue to be serious challenges facing Emerald Onion along with any other organization that is dedicated to privacy IaaS.

IP transit service is exclusively a for-profit business

IP transit is a required part of an ISP. Emerald Onion, Riseup, Calyx, Quintex, etc, need to pay upstream providers who provide the physical transport of the encrypted packets that we transit as part of the Tor network. This transit service is expensive. For example, 1Gbps of service in a residential setting can cost around $80 per month in Seattle, and 1Gbps of service in a datacenter can easily cost $800 per month. This dramatic cost difference is because of capitalism — it is presumed that a service provider in a datacenter environment is going to be profiting off of this service. Upstream providers don’t care one bit that Emerald Onion is a 501(c)3 not-for-profit supporting human rights.

Little options for trustworthy, open source hardware, particularly networking equipment

Emerald Onion is using general-purpose computing devices (currently low-power Intel Xeon D) with BSD operating systems. It is a priority for us to be using trustworthy compute infrastructure, so we are at least ensuring that the kernels and applications that we use are free/libre open source software. We hope to transition to free/libre open source hardware and firmware as soon as we can, but we also have to be concerned with compatibility and stability with HardenedBSD/FreeBSD, and the cost of this hardware. We know that options exist for free/libre open source hardware, but this is still a very new and maturing market. To further complicate this prioritized need for trustworthy compute infrastructure, Emerald Onion has particular interest in 10Gbps networking for both the LAN and WAN.

One day, we’d like to be able to support 40Gbps and 100Gbps; however, we are not aware of any free/libre open source hardware and firmware that supports 40Gbps or 100Gbps networking.

High cost of network redundancy

Our proof-of-concept work has focused on low-cost options. This means we do not currently have redundancy at our LAN or WAN layers. Network redundancy for Emerald Onion, at minimum, would entail having not just one expensive IP transit link, but two, ideally from different upstream providers, which means two edge routers and two links to each of our Internet Exchange Points. This would also mean that we have to add redundant LAN switching, and all of this means increasing our rack space and power requirements. Basically, we would have to more than double our recurring costs to be able to have this level of infrastructure stability. While Tor itself is highly resistant to network changes, the more capacity that Emerald Onion, and other large Tor operators support, the greater the negative impact we would impose on the Tor network whenever we have to perform hardware, firmware, kernel, and application updates.

IPv4 scopes for exit operators

As an exit relay operator, Emerald Onion must own and operate our own IPv4 address space for efficiently handling abuse communications from other service providers and law enforcement. Additionally, relay operators who peer directly with other service providers in Internet Exchange Points (IXPs) who have their own Autonomous System Number (ASNs) also require their own IP space. The entire world ran out of IPv4 address scopes to hand out to new and existing service providers a few years ago, and this is a blocker for any new Tor operator who is working to achieve the same level of stability that Emerald Onion is working to achieve.

Tor exit relaying currently depends in IPv4 connectivity between Tor routers (middle relay to exit relay traffic, for example). To be given an exit flag, a static IPv4 address is ideal for the Tor network (dynamic IP addresses would require a few hours delay of client discovery in the consensus).

Tor exit operators would not need their own Regional Internet Registry (RIR) -provisioned IPv4 address space if the exit flag could be given to IPv6-only operators, but this is not currently possible. ORPort connections (inter-tor circuit connections from middle relays, for example) cannot usually generate abuse, so IPv4 scope leasing is an easier option if IPv6-only exiting was a possibility.

One idea that Emerald Onion has had is that it may be possible to make proposals to large organizations, including universities, that are sitting on very large IPv4 scopes. We think that these organizations might be willing to donate small (/24) scopes to not-for-profit Tor network operators.

Opportunities

Surveillance and latency minimization

Seattle is home to a very large telecommunications hub called the Westin Building Exchange (WBE). We know that this building has National Security Agency (NSA) taps on I/O connections that are likely to facilitate traffic to regions like China and Russia. Additionally, WBE hosts several of the Internet’s DNS Root Servers, several of which are part of the Seattle Internet Exchange (SIX).

Emerald Onion went through the process of securing our own ASN, IPv6, and IPv4 scopes from American Registry of Internet Numbers (ARIN). We needed these things to connect to the SIX. Connecting to the SIX means that we are physically and directly connected to as many as 280 other service providers. We made this a priority because direct peering, using Border Gateway Protocol (BGP), minimizes the amount of clear-net switches and routers that a Tor user’s exit traffic has to travel through to reach its final destination. Every switch or router that Tor traffic has to traverse is an opportunity for surveillance and adds latency.

This strategy for Tor exit router placement is also ideal considering DNS. Being that multiple DNS Root Servers are directly peered with Emerald Onion, this further minimizes a global persistent adversary’s ability to spy on what Tor users are doing.

Statistically, due to requirements in the Tor protocol, individual Tor circuits bounce around multiple countries before they exit the network. This means that a non-trivial amount of the traffic that Emerald Onion, and any other United States exit operator facilitates, comes from a middle relay not within the United States. In tandem, generally speaking, a non-trivial amount of Tor exit traffic is destined to American services like Akamai, Cloudflare, Facebook, Google, and DNS Root Servers. These two likelihoods, together, means the following:

Tor exit traffic that is destined to service providers in the United States is best served, in terms of surveillance and latency minimization, by Tor exit operators that have exit relays connected to IXPs in datacenters along the coasts of the United States where undersea cables physically terminate, presuming that popular service providers like Akamai, Cloudflare, Facebook, Google, and DNS Root Servers are direct peers. This, in theory, minimizes the opportunity for American-sponsored traffic analysis, data retention, and surveillance, in addition to any other global persistent adversary who may have compromised network equipment within IXPs.

Emerald Onion has already compiled an initial list of IXPs around the United States. We continue to work on a list of qualities that an IXP should have that is a ideal for a Tor network operator:

  • Number of participants — This is important because if there is a peering link (using BGP) with another service provider, the opportunities for surveillance are minimized. For obvious reasons, the amount of direct peers is as important as the popularity of said services.
  • Access to specific participants — This is important because, for example, peering agreements with large providers such as Akamai, Cloudflare, Facebook, Google, and DNS Root Servers minimize the opportunity for surveillance while minimizing latency.
  • Nonprofit and affordable — A large number of IXPs are for-profit and thus have high up-front and high recurring costs for connectivity, in addition to setup fees and recurring fees for copper or fiber maintenance.
  • Geo-location — This is important because of, at least, location diversity, peer diversity, and direct connections with international undersea cables are focal points for the facilitators of global passive surveillance.
  • Prohibits network surveillance — The Seattle Internet Exchange, for example, has a stated policy that prohibits surveillance on peering links. One day, we hope that the SIX, and other public-benefit IXPs, will also publish a regular transparency report.

Funding

Emerald Onion has been in operation for 10 months. We wouldn’t exist, as we are today, without the generous startup grant of $5,000 from Tor Servers. We also would not still be around without the continuous donations from our Directors who personally donate as much as $350 each month. We currently require roughly $700 per month to operate, largely due to our service contract with our co-location provider who is also our upstream transit ISP.

Going back to the beginning of this article, the Tor network is a privacy-focused IaaS. Sustainability is a constant issue for Tor network operators, especially for operators who preemptively tackle legal and long-term operational challenges. We need help. There is no easy answer for funding. Grant writing and grant management is not a trivial task, nor is sustaining a 501(c)3 not-for-profit purely based on part-time volunteer work. Emerald Onion is incredibly lucky to have a few people who regularly donate large amounts of money and time to keep the organization online, but this is not sustainable.

The operations model that Emerald Onion has created, however, is scalable if properly funded. If we were provided between between $7,000 to $10,000 per month, we could multiply our capacity by a factor of 10. If we had a pool of funding that supported 10 independent Tor network operators in the United States (there are over 100 IXPs in the United States), we could dramatically bolster the capacity and stability of the Tor network while also minimizing network surveillance opportunities and network latency.

Conclusion

I hope that this article begins to shed light on the challenges facing privacy IaaS providers like the thousands of operators that make up the Tor network. Emerald Onion is going to continue to educate others on these topics, attempt to find and create solutions for these challenges, and continue to encourage hacker communities around the United States to build their own privacy-focused not-for-profit ISP.

DNS for Tor Exit Relaying

One of the major pieces of infrastructure run by Tor Exit Nodes is DNS. DNS is the system that translates human readable names, like emeraldonion.org, into IP addresses. In Tor, the exit node is the location where this translation takes place. As such, DNS has been recognized as one of the places where centralization or attacks could be performed that could affect the integrity of the Tor network. To serve our users well, we want to mitigate the risks of compromise and surveillance as we resolve names on behalf of our users. It’s these principles that direct how we structure our DNS resolution.

 

Emerald Onion currently uses pfSense, which uses Unbound for DNS. Per our architectural design, we run our own recursive DNS server, meaning we query up to the root name servers for DNS resolution, and avoid the cache any upstream ISPs offering us DNS resolvers. This also means we query the authoritative resolvers directly, minimizing the number of additional parties able to observe domain resolutions coming from our users.

General Settings:

  • We use DNS Resolver and disable the DNS Forwarder
  • Only bind the DNS listener to the NIC the Tor server is connected to and localhost.
  • Only bind the DNS outgoing interface to the NIC that carries our public IP. If you use BGP, do NOT bind DNS to the interface used to connect to BGP peers.
  • Enable DNSSEC support
  • Disable DNS Query Forwarding
  • We don’t use DHCP, leave DHCP Registration disabled
  • Same goes with Static DHCP

Custom options:

prefer-ip6: yes
hide-trustanchor: yes
harden-large-queries: yes
harden-algo-downgrade: yes
qname-minimisation-strict: yes
ignore-cd-flag: yes

The most important of these options is qname-minimization, which means that when we perform a resolution like www.emeraldonion.org, we ask the root name servers only for who controls .org, the Org name servers, who controls emeraldonion.org, and only ask emerald onion’s name servers for the IP of www.emeraldonion.org. This helps to protect against our traffic resolutions being swept into the various “passive DNS” feeds that have been commoditized around the network.

Of the other custom options, the bulk are related to DNSSEC security.

 

Advanced Settings:

  • Hide Identity
  • Hide Version
  • Use Prefetch Support
  • Use Prefetch DNS Keys
  • Harden DNSSEC data
  • Leave the tuning values alone for now (Things like cache size, buffers, queues, jostle, etc)
  • Log Level is 1, which is pretty low.
  • Leave the rest alone.

Hiding the identity and version helps prevent the leakage of information that could be used in attacks against us. Prefetch Support changes how the DNS server fetches DNS records. Without it, it fetches the DNS record at the time of the request. With Prefetch Support, it refreshes the DNS entries as each record’s TTL expires helping to further obfuscate requests and makes it harder for specific Tor request correlation attacks.

Access Lists:

We don’t use this, but if you want to work with the Access Lists, that should be fine, just keep it in mind when troubleshooting.

DNSSEC is now fully implemented for our forward and reverse lookup zones

Last month (July 2017) we moved our DNS zone management to the Google Cloud Platform since our domains were already registered with Google. After applying for the DNSSEC alpha, we were granted access and turned on DNSSEC for all three of our forward (domain) and reverse (IPv6 and IPv4 scopes) lookup zones. Google’s alpha products come with no SLA, so we took a risk implementing DNSSEC through Google.

Turning on DNSSEC was as easy flipping a switch in the control panel. The last part is adding the DS entries at the Registrar.

In the upper-right hand corner of Zone Details is Registrar Setup. This is where we got our DS entry information.

This DS information translates to a specific Key Tag, Algorithm, Digest Type, and Digest that needs to go into Google Domains (the actual Registrar).

This completed the domain setup. Now we needed to configure DNSSEC for our reverse lookup zones. Because they are direct allocations from ARIN, we needed to copy over the DS details over to ARIN.

View and Manage Your Networks > View & Manage Network (for both our IPv6 and IPv4 scopes) > Actions > Manage Reverse DNS > (select the delegation) > Modify DS Records

String (for our IPv6) parsed:

3600 DS 46756 8 2 5396635C919BAF34F24011FAB2DE251630AE2B8C17F1B69D05BCFDD603510014

String (for our IPv4) parsed:

3600 DS 40286 8 2 54686118794BD67CC76295F3D7F1C269D70EB5646F5DA130CC590AE14B33935F

This completed the ARIN DNSSEC configuration. While Google provided a quick DNS update for validation, ARIN took over 12 hours.