We’re back after a 6.5 day outage

Today (8/1/2017) @ 23:22 Pacific Time, we came back online after being down for 6 days and 12 hours. Our previous configuration where we had two physically separate systems (1 x pfSense router and 1 x Tor router) is gone. The server that was running the Tor router started to experience hardware errors, as reported by kern.log. These errors were traced back to the system board, which eventually caused issues with the disk.

While all of this was happening, we were also down an admin as he was out at defcon. So, juggling that, wanting to restore service and limited funds because to replace the Tor router, we would’ve had to wait for our refund check from the RMA before buying a new system board, we decided to virtualize our infrastructure.

We are now operating on a single server (12 x Core Intel + 32GB RAM) with 1 x pfSense 2.3.4 VM and 1 x Tor Ubuntu 16.04 VM. The system is up and passing Tor traffic.

Tor router configuration v3

01 August 2017

We experienced a catastrophic hardware failure recently which will be detailed in an upcoming blog post. We are back online today with new router IDs and we added two more routers for a total of six Tor routers.

We moved to Google Cloud DNS recently to be able to manage our PTR records for reverse DNS since we have our own IP scopes now. We also moved our forward-lookup zone to Google Cloud DNS. Next on the agenda is setting up DNSSEC.

IPv6 PTR

1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.4.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.5.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.6.1.0.0.0.0.c.8.1.0.0.2.6.2.ip6.arpa.

DNS

2620:18c:0:1100::1 tor01.emeraldonion.org
2620:18c:0:1200::1 tor02.emeraldonion.org
2620:18c:0:1300::1 tor03.emeraldonion.org
2620:18c:0:1400::1 tor04.emeraldonion.org
2620:18c:0:1500::1 tor05.emeraldonion.org
2620:18c:0:1600::1 tor06.emeraldonion.org
23.129.64.11 tor01.emeraldonion.org
23.129.64.12 tor02.emeraldonion.org
23.129.64.13 tor03.emeraldonion.org
23.129.64.14 tor04.emeraldonion.org
23.129.64.15 tor05.emeraldonion.org
23.129.64.16 tor06.emeraldonion.org

Tor router #1

Nickname EmeraldOnion01
Address tor01.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.11
OutboundBindAddressOR 23.129.64.11
DirPort 23.129.64.11:80
ORPort 23.129.64.11:443
ORPort [2620:18c:0:1100::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #2

Nickname EmeraldOnion02
Address tor02.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.12
OutboundBindAddressOR 23.129.64.12
DirPort 23.129.64.12:80
ORPort 23.129.64.12:443
ORPort [2620:18c:0:1200::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #3

Nickname EmeraldOnion03
Address tor03.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.13
OutboundBindAddressOR 23.129.64.13
DirPort 23.129.64.13:80
ORPort 23.129.64.13:443
ORPort [2620:18c:0:1300::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #4

Nickname EmeraldOnion04
Address tor04.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.14
OutboundBindAddressOR 23.129.64.14
DirPort 23.129.64.14:80
ORPort 23.129.64.14:443
ORPort [2620:18c:0:1400::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #5

Nickname EmeraldOnion05
Address tor05.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.15
OutboundBindAddressOR 23.129.64.15
DirPort 23.129.64.15:80
ORPort 23.129.64.15:443
ORPort [2620:18c:0:1500::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #6

Nickname EmeraldOnion06
Address tor06.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.16
OutboundBindAddressOR 23.129.64.16
DirPort 23.129.64.16:80
ORPort 23.129.64.16:443
ORPort [2620:18c:0:1600::1]:443
RelayBandwidthRate 18 MBytes
RelayBandwidthBurst 18 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Starting the processes

sudo service tor@tor01 start
sudo service tor@tor02 start
sudo service tor@tor03 start
sudo service tor@tor04 start
sudo service tor@tor05 start
sudo service tor@tor06 start

Tor router configuration v2

22 July 2017

We are rearchitecting our network by eliminating the use of our ISP-provisioned /27 IP scope in order to utilize our ARIN-assigned /24. Doing so allows us to route across multiple networks with the same ASN, a requirement in order to use our ARIN-assigned IPv6 scope. For network simplification, we are also eliminating the use of NAT.

Tor router #1

Nickname EmeraldOnion01
Address tor01.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.11
OutboundBindAddressOR 23.129.64.11
DirPort 23.129.64.11:80
ORPort 23.129.64.11:443
ORPort [2620:18c:0:1100::1]:443
RelayBandwidthRate 27.5 MBytes
RelayBandwidthBurst 27.5 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #2

Nickname EmeraldOnion02
Address tor02.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.12
OutboundBindAddressOR 23.129.64.12
DirPort 23.129.64.12:80
ORPort 23.129.64.12:443
ORPort [2620:18c:0:1200::1]:443
RelayBandwidthRate 27.5 MBytes
RelayBandwidthBurst 27.5 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #3

Nickname EmeraldOnion03
Address tor03.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.13
OutboundBindAddressOR 23.129.64.13
DirPort 23.129.64.13:80
ORPort 23.129.64.13:443
ORPort [2620:18c:0:1300::1]:443
RelayBandwidthRate 27.5 MBytes
RelayBandwidthBurst 27.5 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router #4

Nickname EmeraldOnion04
Address tor04.emeraldonion.org
ContactInfo abuse_at_emeraldonion_dot_org
OutboundBindAddressExit 23.129.64.14
OutboundBindAddressOR 23.129.64.14
DirPort 23.129.64.14:80
ORPort 23.129.64.14:443
ORPort [2620:18c:0:1400::1]:443
RelayBandwidthRate 27.5 MBytes
RelayBandwidthBurst 27.5 MBytes
IPv6Exit 1
Exitpolicy accept *:*
ExitPolicy accept6 *:*

Tor router configuration v1

2 July 2017

We have started four Tor exit routers to saturate our unmetered-1Gbps, 10Gbps-burstable link.

DNS

216.176.186.131 tor01.emeraldonion.org
216.176.186.132 tor02.emeraldonion.org
216.176.186.133 tor03.emeraldonion.org
216.176.186.134 tor04.emeraldonion.org

Create instances

sudo tor-instance-create tor01
sudo tor-instance-create tor02
sudo tor-instance-create tor03
sudo tor-instance-create tor04

Tor router #1

sudo vim /etc/tor/instances/tor01/torrc
Nickname EmeraldOnion01
Address tor01.emeraldonion.org
ContactInfo abuse@emeraldonion.org
OutboundBindAddressExit 10.10.10.101
OutboundBindAddressOR 10.10.10.101
DirPort 216.176.186.131:80 NoListen
DirPort 10.10.10.101:80 NoAdvertise
ORPort 216.176.186.131:443 NoListen
ORPort 10.10.10.101:443 NoAdvertise
#ORPort [2620:18c:0:100::1]:443
RelayBandwidthRate 50 MBytes
RelayBandwidthBurst 50 MBytes
#IPv6Exit 1
Exitpolicy accept *:*
#ExitPolicy accept6 *:*

Tor router #2

sudo vim /etc/tor/instances/tor02/torrc
Nickname EmeraldOnion02
Address tor02.emeraldonion.org
ContactInfo abuse@emeraldonion.org
OutboundBindAddressExit 10.10.10.102
OutboundBindAddressOR 10.10.10.102
DirPort 216.176.186.132:80 NoListen
DirPort 10.10.10.102:80 NoAdvertise
ORPort 216.176.186.132:443 NoListen
ORPort 10.10.10.102:443 NoAdvertise
#ORPort [2620:18c:0:200::1]:443
RelayBandwidthRate 20 MBytes
RelayBandwidthBurst 20 MBytes
#IPv6Exit 1
Exitpolicy accept *:*
#ExitPolicy accept6 *:*

Tor router #3

sudo vim /etc/tor/instances/tor03/torrc
Nickname EmeraldOnion03
Address tor03.emeraldonion.org
ContactInfo abuse@emeraldonion.org
OutboundBindAddressExit 10.10.10.103
OutboundBindAddressOR 10.10.10.103
DirPort 216.176.186.133:80 NoListen
DirPort 10.10.10.103:80 NoAdvertise
ORPort 216.176.186.133:443 NoListen
ORPort 10.10.10.103:443 NoAdvertise
#ORPort [2620:18c:0:300::1]:443
RelayBandwidthRate 20 MBytes
RelayBandwidthBurst 20 MBytes
#IPv6Exit 1
Exitpolicy accept *:*
#ExitPolicy accept6 *:*

Tor router #4

sudo vim /etc/tor/instances/tor04/torrc
Nickname EmeraldOnion04
Address tor04.emeraldonion.org
ContactInfo abuse@emeraldonion.org
OutboundBindAddressExit 10.10.10.104
OutboundBindAddressOR 10.10.10.104
DirPort 216.176.186.134:80 NoListen
DirPort 10.10.10.104:80 NoAdvertise
ORPort 216.176.186.134:443 NoListen
ORPort 10.10.10.104:443 NoAdvertise
#ORPort [2620:18c:0:400::1]:443
RelayBandwidthRate 20 MBytes
RelayBandwidthBurst 20 MBytes
#IPv6Exit 1
Exitpolicy accept *:*
#ExitPolicy accept6 *:*

Start instances

sudo systemctl start tor@tor01
sudo systemctl start tor@tor02
sudo systemctl start tor@tor03
sudo systemctl start tor@tor04

Check logs

sudo journalctl --boot -u tor@tor01.service
sudo journalctl --boot -u tor@tor02.service
sudo journalctl --boot -u tor@tor03.service
sudo journalctl --boot -u tor@tor04.service

Tor changes + reloading

sudo service tor@tor01 reload
sudo service tor@tor02 reload
sudo service tor@tor03 reload
sudo service tor@tor04 reload