Planet

September 09, 2019

Iain R. Learmonth

Face

Spoofing commits to repositories on GitHub

The following has already been reported to GitHub via HackerOne. Someone from GitHub has closed the report as “informative” but told me that it’s a known low-risk issue. As such, while they haven’t explicitly said so, I figure they don’t mind me blogging about it.

Check out this commit in torvalds’ linux.git on GitHub. In case this is fixed, here’s a screenshot of what I see when I look at this link:

GitHub page showing a commit in torvalds/linux with the commit message add super evil code

How did this get past review? It didn’t. You can spoof commits in any repo on GitHub due to the way they handle forks of repositories internally. Instead of copying repositories when forks occur, the objects in the git repository are shared and only the refs are stored per-repository. (GitHub tell me that private repositories are handled differently to avoid private objects leaking out this way. I didn’t verify this but I have no reason to suspect it is not true.)

To reproduce this:

  1. Fork a repository
  2. Push a commit to your fork
  3. Put your commit ref on the end of:
https://github.com/[parent]/[repo]/commit/

That’s all there is to it. You can also add .diff or .patch to the end of the URL and those URLs work too, in the namespace of the parent.

The situation that worries me relates to distribution packaging. Debian has a policy that deltas to packages in the stable repository should be as small as possible, targetting fixes by backporting patches from newer releases.

If you get a bug report on your Debian package with a link to a commit on GitHub, you had better double check that this commit really did come from the upstream author and hasn’t been spoofed in this way. Even if it shows it was authored by the upstream’s GitHub account or email address, this still isn’t proof because this is easily spoofed in git too.

The best defence against being caught out by this is probably signed commits, but if the upstream is not doing that, you can clone the repository from GitHub and check to see that the commit is on a branch that exists in the upstream repository. If the commit is in another fork, the upstream repo won’t have a ref for a branch that contains that commit.

September 09, 2019 08:17 PM

August 31, 2019

Tom Jones

Face

FreeBSD on the NanoPi NEOLTS

The NanoPi NEOLTS is a SBC from FriendlyElec that uses the Allwinner H3 SOC. The NanoPi NEOLTS has a nice selection of hardware including 100Mbit Ethernet, 3 USB Ports and a bunch of exposed GPIO.

FreeBSD on the NanoPi uses GENERICSD image. This image requires a bootloader to be added before it will work. We can prepare a single image to be copied to many SD cards by using a memory disk as an intermediate step.

NanoPi NEOLTS

We need to:

  • Get the latest GENERICSD card image snapshot
  • Install the correct boot loader pkg
  • Create a memory disk
  • Copy the GENERICSD image to memory disk
  • Copy the bootloader to the memory disk
  • Mount the root partition of the sd card image
  • Copy the programs and files we need for the tutorial to the sd card

The latest image is as I write is 13 CURRENT from 20190829:

$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/arm/armv7/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img.xz

We have to decompress the image before we can use it

$ xz -d FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img.xz

Each u-boot bootloader platform has its own package, currently there are 46 different bootloaders in the FreeBSD ports system. We want the u-boot for the nanopi_neo (our target).

$ pkg search nanopi     
u-boot-nanopi-neo2-2019.07     Cross-build das u-boot for model nanopi-neo2
u-boot-nanopi_a64-2019.07      Cross-build das u-boot for model nanopi_a64
u-boot-nanopi_m1plus-2019.07   Cross-build das u-boot for model nanopi_m1plus
u-boot-nanopi_neo-2019.07      Cross-build das u-boot for model nanopi_neo
u-boot-nanopi_neo_air-2019.07  Cross-build das u-boot for model nanopi_neo_air

# pkg install u-boot-nanopi_neo-2019.07

The u-boot-nanopi_neo package contains the binary bootloader we need in u-boot-sunxi-with-spl.bin

$ pkg info -l u-boot-nanopi_neo-2019.07

u-boot-nanopi_neo-2019.07:
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/GPLv2
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/LICENSE
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/catalog.mk
    /usr/local/share/u-boot/u-boot-nanopi_neo/README
    /usr/local/share/u-boot/u-boot-nanopi_neo/boot.scr
    /usr/local/share/u-boot/u-boot-nanopi_neo/metadata
    /usr/local/share/u-boot/u-boot-nanopi_neo/u-boot-sunxi-with-spl.bin

With the GENERICSD image and the bootloader we need to create the memory disk image we will use for staging. First we need to create a large enough backing file.

$ truncate -s 8G nanopi.img
# mdconfig -f nanopi.img
md0

Now we can dd the GENERICSD image to the memory disk

# dd if=FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img of=/dev/md0 bs=1m

We need to dd the bootloader to the start of the SD card, i.e. the entire device and not a partition.

# dd if=/usr/local/share/u-boot/u-boot-nanopi_neo/u-boot-sunxi-with-spl.bin of=/dev/da0 bs=1k seek=8 conv=sync

With the memory disk attached we can interact with the image file as if it were a real USB drive or SD card.

$ gpart show md0
=>      63  16777153  md0  MBR  (8.0G)
        63      2016       - free -  (1.0M)
      2079    102312    1  fat32lba  [active]  (50M)
    104391   6187041    2  freebsd  (3.0G)
   6291432  10485784       - free -  (5.0G)

We can mount the root partition of the SD card and modify or add any files we wish:

# mount /dev/md0sa mnt

When we are done changing things we have to disconnect the memory disk:

# sudo mdconfig -d -u md0

Finally we can copy the memory disk to a real sd card using dd:

# sudo dd if=nanopi.img of=/dev/da0 bs=1m

August 31, 2019 12:00 AM

August 14, 2019

Tom Jones

Face

My FreeBSD Development Setup

I do my FreeBSD development using git, tmux, vim and cscope.

I keep a FreeBSD fork on my github, I have forked https://github.com/freebsd/freebsd to https://github.com/adventureloop/freebsd

On my fork I have the freebsd/freebsd repo set as an upstream

$ git remote -v
origin  git@github.com:adventureloop/freebsd.git (fetch)
origin  git@github.com:adventureloop/freebsd.git (push)
upstream        https://github.com/freebsd/freebsd.git (fetch)
upstream        https://github.com/freebsd/freebsd.git (push)

See this article for information on setting this up https://help.github.com/en/articles/configuring-a-remote-for-a-fork

I do all work on branches using worktrees, keeping the master branch clean.

Periodically I sync the master branch with the FreeBSD upstream:

$ cd ~/code/freebsd-dev/freebsd-git
$ git checkout master
$ git fetch upstream
$ git merge upstream/master
$ git push

I have a development setup based on Ian Lapore's arm set up documented on the FreeBSD wiki https://wiki.freebsd.org/FreeBSD/arm/crossbuild

I have a freebsd-dev directory in my code directory. It their I keep a copy of FreeBSD in freebsd-git, and obj directory for build output and a projects directory for in progress code.

$ tree -L2
.
├── diffs
│   ├── D15222.diff
│   └── old
├── dstdir
│   ├── boot
│   ├── METALOG
│   └── usr
├── freebsd-git
│   ├── bin
│   ├── sbin
...
│   └── usr.sbin
├── obj
│   └── usr
├── projects
│   ├── atomicstats
│   ├── axp288
│   ├── bugsquash
│   ├── byebyejumbograms
...

I use git worktrees for ongoing projects. git worktrees allow you to have a shallow file system copy on a git branch in a directory.

When starting a new project I do something like:

$ cd ~/code/freebsd-dev/freebsd-git
$ git worktree add thj/newdevelopment ../projects/newdevelopment master
$ cd ../projects/newdevelopment

Once the worktree is set up I launch a tmux session in the projects directory. Each random idea or itch I have, if there is enough there, ends up with a project worktree and a tmux session.

tmux allows me to have many windows in a session, I have a serious tmux problem. Right now I have 11 sessions with 42 windows across them. This is a good indicator of my focus level.

I do FreeBSD development with cscope and vim. With tmux splits I normally have an open file and I use other cscope instances in tmux windows to search for things I need in the tree.

I do testing in a bhyve vm and leave the serial port in a tmux window somewhere. I follow the setup in the FreeBSD handbook and back each vm with a zfs dataset.

I do FreeBSD kernel builds using a command like:

env MAKEOBJDIRPREFIX=/home/tom/code/freebsd-dev/obj make -j 44 buildkernel \
        -DKERNFAST installkernel \
    -DNO_ROOT DESTDIR=/home/tom/code/freebsd-dev/dstdir

I then ship kernels to the test vm with scp. jhb@ has a nicer method using the bhyve-loader, but I am yet to try it.

When changes are maturing I create reviews for them using arcanist, manu@ has a good article on doing this

August 14, 2019 12:00 AM

April 08, 2019

Iain R. Learmonth

Face

Exploring Tor Exit Lists and ECC Signatures

At some point, I might write up a series of articles on the Tor Directory Protocol and how it is all working, but first there are some things I’m trying to get my head around in order to finish off a new specification for Tor Exit Lists that will include signatures. Tor Exit Lists are produced by an “exit scanner”. We archive exit lists and use them to provide data for ExoneraTor and Onionoo.

Some background: it is possible to configure an exit relay to use an IP address for exit connections that is different from the IP address that appears in the consensus for Onion Routing (OR) connections or directory connections. The IP addresses are useful to be able to say whether a connection from a given IP at a given time was likely to be from a Tor user or not.

Looking over 72 hours of data, we see an average of 7435 distinct IP addresses (IPv4 and IPv6) in each consensus. In the same period, there are an average of 938 distinct IP addresses in the exit lists. Not all addresses found in the consensus will be used for exit connections, even if they belong to relays with the exit flag. Using this list to “block Tor” will result in overblocking, incorrectly blocking over 7 times the number of addresses as would be correctly blocked.

Using the list of addresses in the consensus would also result in underblocking. In this period, we see an average of 73 addresses that are only to be found in the exit lists, and are not found in the consensus. Active measurement is able to show the addresses that are used by exit relays by controlled experimentation as opposed to guesswork.

IP Addresses found in 72 hours of consensuses and exit lists

IP Addresses found in 72 hours of consensuses and exit lists

Up to now, there have not been any signatures on the exit lists used by Tor Metrics. As Tor Metrics are both the primary producers and consumers of this format, we are able to extend it, and so we are currently working on a (not backwards compatible) revision of the specification.

Tor relay “router identity” keys are limited to 1024-bit RSA keys. When you see relays mentioned in tor log files, or are looking up a relay using Relay Search, the fingerprint you see identifying routers is based on this 1024-bit key. We don’t want to implement new code with 1024-bit RSA as that wouldn’t be very future-proof, but luckily Tor is already moving towards another scheme that was first proposed in 2013 and was implemented in tor 0.3.0.1-alpha: Ed25519.

Roughly 29% of the relays running in the Tor network are using pre-0.3.0 versions of tor which do not yet use Ed25519 signatures and so 1024-bit RSA is still the primary mechanism used for identifying routers. Around 15 months ago, work was going on to add the Ed25519 relay identity keys to the consensus but due to performance issues that work was shelved at the time.

So, let’s take a look at “server descriptors” (§2.1.1 dir-spec). These are the signed documents that relays publish to the directory authorities, that the authorities then use for constructing their votes. Server descriptors contain information that the relay is asserting about itself, like what addresses it has and what features it supports. This is also where it publishes the keys that are needed to use it with the Tor protocol (the onion and ntor keys).

For example:

router uoaerg 137.50.19.11 443 0 80
identity-ed25519
-----BEGIN ED25519 CERT-----
AQQABpmpAaxBnHwb8In0z5hnq8RpxXNnV0VSrLeDxBoasZv9ozcqAQAgBAB6FFq4
tV9O7gFcWKb7xZdYLR5XHkAz9MecwXB7Ohnu3Uj2oUaFVDboy1LkzuK4PoGako1X
c3WIO4I1/VHrUugGYgzrXKVNDu5pFNf8ibf7ksBid0Cv4tK6pWMCS6r2agQ=
-----END ED25519 CERT-----
master-key-ed25519 ehRauLVfTu4BXFim+8WXWC0eVx5AM/THnMFwezoZ7t0
platform Tor 0.3.5.8 on Linux
proto Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-5 LinkAuth=1,3 Microdesc=1-2 Relay=1-2
published 2019-04-08 09:43:45
fingerprint CF0C C69D E1E7 E75A 2D99 5FD8 D9FA 7D20 9835 31DA
uptime 1674318
bandwidth 1658880 16588800 2994620
extra-info-digest 0BFA51DEE097612601868748DB058B5377B69F5A 1b2JkVb0ZjpiLZfjGbVfl2RSJNzf//MCbKxeQvuSALM
onion-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALsZpcDVk13jt0588OeA/Lf2PKhTjfQTwjiqau9RLot0UZyA8ee5nlkf
2Rr2R2asT/tMMFI3PvClqJfx+QHnVGZDsXhOuxcNSZBzUqnsxooeejBSuFWzCLlP
VHqqj58cg43GmW4p9EqaitQD9uiP2Ov5gZu30yGy9IFHhhuoYF2ZAgMBAAE=
-----END RSA PUBLIC KEY-----
signing-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBANIzr5wvMsZhSUxACSSTeuOqcjHgRw1iJWOXaTqWwdUXRudBsOR1+RaG
6i9E3UFMcCCMv+haLzVnlOCuUkvhSWIgeEn0RxgpaO41EieXnKcTWWMfPlXQoe2K
j0MhzCaI5YRxFK0AnFovBBX7eV9x+2fASo/+5aaavqBCy7ZFoTQJAgMBAAE=
-----END RSA PUBLIC KEY-----
onion-key-crosscert
-----BEGIN CROSSCERT-----
QCps8gFRHDyPvzWAQZAQlSmjxD3t2R1xdiMJ0w3Yzmu9fDuOFIaJz1mFD0U02Pl5
VU/ZfztQXmhCx9s/1Nt0YIJ1BUkg23Pn9nBkfPkLh+i/scH++cCg/0aGu14AqSbi
B+c2jMOF6fNtdXr1SCaHuPTaAl8Ja6IVozg+dcwJ14g=
-----END CROSSCERT-----
ntor-onion-key-crosscert 0
-----BEGIN ED25519 CERT-----
AQoABpmaAXoUWri1X07uAVxYpvvFl1gtHlceQDP0x5zBcHs6Ge7dADNSKRx3aKBy
adX9gcFBmJ4g7LTGWvKp8n6md6KwtnUAIaJjSr3JTgJ4eZj6gtyH0uzDwxfRpElE
ITqc2DHfzg8=
-----END ED25519 CERT-----
family $E59CC0060074E14CA8E9469999B862C5E1CE49E9
hidden-service-dir
contact 0xF540ABCD Iain R. Learmonth <irl@fsfe.org>
ntor-onion-key etwB4T2yCuYne+BZMBE7IU/s9cWkxwyPYpoNcUJMAHQ=
reject *:*
tunnelled-dir-server
router-sig-ed25519 2o32rFB0FvGO4ajKl65JaQpwDWQ4XZHr2Q4qdncQICK2avcEDNyGC05Ny1f82e59JMT/LyAZ7cvsZO3Z7q5tBA
router-signature
-----BEGIN SIGNATURE-----
pA0gSNGipcCXiCQ5A6vg+bwCR10cWQsIerdo2eTomfSo8LTp+9lQGjtokyjJKK4c
7A0JSG+DH33EogN2t4/J3PH508WU9l5A8xkXIBa/PW8U4ZhHWPj8+tEBQWkCn1Mu
+aWkEo6hCmOyr4c4Qj7ha+1KO507hgfa0XRVPmRNJPY=
-----END SIGNATURE-----

There are three keywords here that we’ll focus on relevant to the use of Ed25519 signatures: identity-ed25519, master-key-ed25519, and router-sig-ed25519.

The first of these, identity-ed25519, contains an Ed25519 certificate (cert-spec). That base64 encoded data isn’t very human friendly but luckily stem has a module for dissecting these.

The master-key-ed25519 keyword line is here for convenience only as this is the same key as is used to sign the certificate for identity-ed25519. This key is also bundled with the certificate, and the specification requires that these keys are the same. Let’s see if that is happening:

>>> from stem.descriptor.certificate import Ed25519Certificate
>>> cert = Ed25519Certificate.parse("""AQQABpmpAaxBnHwb8In0z5hnq8RpxXNnV0VSrLeDxBoasZv9ozcqAQAgBAB6FFq4
... tV9O7gFcWKb7xZdYLR5XHkAz9MecwXB7Ohnu3Uj2oUaFVDboy1LkzuK4PoGako1X
... c3WIO4I1/VHrUugGYgzrXKVNDu5pFNf8ibf7ksBid0Cv4tK6pWMCS6r2agQ=
... """)
>>> cert.extensions
[Ed25519Extension(type=4, flags=[], flag_int=0, data=b'z\x14Z\xb8\xb5_N\xee\x01\\X\xa6\xfb\xc5\x97X-\x1eW\x1e@3\xf4\xc7\x9c\xc1p{:\x19\xee\xdd')]
>>> # We can see here a single extension, with type 4, which is
>>> # the extension type containing the bundled key. The data
>>> # does not match what we see in the master-key-ed25519 line
>>> # because there it is base64 encoded.
>>> from base64 import encodebytes
>>> encodebytes(cert.extensions[0].data).strip(b'\n=')
b'ehRauLVfTu4BXFim+8WXWC0eVx5AM/THnMFwezoZ7t0'
>>> # Now it matches!

As we can extract the identity key successfully from the certificate, let’s ignore the master-key-ed25519 field going forward. In the certificate, the master key is certifying the signing key, which can expire. This is how the offline signing key system works, because the certificate needs to be regenerated only when the signing key is rotated. If the signing key is compromised, it is only useful until it expires.

>>> # the signing key is the key in the certificate
>>> encodebytes(cert.key).strip(b'\n=')
b'rEGcfBvwifTPmGerxGnFc2dXRVKst4PEGhqxm/2jNyo'
>>> # which is a medium term key
>>> cert.expiration.isoformat()
'2019-05-07T01:00:00'

Now we can check that the signature on the descriptor was generated using the signing key in the certificate.

>>> # /tmp/server_descriptor.txt contains the server descriptor quoted above
>>> from stem.descriptor import parse_file
>>> desc = next(parse_file("/tmp/server_descriptor.txt", descriptor_type="server-descriptor 1.0"))
>>> cert.validate(desc)
>>> # not throwing an exception means success

We can check what happens when we use another certificate, which should fail as the signature will have used the wrong key:

>>> cert2 = Ed25519Certificate.parse("""AQQABpmWAVLI15lBP2vLQtbMpvZmBTzFsMIcmN1WC3CNmUaqXn+CAQAgBADi5wOy
... PA8fELwr32F2BkVcd3h8PLnOkQGDa4Edv3PKCeu+KGVoXmY6WnEB2qsYkvsWQk+a
... MRmS18Sbekg0aWPPfmpajEs4kq2dK3eKRL6Mtoad0s4fLkhRYLCmRvQuAA4=""")
>>> cert2.validate(desc)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    cert2.validate(desc)
  File "/usr/local/lib/python3.7/dist-packages/stem/descriptor/certificate.py", line 254, in validate
    raise ValueError('Ed25519KeyCertificate signing key is invalid (%s)' % exc)
ValueError: Ed25519KeyCertificate signing key is invalid (Signature was forged or corrupt)

Looking at the source code for the validate function, it’s not entirely clear to me what this error means, but an error was expected and we got one so that’s good. It would be nice to be able to validate the certificate independently of validating a server descriptor but that doesn’t currently seem to be possible.

I’ve also not entirely figured out yet if it’s comparing the certificate against the key that is contained in the server descriptor or using the keys in the certificate.

April 08, 2019 04:34 PM

April 07, 2019

Iain R. Learmonth

Face

Functional Cryptography

I’m pretty good at buying books and not finding the time to read them. I did manage to finish The Manga Guide to Cryptography and have now started on Introducing Elixir. In order to make sure I’m understanding these topics, I’ve decided to implement the algorithms I’ve read about in the former, using the programming language I’m reading about in the latter.

I was pleased to find that installing Elixir on Debian is really easy:

$ sudo apt install elixir

I’ve not got that far into the Elixir book, but I do already have an implementation of a Caeser shift. I have no idea if this is idiomatic or not, but it works:

defmodule Caesar do
  def shift(message, key \\ 3) do
    shift(to_charlist(message), "", key)
  end

  defp shift([head | tail], interim, key) do
    cp = head + key
    interim = interim <> <<cp>>
    shift(tail, interim, key)
  end

  defp shift([], interim, _key) do
    interim
  end
end

When I run it in iex(1) I get:

iex(57)> Caesar.shift("hello")
"khoor"

I’ve not yet dealt with things like running out of the end of the alphabet yet (z + 3 = ?) which I guess I will do with guards.

It’s a little frustrating that things are not working the way I expect them to, and I’m very concious that there’s a whole load of classes of problem that I would have no idea how to tackle yet. I feel like I’m learning an entirely new thing compared to learning Python after Java where much of the syntax is similar and you have the same paradigms. I think this is a good thing.

April 07, 2019 05:07 PM

April 05, 2019

Iain R. Learmonth

Face

Hard drive failure in my zpool 😞

I have a storage box in my house that stores important documents, backups, VM disk images, photos, a copy of the Tor Metrics archive and other odd things. I’ve put a lot of effort into making sure that it is both reliable and performant. When I was working on a modern CollecTor for Tor Metrics recently, I used this to be able to run the entire history of the Tor network through the prototype replacement to see if I could catch any bugs.

I have had my share of data loss events in my life, but since I’ve found ZFS I have hope that it is possible to avoid, or at least seriously minimise the risk of, any catastrophic data loss events ever happening to me again. ZFS has:

  • cryptographic checksums to validate data integrity
  • mirroring of disks
  • “scrub” function that ensures that the data on disk is actually still good even if you’ve not looked at it yourself in a while

ZFS on its own is not the entire solution though. I also mix-and-match hard drive models to ensure that a systematic fault in a particular model won’t wipe out all my mirrors at once, and I also have scheduled SMART self-tests to detect faults before any data loss has occured.

Unfortunately, one of my drives in my zpool has failed a SMART self-test.

Unfortunately, one of my drives in my zpool has failed a SMART self-test.

This means I now have to treat that drive as “going to fail soon” which means that I don’t have redundancy in my zpool anymore, so I have to act. Fortunately, in September 2017 when my workstation died, I received some donations towards the hardware I use for my open source work and I did buy a spare HDD for this very situation!

At present my zpool setup looks like:

% zpool status flat
  pool: flat
 state: ONLINE
  scan: scrub repaired 0 in 0 days 07:05:28 with 0 errors on Fri Apr  5 07:05:36 2019
config:

	NAME                                            STATE     READ WRITE CKSUM
	flat                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	cache
	  gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0

errors: No known data errors

The drives in the two mirrors are 3TB drives, in each mirror is one WD Red and one Toshiba NAS drive. In this case, it is one of the WD Red drives that has failed and I’ll be replacing it with another WD Red. One important thing to note is that you have to replace the drive with one of equal or greater capacity. In this case it is the same model so the capacity should be the same, but not all X TB drives are going to be the same size.

You’ll notice here that it is saying No known data errors. This is because there hasn’t been any issues with the data yet, it is just a SMART failure, and hopefully by replacing the disk any data error can be avoided entirely.

My plan was to move to a new system soon, with 8 bays. In that system I’ll keep the stripe over 2 mirrors but one mirror will run over 3x 6TB drives with the other remaining on 2x 3TB drives. This incident leaves me with only 1 leftover 3TB drive though so maybe I’ll have to rethink this.

Free space remaining in my zpool

Free space remaining in my zpool

My current machine, an HP MicroServer, does not support hot-swapping the drives so I have to start by powering off the machine and replacing the drive.

% zpool status flat
  pool: flat
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 07:05:28 with 0 errors on Fri Apr  5 07:05:36 2019
config:

	NAME                                            STATE     READ WRITE CKSUM
	flat                                            DEGRADED     0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	  mirror-1                                      DEGRADED     0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    xxxxxxxxxxxxxxxxxxxx                        UNAVAIL      0     0     0  was /dev/gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
	cache
	  gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    ONLINE       0     0     0

errors: No known data errors

The disk that was part of the mirror is now unavailable, but the pool is still functioning as the other disk is still present. This means that there are still no data errors and everything is still running. The only downtime was due to the non-hot-swappableness of my SATA controller.

Through the web interface in FreeNAS, it is possible to now use the new disk to replace the old disk in the mirror: Storage -> View Volumes -> Volume Status (under the table, with the zpool highlighted) -> Replace (with the unavailable disk highlighted).

Running zpool status again:

% zpool status flat
  pool: flat
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  5 16:55:47 2019
	1.30T scanned at 576M/s, 967G issued at 1.12G/s, 4.33T total
	4.73G resilvered, 21.82% done, 0 days 00:51:29 to go
config:

	NAME                                            STATE     READ WRITE CKSUM
	flat                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0
	    gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ONLINE       0     0     0  (resilvering)
	cache
	  gptid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    ONLINE       0     0     0

errors: No known data errors

And everything should be OK again soon, now with the dangerous disk removed and a hopefully more reliable disk installed.

A more optimistic message from FreeNAS

A more optimistic message from FreeNAS

This has put a dent in my plans to upgrade my storage, so for now I’ve added the hard drives I’m looking for to my Amazon wishlist.

As for the drive that failed, I’ll be doing an ATA Secure Erase and then disposing of it. NIST SP 800-88 thinks that ATA Secure Erase is in the same category as degaussing a hard drive and that it is more effective than overwriting the disk with software. ATA Secure Erase is faster too because it’s the hard drive controller doing the work. I just have to hope that my firmware wasn’t replaced with firmware that only fakes the process (or I’ll just do an overwrite anyway to be sure). According to the same NIST document, “for ATA disk drives manufactured after 2001 (over 15 GB) clearing by overwriting the media once is adequate to protect the media from both keyboard and laboratory attack”.


This blog post is also a little experiment. I’ve used a Unicode emoji in the title, and I want to see how various feed aggregators and bots handle that. Sorry if I broke your aggregator or bot.

April 05, 2019 08:13 PM

IETF 104 in Prague

Thanks to support from Article 19, I was able to attend IETF 104 in Prague, Czech Republic this week. Primarily this was to present my Internet Draft which takes safe measurement principles from Tor Metrics work and the Research Safety Board and applies them to Internet Measurement in general.

My IETF badge, complete with additional tag for my nick

My IETF badge, complete with additional tag for my nick

I attended with a free one-day pass for the IETF and free hackathon registration, so more than just the draft presentation happened. During the hackathon I sat at the MAPRG table and worked on PATHspider with Mirja Kühlewind from ETH Zurich. We have the code running again with the latest libraries available in Debian testing and this may become the basis of a future Tor exit scanner (for generating exit lists, and possibly also some bad exit detection). We ran a quick measurement campaign that was reported in the hackathon presentations.

During the hackathon I also spoke to Watson Ladd from Cloudflare about his Roughtime draft which could be interesting for Tor for a number of reasons. One would be for verifying if a consensus is fresh, another would be for Tor Browser to detect if a TLS cert is valid, and another would be providing archive signatures for Tor Metrics. (We’ve started looking at archive signatures since our recent work on modernising CollecTor).

On the Monday, this was the first “real” day of the IETF. The day started off for me at the PEARG meeting. I presented my draft as the first presentation in that session. The feedback was all positive, it seems like having the document is both desirable and timely.

The next presentation was from Ryan Guest at Salesforce. He was talking about privacy considerations for application level logging. I think this would also be a useful draft that compliments my draft on safe measurement, or maybe even becomes part of my draft. I need to follow up with him to see what he wants to do. A future IETF hackathon project might be comparing Tor’s safe logging with whatever guidelines we come up with, and also comparing our web server logs setup.

Nick Sullivan was up next with his presentation on Privacy Pass. It seems like a nice scheme, assuming someone can audit the anti-tagging properties of it. The most interesting thing I took away from it is that federation is being explored which would turn this into a system that isn’t just for Cloudflare.

Amelia Andersdotter and Christoffer Långström then presented on differential privacy. They have been exploring how it can be applied to binary values as opposed to continuous, and how it could be applied to Internet protocols like the QUIC spin bit.

The last research presentation was Martin Schanzenbach presenting on an identity provider based on the GNU Name System. This one was not so interesting for me, but maybe others are interested.

I attended the first part of the Stopping Malware and Researching Threats (SMART) session. There was an update from Symantec based on their ISTR report and I briefly saw the start of a presentation about “Malicious Uses of Evasive Communications and Threats to Privacy“ but had to leave early to attend another meeting. I plan to go back and look through all of the slides from this session later.

The next IETF meeting is directly after the next Tor meeting (I had thought for some reason it directly clashed, but I guess I was wrong). I will plan to remotely participate in PEARG again there and move my draft forwards.

April 05, 2019 02:48 PM

February 12, 2019

Ana Custura

LibreELEC Streaming Camera

This is a blogpost about a Raspberry Pi camera setup on LibreELEC. It includes a step by step tutorial.

If you have the very specific use case of needing to run LibreELEC on a Raspberry Pi to play media while also streaming a Pi camera, look no further. This setup has been running for the past year or so in my house. I’ve rebuilt the Docker container for it recently, which is the perfect excuse for a blogpost.

LibreELEC is a Linux distribution for running media center software Kodi. It will stream films, music, TV shows from your NAS to your TV and has several awesome add-ons. LibreELEC has no add-on for streaming from a camera. Perhaps because the drivers needed for USB cameras are not actually even compiled into the image for the RPi2. I’ve tried this setup using a USB camera and had to build a custom LibreELEC image.

However, there is support for streaming from RPi cameras and there is a Docker add-on, which can be used to run a simple camera streaming app in a container. The app I’ve chosen after some trial and error of which gruelling details I will spare you is mjpeg-streamer.

The container uses the current Raspbian stable image, stretch, on top of which it builds mjpeg-streamer. It then uses an entrypoint script to capture a still image per second from the Pi camera, which the software then turns into a www stream on port 8080.

You can get the container here. For a step by step tutorial on how to deploy this, read on.


Step by step tutorial

Ingredients:
  • Raspberry Pi v2 or v3
  • Micro SD card (>2GB) and SD card adapter
  • Compatible Raspberry Pi camera module
  • Monitor and keyboard
  • Wired connection or wireless dongle if using a RPi 2

Step 1: Download

Download the official LibreELEC image for the Raspberry Pi from the official website. Scroll to ‘Direct Downloads’, select ‘Raspberry Pi v2 and Raspberry Pi v3’ and click the filename link on the page. The filename looks like LibreELEC-RPi2.arm-8.2.5.img.gz at the moment of writing this blog post, yours will most likely be newer.

Step 2: Put image on SD card

I use an SD card adapter and the in-built card reader in my laptop. On Linux, after plugging the SD card in, command sudo dmesg should display the name newly inserted device, similar to the following:

ana@sinopia:~|⇒  sudo dmesg
[...]
[70540.504869] mmc0: new high speed SDHC card at address aaaa
[70540.585060] mmcblk0: mmc0:aaaa SB16G 14.8 GiB (ro)
[70540.590225] mmcblk0: p1

The identifier for SD cards on Linux usually looks like mmcblk. Careful to copy your image on the right device, and not on your local hard drive:

ana@cyan:~|⇒  gunzip LibreELEC-RPi2.arm-8.2.5.img.gz
ana@cyan:~|⇒  sudo dd if=LibreELEC-RPi2.arm-8.2.5.img of=/dev/mmcblk0
549+0 records in
549+0 records out
575668224 bytes (576 MB, 549 MiB) copied, 50.8254 s, 11.3 MB/s

This may take a minute or two. Once dd has finished, load up your MicroSD card in your Raspberry Pi.

Step 3: Configure LibreELEC

Plug a monitor and a keyboard in and turn on your RPi. LibreELEC starts an autoconfiguration wizard when it first boots. This will guide you through setting up location, timezone and most importantly, a network connection needed for the next step. Hit ‘Next’ and follow the wizard.

  • One you connect to a network, the interface will display your IP address. Make sure to take a note of it.
  • Under the initial ‘Sharing and Remote Access’ screen make sure you enable the SSH service.

For security purposes, we will disable SSH after setting up docker as the root password is hardcoded in the image.

Step 4: Download the Docker Add-on

From the left hand-side menu, navigate to Add-ons -> Install from repository -> LibreELEC Add-ons -> Services -> Docker and then select Install.

The interface will notify you once this has installed.

Step 5: Build the Docker container

In order to build and run the docker container, you need to ssh as root into your Pi, using the IP address from Step 3. In this example, my IP is 192.168.0.156.

ana@cyan:~|⇒  ssh root@172.22.152.253
root@172.22.152.253's password

At the password prompt, type ‘libreelec’, the LibreELEC default and hardcoded password.

Next, make a directory and download the Dockerfile and starting script.

LibreELEC:~ # mkdir mpeg
LibreELEC:~ # cd mpeg
LibreELEC:~/mpeg # wget https://raw.githubusercontent.com/ana-cc/dockerstuffs/master/mjpeg-streamer-rpi/Dockerfile
Connecting to raw.githubusercontent.com (151.101.60.133:443)
Dockerfile           100% |************************************************************************************************************************|   917   0:00:00 ETA
LibreELEC:~/mpeg # wget https://raw.githubusercontent.com/ana-cc/dockerstuffs/master/mjpeg-streamer-rpi/stream_from_pi.sh
Connecting to github.com (140.82.118.3:443)
stream_from_pi.sh    100% |************************************************************************************************************************| 55534   0:00:00 ETA

Inspect and (optionally) edit the stream script. The script uses command raspistill to capture an image per second from the Pi camera, which mjpeg-streamer then turns into a www stream on port 8080. You can vary the resolution, quality and time apart these images are taken with the -w -h and sleep values. Here is the script by default:

until raspistill --nopreview -vf -hf -w 640 -h 480 -q 5 -o /tmp/stream/pic.jpg -tl 100 -t 9999999 -th 0:0:0 & LD_LIBRARY_PATH=/usr/local/lib mjpg_streamer -i "input_file.so -f /tmp/stream -n pic.jpg" -o "output_http.so -w /usr/local/www"; do
	sleep 1
done

Finally, build the container and give it an easy to remember tag:

docker build -t mjpeg_container .

This should take a while, at the end your output should be similar to:

[...]
Step 12/13 : COPY stream_from_pi.sh /
 ---> 2299b11e7696
Removing intermediate container de4317561efe
Step 13/13 : ENTRYPOINT /bin/bash /stream_from_pi.sh
 ---> Running in deff3a4ebe15
 ---> b5f669ccd45e
Removing intermediate container deff3a4ebe15
Successfully built b5f669ccd45e

Step 6: Run the docker container

Now for the fun part: running and testing the newly-built container.

We want the Docker process to map the default streaming port to port 8081 of our RPi host (Kodi’s own web interface occupies port 8080 by default):

 docker run -d --privileged -p 8081:8080 mjpeg_container

And now the LibreELEC box will present the stream on port 8081. In a browser of choice, navigate to your IP address on port 8081, for example https//192.168.0.156:8081.

You should see a welcome page like the following:

Your stream can be viewed under the stream tab.

Step 8: Clean up

  • Worth adding an @reboot crontab to run the docker command if your box is subject to, uh, a lot of reboots.

Before ending your ssh session:

LibreELEC:~ # crontab -e

Make sure the file contains the following line:

@reboot docker run -d --privileged -p 8081:8080 mjpeg_container

This tells the box to run the docker container at every reboot so that you don’t have to start it manually.

  • Finally, disable SSH by going to Settings -> Services in your Kodi navigation.

This concludes the tutorial. Happy streaming!

February 12, 2019 12:00 AM

February 07, 2019

Iain R. Learmonth

Face

Privacy-preserving monitoring of an anonymity network (FOSDEM 2019)

This is a transcript of a talk I gave at FOSDEM 2019 in the Monitoring and Observability devroom about the work of Tor Metrics.

Direct links:

Producing this transcript was more work than I had anticipated it would be, and I’ve done this in my free time, so if you find it useful then please do let me know otherwise I probably won’t be doing this again.


I’ll start off by letting you know who I am. Generally this is a different audience for me but I’m hoping that there are some things that we can share here. I work for Tor Project. I work in a team that is currently formed of two people on monitoring the health of the Tor network and performing privacy-preserving measurement of it. Before Tor, I worked on active Internet measurement in an academic environment but I’ve been volunteering with Tor Project since 2015. If you want to contact me afterwards, my email address, or if you want to follow me on the Fediverse there is my WebFinger ID.

So what is Tor? I guess most people have heard of Tor but maybe they don’t know so much about it. Tor is quite a few things, it’s a community of people. We have a paid staff of approximately 47, the number keeps going up but 47 last time I looked. We also have hundereds of volunteer developers that contribute code and we also have relay operators that help to run the Tor network, academics and a lot of people involved organising the community locally. We are registered in the US as a non-profit organisation and the two main things that really come out of Tor Project are the open source software that runs the Tor network and the network itself which is open for anyone to use.

Currently there are an estimated average 2,000,000 users per day. This is estimated and I’ll get to why we don’t have exact numbers.

Most people when they are using Tor will use Tor Browser. This is a bundle of Firefox and a Tor client set up in such a way that it is easy for users to use safely.

When you are using Tor Browser your traffic is proxied through three relays. With a VPN there is only one server in the middle and that server can see either side. It is knows who you are and where you are going so they can spy on you just as your ISP could before. The first step in setting up a Tor connection is that the client needs to know where all of those relays are so it downloads a list of all the relays from a directory server. We’re going to call that directory server Dave. Our user Alice talks to Dave to get a list of the relays that are available.

In the second step, the Tor client forms a circuit through the relays and connects finally to the website that Alice would like to talk to, in this case Bob.

If Alice later decides they want to talk to Jane, they will form a different path through the relays.

We know a lot about these relays. Because the relays need to be public knowledge for people to be able to use them, we can count them quite well. Over time we can see how many relays there are that are announcing themselves and we also have the bridges which are a seperate topic but these are special purpose relays.

Because we have to connect to the relays we know their IP addresses and we know if they have IPv4 or IPv6 addresses so as we want to get better IPv6 support in the Tor network we can track this and see how the network is evolving.

Because we have the IP addresses we can combine those IP addresses with GeoIP databases and that can tell us what country those relays are in with some degree of accuracy. Recently we have written up a blog post about monitoring the diversity of the Tor network. The Tor network is not very useful if all of the relays are in the same datacenter.

We also perform active measurement of these relays so we really analyse these relays because this is where we put a lot of the trust in the Tor network. It is distributed between multiple relays but if all of the relays are malicious then the network is not very useful. We make sure that we’re monitoring its diversity and the relays come in different sizes so we want to know are the big relays spread out or are there just a lot of small relays inflating the absolute counts of relays in a location.

When we look at these two graphs, we can see that the number of relays in Russia is about 250 at the moment but when we look at the top 5 by the actual bandwidth they contribute to the network they drop off and Sweden takes Russia’s place in the top 5 contributing around 4% of the capacity.

The Tor Metrics team, as I mentioned we are two people, and we care about measuring and analysing things in the Tor network. There are 3 or 4 repetitive contributors and then occasionally people will come along with patches or perform a one-off analysis of our data.

We use this data for lots of different use cases. One of which is detecting censorship so if websites are blocked in a country, people may turn to Tor in order to access those websites. In other cases, Tor itself might be censored and then we see a drop in Tor users and then we also see we a rise in the use of the special purpose bridge relays that can be used to circumvent censorship. We can interpret the data in that way.

We can detect attacks against the network. If we suddenly see a huge rise in the number of relays then we can suspect that OK maybe there is something malicious going on here and we can deal with that. We can evaluate effects on how performance changes when we make changes to the software. We have recently made changes to an internal scheduler and the idea there is to reduce congestion at relays and from our metrics we can say we have a good idea that this is working.

Probably one of the more important aspects is being able to take this data and make the case for a more private and secure Internet, not just from a position of I think we should do this, I think it’s the right thing, but here is data and here is facts that cannot be so easily disputed.

We only handle public non-sensitive data. Each analysis goes through a rigorous review and discussion process before publication.

As you might imagine the goals of privacy and anonymity network doesn’t lend itself to easy data gathering and extensive monitoring of the network. The Research Safety Board if you are interested in doing research on Tor or attempting to collect data through Tor can offer advice on how to do that safely. Often this is used by academics that want to study Tor but also the Metrics Team has used it on occasion where we want to get second opinions on deploying new measurements.

What we try and do is follow three key principles: data minimalisation, source aggregation, and transparency.

The first one of these is quite simple and I think with GDPR probably is something people need to think about more even if you don’t have an anonymity network. Having large amounts of data that you don’t have an active use for is a liability and something to be avoided. Given a dataset and given an infinite amount of time that dataset is going to get leaked. The probability increases as you go along. We want to make sure that we are collecting as little detail as possible to answer the questions that we have.

When we collect data we want to aggregate it as soon as we can to make sure that sensitive data exists for as little time as possible. This means usually in the Tor relays themselves before they even report information back to Tor Metrics. They will be aggregating data and then we will aggregate the aggregates. This can also include adding noise, binning values. All of these things can help to protect the individual.

And then being as transparent as possible about our processes so that our users are not surprised when they find out we are doing something, relay operators are not surprised, and academics have a chance to say whoa that’s not good maybe you should think about this.

The example that I’m going to talk about is counting unique users. Users of the Tor network would not expect that we are storing their IP address or anything like this. They come to Tor because they want the anonymity properties. So the easy way, the traditional web analytics, keep a list of the IP addresses and count up the uniques and then you have an idea of the unique users. You could do this and combine with a GeoIP database and get unique users per country and these things. We can’t do this.

So we measure indirectly and in 2010 we produced a technical report on a number of different ways we could do this.

It comes back to Alice talking to Dave. Because every client needs to have a complete view of the entire Tor network, we know that each client will fetch the directory approximately 10 times a day. By measuring how many directory fetches there are we can get an idea of the number of concurrent users there are of the Tor network.

Relays don’t store IP addresses at all, they count the number of directory requests and then those directory requests are reported to a central location. We don’t know how long an average session is so we can’t say we had so many unique users but we can say concurrently we had so many users on average. We get to see trends but we don’t get the exact number.

So here is what our graph looks like. At the moment we have the average 2,000,000 concurrent Tor users. The first peak may have been an attempted attack, as with the second peak. Often things happen and we don’t have full context for them but we can see when things are going wrong and we can also see when things are back to normal afterwards.

This is in a class of problems called the count-distinct problem and these are our methods from 2010 but since then there has been other work in this space.

One example is HyperLogLog. I’m not going to explain this in detail but I’m going to give a high-level overview. Imagine you have a bitfield and you initialise all of these bits to zero. You take an IP address, you take a hash of the IP address, and you look for the position of the leftmost one. How many zeros were there at the start of that string? Say it was 3, you set the third bit in your bitfield. At the end you have a series of ones and zeros and you can get from this to an estimate of the total number that there are.

Every time you set a bit there is 50% chance that that bit would be set given the number of distinct items. (And then 50% chance of that 50% for the second bit, and so on…) There is a very complicated proof in the paper that I don’t have time to go through here but this is one example that actually turns out to be quite accurate for counting unique things.

This was designed for very large datasets where you don’t have enough RAM to keep everything in memory. We have a variant on this problem where even keep 2 IP addresses in memory would, for us, be a very large dataset. We can use this to avoid storing even small datasets.

Private Set-Union Cardinality is another example. In this one you can look at distributed databases and find unique counts within those. Unfortunately this currently requires far too much RAM to actually do the computation for us to use this but over time these methods are evolving and should become feasible, hopefully soon.

And then moving on from just count-distinct, the aggregation of counters. We have counters such as how much bandwidth has been used in the Tor network. We want to aggregate these but we don’t want to release the individual relay counts. We are looking at using a method called PrivCount that allows us to get the aggregate total bandwidth used while keeping the individual relay bandwidth counters secret.

And then there are similar schemes to this, RAPPOR and PROCHLO from Google and Prio that Mozilla have written a blog post about are similar technologies. All of the links here in the slides and are in the page on the FOSDEM schedule so don’t worry about writing these down.

Finally, I’m looking at putting together some guidelines for performing safe measurement on the Internet. This is targetted primarily at academics but also if people wanted to apply this to analytics platforms or monitoring of anything that has users and you want to respect those users’ privacy then there could be some techniques in here that are applicable.

Ok. So that’s all I have if there are any questions?


Q: I have a question about how many users have to be honest so that the network stays secure and private, or relays?

A: At the moment when we are collecting statistics we can see – so as I showed earlier the active measurement – we know how much bandwidth a relay can cope with and then we do some load balancing so we have an idea of what fraction of traffic should go to each relay and if one relay is expecting a certain level of traffic and it has wildly different statistics to another relay then we can say that relay is cheating. There isn’t really any incentive to do this and it’s something we can detect quite easily but we are also working on more robust metrics going forward to avoid this being a point where it could be attacked.

Q: A few days ago I heard that with Tor Metrics you are between 2 and 8 million users but you don’t really know in detail what the real numbers are? Can you talk about the variance and which number is more accurate?

A: The 8 million number comes from the PrivCount paper and they did a small study where they looked at unique IP address over a day where we look at concurrent users. There are two different measurements. What we can say is that we know for certain that there are between 2 million and 25 million unique users per day but we’re not sure where in there we fall and 8 million is a reasonable-ish number but also they measured IP addresses and some countries use a lot of NAT so it could be more than 8 million. It’s tricky but we see the trends.

Q: Your presentation actually implies that you are able to collect more private data than you are doing. It says that the only thing preventing you from collecting private user data is the team’s good will and good intentions. Have I got it wrong? Are there any possibilities for the Tor Project team to collect some private user data?

A: Tor Project does not run the relays. We write the code but there individual relay operators that run the relays and if we were to release code that suddenly collecting lots of private data people would realise and they wouldn’t run that code. There is a step between the development and it being deployed. I think that it’s possible other people could write that code and then run those relays but if they started to run enough relays that it looked suspicious then people would ask questions there too, so there is a distributed trust model with the relays.

Q: You talk about privacy preserving monitoring, but also a couple of years ago we learned that the NSA was able to monitor relays and learn which user was connecting to relays so is there also research to make it so Tor users cannot be targetted for using a relay and never being able to be monitored.

A: Yes, there is! There is a lot of research in this area and one of them is through obfuscation techniques where you can make your traffic look like something else. They are called pluggable transports and they can make your traffic look like all sorts of things.

February 07, 2019 02:00 PM

February 05, 2019

Iain R. Learmonth

Face

Cryptonoise: January 2019

On Thursday 17th January, we held the first Cryptonoise event of 2019. We had a good turn out and kicked off the discussion with a quick browse through Wikipedia’s list of data breaches.

Our first topic of discussion was relating to how we all used passwords and how password reuse can very quickly become problematic if it happens that your password is leaked.

Over time, the probability that any entity holding a large store of sensitive private data will remain both competent enough to protect it adequately and honest enough to want to goes to zero. –@mattblaze

In good news, everyone was already aware of the risks of password reuse. The vast majority of the attendees were using pass as a password manager. One attendee used a method to derive unique passwords for services from a hash function, a salt, a master password, and a memorable name for the service. Between us we were not able to come up with any shortcomings in the strategy except for those that had already been considered but a password manager still looked to be the easier option.

We talked a little about how some passwords are more important than others and that some passwords, such as your email password, might be proxies for other passwords, e.g. by password reset emails. I learned of a feature of pass that I did not know about before that allows you to encrypt different subfolders of your password store to different keys allowing for passwords to be selectively available in different locations. This would allow you to protect the most important passwords while allowing others to be used more freely.

As so many of us are using pass, we also talked about the possibility of future collaboration on pass extensions and swapping hints and tips. Two areas we identified were in obfuscating the accounts list and being able to see how old a password is and so have an idea when it should be rotated.

No one had any set schedule for rotating their passwords, so this is hopefully something that we will be thinking about some more and we might discuss more at future events.

Following from the password manager discussion we talked about the use of two-factor authentication. Some were using email or SMS second factor authentication but not really using cryptographic second factors such as a time-based one time password app on a phone or a hardware token like a YubiKey.

Many of us are using a YubiKey for GPG, which we then use for pass. As only the password would be required to access a service and the YubiKey only unlocks the password, this isn’t really two factor authentication. The oathtool command can be used to generate one time passwords and the key for this can be stored in pass. As the key is just a password that allows you to generate a two factor authentication code this may just be a two password arrangement if your pass database is compromised. If just the password is compromised though this may still provide some security as an attacker cannot use the password alone to log in.

Our last discussion topic was disk encryption. We discussed the trade offs between full disk encryption and encrypting only your home directory. If a machine only has an encrypted home directory then it would still be able to boot and perhaps you would have a chance to locate it if you can access it remotely. An attacker would have the ability to modify the contents of the disk however, including the operating system. Given that an attacker could insert backdoors into a laptop in a number of locations (firmware, hardware implant, rootkit, etc.) it may be that you wouldn’t want the laptop back after it has been suspiciously missing even if you did track it down.

Signed binaries may in the future remove the threat of modification of the disk contents as an attacker’s modifications would just never get run. Of course this is only in the case that the user has some control over the system.

We had a quick session at the end to discuss topics for future sessions this year. So far we have:

  • Securing your communications in hostile networks
  • Securing your communications at home
  • The differences between BSD and Linux
  • Best practices for working with Windows
  • GnuPG, web-of-trust, and keysigning

The next Cryptonoise will be at 19:00 on the 28th February 2019 in the 57North Hacklab Social Room. More details will be sent closer to the time on the org-aberdeen and 57north-discuss mailing lists.

February 05, 2019 04:30 PM

February 01, 2019

Ana Custura

Tor Relay Map

Here is a map that displays Tor relays!

The map also supports searching for the full or partial nickname, IP address fingerprint of a Tor relay. You can also search for relays in specific countries or ASes (country:us, as:3), or with specific flags (flag:exit or flag:authority).

To display all running relays, the search string is running:true. An empty search string will plot all the relays.

The Tor relay information is queried from Onionoo.

This was written a while ago for the only purpose of trying out JS and in the future, this will have other open source nodes added to it, including things like BSD mirrors, Bitcoin nodes etc. For now, the source can be found here.

There are several icons used to display the relays, their meanings can be found here. A legend is still on the TODO list.

Full-page version.

©&fields&fields



February 01, 2019 12:00 AM

January 23, 2019

Ana Custura

Zenburn Theme for the terminal

So today I glanced down to my i3 status bar, and thought man those colours are fugly. By default, these are white, fully saturated neon green and red on black:

‘5 minutes is enough to configure the colors for this, right?’

Some research and 2 hours down the rabbit-hole later, and everything from my text editor to the status bar is Zenburn-themed.

Zenburn is a low-contrast theme meant to ensure your eyes do not fall off when you’re having that mexican-standoff with whatever project you happen to be Vimming.

But most of us spend a lot of time in text editors and on the command line, and Zenburn is a theme for Vim only. This useful and beautiful theme does not have an .Xresources file listed on the official page of Zenburn things.

As I’m a fan of cohesion, this now exists, this blog post will explain how.

First, I extracted the colors from the original zenburn.vim using simple regex:

matches = re.findall(r'#(?:[0-9a-fA-F]{3}){1,2}', line)

I then used yattag, a Python library for creating HTML, and displayed all the unique colors from the file. It looks like this:

#c0bed1
#bc6c4c
#80d4aa
#dfdfbf
#4f4f4f
#2b2b2b
#bc8cbc
#d0d0a0
#e89393
#82a282
#304a3d
#444444
#8f8f8f
#2e3330
#242424
#f8f893
#41363c
#88b090
#f0f0c0
#dccdcc
#121212
#313c36
#efef8f
#bc6c9c
#2f2f2f
#464646
#353535
#ecbcbc
#71d3b4
#dcdccc
#7cac7c
#222222
#93b3a3
#5b605e
#dfdfdf
#000d18
#0f0f0f
#353a37
#ffd7a7
#dc8c6c
#dca3a3
#383838
#1f1f1f
#b2b2a0
#b6bf98
#e3ceab
#688060
#709080
#cc9393
#434343
#dfaf8f
#3a3a39
#acd0b3
#2c2e2e
#bca3a3
#efefaf
#545a4f
#343434
#2c302d
#000000
#a0afa0
#313633
#6c6c9c
#8c8cbc
#9fafaf
#efefef
#d0d0b8
#9ccc9c
#8faf9f
#3f3f3f
#ffffe0
#ffffff
#284f28
#664040
#ccd990
#c3bf9f
#dfcfaf
#e37170
#1c1c1b
#efdcbc
#ccccbc
#233323
#333333
#f18c96
#404040
#161616
#262626
#f0f0b0
#cfcfaf
#f0dfaf
#385f38
#040404
#9ece9e
#cbecd0
#434443
#dfe4cf
#3f4040
#ccdc90
#3d3535
#101010
#f0efd0
#ffcfaf
#7f9f7f
#2e2e2e
#181818
#9f9f9f
#8cd0d3
#cfbfaf

To get the colors to make a bit of sense, some sorting was in order. I spent some time reading this excellent article about why sorting colours is a pain. I then did some quick RGB to HSV conversion and sorting, and voilà:

#000000
#040404
#0f0f0f
#101010
#121212
#161616
#181818
#1f1f1f
#222222
#242424
#262626
#2b2b2b
#2e2e2e
#2f2f2f
#333333
#343434
#353535
#383838
#3f3f3f
#404040
#434343
#444444
#464646
#4f4f4f
#8f8f8f
#9f9f9f
#dfdfdf
#efefef
#ffffff
#3d3535
#bca3a3
#ecbcbc
#dca3a3
#cc9393
#e89393
#664040
#e37170
#dccdcc
#bc6c4c
#dc8c6c
#ffcfaf
#dfaf8f
#cfbfaf
#ffd7a7
#e3ceab
#efdcbc
#dfcfaf
#f0dfaf
#c3bf9f
#f0efd0
#3a3a39
#1c1c1b
#dcdccc
#ccccbc
#b2b2a0
#d0d0b8
#ffffe0
#dfdfbf
#cfcfaf
#f0f0c0
#d0d0a0
#f0f0b0
#efefaf
#efef8f
#f8f893
#ccd990
#ccdc90
#b6bf98
#dfe4cf
#545a4f
#688060
#434443
#a0afa0
#82a282
#7f9f7f
#9ece9e
#9ccc9c
#7cac7c
#233323
#385f38
#284f28
#cbecd0
#acd0b3
#88b090
#2c302d
#2e3330
#353a37
#313633
#313c36
#93b3a3
#8faf9f
#709080
#304a3d
#80d4aa
#5b605e
#71d3b4
#3f4040
#2c2e2e
#9fafaf
#8cd0d3
#000d18
#8c8cbc
#6c6c9c
#c0bed1
#bc8cbc
#bc6c9c
#41363c
#f18c96

This looks better and one can easily see what colors there are.

The next stop was terminal.sexy, which is an amazing terminal theme tool which allows you to vizualize and export your work easily. Here’s our theme:

And here’s how it looks in the terminal:

The i3 window colors, borders and status bar have now also been changed to match:

And for completeness, here’s a Firefox theme to match, because we can:

Starting the new year ready to sysadmin for hours on end in an ill-lit room, happy hacking!

January 23, 2019 12:00 AM

January 16, 2019

Iain R. Learmonth

Face

A Solution for Authoritative DNS

I’ve been thinking about improving my DNS setup. So many things will use e-mail verification as a backup authentication measure that it is starting to show as a real weak point. An Ars Technica article earlier this year talked about how “[f]ederal authorities and private researchers are alerting companies to a wave of domain hijacking attacks that use relatively novel techniques to compromise targets at an almost unprecedented scale.”

The two attacks that are mentioned in that article, changing the nameserver and changing records, are something that DNSSEC could protect against. Records wouldn’t have to be changed on my chosen nameservers, a BGP-hijacking could just give another server the queries for records on my domain instead and then reply with whatever it chooses.

After thinking for a while, my requirements come down to:

  • Offline DNSSEC signing
  • Support for storing signing keys on a HSM (YubiKey)
  • Version control
  • No requirement to run any Internet-facing infrastructure myself

After some searching I discovered GooDNS, a “good” DNS hosting provider. They have an interesting setup that looks to fit all of my requirements. If you’re coming from a more traditional arrangement with either a self-hosted name server or a web panel then this might seem weird, but if you’ve done a little “infrastructure as code” then maybe it is not so weird.

The inital setup must be completed via the web interface. You’ll need to have an hardware security module (HSM) for providing a time based one time password (TOTP), an SSH key and optionally a GPG key as part of the registration. You will need the TOTP to make any changes via the web interface, the SSH key will be used to interact with the git service, and the GPG key will be used for any email correspondance including recovery in the case that you lose your TOTP HSM or password.

You must validate your domain before it will be served from the GooDNS servers. There are two options for this, one for new domains and one “zero-downtime” option that is more complex but may be desirable if your domain is already live. For new domains you can simply update your nameservers at the registrar to validate your domain, for existing domains you can add a TXT record to the current DNS setup that will be validated by GooDNS to allow for the domain to be configured fully before switching the nameservers. Once the domain is validated, you will not need to use the web interface again unless updating contact, security or billing details.

All the DNS configuration is managed in a single git repository. There are three branches in the repository: “master”, “staging” and “production”. These are just the default branches, you can create other branches if you like. The only two that GooDNS will use are the “staging” and “production” branches.

GooDNS provides a script that you can install at /usr/local/bin/git-dns (or elsewhere in your path) which provides some simple helper commands for working with the git repository. The script is extremely readable and so it’s easy enough to understand and write your own scripts if you find yourself needing something a little different.

When you clone your git repository you’ll find one text file on the master branch for each of your configured zones:

irl@computer$ git clone git@goodns.net:irl.git
Cloning into 'irl1'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Receiving objects: 100% (3/3), 22.55 KiB | 11.28 MiB/s, done.
Resolving deltas: 100% (1/1), done.
irl@computer$ ls
irl1.net   learmonth.me
irl@computer$ cat irl1.net
@ IN SOA ns1.irl1.net. hostmaster.irl1.net. (
            _SERIAL_
            28800
            7200
            864000
            86400
            )

@           IN      NS        ns1.goodns.net.
@           IN      NS        ns2.goodns.net.
@           IN      NS        ns3.goodns.net.

In the backend GooDNS is using OpenBSD 6.4 servers with nsd(8). This means that the zone files use the same syntax. If you don’t know what this means then that is fine as the documentation has loads of examples in it that should help you to configure all the record types you might need. If a record type is not yet supported by nsd(8), you can always specify the record manually and it will work just fine.

One thing you might note here is that the string _SERIAL_ appears instead of a serial number. The git-dns script will replace this with a serial number when you are ready to publish the zone file.

I’ll assume that you already have you GPG key and SSH key set up, now let’s set up the DNSSEC signing key. For this, we will use one of the four slots of the YubiKey. You could use either 9a or 9e, but here I’ll use 9e as 9a is already the SSH key for me.

To set up the token, we will need the yubico-piv-tool. Be extremely careful when following these steps especially if you are using a production device. Try to understand the commands before pasting them into the terminal.

First, make sure the slot is empty. You should get an output similar to the following one:

irl@computer$ yubico-piv-tool -s 9e -a status 
CHUID:  ...
CCC:    No data available
PIN tries left: 10

Now we will use git-dns to create our key signing key (KSK):

irl@computer$ git dns kskinit --yubikey-neo
Successfully generated a new private key.
Successfully generated a new self signed certificate.
Found YubiKey NEO.
Slots available:
 (1) 9a - Not empty
 (2) 9e - Empty
Which slot to use for DNSSEC signing key? 2
Successfully imported a new certificate.
CHUID:  ...
CCC:    No data available
Slot 9e:    
    Algorithm:  ECCP256
    Subject DN: CN=irl1.net
    Issuer DN:  CN=irl1.net
    Fingerprint:    97dda8a441a401102328ab6ed4483f08bc3b4e4c91abee8a6e144a6bb07a674c
    Not Before: Feb 01 13:10:10 2019 GMT
    Not After:  Feb 01 13:10:10 2021 GMT
PIN tries left: 10

We can see the public key for this new KSK:

irl@computer$ git dns pubkeys
irl1.net. DNSKEY 256 3 13 UgGYfiNse1qT4GIojG0VGcHByLWqByiafQ8Yt7/Eit2hCPYYcyiE+TX8HP8al/SzCnaA8nOpAkqFgPCI26ydqw==

Next we will create a zone signing key (ZSK). These are stored in the keys/ folder of your git repository but are not version controlled. You can optionally encrypt these with GnuPG (and so requiring the YubiKey to sign zones) but I’ve not done that here. Operations using slot 9e do not require the PIN so leaving the YubiKey connected to the computer is pretty much the same as leaving the KSK on the disk. Maybe a future YubiKey will not have this restriction or will add more slots.

irl@computer$ git dns zskinit
Created ./keys/
Successfully generated a new private key.
irl@computer$ git dns pubkeys
irl1.net. DNSKEY 256 3 13 UgGYfiNse1qT4GIojG0VGcHByLWqByiafQ8Yt7/Eit2hCPYYcyiE+TX8HP8al/SzCnaA8nOpAkqFgPCI26ydqw=
irl1.net. DNSKEY 257 3 13 kS7DoH7fxDsuH8o1vkvNkRcMRfTbhLqAZdaT2SRdxjRwZSCThxxpZ3S750anoPHV048FFpDrS8Jof08D2Gqj9w==

Now we can go to our domain registrar and add DS records to the registry for our domain using the public keys. First though, we should actually sign the zone. To create a signed zone:

irl@computer$ git dns signall
Signing irl1.net...
Signing learmonth.me...
[production 51da0f0] Signed all zone files at 2019-02-01 13:28:02
 2 files changed, 6 insertions(+), 0 deletions(-)

You’ll notice that all the zones were signed although we only created one set of keys. Set ups where you have one shared KSK and individual ZSK per zone are possible but they provide questionable additional security. Reducing the number of keys required for DNSSEC helps to keep them all under control.

To make these changes live, all that is needed is to push the production branch. To keep things tidy, and to keep a backup of your sources, you can push the master branch too. git-dns provides a helper function for this:

irl@computer$ git dns push
Pushing master...done
Pushing production...done
Pushing staging...done

If I now edit a zone file on the master branch and want to try out the zone before making it live, all I need to do is:

irl@computer$ git dns signall --staging
Signing irl1.net...
Signing learmonth.me...
[staging 72ea1fc] Signed all zone files at 2019-02-01 13:30:12
 2 files changed, 8 insertions(+), 0 deletions(-)
irl@computer$ git dns push
Pushing master...done
Pushing production...done
Pushing staging...done

If I now use the staging resolver or lookup records at irl1.net.staging.goodns.net then I’ll see the zone live. The staging resolver is a really cool idea for development and testing. They give you a couple of unique IPv6 addresses just for you that will serve your staging zone files and act as a resolver for everything else. You just have to plug these into your staging environment and everything is ready to go. In the future they are planning to allow you to have more than one staging environment too.

All that is left to do is ensure that your zone signatures stay fresh. This is easy to achieve with a cron job:

0 3 * * * /usr/local/bin/git-dns cron --repository=/srv/dns/irl1.net --quiet

I monitor the records independently and disable the mail output from this command but you might want to drop the --quiet if you’d like to get mails from cron on errors/warnings.

On the GooDNS blog they talk about adding an Onion service for the git server in the future so that they do not have logs that could show the location of your DNSSEC signing keys, which allows you to have even greater protection. They already support performing the git push via Tor but the addition of the Onion service would make it faster and more reliable.


Unfortunately, GooDNS is entirely fictional and you can’t actually manage your DNS in this way, but wouldn’t it be nice? This post has drawn inspiration from the following:

January 16, 2019 04:30 PM

November 14, 2018

Ana Custura

A tiny Nagios plugin to check DNSSEC RRSIG expiry

I have used both Icinga and Nagios to monitor a variety of infrastructures, either at work, at home and anywhere in between. These share a variety of very useful plugins for monitoring lots and lots of standard stuff, from basic ping checks to filesystem mounts. Occasionally I had to write the odd script to monitor something ridiculously niche, like split-brain HSRP status between pairs of Cisco routers.

This one is a plugin for checking RRSIG expiry dates on DNSSEC records. There are other plugins out there that do exactly this, but I could not find anything that would not depend on a ton of other packages to do it (libnet-dns-sec-perl anyone?). So I decided to write something that has minimal footprint, ended up with a python wrapper around dig. Of course, this depends on dig (which should come with the plugin installation), as well as python. The python part is optional if you rewrite this in bash.

With regards to RRSIGs and DNSSEC:

RRSIGs = signature records for a zone which contain a cryptographic signature used for validating a DNSSEC response. When you sign a zone, the signature records expire after 30 days. There are recommendations for resigning a zone once a day. This is why the plugin default is to warn if the signature expires in less than 29 days (of course warning and critical thresholds can be changed with command line arguments -w and -c; the server to check against can also be specified with -s):

~./check_dns_rrsig.py cloudflare.com
Signature expires in 1 days

Here is the plugin!

This is very, very basic and will work if all you want is something to tell you when an RRSIG expires, without any other overhead.

Happy monitoring!

November 14, 2018 12:00 AM

November 12, 2018

Ana Custura

Unbound on a fresh FreeBSD install

I’ve recenty encountered a weird problem where unbound would not work on a fresh FreeBSD install on an APU3 board. Online research lead to finding a bunch of posts complaining about this, but no leads. As it turns out, my problem was the clock.

But first things first. Installing FreeBSD on an APU board is very easy, at the boot shell enter the following commands:

>set boot_serial=YES
>set comconsole_speed=115200
>set console=comconsole
#this delays the boot by 10s to allow the USB controller to come up
>kern.cam.boot_delay="10000"

Of course, remembering to drop into a shell at the end of the installation, and edit /boot/loader.conf so that we can use it over the console afterwards:

boot_serial="YES"
comconsole_speed="115200"
console="comconsole"
kern.cam.boot_delay="10000"
amdtemp_load="YES" #this loads the driver for the temperature sensor embedded in the CPU. It is equivalent to the km driver in OpenBSD. Yay sensors!

Now, the APU board does not have an on-board clock. I did not set the time/date during installation (lazy), as NTP was going to be set up anyway. So, at this stage in /etc/ntp.conf, I replaced pool 0.freebsd.pool.ntp.org iburst with the name of my local NTP server.

Reboot after successful installation, cue unbound not working. General DNS failure. As unbound is DNSSEC enabled, run:

unbound-anchor

This sets up the root trust anchor for DNSSEC validation, which was part of the solution to this issue.

But unbound was still not working. Do not underestimate actually checking the results of any commands run:

cat /var/unbound/root.key

The file contained no valid key, thus solving the mistery:

  • DNS lookups do not work as there is no valid key here, due to our system clock being, well, off by a few years.
  • And of course NTP does not sync the date… as we gave it a hostname, which needs to be looked up in DNS.

A quick ntpdate <ip-address> followed by running unbound-anchor again solves the issue.

So frustrating.

November 12, 2018 12:00 AM

November 06, 2018

hibby

Face

Disk Destroyer

I finally fell into the dd trap. I dd’d a FreeBSD Beaglebone black image to /dev/sdb, where my home directory lives. It should have been /dev/mmcblk0, but I trusted tab completion and didn’t check dmesg. After a quick check of gparted, it was obvious what had happened. I’ve nuked the partition table on the drive. Well done. The data in /home/hibby was intact, however. My home directory was still responding, I could load new files ith no corruption.

November 06, 2018 07:00 PM

October 28, 2018

Iain R. Learmonth

Face

OpenBSD with GPS synchronised NTP

I wrote on Monday about how I’ve swapped my home router for an OpenBSD box. One of the fun things I’ve done with this box is configure it as a network time server using ntpd(8).

Synchronising time with servers on the Internet isn’t that exciting, but I’ve had a USB GPS dongle sitting doing nothing for a while. I originally purchased it to use with amateur radio projects but I haven’t done much radio recently. It looks like you can pick these up on eBay for around £11.00 and the newer ones also support GLONASS (mine only does GPS as far as I can tell).

When you attach this to OpenBSD, it is recognised as:

umodem0 at uhub0 port 4 configuration 1 interface 0 "u-blox AG - www.u-blox.com u-blox 6  -  GPS Receiver" rev 1.10/7.03 addr 2
umodem0: data interface 1, has CM over data, has no break
umodem0: status change notification available
ucom0 at umodem0

In Linux, I would use gpsd which is userland software and overkill for what is needed to get time out of the device. Instead, OpenBSD provides the nmea(4) line discipline that can be attached to the serial port with only a single line of configuration in /etc/ttys:

cuaU0   "/sbin/ldattach nmea"   unknown on softcar

Now when I reboot and run sysctl hw.sensors:

hw.sensors.nmea0.indicator0=On (Signal), OK
hw.sensors.nmea0.timedelta0=-0.001002 secs (GPS autonomous), OK, Sun Oct 28 17:08:04.998
hw.sensors.nmea0.angle0=57.3748 degrees (Latitude), OK
hw.sensors.nmea0.angle1=-2.3849 degrees (Longitude), OK

(Update 2019-04-04: You don’t actually have to reboot. One reader has pointed out that you can instead kill -s HUP 1 and init(8) will re-read /etc/ttys.)

This was so incredibly simple I couldn’t believe that was it. If you have ntpd enabled, it will automatically pick up this new sensor too and start using it. I wanted to give a greater weight to my GPS clock than to random servers on the Internet though so I did add a line to /etc/ntpd.conf:

sensor nmea0 weight 5 refid GPS

By default all sources have a weight of 1. Giving the GPS a weight of 5 gives it 5× the influence over the clock compared to servers from pool.ntp.org. The reference seems to be passed to clients but I’m not entirely sure yet what its significance is.

In RFC2132 there is a DHCP option defined for advertising NTP servers, so I advertised the server via this mechanism. Thinking more about it, I’m hoping that nothing is automatically configuring itself using this DHCP option as that seems like something that shouldn’t get reconfigured just because you joined a network. There’s a lot of different options defined in this document and I wonder just how many of them are actually used/useful.

I did spot that I can advertise my RFC868 time server too, so I set up both that and a daytime server using inetd(8) by adding to my /etc/inetd.conf:

daytime stream tcp nowait nobody internal
time stream tcp nowait nobody internal

October 28, 2018 05:00 PM

October 22, 2018

Iain R. Learmonth

Face

OpenBSD with PlusNet VDSL

For a long time we’ve had a Cisco 887VA acting as our VDSL modem and gateway. We got this for the old flat when we moved in there in 2016 and took it with us to where we live now. It’s been quite reliable but the Ethernet interfaces are only 10/100Mbps and there are some limitations to the software where either features are hidden behind additional licenses or they are missing altogether. The software was last updated in 2016 and there’s no easy way to get hold of later firmware.

The replacement for this box is a PC Engines apu3c4. This board has an AMD Embedded G series CPU, 4GB RAM and 3 Gigabit Ethernet interfaces. For storage I have fitted a 16GB mSATA SSD and I’m planning to also fit a 4G modem to allow failover in the event of a broadband outage (as happened once before). This box is running OpenBSD 6.4 which was incredibly easy to install via the serial console, booted from a USB stick.

When the prompt appears after booting from the USB stick:

boot> stty com0 115200
boot> set tty com0
boot> boot

But wait, something is missing here. The Cisco box has a VDSL modem but the APU board does not. The original plan here was to use a Huawei HG612 as the modem and then bridge the connection to the APU board. Unfortunately, I had installed the one I had got for this purpose in the hackerspace, as they now have their sponsored VDSL line installed from Converged. Another hackerspace member had said they would provide me with a replacement, but when I went to install it, it was in fact an ECI Telecom B-FOCus V-2FUb/r Rev.B and not an HG612. These two devices look almost identical if you’re not paying attention. So the temporary solution is to use the Cisco box as a bridge.

The relevant configuration:

no ip routing
interface Ethernet0
 no ip address
end
interface Ethernet0.101
 encapsulation dot1Q 101
 bridge-group 1
end
interface Vlan200
 no ip address
 bridge-group 1
end
interface FastEthernet3
 switchport access vlan 200
 no ip address
end
bridge 1 protocol ieee

Essentially what this is doing is bridging the Ethernet interface that used to be used by the Cisco box for the PPPoE tunnel to FastEthernet3 instead. By connecting a cable between FastEthernet3 and em0 on the APU board (first Gigabit Ethernet port) it is then possible to terminate the PPPoE tunnel on the APU board instead.

In /etc/hostname.em0:

up

In /etc/hostname.pppoe0:

inet 0.0.0.0 255.255.255.255 NONE \
        mtu 1492 \
        pppoedev em0 \
        authproto chap \
        authname 'username@plus.net' \
        authkey 'password' \
        peerproto chap \
        peerflag callin \
        up
dest 0.0.0.1
!/sbin/route add default -ifp pppoe0 0.0.0.1

The man pages for hostname.if(5), em(4) and pppoe(4) can give you more details on what is going on here. If you’re not doing this on an APU board you might find that your Ethernet interface is named differently in OpenBSD. Interface names are based on the driver they are using.

It took me a while to debug this because the PPPoE driver was attempting to authenticate the access concentrator. Adding peerflag callin fixed this:

The `callin’ flag will require the remote peer to authenticate only when he’s calling in, but not when the peer is called by the local client.

Annoyingly this is only described in ifconfig(8) and so I didn’t see it the first time reading through the documentation. Some combination of DuckDuckGo and the online man page browser did get me there in the end.

So now I have an OpenBSD box as my home network’s gateway. I’ve configured dhcpd(8), ntpd(8) (using an nmea(4) GPS device as a time source) and a bunch of other things that may or may not become other blog posts. I’m expecting the replacement HG612 modem will arrive soon and then the Cisco box will probably find its way to eBay.

October 22, 2018 08:30 PM

October 03, 2018

Tom Jones

Face

EuroBSDCon Bucharest Romania

The Wikitravel page for Bucharest has some scary warnings about taxis. I didn't heaer any horror stories from conference goers, but there was a large variation in prices for the same journey.

He held a two day DevSummit before the conference proper. A DevSummit is a chance to talk through technical issues and hash things out face to face. We did some planning for FreeBSD 13 with the idea of setting GGoals for the release.

We tried to match a bit of a hackathon with the DevSummit, but the tutorial schedules meant we couldn't focus the time very well and it was broken up.

EuroBSDCon

Day One:

  • Keynote1: Lightweight virtualization with LightVM and Unikraft
  • Hacking together a FreeBSD presentation streaming box – For as little as possible
    • That was me, I thought it was quite good :D
  • The Evolution of FreeBSD Governance
  • Using Boot Environments at Scale
  • The End of DNS as we know it
  • Keynote2: Some computing and networking historical perspectives
    • Ron's keynote was unreal and it is a massive shame that this sessions wasn't recorded. Ron has a ton of experience with working with network systems since 1976, he shared some stories and anecdotes. The one closest to my heart was pirating away an IMP front pannel and saving it from the scrappers. If you get a chance to see Ron speak you should jump up and down at it.

Day Two:

  • Taking NetBSD kernel bug roast to the next level : Kernel Sanitizers
  • Livepatching FreeBSD kernel
    • This was an interesting study into how many different platforms do live patching. The FreeBSD way to do live patching could be simplified to 'use dtrace fbt probes'. Which is super reductive of all of the work invovled, but it shows the power of the system we have with dtrace.
  • Profiling Packet Processing: Performance & Peculiarities
  • Debugging lessons learned as a newbie fixing NetBSD
    • Maya is a terrifying person. Somehow she manages to hack productivly across the entire range of the stack and across many different architectures. There were many debuggin gems in here, I hope she continues to present on this the information was great.
  • FreeBSD/VPC: a new kernel subsystem for cloud workloads
  • FreeBSD on IBM PowerNV
    • An recount of the porting work Semihalf did to POWER8. Interesting, I hope it is also sumbitted to AsiaBSDCon. There need to be more written account of bringing up on different architectures.

Day Two concluded with announcing the location of the next EuroBSDCon, Lillehammer Norway.

October 03, 2018 12:00 AM

Presentations with mdp

It feels like work is just a constant stream of preparing, travelling for and giving presentations. Brief words and pictures is an excellent for conveying information between small groups of humans. All of these presentations I write in keynote, keynote manages to be light weight, powerful and not horrific to use. As a bonus, my boss feels at home in keynote and is happy to make edits there.

The keynote workflow does not match well to how I think. When dreaming up a presentation I want to shit of a stream of conciousness and have it magically become slides in the right shape.

I might write a series of headings like:

# intro
# who
# meat 
# details
# questions?

I will iterate on these to add bodies, details and more slides.

For quite a while I have wanted a system where I could write plain text and have it become slides. I [wrote][3] about the [sent][4] tool from suckless, but in the end I found it wanting. I have also considered just showing screens of text, but a nightmare DEFCON wireess village talk by Hak5 scared me away. They attempted to just present using just a plain text file and less, but the window size got out of whack and it all fell apart.

Enter mdp

mdp is a terminal presentation program, it takes slides it approximately markdown and takes over the entire terminal as its presentation surface.

Intrigued I used an opportunity to speak at a [local tech event][5] to try out mdp. [The slides][6] from that presentation can be found on [my talks page][7] and overall I thought mdp worked quite well.

I was able to draft in the stream of conciousness style I want, getting the bulk of the slides written very quickly. Adding diagrams required resorting to ASCII art which isn't so bad, I like [ascii][10] [art][11]. mdp worked great in practice, I had to find readable dimensions for the text by trial and error, but overall it went well.

Plain text as a format does have some major downsides, mdp has a way to encode builds for text (see below), but I couldn't use it with my tools. ASCII art diagrams also meant that the builds I did use were eggregious to maintain, any modification required manual propigation through the build chain.

mdp does not support a portable output format. You may say the source markdown is an excellent format for portability, but I find it lacks the crispness of having a single slide in view at once.

I wanted to be able to point at a viewable copy of my slides and so I hacked together some tools to export the mdp presentation to html, but for this I had to sacrifice the built in build mechanism of mdp

Finally there was no way to include images in the mdp presentation let alone the sacride gif format required to correctly convey nyan cat. I played with some terminal graphics viewers, but none of them worked well and after a while I started to think 'what is the point of reinventing everything'.

Drafting the presentation in markdown fit very well with my work flow, but the difficulties in getting a complete presentation with mdp meant that I didn't want to use it for future presentations.

Exporting to html

Getting html of the mdp presentation hinged on a complete hack. There is a tool I had seen in the past that can output a html dump of a tmux session unsurprisingly called [tmux2html][12]. With some playing around I was able to automate a tmux session to work through the slides and use tmux2html to grab each slide as a frame.

Finding the number of slides in the deck required splitting on the slide seperator from the markdown, this ruled out using the built in build mechanism as I would end up with the wrong number of slides.

The output script runs through the markdown to find the number of slides then uses tmux send-keys to control moving through the deck.

#!/bin/sh

set -e 

command -v tmux >/dev/null 2>&1 || { echo >&2 "I require tmux but it's not installed.  Aborting."; exit 1; }
command -v tmux2html >/dev/null 2>&1 || { echo >&2 "I require tmux2html but it's not installed.  Aborting."; exit 1; }
command -v mdp >/dev/null 2>&1 || { echo >&2 "I require mdp but it's not installed.  Aborting."; exit 1; }

if [ -z "$1" ]
  then
    echo "tohtml presentatin.md [outfile.html]"
    exit
fi

file=$1
outfile=outfile.html

if [ ! -z "$2" ]
  then
  outfile=$2
fi

javascript="<script>function page(){var e=!1,n=document.getElementsByClassName('tmux-html'),l=0; document.onkeydown=function(t){if(t=t||window.event,key=t.keyCode,e)if(13==key){e=!1,l=0;for(var i=0;i<n.length;i++)n[i].style.display='inline'}else{37==key&&--l<0&&(l=0),39==key&&++l>=n.length&&(l=n.length-1);for(i=0;i<n.length;i++)n[i].style.display='none';n[l].style.display='inline'}else if(13==key){e=!0,n[0].style.display='inline',l=0;for(i=1;i<n.length;i++)n[i].style.display='none'}}}window.onload=function(){page()};</script>"

tmpfile=tmpfilenamefilething
tmux='mdptohtmlconverstionsession'

slides=`grep -e "^---" $file | wc -l`

tmux new-session -s $tmux -d -x 96 -y 25

tmux send-keys -t $tmux "mdp $file"
tmux send-keys -t $tmux "Enter"

tmux send-keys -t $tmux 'g'
tmux2html -o $tmpfile $tmux 1>/dev/null

# insert javascript
lines=`cat $tmpfile | wc -l`
styleend=`cat -n $tmpfile | grep -e "</style>" | awk '{print \$1}'`
head -n $styleend $tmpfile > $outfile
echo $javascript >> $outfile
tail -n $((lines-styleend)) $tmpfile >> $outfile
mv $outfile $tmpfile

# remove closing tag
lines=`cat $tmpfile | wc -l `
end=`tail -n 1 $tmpfile`
head -n $((lines-1)) $tmpfile > $outfile

echo turning $file into $((slides+1)) slides 

i=1
while [ $i -lt $((slides+1)) ]
do
    printf "\rSlide $i"
    tmux send-keys -t $tmux 'j'

    tmux2html -o $tmpfile $tmux 1>/dev/null 
    grep -e "^<div" $tmpfile >> $outfile
    (( i++ ))
done

echo $end >> $outfile
tmux kill-session -t $tmux 
rm $tmpfile
printf "\rwritten to $outfile \n"

[If you view the presentation page][6] you will see the entire slide deck, this was the first output I got from this script. All the slides in a nice order. After a little pondering I wrote up some javascript to give controls, if you hit enter it will go from all slides to single slide. Arrow keys in single slide mode will allow you to move through the slide deck. The unminified javascript for this is below.

function page() 
{
    var presenting = false
    var elements = document.getElementsByClassName('tmux-html');
    var current = 0;

    document.onkeydown = function(evt) {
        evt = evt || window.event;
        key = evt.keyCode 

        if (presenting) {
            if (key == 13) {
                presenting = false;
                current = 0;
                for (var i = 0; i < elements.length;i++)
                    elements[i].style.display='inline'
            } else {
                if (key == 37) {    //left
                    current--;
                    if (current < 0)
                        current = 0;
                }
                if (key == 39) {    //right
                    current++;
                    if (current >= elements.length)
                        current = elements.length-1;
                }
                for (var i = 0; i < elements.length;i++)
                    elements[i].style.display='none'
                elements[current].style.display='inline'
            }
        } else {
            if (key == 13) {
                presenting = true;
                elements[0].style.display='inline'

                current = 0;
                for (var i = 1; i < elements.length;i++)
                    elements[i].style.display='none'
            }
        }
    };
}

window.onload = function () {
   page();
}

[3]: sent blog post [4]: sent link [5]: techmeetup aberdeen [6]: mdp slides [7]: talks page

[10]: ietf ascii art guidelines [11]: draft cco [12]: tmux to html

October 03, 2018 12:00 AM

Subscriptions

Last updated:
October 20, 2019 10:05 AM
All times are UTC.

Powered by:
Planet

Planetarium: