What every IT person needs to know about OpenBSD

class: center, middle

# What every IT person needs to know about OpenBSD

## How to have fun with the world’s most important free software project

[![OpenBSD 6.9 Puffy](puffy69.jpg)](https://www.openbsd.org/)

This presentation: [home.nuug.no/~peter/openbsd_needtoknow/](https://home.nuug.no/~peter/openbsd_needtoknow/)
---
# The World’s Most Important Free Software Project

"Functional, free and secure by default", [OpenBSD](https://www.openbsd.org/) has been around for 25 years (started October 1995)

* OpenBSD is *[proactively secure](https://www.openbsd.org/security.html#goals)* with only 2 remote holes in default install in 20+ years

* OpenBSD pioneered use of *strong cryptography*, first free system to ship with IPSec (thereby subject to US export control in early years)

* OpenBSD pioneered and is still leading in *[code audit](http://www.openbsd.org/security.html#process)*, fix similar bugs everywhere

* OpenBSD has all security enhancements *[enabled by default](http://www.openbsd.org/security.html#default)*; hard to disable

* OpenBSD is open source, free software and enables *[independent verification](http://www.openbsd.org/security.html#watching)* (should be a strong hint for app developers too)

<div class="footer"><img src="banner1.gif"></div>
---
# The World’s Most Important Free Software Project

* OpenBSD has a high profile quality image based on code quality and real world use

* OpenBSD is upstream (origin) for several widely used pieces of software (OpenSSH, OpenBGPD, PF, OpenSMTPd, LibreSSL, iked, mandoc and others, for complete list see [www.openbsd.org/innovations.html](https://www.openbsd.org/innovations.html))

* OpenBSD has been ‘growing up in public’ with code generally accessible via [anonymous CVS](https://www.openbsd.org/anoncvs.html) (the first of its kind) since 1995 – transparent process, development discussions on public tech@ mailing list

* Developers would do well to study high quality (mainly) C source and how the project runs a 6 month release cycle like clockwork (with only a few notable exceptions).

<div class="footer"><img src="banner1.gif"></div>
---
# whoami (Who am I?)

Peter N. M. Hansteen <peter@bsdly.net>

OpenBSD user since OpenBSD 2.5 (1999)

Security Engineer going on cloud expert -- Unix/Linux etc sysadmin, networker

Wrote [The Book of PF](https://www.nostarch.com/pf3) based on experience and conference lectures (tutorials)

Blog at [bsdly.blogspot.com](https://bsdly.blogspot.com) about (lack of) sanity in IT

Yes, I'll do another book any decade now

---
# OpenBSD: How it all started

OpenBSD's origin story is really the history of the Internet itself.

OpenBSD is descended from 'BSD Unix' - Berkeley Software Distribution

Tthe *Berkeley Software Distribution* aka *BSD* lived at the CSRG, UC Berkeley from 1974 (also Ken Thompson's 1975 sabbatical) -> active development until c 1992

Became over time the complete OS *BSD Unix*, and was the home of the TCP/IP reference implementation

CSRG's DARPA research funding ended c 1992, initial 386BSD (Jolitz et al 1991-1992) spawned [FreeBSD](https://www.freebsd.org/) (BSD on PC architecture) and [NetBSD](https://www.netbsd.org/) (BSD on anything), [BSDI](https://en.wikipedia.org/wiki/Berkeley_Software_Design) (commercial - 1-800-ITS-UNIX generated USL vs BSDI lawsuit, settled 1994)

[OpenBSD](https://www.openbsd.org/) forked from NetBSD 1995 (tree created Oct 18, 1995)

The first project to provide [anonymous, public CVS](http://www.openbsd.org/anoncvs.html) - *See code change in real time!*

<div class="footer"><img src="banner1.gif"></div>
---
# You are already an OpenBSD user!

You may not know it, but you are. OpenBSD code is in

* macOS, iOS
* Blackberry
* Android
* Solaris
* Cisco networking products (and likely others)
* All Linux distributions, all Unixes

... and Microsoft put [OpenSSH](https://www.openssh.com/) in base on recent Windows versions

<div class="footer"><img src="banner1.gif"></div>
---
# OpenBSD: Code audit and security evolution

Early and continuing emphasis on security - code audit started 1995 and still ongoing:

* Assume hostile environment
* Look for unsafe behaviors
* Find one bug, fix similar bugs everywhere in the tree
* *(repeat ...)*

*"Where it is possible to spot damage, fail hard"*

Lead to exploit mitigation techniques (W^X, privsep, ASLR, see eg [www.openbsd.org/papers/ven05-deraadt/index.html](https://www.openbsd.org/papers/ven05-deraadt/index.html) and the 10 years later [www.openbsd.org/papers/ru13-deraadt](https://www.openbsd.org/papers/ru13-deraadt))

First free OS with *strong crypto* in base, illegal to re-export from the US early on(!)

---
# Why use OpenBSD? Proactive security

*All of these have been [enabled by default](http://www.openbsd.org/security.html#default) for 10+ years:*

**Exploit mitigation**

* *Address space randomization* (aka ASLR) no fixed jump targets or gaps

* *W^X* memory can be writable XOR executable

* *Guard pages* 'fence-like' unreadable, unwritable page after [malloc()](http://man.openbsd.org/malloc)ed chunks, detect overruns

* *Privilege separation* daemons run bulk of their code as different non-privileged users (most in chroot without shell) - [sshd](https://man.openbsd.org/sshd) was the first, the rest followed

* *Privilege revocation* privsep'd daemons drop privilege as soon as possible

<div class="footer"><img src="banner1.gif"></div>
---
# Why use OpenBSD? Proactive security (cont)

**Exploit mitigation (cont)**

* *chroot* jail -- daemons run in restricted environment ($HOME /var/empty, no shell)

* *ProPolice* random stack gap inserted, fixed returns fail

Newer developments include:

* OpenBSD 5.9 introduced *[pledge(2)](https://man.openbsd.org/pledge)* syscall to restrict program behavior to predeclared profile

* OpenBSD 6.4 introduced *[unveil(2)](https://man.openbsd.org/unveil)* syscall to restrict file system access to predeclared profile

* OpenBSD 6.2 introduced *KARL* (kernel address randomized link) - kernel relinked with randomized layout for each boot, see the [undeadly.org article](http://undeadly.org/cgi?action=article&sid=20170613041706) or the [tech@ message](https://marc.info/?l=openbsd-tech&m=149732026405941) (kernel object files grew base by ~300MB)

<div class="footer"><img src="banner1.gif"></div>
---
# OpenBSD: Usable, Portable and secure

*Secure, correct code* that runs on 14 hardware [platforms](https://www.openbsd.org/plat.html)

*Usable system* - sane defaults, complete and readable documentation ([man pages](https://man.openbsd.org/) -- commit new features w/o matching man page not allowed -- plus [FAQ](https://www.openbsd.org/faq/) and other guides)

*Free to use* - short (2 paragraph) BSD license preferred (see [www.openbsd.org/policy.html](https://www.openbsd.org/policy.html)), also see [www.openbsd.org/goals.html](https://www.openbsd.org/goals.html)

By a small team: Approx 100 active developers (historically '~356 hackers'), see [marc.info/?l=openbsd-misc&m=144515087006177&w=2](http://marc.info/?l=openbsd-misc&m=144515087006177&w=2)

Coordinated by a Canadian in Canada, US export restrictions do not apply.

*Release every six months* (May and November), most recently [OpenBSD 6.9](https://www.openbsd.org/69.html) May 1, 2021 (50th release, 25 years), with [OpenBSD 7.0](https://www.openbsd.org/69.html) slated for November 1, 2021

Emphasis on security and usability (yes, devs run it on their laptops).

<div class="footer"><img src="banner1.gif"></div>
---
# Why use OpenBSD?

It's a UNIX. A real UNIX (no **systemd**)

Uncluttered base system - full install w/X, compilers etc is approx 962MB total on amd64

No services enabled (listening) by default. Installer asks if you want to run [sshd(8)](https://man.openbsd.org/sshd)

Full featured system (c, c++ (clang or gcc), perl and tools in base)

Supports modern to fairly ancient hardware across [14 platforms](https://www.openbsd.org/plat.html) (alpha, amd64, arm64, armv7, hppa, i386, landisk, loongson, luna88k, macppc, octeon, powerpc64, riscv64, sparc64)

<div class="footer"><img src="banner1.gif"></div>
---
# OpenBSD: Ported software goes under /usr/local

Ported software in package system (*11310* packages prebuilt on amd64 for [OpenBSD 6.9](https://www.openbsd.org/69.html)), mips64 had 8182 and mips64el is still building.

Clear separation of base system and packages. Use of packages encouraged, but if you want to build from checked out **ports** source (make install), you can.

See [You've Installed It. Now What? Packages!](https://bsdly.blogspot.com/2013/04/youve-installed-it-now-what-packages.html).

<div class="footer"><img src="banner1.gif"></div>
---
# The installer was always good, got better

I came to [OpenBSD](https://www.openbsd.org/) from mainly Linux and FreeBSD 20+ years back.

Came for the security focus, ordered the 2.5 CD set and the [wireframe daemon shirt](https://www.bsdly.net/~peter/blogpix/tshirt-2.jpg).

Installed at first, *experimentally*, on even-then-crappy hardware (80386/33MHz, 8MB RAM, 100MB IDE disk).

*The thing worked!*

*Everything made sense!*

*Everything had a (readable) man page!*

I later heard about "no commit without documentation" rule + anonymous CVS -- *follow commits in real time!*
<div class="footer"><img src="banner1.gif"></div>
---
# The installer got better

After all those years, the installer is still stubbornly *text-only*, portably *the same across [hardware platforms](http://www.openbsd.org/plat.html)*, and has spawned

* repeatable, scriptable ([autoinstall(8)](https://man.openbsd.org/autoinstall) from [OpenBSD 5.5](https://www.openbsd.org/55.html))

* [sysupgrade(8)](https://man.openbsd.org/sysupgrade) from [OpenBSD 6.6](https://www.openbsd.org/66.html)) for (almost) hands-off upgrades snapshot to snapshot or one release to the next

and still incremental improvements (of course) in almost every release.

<div class="footer"><img src="banner1.gif"></div>
---
# Something for the laptop: hardware support

I got myself a new laptop late May 2021, and went from **"Oh sh*t, the SSD isn't recognized"** followed by

To **full support of everything** -- Intel Core 11th gen evo -- within days, and will be in [OpenBSD 7.0](https://www.openbsd.org/70.html)

I *did* run kernels from jsg@'s drm510 branch for a couple of weeks, then back to regular [sysupgrade](https://man.openbsd.org/sysupgrade)s.

The SSD thing was down to a BIOS option -- **pseudo-RAID on a single device does not make sense**

The gory details are at [bsdly.blogspot.com](https://bsdly.blogspot.com/2021/07/the-impending-doom-of-your-operating.html) or the .no IT mag [digi.no](https://www.digi.no/artikler/debatt-nylige-og-ikke-helt-nylige-endringer-i-openbsd-som-gjor-livet-bedre/512893) (as a 3 part series in *Norwegian*)

<div class="footer"><img src="banner1.gif"></div>
---
# Why OpenBSD? IPSEC

OpenBSD pioneered IPSEC in general, with IPSEC in the base system since OpenBSD 2.1 (1997)

Major usability upgrade in OpenBSD 3.8 – [ipsecctl(8)](https://man.openbsd.org/ipsecctl) and [/etc/ipsec.conf](https://man.openbsd.org/ipsec.conf) -

```shell 
        # Set up two flows:
        # First between the machines 192.168.3.14 and 192.168.3.100
        # Second between the networks 192.168.7.0/24 and 192.168.8.0/24
        flow esp from 192.168.3.14 to 192.168.3.100
        flow esp from 192.168.7.0/24 to 192.168.8.0/24 peer 192.168.3.12
```

(Compare with others, eg Microsoft’s 36 dialogs and counting - [www.openbsd.org/papers/asiabsdcon07-ipsec/index.html](https://www.openbsd.org/papers/asiabsdcon07-ipsec/index.html))

IKE v2 support in [iked(8)](https://man.openbsd.org/iked) ([OpenIKED](https://www.openiked.org/))

*«IPSEC shouldn’t be this hard. The defaults should make sense.»*

*Bonus: [ikectl(8)](https://man.openbsd.org/ikectl) generates config for Windows and macOS clients too*

<div class="footer"><img src="banner1.gif"></div>
---
# The thing that lured me in

In 2001, I was longing for something saner than Linux' *iptables*, still only experimenting with OpenBSD and IPF.

Then the IPF license was not *actually* free, needed to be replaced.

With [OpenBSD 3.0](https://www.openbsd.org/30.html). PF was born, a '**working prototype**' that performed better than IPF.

That had me dive in, starting the process that had me produce [The Book of PF](https://nostarch.com/pf3) and numerous [blog posts](https://bsdly.blogspot.com/). More about that later.

The IPF incident lead to a *license audit*, summed up by **deraadt@** on [misc@](https://marc.info/?l=openbsd-misc&m=104570938124454&w=2): Lots of problems fixed, including no license or no copyright.

The relevant developers were tracked down, a frequent response was

### "Say what? Are people still using this?"
<div class="footer"><img src="banner1.gif"></div>
---
# SSH, open and better

PF was written from scratch to replace non-free code. But it was not the first *nonlibreectomy* performed by the OpenBSD project.

The by the late 1990s, the original SSH was turning commercial and proprietary.

OpenBSD developers dug out the last free version, re-implemented more modern features and got rid of bugs.

[OpenSSH](https://www.openssh.com/) debuted in [OpenBSD 2.6](https://www.openbsd.org/26.html). It soon grew a *-portable* variant which has been widely ported.
In recent years, OpenSSH has a steady high-ninteties percent market share.

With time, *telnetd* became irrelevant and was removed in [OpenBSD 3.8](https://www.openbsd.org/38.html).

OpenSSH was the first daemon to become *privilege separated* in an overhaul up to [OpenBSD 3.2](https://www.openbsd.org/32.html).
<div class="footer"><img src="banner1.gif"></div>
---
# And yes, that packet filter

I must confess:

- PF has been an important part of my life since the early noughties

- I have contributed to the (popular, wrong) perception that OpenBSD is mainly a firewall OS

But lots of *useful* and *fun* tools and features sprung from PF, some were ported or imitated elsewhere, some remain OpenBSD only.

My favorites follow.
<div class="footer"><img src="banner1.gif"></div>
---
# Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3

I was already a mail server admin back when I found OpenBSD and PF.

When [OpenBSD 3.3](https://www.openbsd.org/33.html) dropped with [spamd(8)](https://man.openbsd.org/spamd) the spam deferral daemon, featuring blocklists and stuttering (1 byte per second), I loved the idea.

Just minor surgery to [pf.conf](https://man.openbsd.org/pf.conf) and spamd config, and the FreeBSD mail server with spamassassin and clamav grew noticeably quieter.
<div class="footer"><img src="banner1.gif"></div>
---
# Going grey, then trapping

The next big thing in [spamd(8)](https://man.openbsd.org/spamd) was a greylisting mode. The mail server's fans grew even quieter.

The came **greytrapping**: any host that tries to deliver to *known bad*, non existent addresses in our domains are added to the blocklist, get the 10 bytes per sec treatment for 24 hours.

I do my own trapping:

* harvest spamtraps from bounce entries in mail server logs
* by July 2007, I started to [publish](https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html) both the *trapped* IP addresses and the *spamtraps*.

*Trapped* IP addresses typically in the 3000 to 5000 range, exported once per hour, has been past the 20' mark

Bizarrely, number of *spamtraps* is now over 270', still growing. It's absurd but fun, see [Badness, Enumerated by Robots](https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html) for details.
<div class="footer"><img src="banner1.gif"></div>
---
# The brutes, the password gropers and the state tracking options

If you run any internet facing service with a login option, your logs will contain noise from *password guessing*, aka *password gropers*.

OpenBSD 3.6-current introduced *state tracking options*: act on state data, set and enforce limits by actions, such as

```
table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
        flags S/SA keep state \
	(max-src-conn 100, max-src-conn-rate 15/5, \
         overload <bruteforce> flush global)
```
=> more than 100 connections *or* more than 15 new over 5 seconds -> added to the table, blocked and connections cut.
<div class="footer"><img src="banner1.gif"></div>
---
# State tracking, generally  do expire

Useful for other services too. A few examples for inspiration can be found in the article [Forcing the password gropers through a smaller hole with OpenBSD's PF queues](https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html).

**Do** remember to *expire* entries after a while -- I now favor a few weeks over the classical 24 hours.

Like *greytrapping*, this lets you build a configuration that *adapts* to network conditions and *learns* from the traffic it sees.

The *buzzwordability* potential is enormous, I'm puzzled as to why none of the other **big names** imitated this and marketed to the max.

---
# We went to modern queueing

Traffic shaping in OpenBSD used to be *ALTQ*, code labeled experimental for 15 years, several different algorithms, the syntax was '*inelegant at best*'-

Replaced with a not-small diff in [OpenBSD 5.5](https://www.openbsd.org/55.html), simpler scheme w/ priorities + one base algorithm, *Hierarchical Fair Service Curve* (HFSC), saner syntax:

```
queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K max 18M default
             queue qweb parent main bandwidth 9600K min 6000K max 18M
             queue qpri parent main bandwidth 700K min 100K max 1200K
             queue qdns parent main bandwidth 200K min 12K burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300
```

Queue assignment with *match* or *pass* rules. (Yes, I *do* punish spammers extra here).

Triggered the third edition of [The Book of PF](https://nostarch.com/pf3), with new and old plus conversion tips.

*ALTQ* lives on in FreeBSD + NetBSD, removed from OpenBSD in [OpenBSD 5.6](https://www.openbsd.org/56.html).
<div class="footer"><img src="banner1.gif"></div>

---
# pflow(4) offers network insights lite

If you run a network, you will at times need to look into what the traffic there looks like.

If nothing else to see which endpoints are involved, the amount of data transferred, which protocols and so forth.

For some purposes just the metadata is all you need, and that's where *NetFlow* or *IPFIX* comes in.

In [OpenBSD 4.5](https://www.openbsd.org/45.html), the tools for a Netflow *sensor* were introduced with the [pflow(4)](https://man.openbsd.org/pflow) virtual network device and the **pflow** state tracking option.

Set up the interface, add the pflow option to your rules, and collect with tools such as [nfsen](http://nfsen.sourceforge.net/) or [flow-tools](https://code.google.com/archive/p/flow-tools/) in packages.

My story of a noisy network and how pflow helped is [Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen](https://bsdly.blogspot.com/2014/02/yes-you-too-can-be-evil-network.html).
<div class="footer"><img src="banner1.gif"></div>
---
# LibreSSL, the great deobfuscation

The [Heartbleed](https://en.wikipedia.org/wiki/Heartbleed) bug was **not** the reason why [LibreSSL](https://www.libressl.org) now exists.

OpenBSD developers had felt physical agony at reading the OpenSSL code for years.

The pain levels reached a critical threshold ("Why, oh *why* is sh*t code like that in our tree?")

A few core OpenBSD hackers decided a few weeks before Heartbleed to fork the OpenSSL code and [*flense*](https://en.wikipedia.org/wiki/Flensing) the code of bugs and misfeatures.

At first, unfixed bugs from **openssl.org**'s request tracker.

In parallell, **jsing@** reformatted the code to less eye-stinging form so the thing was approaching readable.
<div class="footer"><img src="banner1.gif"></div>
---
# LibreSSL, the great deobfuscation

With readable code, the digging turned up

* Code was never deleted, no matter how obsolete or irrelevant

* bypassed malloc, wrote their own instead, never did free memory but reused it LIFO, and stuck private data in logs

* written in "OpenSSL C", according to **beck@** a dialect of "worst common denominator".

Details to be found in Bob Beck's BSDCan talk on the first 30 days of LibreSSL (the start of code flensing)

The default TLS library since [OpenBSD 5.6](https://www.openbsd.org/56.html). Offers a **-portable** variant for porting.

In my experience, *"Just Works™"*, likely a healthier alternative than its predecessor.
<div class="footer"><img src="banner1.gif"></div>

---
# This was my list of life improving OpenBSD events - I'd love to hear yours

As I warned earlier, this has been about *my* personal list of OpenBSD events that I remember fondly.

I am sure *your* list is at least a little different.

I am sure there are things from the [innovations](https://www.openbsd.org/innovations.html) page that I have simply forgotten about.

I would love to hear about your favorite [OpenBSD](https://www.openbsd.org/) moments.

<div class="footer"><img src="banner1.gif"></div>
---
# More items for your OpenBSD reading

[www.openbsd.org](https://www.openbsd.org/)  The official OpenBSD website – to donate: [www.openbsd.org/donations.html](https://www.openbsd.org/donations.html) and please do donate, corporates may prefer [www.openbsdfoundation.org](https://www.openbsdfoundation.org/)  - a Canadian non-profit

[undeadly.org](https://undeadly.org/)  - The OpenBSD Journal news site

[bsdly.blogspot.com](https://bsdly.blogspot.com/)  - My rant^H^H^H^Hblog posts

[flak.tedunangst.com/](https://flak.tedunangst.com/)  tedu@ on developments

Michael W Lucas: [Absolute OpenBSD, 2nd ed.](https://www.nostarch.com/openbsd2e)

Peter N. M. Hansteen: [The Book of PF, 3rd ed.](https://www.nostarch.com/pf3)

Henning Brauer: [OpenBSD sucks (… least)](https://quigon.bsws.de/papers/2015/eurobsdcon/)

This presentation: [home.nuug.no/~peter/openbsd_needtoknow/](https://home.nuug.no/~peter/openbsd_needtoknow/)

[Recent and not so recent changes in OpenBSD that make life better (and may turn up elsewhere too)](https://bsdly.blogspot.com/2021/08/recent-and-not-so-recent-changes-in.html) - full text article

---

class: center
# Thank you!

<div align="center"><img src="tshirt-4.gif"></div>
 
 .

Full text of this talk: [What every IT person needs to know about OpenBSD](https://bsdly.blogspot.com/2021/09/what-every-it-person-needs-to-know.html)