Fixing for the people who really do not get it

For the impatient:

table <localwhite> file "/etc/mail/whitelist.txt"

Disable rdr by putting this at the top of the rdr block:

no rdr proto tcp from <localwhite> to $mailservers port smtp

Edit whitelist.txt, reload rule set or replace <localwhite> table contents using pfctl; all table options available.