Once you've set up IPSec, you can do your filtering on the enc interfaces:
pass on enc0 from $allowedsource to $sechosts port $allowedin pass on enc0 from $myhosts to $remotedest port $remoteports