Set the sizes of memory pools
$ sudo pfctl -sm states hard limit 10000 src-nodes hard limit 10000 frags hard limit 5000 tables hard limit 1000 table-entries hard limit 200000
You can adjust those in pf.conf,
set limit states 25000 set limit table-entries 300000
or
set limit { states 25000, src-nodes 25000, table-entries 300000 }
Note: limited by available kernel memory