Tighten a bit for ssh, differentiate:
/etc/pf.conf
table <bruteforce> persist block quick from <bruteforce> # tighter for ssh pass quick proto tcp to port ssh \ keep state (max-src-conn 15, max-src-conn-rate 5/3, \ overload <bruteforce> flush global) pass inet proto tcp to $int_if:network port $tcp_services \ keep state (max-src-conn 100, max-src-conn-rate 15/5, \ overload <bruteforce> flush global)