There are a couple of data structures involved in pf that need protection. A abstract view of them. They are at the top level:
It is my plan to protect most of those with a "global" read mostly lock, which is very efficient and recurseable! As shown here.
The individual items referenced from these rulesets and tables still need to be protected individually for they contain things like statistic counters that can not be kept in order without proper locking. I intend to use per-rule / per-state mutexes which will be obtained as soon as the match is found. This does not mean, however, that there will be a unique mutex for every rule/state, but rather that every rule/state will be "hashed" to one mutex out of a array of X mutexes - where X will be somewhere in the neighbourhood of MAXCPU-2*MAXCPU, depending on how well we can hash. For rulesets skip steps should be a good indicator of how to devide mutexes. Especially the interface skip step, as this different interfaces are the (only) source of concurrency.
Detail views of this for scrub and normal rulesets are linked here.
The state table is special in that it is far more volatile than the rulesets. A read/write lock will be used to protect the main state table as shown here. This has also implications for the state purge timeout. If done carefully, it is possible to run with a read lock for most of the time. More on this as I find time.
Open issues at this time are:
The reassembly queue(s) need probably be protected by a single mutex in the hope that reassembly is relatively seldom. It's rather difficult to work around this serialization point anyways.
For the recursion all mutexes have to be dropped, but being able to keep the global read mostly lock would be beneficial to keep things in order and make sure the rule/state we are referencing is still available after we come back.
Source tracking is mostly the same as "normal" states, but lock order might be a problem.
Tables already are using the locks obtained from the radix code. This needs to be revisited at some point, but seems okay for the moment. Again lock order is a concern!
The ioctl path won't change much. A global sx lock already protects all memory. i.e. state-, rule- and table-memory is only freed while holding the an exclusive lock. This way we can ensure copyout operations won't fault. Instead of the global mutex we use now, we will pick up the read mostly and (if neccessary) the read write locks for the tables we are interested in.
This is all for the moment, I will try to update this document as I make progress. Feel free to send comments.
--Max
