To any users trying to compile the ModSecurity module for nginx 1.21.5 and up, there are some changes to be made according to this github issue..
The issue is related to a change in nginx (now nginx is built with the PCRE2 library by default).
PCRE2 support must be added to the library (libmodsecurity) and then to the connector. Applying just the connector’s PR will lead to enormous memory leaks in regex processing.
Long story short: use --without-pcre2 configure argument when building ModSecurity-nginx V3 connector module.
So your full module configure line should look like this:
./configure --with-compat --without-pcre2 --add-dynamic-module=/usr/local/src/ModSecurity-nginx