=======
vmod_re
=======

-------------------------------------------------------------------------
Varnish Module for Regular Expression Matching with Subexpression Capture
-------------------------------------------------------------------------

DESCRIPTION
===========

.. _regsub(): https://varnish-cache.org/docs/trunk/reference/vcl.html#regsub-str-regex-sub

.. _regsuball(): https://varnish-cache.org/docs/trunk/reference/vcl.html#regsuball-str-regex-sub

.. _beresp.filters: https://varnish-cache.org/docs/trunk/reference/vcl-var.html#beresp-filters

.. _resp.filters: https://varnish-cache.org/docs/trunk/reference/vcl-var.html#resp-filters

Varnish Module (VMOD) for matching strings against regular expressions,
and for extracting captured substrings after matches.

Regular expression matching as implemented by the VMOD is equivalent
to VCL's infix operator ``~``. The VMOD is motivated by the fact that
backreference capture in standard VCL requires verbose and suboptimal
use of the `regsub()`_ or `regsuball()`_ functions. For example, this
common idiom in VCL captures a string of digits following the
substring ``"bar"`` from one request header into another::

	sub vcl_recv {
		if (req.http.Foo ~ "bar\d+")) {
		   set req.http.Baz = regsub(req.http.Foo,
                                             "^.*bar(\d+).*$", "\1");

		}
	}

It requires two regex executions when a match is found, the second one
less efficient than the first (since it must match the entire string
to be replaced while capturing a substring), and is just cumbersome.

The equivalent solution with the VMOD looks like this::

	import re;

	sub vcl_init {
		new myregex = re.regex("bar(\d+)");
	}

	sub vcl_recv {
		if (myregex.match(req.http.Foo)) {
		   set req.http.Baz = myregex.backref(1);
		}
	}

For an example on body matching, see `xregex.match_body()`_.

The object is created at VCL initialization with the regex containing
the capture expression, only describing the substring to be
matched. When a match with the ``match`` or ``match_body`` method
succeeds, then a captured string can be obtained from the ``backref``
method.

Calls to the ``backref`` method refer back to the most recent call to ``match``
or ``match_body`` for the same object in the same task scope; that is, in the
same client or backend context. For example if ``match`` is called for an object
in one of the ``vcl_backend_*`` subroutines and returns ``true``, then
subsequent calls to ``backref`` in the same backend scope extract substrings
from the matched substring. For an unsuccessful match, all back references are
cleared.

By setting the ``asfilter`` parameter to true, a regex object can also
be configured to add a filter for performing substitutions on
bodies. See `xregex.substitute_match()`_ for details and examples.

The VMOD also supports dynamic regex matching with the ``match_dyn``
and ``backref_dyn`` functions::

	import re;

	sub vcl_backend_response {
		if (re.match_dyn(beresp.http.Bar + "(\d+)",
		                      req.http.Foo)) {
		   set beresp.http.Baz = re.backref_dyn(1);
		}
	}

In ``match_dyn``, the regex in the first argument is compiled when it
is called, and matched against the string in the second
argument. Subsequent calls to ``backref_dyn`` extract substrings from
the matched string for the most recent successful call to
``match_dyn`` in the same task scope.

As with the constructor, the regex argument to ``match_dyn`` should
contain any capturing expressions needed for calls to ``backref_dyn``.

``match_dyn`` makes it possible to construct regexen whose contents
are not fully known until runtime, but ``match`` is more efficient,
since it re-uses the compiled expression obtained at VCL
initialization. So if you are matching against a fixed pattern that
never changes during the lifetime of VCL, use ``match``.

REQUIREMENTS
============

The VMOD requires the Varnish since version 6.0.0 or the master
branch. See the project repository for versions that are compatible
with other versions of Varnish.

INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the project source repository.

LIMITATIONS
===========

The VMOD allocates memory for captured subexpressions from Varnish
workspaces, whose sizes are determined by the runtime parameters
``workspace_backend``, for calls within the ``vcl_backend_*``
subroutines, and ``workspace_client``, for the other VCL subs. The
VMOD copies the string to be matched into the workspace, if it's not
already in the workspace, and also uses workspace to save data about
backreferences.

For typical usage, the default workspace sizes are probably enough;
but if you are matching against many, long strings in each client or
backend context, you might need to increase the Varnish parameters for
workspace sizes. If the VMOD cannot allocate enough workspace, then a
``VCL_error`` message is emitted, and the match methods as well as
``backref`` will fail. (If you're just using the regexen for matching
and not to capture backrefs, then you might as well just use the
standard VCL operators ``~`` and ``!~``, and save the workspace.)

``backref`` can extract up to 10 subexpressions, in addition to the
full expression indicated by backref 0. If a ``match`` or
``match_dyn`` operation would have resulted in more than 11 captures
(10 substrings and the full string), then a ``VCL_Error`` message is
emitted to the Varnish log, and the captures are limited to 11.

SEE ALSO
========

* varnishd(1)
* vcl(7)
* pcre(3)
* source repository: https://code.uplex.de/uplex-varnish/libvmod-re


COPYRIGHT
=========

::

  Copyright 2014-2023 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  This document is licensed under the same conditions as the libvmod-re
  project. See LICENSE for details.
 
  Authors: Geoffrey Simmons <geoffrey.simmons@uplex.de>
           Nils Goroll <nils.goroll@uplex.de>
