CL-PPCRE - Portable Perl-compatible regular expressions for Common Lisp

 

Abstract

CL-PPCRE is a portable regular expression library for Common Lisp which has the following features:

• It is compatible with Perl.
• It is pretty fast.
• It is portable between ANSI-compliant Common Lisp implementations.
• It is thread-safe.
• In addition to specifying regular expressions as strings like in Perl you can also use S-expressions.
• It comes with a BSD-style license so you can basically do with it whatever you want.

CL-PPCRE has been used successfully in various applications like BioBike, clutu, LoGS, CafeSpot, Eboy, or The Regex Coach.

Download shortcut: http://weitz.de/files/cl-ppcre.tar.gz.

Contents

1. Download and installation
2. Support and mailing lists
3. The CL-PPCRE dictionary
   1. Scanning
      1. create-scanner (for Perl regex strings)
      2. create-scanner (for parse trees)
      3. scan
      4. scan-to-strings
      5. register-groups-bind
      6. do-scans
      7. do-matches
      8. do-matches-as-strings
      9. do-register-groups
      10. all-matches
      11. all-matches-as-strings
   2. Splitting and replacing
      1. split
      2. regex-replace
      3. regex-replace-all
   3. Modifying scanner behaviour
      1. *property-resolver*
      2. parse-tree-synonym
      3. define-parse-tree-synonym
      4. *regex-char-code-limit*
      5. *use-bmh-matchers*
      6. *optimize-char-classes*
      7. *allow-quoting*
      8. *allow-named-registers*
   4. Miscellaneous
      1. parse-string
      2. create-optimized-test-function
      3. quote-meta-chars
      4. regex-apropos
      5. regex-apropos-list
   5. Conditions
      1. ppcre-error
      2. ppcre-invocation-error
      3. ppcre-syntax-error
      4. ppcre-syntax-error-string
      5. ppcre-syntax-error-pos
4. Unicode properties
   1. unicode-property-resolver
5. Filters
6. Compatibility with Perl
   1. Empty strings instead of undef in $1, $2, etc.
   2. Strange scoping of embedded modifiers
   3. Inconsistent capturing of $1, $2, etc.
   4. Captured groups not available outside of look-aheads and look-behinds
   5. Alternations don't always work from left to right
   6. Different names for Unicode properties
   7. "\r" doesn't work with MCL
   8. What about "\w"?
7. Bugs and problems
   1. "\Q" doesn't work, or does it?
   2. Backslashes may confuse you...
8. AllegroCL compatibility mode
9. Hints, comments, performance considerations
10. Acknowledgements

Download and installation

CL-PPCRE comes with a system definition for ASDF and you compile and load it in the usual way. There are no dependencies (except that the test suite which is not needed for normal operation depends on FLEXI-STREAMS).

CL-PPCRE is integrated into the package/port systems of Debian, Gentoo, and FreeBSD, but before you install it from there, you should check if they actually offer the latest release. Installation via ASDF-Install should as well be possible.

You can run a test suite which tests most aspects of the library with

(asdf:oos 'asdf:test-op :cl-ppcre)

Luís Oliveira maintains a darcs repository of CL-PPCRE at http://common-lisp.net/~loliveira/ediware/.
 

Support and mailing lists

If you want to send patches, please read this first.
 

The CL-PPCRE dictionary

Scanning

[Method]
create-scanner (string string)&key case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive => scanner, register-names

Accepts a string which is a regular expression in Perl syntax and returns a closure which will scan strings for this regular expression. The second value is only returned if *ALLOW-NAMED-REGISTERS* is true. It represents a list of strings mapping registers to their respective names - the first element stands for first register, the second element for second register, etc. You have to store this value if you want to map a register number to its name later as scanner doesn't capture any information about register names. If a register isn't named, it has NIL as its name.

The mode keyword arguments are equivalent to the "imsx" modifiers in Perl. The destructive keyword will be ignored.

The function accepts most of the regex syntax of Perl 5.8 as described in man perlre including extended features like non-greedy repetitions, positive and negative look-ahead and look-behind assertions, "standalone" subexpressions, and conditional subpatterns. The following Perl features are (currently) not supported:

• (?{ code }) and (??{ code }) because they obviously don't make sense in Lisp.
• \N{name} (named characters), \x{263a} (wide hex characters), \l, \u, \L, and \U because they're actually not part of Perl's regex syntax - but see CL-INTERPOL.
• \X (extended Unicode), and \C (single character). But you can of course use all characters supported by your CL implementation.
• Posix character classes like [[:alpha]]. Use Unicode properties instead.
• \G for Perl's pos() because we don't have it.

Note, however, that \t, \n, \r, \f, \a, \e, \033 (octal character codes), \x1B (hexadecimal character codes), \c[ (control characters), \w, \W, \s, \S, \d, \D, \b, \B, \A, \Z, and \z are supported.

Since version 0.6.0, CL-PPCRE also supports Perl's \Q and \E - see *ALLOW-QUOTING* below. Make sure you also read the relevant section in "Bugs and problems."

Since version 1.3.0, CL-PPCRE offers support for AllegroCL's (?<name>"<regex>") named registers and \k<name> back-references syntax, have a look at *ALLOW-NAMED-REGISTERS* for details.

Since version 2.0.0, CL-PPCRE supports named properties (\p and \P), but only the long form with braces is supported, i.e. \p{Letter} and \p{L} will work while \pL won't.

The keyword arguments are just for your convenience. You can always use embedded modifiers like "(?i-s)" instead.

[Method]
create-scanner (function function)&key case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive => scanner

In this case function should be a scanner returned by another invocation of CREATE-SCANNER. It will be returned as is. You can't use any of the keyword arguments because the scanner has already been created and is immutable.

[Method]
create-scanner (parse-tree t)&key case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive => scanner, register-names

This is similar to CREATE-SCANNER for regex strings above but accepts a parse tree as its first argument. A parse tree is an S-expression conforming to the following syntax:

• Every string and character is a parse tree and is treated literally as a part of the regular expression, i.e. parentheses, brackets, asterisks and such aren't special.
• The symbol :VOID is equivalent to the empty string.
• The symbol :EVERYTHING is equivalent to Perl's dot, i.e it matches everything (except maybe a newline character depending on the mode).
• The symbols :WORD-BOUNDARY and :NON-WORD-BOUNDARY are equivalent to Perl's "\b" and "\B".
• The symbols :DIGIT-CLASS, :NON-DIGIT-CLASS, :WORD-CHAR-CLASS, :NON-WORD-CHAR-CLASS, :WHITESPACE-CHAR-CLASS, and :NON-WHITESPACE-CHAR-CLASS are equivalent to Perl's special character classes "\d", "\D", "\w", "\W", "\s", and "\S" respectively.
• The symbols :START-ANCHOR, :END-ANCHOR, :MODELESS-START-ANCHOR, :MODELESS-END-ANCHOR, and :MODELESS-END-ANCHOR-NO-NEWLINE are equivalent to Perl's "^", "$", "\A", "\Z", and "\z" respectively.
• The symbols :CASE-INSENSITIVE-P, :CASE-SENSITIVE-P, :MULTI-LINE-MODE-P, :NOT-MULTI-LINE-MODE-P, :SINGLE-LINE-MODE-P, and :NOT-SINGLE-LINE-MODE-P are equivalent to Perl's embedded modifiers "(?i)", "(?-i)", "(?m)", "(?-m)", "(?s)", and "(?-s)". As usual, changes applied to modes are kept local to the innermost enclosing grouping or clustering construct.
• All other symbols will signal an error of type PPCRE-SYNTAX-ERROR unless they are defined to be parse tree synonyms.
• (:FLAGS {<modifier>}*) where <modifier> is one of the modifier symbols from above is used to group modifier symbols. The modifiers are applied from left to right. (This construct is obviously redundant. It is only there because it's used by the parser.)
• (:SEQUENCE {<parse-tree>}*) means a sequence of parse trees, i.e. the parse trees must match one after another. Example: (:SEQUENCE #\f #\o #\o) is equivalent to the parse tree "foo".
• (:GROUP {<parse-tree>}*) is like :SEQUENCE but changes applied to modifier flags (see above) are kept local to the parse trees enclosed by this construct. Think of it as the S-expression variant of Perl's "(?:<pattern>)" construct.
• (:ALTERNATION {<parse-tree>}*) means an alternation of parse trees, i.e. one of the parse trees must match. Example: (:ALTERNATION #\b #\a #\z) is equivalent to the Perl regex string "b|a|z".
• (:BRANCH <test> <parse-tree>) is for conditional regular expressions. <test> is either a number which stands for a register or a parse tree which is a look-ahead or look-behind assertion. See the entry for (?(<condition>)<yes-pattern>|<no-pattern>) in man perlre for the semantics of this construct. If <parse-tree> is an alternation is must enclose exactly one or two parse trees where the second one (if present) will be treated as the "no-pattern" - in all other cases <parse-tree> will be treated as the "yes-pattern".
• (:POSITIVE-LOOKAHEAD|:NEGATIVE-LOOKAHEAD|:POSITIVE-LOOKBEHIND|:NEGATIVE-LOOKBEHIND <parse-tree>) should be pretty obvious...
• (:GREEDY-REPETITION|:NON-GREEDY-REPETITION <min> <max> <parse-tree>) where <min> is a non-negative integer and <max> is either a non-negative integer not smaller than <min> or NIL will result in a regular expression which tries to match <parse-tree> at least <min> times and at most <max> times (or as often as possible if <max> is NIL). So, e.g., (:NON-GREEDY-REPETITION 0 1 "ab") is equivalent to the Perl regex string "(?:ab)??".
• (:STANDALONE <parse-tree>) is an "independent" subexpression, i.e. (:STANDALONE "bar") is equivalent to the Perl regex string "(?>bar)".
• (:REGISTER <parse-tree>) is a capturing register group. As usual, registers are counted from left to right beginning with 1.
• (:NAMED-REGISTER <name> <parse-tree>) is a named capturing register group. Acts as :REGISTER, but assigns <name> to a register too. This <name> can be later referred to via :BACK-REFERENCE. Names are case-sensitive and don't need to be unique. See *ALLOW-NAMED-REGISTERS* for details.
• (:BACK-REFERENCE <ref>) is a back-reference to a register group. <ref> is a positive integer or a string denoting a register name. If there are several registers with the same name, the regex engine tries to successfully match at least of them, starting with the most recently seen register continuing to the least recently seen one, until a match is found. See *ALLOW-NAMED-REGISTERS* for more information.
• (:PROPERTY|:INVERTED-PROPERTY <property>) is a named property (or its inverse) with <property> being a function designator or a string which must be resolved by *PROPERTY-RESOLVER*.
• (:FILTER <function> &optional <length>) where <function> is a function designator and <length> is a non-negative integer or NIL is a user-defined filter.
• (:REGEX <string>) where <string> is an embedded regular expression in Perl syntax.
• (:CHAR-CLASS|:INVERTED-CHAR-CLASS {<item>}*) where <item> is either a character, a character range, a named property (see above), or a symbol for a special character class (see above) will be translated into a (one character wide) character class. A character range looks like (:RANGE <char1> <char2>) where <char1> and <char2> are characters such that (CHAR<= <char1> <char2>) is true. Example: (:INVERTED-CHAR-CLASS #\a (:RANGE #\D #\G) :DIGIT-CLASS) is equivalent to the Perl regex string "[^aD-G\d]".

Because CREATE-SCANNER is defined as a generic function which dispatches on its first argument there's a certain ambiguity: Although strings are valid parse trees they will be interpreted as Perl regex strings when given to CREATE-SCANNER. To circumvent this you can always use the equivalent parse tree (:GROUP <string>) instead.

Note that CREATE-SCANNER doesn't always check for the well-formedness of its first argument, i.e. you are expected to provide correct parse trees.

The usage of the keyword argument extended-mode obviously doesn't make sense if CREATE-SCANNER is applied to parse trees and will signal an error.

If destructive is not NIL (the default is NIL), the function is allowed to destructively modify parse-tree while creating the scanner.

If you want to find out how parse trees are related to Perl regex strings, you should play around with PARSE-STRING:

* (parse-string "(ab)*")
(:GREEDY-REPETITION 0 NIL (:REGISTER "ab"))

* (parse-string "(a(b))")
(:REGISTER (:SEQUENCE #\a (:REGISTER #\b)))

* (parse-string "(?:abc){3,5}")
(:GREEDY-REPETITION 3 5 (:GROUP "abc"))
;; (:GREEDY-REPETITION 3 5 "abc") would also be OK

* (parse-string "a(?i)b(?-i)c")
(:SEQUENCE #\a
 (:SEQUENCE (:FLAGS :CASE-INSENSITIVE-P)
  (:SEQUENCE #\b (:SEQUENCE (:FLAGS :CASE-SENSITIVE-P) #\c))))
;; same as (:SEQUENCE #\a :CASE-INSENSITIVE-P #\b :CASE-SENSITIVE-P #\c)

* (parse-string "(?=a)b")
(:SEQUENCE (:POSITIVE-LOOKAHEAD #\a) #\b)

For the rest of the dictionary, regex can always be a string (which is interpreted as a Perl regular expression), a parse tree, or a scanner created by CREATE-SCANNER. The start and end keyword parameters are always used as in SCAN.

[Generic Function]
scan regex target-string &key start end => match-start, match-end, reg-starts, reg-ends

Searches the string target-string from start (which defaults to 0) to end (which default to the length of target-string) and tries to match regex. On success returns four values - the start of the match, the end of the match, and two arrays denoting the beginnings and ends of register matches. On failure returns NIL. target-string will be coerced to a simple string if it isn't one already. (There's another keyword parameter real-start-pos. This one should never be set from user code - it is only used internally.)

SCAN acts as if the part of target-string between start and end were a standalone string, i.e. look-aheads and look-behinds can't look beyond these boundaries.

* (scan "(a)*b" "xaaabd")
1
5
#(3)
#(4)

* (scan "(a)*b" "xaaabd" :start 1)
1
5
#(3)
#(4)

* (scan "(a)*b" "xaaabd" :start 2)
2
5
#(3)
#(4)

* (scan "(a)*b" "xaaabd" :end 4)
NIL

* (scan '(:greedy-repetition 0 nil #\b) "bbbc")
0
3
#()
#()

* (scan '(:greedy-repetition 4 6 #\b) "bbbc")
NIL

* (let ((s (create-scanner "(([a-c])+)x")))
    (scan s "abcxy"))
0
4
#(0 2)
#(3 3)

[Function]
scan-to-strings regex target-string &key start end sharedp => match, regs

Like SCAN but returns substrings of target-string instead of positions, i.e. this function returns two values on success: the whole match as a string plus an array of substrings (or NILs) corresponding to the matched registers. If sharedp is true, the substrings may share structure with target-string.

* (scan-to-strings "[^b]*b" "aaabd")
"aaab"
#()

* (scan-to-strings "([^b])*b" "aaabd")
"aaab"
#("a")

* (scan-to-strings "(([^b])*)b" "aaabd")
"aaab"
#("aaa" "a")

[Macro]
register-groups-bind var-list (regex target-string &key start end sharedp) declaration* statement* => result*

Evaluates statement* with the variables in var-list bound to the corresponding register groups after target-string has been matched against regex, i.e. each variable is either bound to a string or to NIL. As a shortcut, the elements of var-list can also be lists of the form (FN VAR) where VAR is the variable symbol and FN is a function designator (which is evaluated) denoting a function which is to be applied to the string before the result is bound to VAR. To make this even more convenient the form (FN VAR1 ...VARn) can be used as an abbreviation for (FN VAR1) ... (FN VARn).

If there is no match, the statement* forms are not executed. For each element of var-list which is NIL there's no binding to the corresponding register group. The number of variables in var-list must not be greater than the number of register groups. If sharedp is true, the substrings may share structure with target-string.

* (register-groups-bind (first second third fourth)
      ("((a)|(b)|(c))+" "abababc" :sharedp t)
    (list first second third fourth))
("c" "a" "b" "c")

* (register-groups-bind (nil second third fourth)
      ;; note that we don't bind the first and fifth register group
      ("((a)|(b)|(c))()+" "abababc" :start 6)
    (list second third fourth))
(NIL NIL "c")

* (register-groups-bind (first)
      ("(a|b)+" "accc" :start 1)
    (format t "This will not be printed: ~A" first))
NIL

* (register-groups-bind (fname lname (#'parse-integer date month year))
      ("(\\w+)\\s+(\\w+)\\s+(\\d{1,2})\\.(\\d{1,2})\\.(\\d{4})" "Frank Zappa 21.12.1940")
    (list fname lname (encode-universal-time 0 0 0 date month year 0)))
("Frank" "Zappa" 1292889600)

[Macro]
do-scans (match-start match-end reg-starts reg-ends regex target-string &optional result-form &key start end) declaration* statement* => result*

A macro which iterates over target-string and tries to match regex as often as possible evaluating statement* with match-start, match-end, reg-starts, and reg-ends bound to the four return values of each match (see SCAN) in turn. After the last match, returns result-form if provided or NIL otherwise. An implicit block named NIL surrounds DO-SCANS; RETURN may be used to terminate the loop immediately. If regex matches an empty string, the scan is continued one position behind this match.

This is the most general macro to iterate over all matches in a target string. See the source code of DO-MATCHES, ALL-MATCHES, SPLIT, or REGEX-REPLACE-ALL for examples of its usage.

[Macro]
do-matches (match-start match-end regex target-string &optional result-form &key start end) declaration* statement* => result*

Like DO-SCANS but doesn't bind variables to the register arrays.

* (defun foo (regex target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (do-matches (s e regex target-string nil :start start :end end)
        (incf sum (- e s)))
      (format t "~,2F% of the string was inside of a match~%"
                ;; note: doesn't check for division by zero
                (float (* 100 (/ sum (- end start)))))))

FOO

* (foo "a" "abcabcabc")
33.33% of the string was inside of a match
NIL
* (foo "aa|b" "aacabcbbc")
55.56% of the string was inside of a match
NIL

[Macro]
do-matches-as-strings (match-var regex target-string &optional result-form &key start end sharedp) declaration* statement* => result*

Like DO-MATCHES but binds match-var to the substring of target-string corresponding to each match in turn. If sharedp is true, the substrings may share structure with target-string.

* (defun crossfoot (target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (do-matches-as-strings (m :digit-class
                                         target-string nil
                                         :start start :end end)
        (incf sum (parse-integer m)))
      (if (< sum 10)
        sum
        (crossfoot (format nil "~A" sum)))))

CROSSFOOT

* (crossfoot "bar")
0

* (crossfoot "a3x")
3

* (crossfoot "12345")
6

Of course, in real life you would do this with DO-MATCHES and use the start and end keyword parameters of PARSE-INTEGER.

[Macro]
do-register-groups var-list (regex target-string &optional result-form &key start end sharedp) declaration* statement* => result*

Iterates over target-string and tries to match regex as often as possible evaluating statement* with the variables in var-list bound to the corresponding register groups for each match in turn, i.e. each variable is either bound to a string or to NIL. You can use the same shortcuts and abbreviations as in REGISTER-GROUPS-BIND. The number of variables in var-list must not be greater than the number of register groups. For each element of var-list which is NIL there's no binding to the corresponding register group. After the last match, returns result-form if provided or NIL otherwise. An implicit block named NIL surrounds DO-REGISTER-GROUPS; RETURN may be used to terminate the loop immediately. If regex matches an empty string, the scan is continued one position behind this match. If sharedp is true, the substrings may share structure with target-string.

* (do-register-groups (first second third fourth)
      ("((a)|(b)|(c))" "abababc" nil :start 2 :sharedp t)
    (print (list first second third fourth)))
("a" "a" NIL NIL) 
("b" NIL "b" NIL) 
("a" "a" NIL NIL) 
("b" NIL "b" NIL) 
("c" NIL NIL "c")
NIL

* (let (result)
    (do-register-groups ((#'parse-integer n) (#'intern sign) whitespace)
        ("(\\d+)|(\\+|-|\\*|/)|(\\s+)" "12*15 - 42/3")
      (unless whitespace
        (push (or n sign) result)))
    (nreverse result))
(12 * 15 - 42 / 3)

[Function]
all-matches regex target-string &key start end => list

Returns a list containing the start and end positions of all matches of regex against target-string, i.e. if there are N matches the list contains (* 2 N) elements. If regex matches an empty string the scan is continued one position behind this match.

* (all-matches "a" "foo bar baz")
(5 6 9 10)

* (all-matches "\\w*" "foo bar baz")
(0 3 3 3 4 7 7 7 8 11 11 11)

[Function]
all-matches-as-strings regex target-string &key start end sharedp => list

Like ALL-MATCHES but returns a list of substrings instead. If sharedp is true, the substrings may share structure with target-string.

* (all-matches-as-strings "a" "foo bar baz")
("a" "a")

* (all-matches-as-strings "\\w*" "foo bar baz")
("foo" "" "bar" "" "baz" "")

Splitting and replacing

[Function]
split regex target-string &key start end limit with-registers-p omit-unmatched-p sharedp => list

Matches regex against target-string as often as possible and returns a list of the substrings between the matches. If with-registers-p is true, substrings corresponding to matched registers are inserted into the list as well. If omit-unmatched-p is true, unmatched registers will simply be left out, otherwise they will show up as NIL. limit limits the number of elements returned - registers aren't counted. If limit is NIL (or 0 which is equivalent), trailing empty strings are removed from the result list. If regex matches an empty string, the scan is continued one position behind this match. If sharedp is true, the substrings may share structure with target-string.

This function also tries hard to be Perl-compatible - thus the somewhat peculiar behaviour.

* (split "\\s+" "foo   bar baz
frob")
("foo" "bar" "baz" "frob")

* (split "\\s*" "foo bar   baz")
("f" "o" "o" "b" "a" "r" "b" "a" "z")

* (split "(\\s+)" "foo bar   baz")
("foo" "bar" "baz")

* (split "(\\s+)" "foo bar   baz" :with-registers-p t)
("foo" " " "bar" "   " "baz")

* (split "(\\s)(\\s*)" "foo bar   baz" :with-registers-p t)
("foo" " " "" "bar" " " "  " "baz")

* (split "(,)|(;)" "foo,bar;baz" :with-registers-p t)
("foo" "," NIL "bar" NIL ";" "baz")

* (split "(,)|(;)" "foo,bar;baz" :with-registers-p t :omit-unmatched-p t)
("foo" "," "bar" ";" "baz")

* (split ":" "a:b:c:d:e:f:g::")
("a" "b" "c" "d" "e" "f" "g")

* (split ":" "a:b:c:d:e:f:g::" :limit 1)
("a:b:c:d:e:f:g::")

* (split ":" "a:b:c:d:e:f:g::" :limit 2)
("a" "b:c:d:e:f:g::")

* (split ":" "a:b:c:d:e:f:g::" :limit 3)
("a" "b" "c:d:e:f:g::")

* (split ":" "a:b:c:d:e:f:g::" :limit 1000)
("a" "b" "c" "d" "e" "f" "g" "" "")

[Function]
regex-replace regex target-string replacement &key start end preserve-case simple-calls element-type => string, matchp

Try to match target-string between start and end against regex and replace the first match with replacement. Two values are returned; the modified string, and T if regex matched or NIL otherwise.

replacement can be a string which may contain the special substrings "\&" for the whole match, "\`" for the part of target-string before the match, "\'" for the part of target-string after the match, "\N" or "\{N}" for the Nth register where N is a positive integer.

replacement can also be a function designator in which case the match will be replaced with the result of calling the function designated by replacement with the arguments target-string, start, end, match-start, match-end, reg-starts, and reg-ends. (reg-starts and reg-ends are arrays holding the start and end positions of matched registers (or NIL) - the meaning of the other arguments should be obvious.)

If simple-calls is true, a function designated by replacement will instead be called with the arguments match, register-1, ..., register-n where match is the whole match as a string and register-1 to register-n are the matched registers, also as strings (or NIL). Note that these strings share structure with target-string so you must not modify them.

Finally, replacement can be a list where each element is a string (which will be inserted verbatim), one of the symbols :match, :before-match, or :after-match (corresponding to "\&", "\`", and "\'" above), an integer N (representing register (1+ N)), or a function designator.

If preserve-case is true (default is NIL), the replacement will try to preserve the case (all upper case, all lower case, or capitalized) of the match. The result will always be a fresh string, even if regex doesn't match.

element-type specifies the array element type of the string which is returned, the default is LW:SIMPLE-CHAR for LispWorks and CHARACTER for other Lisps.

* (regex-replace "fo+" "foo bar" "frob")
"frob bar"
T

* (regex-replace "fo+" "FOO bar" "frob")
"FOO bar"
NIL

* (regex-replace "(?i)fo+" "FOO bar" "frob")
"frob bar"
T

* (regex-replace "(?i)fo+" "FOO bar" "frob" :preserve-case t)
"FROB bar"
T

* (regex-replace "(?i)fo+" "Foo bar" "frob" :preserve-case t)
"Frob bar"
T

* (regex-replace "bar" "foo bar baz" "[frob (was '\\&' between '\\`' and '\\'')]")
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"
T

* (regex-replace "bar" "foo bar baz"
                          '("[frob (was '" :match "' between '" :before-match "' and '" :after-match "')]"))
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"
T

* (regex-replace "(be)(nev)(o)(lent)"
                          "benevolent: adj. generous, kind"
                          #'(lambda (match &rest registers)
                              (format nil "~A [~{~A~^.~}]" match registers))
                          :simple-calls t)
"benevolent [be.nev.o.lent]: adj. generous, kind"
T

[Function]
regex-replace-all regex target-string replacement &key start end preserve-case simple-calls element-type => string, matchp

Like REGEX-REPLACE but replaces all matches.

* (regex-replace-all "(?i)fo+" "foo Fooo FOOOO bar" "frob" :preserve-case t)
"frob Frob FROB bar"
T

* (regex-replace-all "(?i)f(o+)" "foo Fooo FOOOO bar" "fr\\1b" :preserve-case t)
"froob Frooob FROOOOB bar"
T

* (let ((qp-regex (create-scanner "[\\x80-\\xff]")))
    (defun encode-quoted-printable (string)
      "Converts 8-bit string to quoted-printable representation."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "=~2,'0x" (char-code (char target-string match-start)))))
        (regex-replace-all qp-regex string #'convert))))
Converted ENCODE-QUOTED-PRINTABLE.
ENCODE-QUOTED-PRINTABLE

* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"
T

* (let ((url-regex (create-scanner "[^a-zA-Z0-9_\\-.]")))
    (defun url-encode (string)
      "URL-encodes a string."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "%~2,'0x" (char-code (char target-string match-start)))))
        (regex-replace-all url-regex string #'convert))))
Converted URL-ENCODE.
URL-ENCODE

* (url-encode "Fête Sørensen naïve Hühner Straße")
"F%EAte%20S%F8rensen%20na%EFve%20H%FChner%20Stra%DFe"
T

* (defun how-many (target-string start end match-start match-end reg-starts reg-ends)
    (declare (ignore start end match-start match-end))
    (format nil "~A" (- (svref reg-ends 0)
                        (svref reg-starts 0))))
HOW-MANY

* (regex-replace-all "{(.+?)}"
                              "foo{...}bar{.....}{..}baz{....}frob"
                              (list "[" 'how-many " dots]"))
"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"
T

* (let ((qp-regex (create-scanner "[\\x80-\\xff]")))
    (defun encode-quoted-printable (string)
      "Converts 8-bit string to quoted-printable representation.
Version using SIMPLE-CALLS keyword argument."
      ;; ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (match)
               (format nil "=~2,'0x" (char-code (char match 0)))))
        (regex-replace-all qp-regex string #'convert
                                    :simple-calls t))))

Converted ENCODE-QUOTED-PRINTABLE.
ENCODE-QUOTED-PRINTABLE

* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"
T

* (defun how-many (match first-register)
    (declare (ignore match))
    (format nil "~A" (length first-register)))
HOW-MANY

* (regex-replace-all "{(.+?)}"
                              "foo{...}bar{.....}{..}baz{....}frob"
                              (list "[" 'how-many " dots]")
                              :simple-calls t)

"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"
T

Modifying scanner behaviour

[Special variable]
*property-resolver*

This is the designator for a function responsible for resolving named properties like \p{Number}. If CL-PPCRE encounters a \p or a \P it expects to see an opening curly brace immediately afterwards and will then read everything following that brace until it sees a closing curly brace. The resolver function will be called with this string and must return a corresponding unary test function which accepts a character as its argument and returns a true value if and only if the character has the named property. If the resolver returns NIL instead, it signals that a property of that name is unknown.

* (labels ((char-code-odd-p (char)
             (oddp (char-code char)))
           (char-code-even-p (char)
             (evenp (char-code char)))
           (resolver (name)
             (cond ((string= name "odd") #'char-code-odd-p)
                   ((string= name "even") #'char-code-even-p)
                   ((string= name "true") (constantly t))
                   (t (error "Can't resolve ~S." name)))))
    (let ((*property-resolver* #'resolver))
      ;; quiz question - why do we need CREATE-SCANNER here?
      (list (regex-replace-all (create-scanner "\\p{odd}") "abcd" "+")
            (regex-replace-all (create-scanner "\\p{even}") "abcd" "+")
            (regex-replace-all (create-scanner "\\p{true}") "abcd" "+"))))
("+b+d" "a+c+" "++++")

If the value of *PROPERTY-RESOLVER* is NIL (which is the default), \p and \P in regex strings will simply be treated like p or P as in CL-PPCRE 1.4.1 and earlier. Note that this does not affect the validity of (:PROPERTY <name>) parts in S-expression syntax.

[Accessor]
parse-tree-synonym symbol => parse-tree
(setf (parse-tree-synonym symbol) new-parse-tree)

Any symbol (unless it's a keyword with a special meaning in parse trees) can be made a "synonym", i.e. an abbreviation, for another parse tree by this accessor. PARSE-TREE-SYNONYM returns NIL if symbol isn't a synonym yet.

* (parse-string "a*b+")
(:SEQUENCE (:GREEDY-REPETITION 0 NIL #\a) (:GREEDY-REPETITION 1 NIL #\b))

* (defun my-repetition (char min)
    `(:greedy-repetition ,min nil ,char))
MY-REPETITION

* (setf (parse-tree-synonym 'a*) (my-repetition #\a 0))
(:GREEDY-REPETITION 0 NIL #\a)

* (setf (parse-tree-synonym 'b+) (my-repetition #\b 1))
(:GREEDY-REPETITION 1 NIL #\b)

* (let ((scanner (create-scanner '(:sequence a* b+))))
    (dolist (string '("ab" "b" "aab" "a" "x"))
      (print (scan scanner string)))
    (values))
0
0
0
NIL
NIL

* (parse-tree-synonym 'a*)
(:GREEDY-REPETITION 0 NIL #\a)

* (parse-tree-synonym 'a+)
NIL

[Macro]
define-parse-tree-synonym name parse-tree => parse-tree

This is a convenience macro for parse tree synonyms defined as

(defmacro define-parse-tree-synonym (name parse-tree)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (setf (parse-tree-synonym ',name) ',parse-tree)))

so you can write code like this:

(define-parse-tree-synonym a-z
  (:char-class (:range #\a #\z) (:range #\A #\Z)))

(define-parse-tree-synonym a-z*
  (:greedy-repetition 0 nil a-z))

(defun ascii-char-tester (string)
  (scan '(:sequence :start-anchor a-z* :end-anchor)
        string))

[Special variable]
*regex-char-code-limit*

This variable controls whether scanners take into account all characters of your CL implementation or only those the CHAR-CODE of which is not larger than its value. The default is CHAR-CODE-LIMIT, and you might see significant speed and space improvements during scanner creation if, say, your target strings only contain ISO-8859-1 characters and you're using a Lisp implementation where CHAR-CODE-LIMIT has a value much higher than 256. The test suite will automatically set *REGEX-CHAR-CODE-LIMIT* to 256 while you're running the default test.

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN and other functions, some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *REGEX-CHAR-CODE-LIMIT* is bound at that time. The default value should always yield correct results unless you play dirty tricks with implementation-dependent behaviour, though.

[Special variable]
*use-bmh-matchers*

Usually, the scanners created by CREATE-SCANNER (or implicitly by other functions and macros) will use the standard function SEARCH to check for constant strings at the start or end of the regular expression. If *USE-BMH-MATCHERS* is true (the default is NIL), fast Boyer-Moore-Horspool matchers will be used instead. This will usually be faster but can make the scanners considerably bigger. Per BMH matcher - there can be up to two per scanner - a fixnum array of size *REGEX-CHAR-CODE-LIMIT* is allocated and closed over.

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN and other functions, some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *USE-BMH-MATCHERS* is bound at that time.

[Special variable]
*optimize-char-classes*

Whether character classes should be compiled into look-ups into O(1) data structures. This is usually fast but will be costly in terms of scanner creation time and might be costly in terms of size if *REGEX-CHAR-CODE-LIMIT* is high. This value will be used as the kind keyword argument to CREATE-OPTIMIZED-TEST-FUNCTION - see there for the possible non-NIL values. The default value (NIL) should usually be fine unless you're sure that you absolutely have to optimize some character classes for speed.

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN and other functions, some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *OPTIMIZE-CHAR-CLASSES* is bound at that time.

[Special variable]
*allow-quoting*

If this value is true (the default is NIL), CL-PPCRE will support \Q and \E in regex strings to quote (disable) metacharacters. Note that this entails a slight performance penalty when creating scanners because (a copy of) the regex string is modified (probably more than once) before it is fed to the parser. Also, the parser's syntax error messages will complain about the converted string and not about the original regex string.

* (scan "^a+$" "a+")
NIL

* (let ((*allow-quoting* t))
    ;;we use CREATE-SCANNER because of Lisps like SBCL that don't have an interpreter
    (scan (create-scanner "^\\Qa+\\E$") "a+"))
0
2
#()
#()

* (let ((*allow-quoting* t))
    (scan (create-scanner "\\Qa()\\E(?#comment\\Q)a**b") "()ab"))

Quantifier '*' not allowed at position 19 in string "a\\(\\)(?#commentQ)a**b"

Note how in the last example the regex string in the error message is different from the first argument to the SCAN function. Also note that the second example might be easier to understand (and Lisp-ier) if you write it like this:

* (scan '(:sequence :start-anchor
                    "a+" ;; no quoting necessary
                    :end-anchor)
        "a+")
0
2
#()
#()

Make sure you also read the relevant section in "Bugs and problems."

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN and other functions, some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *ALLOW-QUOTING* is bound at that time.

[Special variable]
*allow-named-registers*

If this value is true (the default is NIL), CL-PPCRE will support (?<name>"<regex>") and \k<name> in regex strings to provide named registers and back-references as in AllegroCL. name is has to start with a letter and can contain only alphanumeric characters or minus sign. Names of registers are matched case-sensitively. The parse tree syntax is not affected by the *ALLOW-NAMED-REGISTERS* switch, :NAMED-REGISTER and :BACK-REFERENCE forms are always resolved as expected. There are also no restrictions on register names in this syntax except that they have to be strings.

;; Perl compatible mode (*ALLOW-NAMED-REGISTERS* is NIL)
* (create-scanner "(?<reg>.*)")
Character 'r' may not follow '(?<' at position 3 in string "(?<reg>)"

;; just unescapes "\\k"
* (parse-string "\\k<reg>")
"k<reg>"

* (setq *allow-named-registers* t)
T

* (create-scanner "((?<small>[a-z]*)(?<big>[A-Z]*))")
#<CLOSURE (LAMBDA (STRING CL-PPCRE::START CL-PPCRE::END)) {AD75BFD}>
(NIL "small" "big")

;; the scanner doesn't capture any information about named groups -
;; you have to store the second value returned from CREATE-SCANNER yourself
* (scan * "aaaBBB")
0
6
#(0 0 3)
#(6 3 6)

;; parse tree syntax
* (parse-string "((?<small>[a-z]*)(?<big>[A-Z]*))")
(:REGISTER
 (:SEQUENCE
  (:NAMED-REGISTER "small"
   (:GREEDY-REPETITION 0 NIL (:CHAR-CLASS (:RANGE #\a #\z))))
  (:NAMED-REGISTER "big"
   (:GREEDY-REPETITION 0 NIL (:CHAR-CLASS (:RANGE #\A #\Z))))))

* (create-scanner *)
#<CLOSURE (LAMBDA (STRING CL-PPCRE::START CL-PPCRE::END)) {B158E3D}>
(NIL "small" "big")

;; multiple-choice back-reference
* (scan "^(?<reg>[ab])(?<reg>[12])\\k<reg>\\k<reg>$" "a1aa")
0
4
#(0 1)
#(1 2)

* (scan "^(?<reg>[ab])(?<reg>[12])\\k<reg>\\k<reg>$" "a22a")
0
4
#(0 1)
#(1 2)

;; demonstrating most-recently-seen-register-first property of back-reference;
;; "greedy" regex (analogous to "aa?")
* (scan "^(?<reg>)(?<reg>a)(\\k<reg>)" "a")
0
1
#(0 0 1)
#(0 1 1)

* (scan "^(?<reg>)(?<reg>a)(\\k<reg>)" "aa")
0
2
#(0 0 1)
#(0 1 2)

;; switched groups
;; "lazy" regex (analogous to "aa??")
* (scan "^(?<reg>a)(?<reg>)(\\k<reg>)" "a")
0
1
#(0 1 1)
#(1 1 1)

;; scanner ignores the second "a"
* (scan "^(?<reg>a)(?<reg>)(\\k<reg>)" "aa")
0
1
#(0 1 1)
#(1 1 1)

;; "aa" will be matched only when forced by adding "$" at the end
* (scan "^(?<reg>a)(?<reg>)(\\k<reg>)$" "aa")
0
2
#(0 1 1)
#(1 1 2)

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN and other functions, some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *ALLOW-NAMED-REGISTERS* is bound at that time.

Miscellaneous

[Function]
parse-string string => parse-tree

Converts the regex string string into a parse tree. Note that the result is usually one possible way of creating an equivalent parse tree and not necessarily the "canonical" one. Specifically, the parse tree might contain redundant parts which are supposed to be excised when a scanner is created.

[Function]
create-optimized-test-function test-function &key start end kind => function

Given a unary test function test-function which is applicable to characters returns a function which yields the same boolean results for all characters with character codes from start to (excluding) end. If kind is NIL, test-function will simply be returned. Otherwise, kind should be one of:

:HASH-TABLE

The function builds a hash table representing all characters which satisfy the test and returns a closure which checks if a character is in that hash table.

:CHARSET

Instead of a hash table the function uses a "charset" which is a data structure using non-linear hashing and optimized to represent (sparse) sets of characters in a fast and space-efficient way (contributed by Nikodemus Siivola).

:CHARMAP

Instead of a hash table the function uses a bit vector to represent the set of characters.

You can also use :HASH-TABLE* or :CHARSET* which are like :HASH-TABLE and :CHARSET but use the complement of the set if the set contains more than half of all characters between start and end. This saves space but needs an additional pass across all characters to create the data structure. There is no corresponding :CHARMAP* kind as the bit vectors are already created to cover the smallest possible interval which contains either the set or its complement.

See also *OPTIMIZE-CHAR-CLASSES*.

[Function]
quote-meta-chars string => string'

This is a simple utility function used when *ALLOW-QUOTING* is true. It returns a string STRING' where all non-word characters (everything except ASCII characters, digits and underline) of STRING are quoted by prepending a backslash similar to Perl's quotemeta function. It always returns a fresh string.

* (quote-meta-chars "[a-z]*")
"\\[a\\-z\\]\\*"

[Function]
regex-apropos regex &optional packages &key case-insensitive => list

Like APROPOS but searches for interned symbols which match the regular expression regex. The output is implementation-dependent. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Here are examples for CMUCL:

* *package*
#<The COMMON-LISP-USER package, 16/21 internal, 0/9 external>

* (defun foo (n &optional (k 0)) (+ 3 n k))
FOO

* (defparameter foo "bar")
FOO

* (defparameter |foobar| 42)
|foobar|

* (defparameter fooboo 43)
FOOBOO

* (defclass frobar () ())
#<STANDARD-CLASS FROBAR {4874E625}>

* (regex-apropos "foo(?:bar)?")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (regex-apropos "(?:foo|fro)bar")
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (regex-apropos "(?:foo|fro)bar" 'cl-user)
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (regex-apropos "(?:foo|fro)bar" '(pcl ext))
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]

* (regex-apropos "foo")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (regex-apropos "foo" nil :case-insensitive nil)
|foobar| [variable] value: 42

[Function]
regex-apropos-list regex &optional packages &key upcase => list

Like APROPOS-LIST but searches for interned symbols which match the regular expression regex. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Example (continued from above):

* (regex-apropos-list "foo(?:bar)?")
(|foobar| FOOBOO FOO)

Conditions

[Condition type]
ppcre-error

Every error signaled by CL-PPCRE is of type PPCRE-ERROR. This is a direct subtype of SIMPLE-ERROR without any additional slots or options.

[Condition type]
ppcre-invocation-error

Errors of type PPCRE-INVOCATION-ERROR are signaled if one of the exported functions of CL-PPCRE is called with wrong or inconsistent arguments. This is a direct subtype of PPCRE-ERROR without any additional slots or options.

[Condition type]
ppcre-syntax-error

An error of type PPCRE-SYNTAX-ERROR is signaled if CL-PPCRE's parser encounters an error when trying to parse a regex string or to convert a parse tree into its internal representation. This is a direct subtype of PPCRE-ERROR with two additional slots. These denote the regex string which HTML-PPCRE was parsing and the position within the string where the error occurred. If the error happens while CL-PPCRE is converting a parse tree, both of these slots contain NIL. (See the next two entries on how to access these slots.)

As many syntax errors can't be detected before the parser is at the end of the stream, the row and column usually denote the last position where the parser was happy and not the position where it gave up.

* (handler-case
    (scan "foo**x" "fooox")
    (ppcre-syntax-error (condition)
      (format t "Houston, we've got a problem with the string ~S:~%~
                 Looks like something went wrong at position ~A.~%~
                 The last message we received was \"~?\"."
              (ppcre-syntax-error-string condition)
              (ppcre-syntax-error-pos condition)
              (simple-condition-format-control condition)
              (simple-condition-format-arguments condition))
      (values)))
Houston, we've got a problem with the string "foo**x":
Looks like something went wrong at position 4.
The last message we received was "Quantifier '*' not allowed.".

[Function]
ppcre-syntax-error-string condition => string

If condition is a condition of type PPCRE-SYNTAX-ERROR, this function will return the string the parser was parsing when the error was encountered (or NIL if the error happened while trying to convert a parse tree). This might be particularly useful when *ALLOW-QUOTING* is true because in this case the offending string might not be the one you gave to the CREATE-SCANNER function.

[Function]
ppcre-syntax-error-pos condition => number

If condition is a condition of type PPCRE-SYNTAX-ERROR, this function will return the position within the string where the error occurred (or NIL if the error happened while trying to convert a parse tree).

Unicode properties

(asdf:oos 'asdf:load-op :cl-ppcre-unicode)

See the CL-UNICODE documentation for information about the supported Unicode properties and how they are named.

[Function]
unicode-property-resolver property-name => function-or-nil

A property resolver which understands Unicode properties using CL-UNICODE's PROPERTY-TEST function. This resolver is automatically installed in *PROPERTY-RESOLVER* when the CL-PPCRE-UNICODE system is loaded.

* (scan-to-strings "\\p{Script:Latin}+" "0+AB_*")
"AB"
#()

Note that this symbol is exported from the CL-PPCRE-UNICODE package and not from the CL-PPCRE package.

Filters

A filter is defined by its filter function which must be a function of one argument. During the parsing process this function might be called once or several times or it might not be called at all. If it's called, its argument is an integer pos which is the current position within the target string. The filter can either return NIL (which means that the subexpression represented by this filter didn't match) or an integer not smaller than pos for success. A zero-length assertion should return pos itself while a filter which wants to consume N characters should return (+ POS N).

If you supply the optional value length and it is not NIL, then this is a promise to the regex engine that your filter will always consume exactly length characters. The regex engine might use this information for optimization purposes but it is otherwise irrelevant to the outcome of the matching process.

The filter function can access the following special variables from its code body:

CL-PPCRE::*STRING*

The target (a string) of the current matching process.

CL-PPCRE::*START-POS* and CL-PPCRE::*END-POS*

The start and end (integers) indices of the current matching process. These correspond to the START and END keyword parameters of SCAN.

CL-PPCRE::*REAL-START-POS*

The initial starting position. This is only relevant for repeated scans (as in DO-SCANS) where CL-PPCRE::*START-POS* will be moved forward while CL-PPCRE::*REAL-START-POS* won't. For normal scans the value of this variable is NIL.

CL-PPCRE::*REG-STARTS* and CL-PPCRE::*REG-ENDS*

Two simple vectors which denote the start and end indices of registers within the regular expression. The first register is indexed by 0. If a register hasn't matched yet, then its corresponding entry in CL-PPCRE::*REG-STARTS* is NIL.

Note that the names of the variables are not exported from the CL-PPCRE package because there's no explicit guarantee that they will be available in future releases. (Although after so many years it is very unlikely that they'll go away...)

* (defun my-info-filter (pos)
    "Show some info about the matching process."
    (format t "Called at position ~A~%" pos)
    (loop with dim = (array-dimension cl-ppcre::*reg-starts* 0)
          for i below dim
          for reg-start = (aref cl-ppcre::*reg-starts* i)
          for reg-end = (aref cl-ppcre::*reg-ends* i)
          do (format t "Register ~A is currently " (1+ i))
          when reg-start
               (write-string cl-ppcre::*string* nil
            do (write-char #\')
               (write-string cl-ppcre::*string* nil
                     :start reg-start :end reg-end)
               (write-char #\')
          else
            do (write-string "unbound")
          do (terpri))
    (terpri)
    pos)
MY-INFO-FILTER

* (scan '(:sequence
           (:register
             (:greedy-repetition 0 nil
                                 (:char-class (:range #\a #\z))))
           (:filter my-info-filter 0) "X")
        "bYcdeX")
Called at position 1
Register 1 is currently 'b'

Called at position 0
Register 1 is currently ''

Called at position 1
Register 1 is currently ''

Called at position 5
Register 1 is currently 'cde'

2
6
#(2)
#(5)

* (scan '(:sequence
           (:register
             (:greedy-repetition 0 nil
                                 (:char-class (:range #\a #\z))))
           (:filter my-info-filter 0) "X")
        "bYcdeZ")
NIL

* (defun my-weird-filter (pos)
    "Only match at this point if either pos is odd and the character
  we're looking at is lowercase or if pos is even and the next two
  characters we're looking at are uppercase. Consume these characters if
  there's a match."
    (format t "Trying at position ~A~%" pos)
    (cond ((and (oddp pos)
                (< pos cl-ppcre::*end-pos*)
                (lower-case-p (char cl-ppcre::*string* pos)))
           (1+ pos))
          ((and (evenp pos)
                (< (1+ pos) cl-ppcre::*end-pos*)
                (upper-case-p (char cl-ppcre::*string* pos))
                (upper-case-p (char cl-ppcre::*string* (1+ pos))))
           (+ pos 2))
          (t nil)))
MY-WEIRD-FILTER

* (defparameter *weird-regex*
                `(:sequence "+" (:filter ,#'my-weird-filter) "+"))
*WEIRD-REGEX*

* (scan *weird-regex* "+A++a+AA+")
Trying at position 1
Trying at position 3
Trying at position 4
Trying at position 6
5
9
#()
#()

* (fmakunbound 'my-weird-filter)
MY-WEIRD-FILTER

* (scan *weird-regex* "+A++a+AA+")
Trying at position 1
Trying at position 3
Trying at position 4
Trying at position 6
5
9
#()
#()

For more ideas about what you can do with filters see this thread on the mailing list.
 

Compatibility with Perl

Empty strings instead of undef in $1, $2, etc.

Strange scoping of embedded modifiers

Inconsistent capturing of $1, $2, etc.

Captured groups not available outside of look-aheads and look-behinds

Alternations don't always work from left to right

Different names for Unicode properties

"\r" doesn't work with MCL

What about "\w"?

Bugs and problems

"\Q" doesn't work, or does it?

#!/usr/bin/perl -l

$a = '\E*';
print 1
  if '\E*\E*' =~ /(?:\Q$a\E){2}/;

* (let ((*allow-quoting* t)
        (a "\\E*"))
    (scan (concatenate 'string "(?:\\Q" a "\\E){2}") "\\E*\\E*"))
Quantifier '*' not allowed at position 3 in string "(?:*\\E){2}"

What gives? "\\Q...\\E" in CL-PPCRE should only be used in literal strings. If you want to quote arbitrary strings, try CL-INTERPOL or use QUOTE-META-CHARS:

* (let ((a "\\E*"))
    (scan (concatenate 'string "(?:" (quote-meta-chars a) "){2}") "\\E*\\E*"))
0
6
#()
#()

* (let ((a "\\E*"))
    (scan `(:greedy-repetition 2 2 ,a) "\\E*\\E*"))
0
6
#()
#()

Backslashes may confuse you...

* (let ((a "y\\y"))
    (scan a a))
NIL

'y\y' =~ /y\y/;

$a = 'y\y';
$a =~ /$a/;

Still confused? You might want to try CL-INTERPOL.
 

AllegroCL compatibility mode

The advantage of this mode is that you'll get a much smaller image and most likely faster code. (But note that CL-PPCRE needs to do a small amount of work to massage AllegroCL's output into the format expected by CL-PPCRE.) The downside is that your code won't be fully compatible with CL-PPCRE anymore. Here are some of the differences (most of which probably don't matter very often):

• The AllegroCL engine doesn't offer parse tree synonyms and filters.
• The AllegroCL engine will choke on some regular expressions involving curly braces that are accepted by Perl and CL-PPCRE's native engine.
• The AllegroCL engine's case-folding mode switch (which is used instead of CL-PPCRE's :CASE-INSENSITIVE keyword parameter) is currently only effective for ASCII characters.
• The AllegroCL engine doesn't support quoting of metacharacters.
• In AllegroCL compatibility mode compiled regular expressions (as returned by CREATE-SCANNER) aren't functions but structures.
• The AllegroCL engine doesn't support named properties.

To use the AllegroCL compatibility mode you have to

(push :use-acl-regexp2-engine *features*)

Hints, comments, performance considerations

• A lot of hackers (especially users of Perl and other scripting languages) think that regular expressions are the greatest thing since slice bread and use it for almost everything. That is just plain wrong. Other hackers (especially Lispers) tend to think that regular expressions are the work of the devil and try to avoid them at all cost. That's also wrong. Regular expressions are a handy and useful addition to your toolkit which you should use when appropriate - you should just try to figure out first if they're appropriate for the task at hand.
• If you're concerned about the string syntax of regular expressions which can look like line noise and is really hard to read for long expressions, consider using CL-PPCRE's S-expression syntax instead. It is less error-prone and you don't have to worry about escaping characters. It is also easier to manipulate programmatically.
• For alternations, order is important. The general rule is that the regex engine tries from left to right and tries to match as much as possible.

  CL-USER 1 > (scan-to-strings "<=|<" "<=")
  "<="
  #()
  
  CL-USER 2 > (scan-to-strings "<|<=" "<=")
  "<"
  #()
  

• CL-PPCRE uses compiler macros to pre-compile scanners at load time if possible. This happens if the compiler can determine that the regular expression (no matter if it's a string or an S-expression) is constant at compile time and is intended to save the time for creating scanners at execution time (probably creating the same scanner over and over in a loop). Make sure you don't prevent the compiler from helping you. For example, a definition like this one is usually not a good idea:

  (defun regex-match (regex target)
    ;; don't do that!
    (scan regex target))
  

• If you want to search for a substring in a large string or if you search for the same string very often, SCAN will usually be faster than Common Lisp's SEARCH if you use BMH matchers. However, this only makes sense if scanner creation time is not the limiting factor, i.e. if the search target is very large or if you're using the same scanner very often.
• Complementary to the last hint, don't use regular expressions for one-time searches for constant strings. That's a terrible waste of resources.
• *USE-BMH-MATCHERS* together with a large value for *REGEX-CHAR-CODE-LIMIT* can lead to huge scanners.
• A character class is by default translated into a sequence of tests exactly as you might expect. For example, "[af-l\\d]" means to test if the character is equal to #\a, then to test if it's between #\f and #\l, then if it's a digit. There's by default no attempt to remove redundancy (as in "[a-ge-kf]") or to otherwise optimize these tests for speed. However, you can play with *OPTIMIZE-CHAR-CLASSES* if you've identified character classes as a bottleneck and want to make sure that you have O(1) test functions.
• If you know that the expression you're looking for is anchored, use anchors in your regex. This can help the engine a lot to make your scanners more efficient.
• In addition to anchors, constant strings at the start or end of a regular expression can help the engine to quickly scan a string. Note that for example "(a-d|aebf)" and "ab(cd|ef)" are equivalent, but only the second form has a constant start the regex engine can recognize.
• Try to avoid alternations if possible or at least factor them out as in the example above.
• If neither anchors nor constant strings are in sight, maybe "standalone" (sometimes also called "possessive") regular expressions can be helpful. Try the following:

  (let ((target (make-string 10000 :initial-element #\a))
        (scanner-1 (create-scanner "a*\\d"))
        (scanner-2 (create-scanner "(?>a*)\\d")))
    (time (scan scanner-1 target))
    (time (scan scanner-2 target)))
  

• Consider using "single-line mode" if it makes sense for your task. By default (following Perl's practice), a dot means to search for any character except line breaks. In single-line mode a dot searches for any character which in some cases means that large parts of the target can actually be skipped. This can be vastly more efficient for large targets.
• Don't use capturing register groups where a non-capturing group would do, i.e. only use registers if you need to refer to them later. If you use a register, each scan process needs to allocate space for it and update its contents (possibly many times) until it's finished. (In Perl parlance - use "(?:foo)" instead of "(foo)" whenever possible.)
• In addition to what has been said in the last hint, note that Perl semantics force the regex engine to report the last match for each register. This implies for example that "([a-c])+" and "[a-c]*([a-c])" have exactly the same semantics but completely different performance characteristics. (Actually, in some cases CL-PPCRE automatically converts expressions from the first type into the second type. That's not always possible, though, and you shouldn't rely on it.)
• By default, repetitions are "greedy" in Perl (and thus in CL-PPCRE). This has an impact on performance and also on the actual outcome of a scan. Look at your repetitions and ponder if a greedy repetition is really what you want.

Acknowledgements

The list of people who participated in this project in one way or the other has grown too long to maintain it here. See the ChangeLog for all the people who helped with patches, bug reports, or in other ways. Thanks to all of them!

Thanks to the guys at "Café Olé" in Hamburg where I wrote most of the 0.1.0 release and thanks to my wife for lending me her PowerBook to test early versions of CL-PPCRE with MCL and OpenMCL.

$Header: /usr/local/cvsrep/cl-ppcre/doc/index.html,v 1.198 2008/09/02 08:45:32 edi Exp $

BACK TO MY HOMEPAGE