Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the esrap-liquid Reference Manual, version 2.3, generated automatically by Declt version 4.0 beta 2 "William Riker" on Thu Sep 15 04:41:42 2022 GMT+0.
Next: Systems, Previous: The esrap-liquid Reference Manual, Up: The esrap-liquid Reference Manual [Contents][Index]
MAJOR API CHANGE IN VERSIONS 2.*: we no longer use codewalker, hence rule definitions now compile much faster. The price to pay is that sometimes you need to specify manually, that something is a sub-rule (see V-macrolet in source and section "API Change" in this README).
Packrat parser generator, with possibility to use full Common Lisp when defining parsing rules.
It started as a fork of ESRAP by Nikodemus Siivola (https://github.com/nikodemus/esrap), which is a DSL with (somewhat) more rigid syntax, but (as of now) nicer error reports.
Original idea of a packrat parser generator is described in this article:
Bryan Ford, 2002, "Packrat Parsing: a Practical Linear Time Algorithm with Backtracking".
http://pdos.csail.mit.edu/~baford/packrat/thesis/
Operates both on strings and streams -- see section "Streaming" below.
Examples:
; plain characters match to precisely that character in text
ESRAP-LIQUID> (parse '#\a "a")
#\a
1
ESRAP-LIQUID> (parse '#\a "b")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; same goes for plain strings
ESRAP-LIQUID> (parse '"foo" "foo")
"foo"
3
ESRAP-LIQUID> (parse '"foo" "bar")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; CHARACTER matches any single character
ESRAP-LIQUID> (parse 'character "a")
#\a
1
ESRAP-LIQUID> (parse 'character "b")
#\b
1
; (ANY-STRING <LENGTH>) matches any string of a given length character
ESRAP-LIQUID> (parse '(any-string 3) "foo")
"foo"
3
ESRAP-LIQUID> (parse '(any-string 3) "bar")
"bar"
3
ESRAP-LIQUID> (parse '(any-string 3) "caboom!")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; (!! <EXPR>) matches, whenever EXPR fails and consumes one character
ESRAP-LIQUID> (parse '(!! "foo") "bar" :junk-allowed t)
#\b
1
ESRAP-LIQUID> (parse '(!! "foo") "foo")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; (|| &rest <EXPRS>) is an ordered choice, matches, whenever one of EXPRS succeeds, and outputs it
ESRAP-LIQUID> (parse '(|| #\a #\b) "a")
#\a
1
ESRAP-LIQUID> (parse '(|| #\a #\b) "b")
#\b
1
ESRAP-LIQUID> (parse '(|| #\a #\b) "c")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; (times <EXPR> &key from upto exactly) greedy matches expression multiple times, returns values as a list.
; If :UPTO or :EXACTLY is given, then consumes only that much exprs, even if there are more
; if :FROM is given, fails if consumed less than :FROM expressions
ESRAP-LIQUID> (parse '(times #\a) "")
NIL
0
ESRAP-LIQUID> (parse '(times #\a) "aaa")
(#\a #\a #\a)
3
ESRAP-LIQUID> (parse '(times #\a :exactly 6) "aaaaaa")
(#\a #\a #\a #\a #\a #\a)
6
ESRAP-LIQUID> (parse '(times #\a :exactly 6) "aaa")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
; (postimes <EXPR>) is an alias for (times <EXPR> :from 1)
ESRAP-LIQUID> (parse '(postimes #\a) "")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
ESRAP-LIQUID> (parse '(postimes #\a) "aaa")
(#\a #\a #\a)
3
; (? <EXPR>) returns result of parsing of EXPR, when parsing succeeds, and NIL otherwise, does not fail
ESRAP-LIQUID> (parse '(? #\a) "")
NIL
0
ESRAP-LIQUID> (parse '(? #\a) "a")
#\a
1
Other operators, defined by ESRAP-LIQUID, include:
Typical idioms:
; succeed, when all subexpressions succeed, return list of those subexpressions
; Note that now you need to explicitly write (V ...) in order to indicate that
; character means "try to parse this character"
ESRAP-LIQUID> (parse '(list (v #\a) (v #\b) (v #\c)) "abc")
(#\a #\b #\c)
3
; succeed, when all subexpression succeed, return only last subexpression
ESRAP-LIQUID> (parse '(progn (v #\a) (v #\b) (v #\c)) "abc")
#\c
3
; succeed, when all subexpression succeed, return only first subexpression
ESRAP-LIQUID> (parse '(prog1 (v #\a) (v #\b) (v #\c)) "abc")
#\a
3
For more examples, see example-sexp.lisp, example-symbol-table.lisp, example-very-context-sensitive.lisp. For more real-life examples see my YAML parser https://github.com/mabragor/cl-yaclyaml. The parsing part uses ESRAP-LIQUID extensively, in particular, in ways different from traditional ESRAP.
Of course, if you could only write one-liners with PARSE, it won't be interesting at all So, you can define new rules using DEFRULE macro.
ESRAP-LIQUID> (defrule foo+ ()
(postimes "foo"))
ESRAP-LIQUID> (parse 'foo+ "foofoofoo")
("foo" "foo" "foo")
; simple arguments to rules are also possible
ESRAP-LIQUID> (defrule foo-times (times)
(times "foo" :exactly times))
ESRAP-LIQUID> (parse '(v foo-times 3) "foofoofoo")
("foo" "foo" "foo")
ESRAP-LIQUID> (parse '(v foo-times 4) "foofoofoo")
#<ESRAP-LIQUID::SIMPLE-ESRAP-ERROR "~a~%">.
Usually you don't want to mix parsing rules for different projects -- you want a mechanism to scope your rules in some way. For that you can use DEFINE-ESRAP-ENV macro.
(define-esrap-env foobar)
;; Now we can define rules in 'foobar' scope
(define-foobar-rule asdf ()
... ; rule definition here)
DEFINE-ESRAP-ENV has a MAINLY-NON-CONTEXT key. If it's T, then DEFINE-$(NAME)-RULE expands into DEF-NOCONTEXT-RULE, rather than DEFRULE. It's still possible to get context-sensitive rules by using DEFINE-C-$(NAME)-RULE. Vice versa, if MAINLY-NON-CONTEXT key is NIL (default), then non-context-sensitive rules can be defined using DEFINE-NC-$(NAME)-RULE.
Analogously to capturing groups in regexps, it is possible to capture results of parsing of named rules, to aid destructuring.
Example: instead of clumsy
(define-rule dressed-rule-clumsy ()
(prog1 (progn (v "foo") (v "bar") (v "baz")
(v meat))
(v "rest1") (v "rest2") (v "rest3")))
you may write something like
(define-rule dressed-rule-elegant ()
(v "foo") (v "bar") (v "baz") (cap 1 meat) (v "rest1") (v "rest2") (v "rest3")
(recap 1))
I.e. result of parsing of rule with name MEAT is stashed with CAP macro and then accessed using RECAP macro.
Difference between RECAP and RECAP? is that while the former fails parsing if the requested key was not captured, the latter just produces NIL.
See tests for examples of usage. Also see CL-MIZAR parsing.lisp, where this is used a lot.
It's possible to parse both strings and streams. Main function for parsing streams is PARSE-STREAM.
Example of usage:
(parse-stream 'some-rule (make-string-input-stream "foobar") :junk-allowed t)
In general, both PARSE and PARSE-STREAMS are just wrappers around more general PARSE-TOKEN-ITER -- which accepts an iterator of tokens. Here, iterators are implemented "pythonic way", i.e. they are classes with NEXT-ITER method (the next method in Python), that throw STOP-ITERATION condition (the StopIteration exception in Python) when there are no more values.
Thus, if you want ESRAP-LIQUID to parse something other than string or stream, just grep for how PARSE-TOKEN-ITER is used.
Change : PARSE-TOKEN-ITER now wraps the iterator into esrap caching iterator, so you don't have to do it manually. Also now ESRAP-ENV makes $(ENVNAME)-PARSE-TOKEN-ITER function accessible.
-- Since Lisp streams only allow to unread one character, and ESRAP-LIQUID in general does quite a lot of look-ahead, it's not safe to use a stream somewhere else after it was fed to PARSE-STREAM (even when :junk-allowed is set to T). This is because you don't know, what state the stream ends up in.
When you define esrap environment using DEFINE-ESRAP-ENV, say, like
(define-esrap-env foo)
you get both FOO-PARSE and FOO-PARSE-STREAM -- environment versions of main parsing functions.
In versions 1.* of ESRAP-LIQUID, we used the following implicit conventions, when defining rules:
These conventions are very handy when defining rules. However, the way they were implemented, is not optimal:
So now these conventions are not valid in the whole scope of DEFRULE, but only inside special macrolets. This allowed not to depend on codewalker and on special reader conventions. Now conventions are like this:
(defrule foo ()
...
(v #\a) ; parse literal char #\a
(v "asdf") ; parse literal string "asdf"
(v sub-rule x) ; parse a sub-rule, giving it an argument X (optional)
...)
(times #\a) ; will be correctly understood as "parse 0 or more characters 'A'"
(progn (v a) (v b) (v c)) ; parse sub-rules a, b and c
(list (v d) (v e) f) ; parse sub-rules d and e, return variable f
Update: it turned out that in practice one most commonly uses V-macrolet inside PROGN, LIST and PROG1 forms. So now ESRAP-LIQUID defines PROGN-V, LIST-V and PROG1-V versions, which work exactly as their non-V analogs, but wrap their sub-forms in DESCEND-WITH-RULE, where needed.
Thus, to make a transition of your ESRAP-LIQUID-using code to 2.* API, you need:
Next: Modules, Previous: Introduction, Up: The esrap-liquid Reference Manual [Contents][Index]
The main system appears first, followed by any subsystem dependency.
A Packrat / Parsing Grammar / TDPL parser for Common Lisp.
GPL
2.3
Next: Files, Previous: Systems, Up: The esrap-liquid Reference Manual [Contents][Index]
Modules are listed depth-first from the system components tree.
esrap-liquid (system).
Next: Packages, Previous: Modules, Up: The esrap-liquid Reference Manual [Contents][Index]
Files are sorted by type and then listed depth-first from the systems components trees.
Next: esrap-liquid/src/package.lisp, Previous: Lisp, Up: Lisp [Contents][Index]
esrap-liquid (system).
Next: esrap-liquid/src/conditions.lisp, Previous: esrap-liquid/esrap-liquid.asd, Up: Lisp [Contents][Index]
src (module).
Next: esrap-liquid/src/miscellany.lisp, Previous: esrap-liquid/src/package.lisp, Up: Lisp [Contents][Index]
package.lisp (file).
src (module).
Next: esrap-liquid/src/iterators.lisp, Previous: esrap-liquid/src/conditions.lisp, Up: Lisp [Contents][Index]
conditions.lisp (file).
src (module).
text (function).
Next: esrap-liquid/src/rule-storage.lisp, Previous: esrap-liquid/src/miscellany.lisp, Up: Lisp [Contents][Index]
miscellany.lisp (file).
src (module).
Next: esrap-liquid/src/memoization.lisp, Previous: esrap-liquid/src/iterators.lisp, Up: Lisp [Contents][Index]
iterators.lisp (file).
src (module).
Next: esrap-liquid/src/macro.lisp, Previous: esrap-liquid/src/rule-storage.lisp, Up: Lisp [Contents][Index]
rule-storage.lisp (file).
src (module).
register-context (macro).
Next: esrap-liquid/src/esrap.lisp, Previous: esrap-liquid/src/memoization.lisp, Up: Lisp [Contents][Index]
memoization.lisp (file).
src (module).
Next: esrap-liquid/src/basic-rules.lisp, Previous: esrap-liquid/src/macro.lisp, Up: Lisp [Contents][Index]
macro.lisp (file).
src (module).
Next: esrap-liquid/src/esrap-env.lisp, Previous: esrap-liquid/src/esrap.lisp, Up: Lisp [Contents][Index]
esrap.lisp (file).
src (module).
any-string (macro).
Next: esrap-liquid/src/iter-extensions.lisp, Previous: esrap-liquid/src/basic-rules.lisp, Up: Lisp [Contents][Index]
basic-rules.lisp (file).
src (module).
Previous: esrap-liquid/src/esrap-env.lisp, Up: Lisp [Contents][Index]
esrap-env.lisp (file).
src (module).
mk-tokenizer (function).
Next: esrap-liquid/example-symbol-table.lisp, Previous: Static, Up: Static [Contents][Index]
src (module).
esrap-liquid (system).
Next: esrap-liquid/README, Previous: esrap-liquid/example-sexp.lisp, Up: Static [Contents][Index]
example-sexp.lisp (file).
esrap-liquid (system).
Previous: esrap-liquid/example-symbol-table.lisp, Up: Static [Contents][Index]
example-symbol-table.lisp (file).
esrap-liquid (system).
Next: Definitions, Previous: Files, Up: The esrap-liquid Reference Manual [Contents][Index]
Packages are listed by definition order.
Next: esrap-liquid-system, Previous: Packages, Up: Packages [Contents][Index]
Previous: esrap-liquid, Up: Packages [Contents][Index]
Next: Indexes, Previous: Packages, Up: The esrap-liquid Reference Manual [Contents][Index]
Definitions are sorted by export status, category, package, and then by lexicographic order.
Next: Internals, Previous: Definitions, Up: Definitions [Contents][Index]
Next: Ordinary functions, Previous: Public Interface, Up: Public Interface [Contents][Index]
Succeeds, whenever parsing of EXPR fails. Does not consume, returns NIL, for compatibility with TEXT
Succeeds, whenever parsing of EXPR fails. Consumes, assumes than EXPR parses just one character.
LIST with automatic descent wrapping.
PROG1 with automatic descent wrapping.
Prog Middle.
PROGN with automatic descent wrapping.
Next: Standalone methods, Previous: Macros, Up: Public Interface [Contents][Index]
Parses TEXT using EXPRESSION from START to END. Incomplete parses are allowed only if JUNK-ALLOWED is true.
Parses STREAM using EXPRESSION. Incomplete parses are allowed if JUNK-ALLOWED is true.
Arguments must be strings, or lists whose leaves are strings. Catenates all the strings in arguments into a single string.
Previous: Ordinary functions, Up: Public Interface [Contents][Index]
Previous: Public Interface, Up: Definitions [Contents][Index]
Assoc list used to capture temporary variables
Next: Ordinary functions, Previous: Special variables, Up: Internals [Contents][Index]
Extra level of indirection in *CAP-STASH* is needed to be able to insert new values in it
Next: Generic functions, Previous: Macros, Up: Internals [Contents][Index]
If we actually only use half of the buffer, shrink it in half.
Relative fast-forward
Relative rewind
Next: Conditions, Previous: Ordinary functions, Up: Internals [Contents][Index]
Pop the last element of the buffer, but not before the start pointer
Place object in the end of the buffer, necessarily increasing its size
text.
Accessor for cached parsing results.
Shrink buffer (and cache) the hard way, actually reallocating the buffer and cleaning up cache
path.
Main method of iteration protocol
Shrink buffer (and cache) the soft way, just moving the start pointer
T if the given iter is at the start. True by default.
Next: Classes, Previous: Generic functions, Up: Internals [Contents][Index]
error.
(quote nil)
:msg
Signaled when an Esrap parse fails. Use ESRAP-ERROR-TEXT to obtain the string that was being parsed, and ESRAP-ERROR-POSITION the position at which the error occurred.
parse-error.
(quote nil)
:text
This slot is read-only.
common-lisp.
(quote nil)
:position
This slot is read-only.
(quote nil)
:reason
This slot is read-only.
Signaled when left recursion is detected during Esrap parsing. LEFT-RECURSION-NONTERMINAL names the symbol for which left recursion was detected, and LEFT-RECURSION-PATH lists nonterminals of which the left recursion cycle consists.
:rule-stack
error.
Previous: Conditions, Up: Internals [Contents][Index]
common-lisp.
(error "steam is required argument")
:stream
Previous: Definitions, Up: The esrap-liquid Reference Manual [Contents][Index]
| Jump to: | !
%
&
(
-
<
?
∅
A B C D E F G H I J L M N P R S T W |
|---|
| Jump to: | !
%
&
(
-
<
?
∅
A B C D E F G H I J L M N P R S T W |
|---|
Next: Data types, Previous: Functions, Up: Indexes [Contents][Index]
| Jump to: | *
B C M N P R S T V |
|---|
| Jump to: | *
B C M N P R S T V |
|---|
| Jump to: | B C E F I L M P R S |
|---|
| Jump to: | B C E F I L M P R S |
|---|