Update Lemon documentation. Patches from sgbeal.
FossilOrigin-Name: f5dc83442bf010bc4083e083b3a1acbb9918b7e685ca676dd899a0e09df196bc
This commit is contained in:
+125
-127
@@ -27,7 +27,7 @@ parser generator.</p>
|
||||
|
||||
<p>The language parser code created by Lemon is very robust and
|
||||
is well-suited for use in internet-facing applications that need to
|
||||
safely process maliciously crafted inputs.
|
||||
safely process maliciously crafted inputs.</p>
|
||||
|
||||
<p>The "lemon.exe" command-line tool itself works great when given a valid
|
||||
input grammar file and almost always gives helpful
|
||||
@@ -48,36 +48,36 @@ To summarize:</p>
|
||||
<p>The main goal of Lemon is to translate a context free grammar (CFG)
|
||||
for a particular language into C code that implements a parser for
|
||||
that language.
|
||||
The program has two inputs:
|
||||
The program has two inputs:</p>
|
||||
<ul>
|
||||
<li>The grammar specification.
|
||||
<li>A parser template file.
|
||||
</ul>
|
||||
Typically, only the grammar specification is supplied by the programmer.
|
||||
<p>Typically, only the grammar specification is supplied by the programmer.
|
||||
Lemon comes with a default parser template which works fine for most
|
||||
applications. But the user is free to substitute a different parser
|
||||
template if desired.</p>
|
||||
|
||||
<p>Depending on command-line options, Lemon will generate up to
|
||||
three output files.
|
||||
three output files.</p>
|
||||
<ul>
|
||||
<li>C code to implement the parser.
|
||||
<li>A header file defining an integer ID for each terminal symbol.
|
||||
<li>An information file that describes the states of the generated parser
|
||||
automaton.
|
||||
</ul>
|
||||
By default, all three of these output files are generated.
|
||||
<p>By default, all three of these output files are generated.
|
||||
The header file is suppressed if the "-m" command-line option is
|
||||
used and the report file is omitted when "-q" is selected.</p>
|
||||
|
||||
<p>The grammar specification file uses a ".y" suffix, by convention.
|
||||
In the examples used in this document, we'll assume the name of the
|
||||
grammar file is "gram.y". A typical use of Lemon would be the
|
||||
following command:
|
||||
following command:</p>
|
||||
<pre>
|
||||
lemon gram.y
|
||||
</pre>
|
||||
This command will generate three output files named "gram.c",
|
||||
<p>This command will generate three output files named "gram.c",
|
||||
"gram.h" and "gram.out".
|
||||
The first is C code to implement the parser. The second
|
||||
is the header file that defines numerical values for all
|
||||
@@ -88,11 +88,11 @@ the states used by the parser automaton.</p>
|
||||
|
||||
<p>The behavior of Lemon can be modified using command-line options.
|
||||
You can obtain a list of the available command-line options together
|
||||
with a brief explanation of what each does by typing
|
||||
with a brief explanation of what each does by typing</p>
|
||||
<pre>
|
||||
lemon "-?"
|
||||
</pre>
|
||||
As of this writing, the following command-line options are supported:
|
||||
<p>As of this writing, the following command-line options are supported:</p>
|
||||
<ul>
|
||||
<li><b>-b</b>
|
||||
Show only the basis for each parser state in the report file.
|
||||
@@ -144,11 +144,11 @@ complete system.</p>
|
||||
|
||||
<p>Before a program begins using a Lemon-generated parser, the program
|
||||
must first create the parser.
|
||||
A new parser is created as follows:
|
||||
A new parser is created as follows:</p>
|
||||
<pre>
|
||||
void *pParser = ParseAlloc( malloc );
|
||||
</pre>
|
||||
The ParseAlloc() routine allocates and initializes a new parser and
|
||||
<p>The ParseAlloc() routine allocates and initializes a new parser and
|
||||
returns a pointer to it.
|
||||
The actual data structure used to represent a parser is opaque —
|
||||
its internal structure is not visible or usable by the calling routine.
|
||||
@@ -158,22 +158,22 @@ The sole argument to the ParseAlloc() routine is a pointer to the
|
||||
subroutine used to allocate memory. Typically this means malloc().</p>
|
||||
|
||||
<p>After a program is finished using a parser, it can reclaim all
|
||||
memory allocated by that parser by calling
|
||||
memory allocated by that parser by calling</p>
|
||||
<pre>
|
||||
ParseFree(pParser, free);
|
||||
</pre>
|
||||
The first argument is the same pointer returned by ParseAlloc(). The
|
||||
<p>The first argument is the same pointer returned by ParseAlloc(). The
|
||||
second argument is a pointer to the function used to release bulk
|
||||
memory back to the system.</p>
|
||||
|
||||
<p>After a parser has been allocated using ParseAlloc(), the programmer
|
||||
must supply the parser with a sequence of tokens (terminal symbols) to
|
||||
be parsed. This is accomplished by calling the following function
|
||||
once for each token:
|
||||
once for each token:<p>
|
||||
<pre>
|
||||
Parse(pParser, hTokenID, sTokenData, pArg);
|
||||
</pre>
|
||||
The first argument to the Parse() routine is the pointer returned by
|
||||
<p>The first argument to the Parse() routine is the pointer returned by
|
||||
ParseAlloc().
|
||||
The second argument is a small positive integer that tells the parser the
|
||||
type of the next token in the data stream.
|
||||
@@ -199,7 +199,7 @@ This is a convenient mechanism for passing state information down
|
||||
to the action routines without having to use global variables.</p>
|
||||
|
||||
<p>A typical use of a Lemon parser might look something like the
|
||||
following:
|
||||
following:</p>
|
||||
<pre>
|
||||
1 ParseTree *ParseFile(const char *zFilename){
|
||||
2 Tokenizer *pTokenizer;
|
||||
@@ -220,7 +220,7 @@ following:
|
||||
17 return sState.treeRoot;
|
||||
18 }
|
||||
</pre>
|
||||
This example shows a user-written routine that parses a file of
|
||||
<p>This example shows a user-written routine that parses a file of
|
||||
text and returns a pointer to the parse tree.
|
||||
(All error-handling code is omitted from this example to keep it
|
||||
simple.)
|
||||
@@ -232,7 +232,7 @@ integer variable hTokenId. The sToken variable is assumed to be
|
||||
some kind of structure that contains details about each token,
|
||||
such as its complete text, what line it occurs on, etc.</p>
|
||||
|
||||
<p>This example also assumes the existence of structure of type
|
||||
<p>This example also assumes the existence of a structure of type
|
||||
ParserState that holds state information about a particular parse.
|
||||
An instance of such a structure is created on line 6 and initialized
|
||||
on line 10. A pointer to this structure is passed into the Parse()
|
||||
@@ -243,7 +243,7 @@ appropriate. In the example, we note that the treeRoot field of
|
||||
the ParserState structure is left pointing to the root of the parse
|
||||
tree.</p>
|
||||
|
||||
<p>The core of this example as it relates to Lemon is as follows:
|
||||
<p>The core of this example as it relates to Lemon is as follows:</p>
|
||||
<pre>
|
||||
ParseFile(){
|
||||
pParser = ParseAlloc( malloc );
|
||||
@@ -254,7 +254,7 @@ tree.</p>
|
||||
ParseFree(pParser, free );
|
||||
}
|
||||
</pre>
|
||||
Basically, what a program has to do to use a Lemon-generated parser
|
||||
<p>Basically, what a program has to do to use a Lemon-generated parser
|
||||
is first create the parser, then send it lots of tokens obtained by
|
||||
tokenizing an input source. When the end of input is reached, the
|
||||
Parse() routine should be called one last time with a token type
|
||||
@@ -265,11 +265,11 @@ parser by calling ParseFree().</p>
|
||||
<p>There is one other interface routine that should be mentioned
|
||||
before we move on.
|
||||
The ParseTrace() function can be used to generate debugging output
|
||||
from the parser. A prototype for this routine is as follows:
|
||||
from the parser. A prototype for this routine is as follows:</p>
|
||||
<pre>
|
||||
ParseTrace(FILE *stream, char *zPrefix);
|
||||
</pre>
|
||||
After this routine is called, a short (one-line) message is written
|
||||
<p>After this routine is called, a short (one-line) message is written
|
||||
to the designated output stream every time the parser changes states
|
||||
or calls an action routine. Each such message is prefaced using
|
||||
the text given by zPrefix. This debugging output can be turned off
|
||||
@@ -279,7 +279,7 @@ by calling ParseTrace() again with a first argument of NULL (0).</p>
|
||||
|
||||
<p>Programmers who have previously used the yacc or bison parser
|
||||
generator will notice several important differences between yacc and/or
|
||||
bison and Lemon.
|
||||
bison and Lemon.</p>
|
||||
<ul>
|
||||
<li>In yacc and bison, the parser calls the tokenizer. In Lemon,
|
||||
the tokenizer calls the parser.
|
||||
@@ -288,7 +288,7 @@ bison and Lemon.
|
||||
<li>Lemon allows multiple parsers to be running simultaneously. Yacc
|
||||
and bison do not.
|
||||
</ul>
|
||||
These differences may cause some initial confusion for programmers
|
||||
<p>These differences may cause some initial confusion for programmers
|
||||
with prior yacc and bison experience.
|
||||
But after years of experience using Lemon, I firmly
|
||||
believe that the Lemon way of doing things is better.</p>
|
||||
@@ -307,11 +307,11 @@ specifies additional information Lemon requires to do its job.
|
||||
Most of the work in using Lemon is in writing an appropriate
|
||||
grammar file.</p>
|
||||
|
||||
<p>The grammar file for Lemon is, for the most part, free format.
|
||||
<p>The grammar file for Lemon is, for the most part, a free format.
|
||||
It does not have sections or divisions like yacc or bison. Any
|
||||
declaration can occur at any point in the file.
|
||||
Lemon ignores whitespace (except where it is needed to separate
|
||||
tokens), and it honors the same commenting conventions as C and C++.</p>
|
||||
declaration can occur at any point in the file. Lemon ignores
|
||||
whitespace (except where it is needed to separate tokens), and it
|
||||
honors the same commenting conventions as C and C++.</p>
|
||||
|
||||
<h3>Terminals and Nonterminals</h3>
|
||||
|
||||
@@ -351,14 +351,13 @@ Rules can occur in any order, except that the left-hand side of the
|
||||
first rule is assumed to be the start symbol for the grammar (unless
|
||||
specified otherwise using the <tt><a href='#start_symbol'>%start_symbol</a></tt>
|
||||
directive described below.)
|
||||
A typical sequence of grammar rules might look something like this:
|
||||
A typical sequence of grammar rules might look something like this:</p>
|
||||
<pre>
|
||||
expr ::= expr PLUS expr.
|
||||
expr ::= expr TIMES expr.
|
||||
expr ::= LPAREN expr RPAREN.
|
||||
expr ::= VALUE.
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
<p>There is one non-terminal in this example, "expr", and five
|
||||
terminal symbols or tokens: "PLUS", "TIMES", "LPAREN",
|
||||
@@ -370,11 +369,10 @@ by the parser.
|
||||
In Lemon, this action is specified by putting the C code (contained
|
||||
within curly braces <tt>{...}</tt>) immediately after the
|
||||
period that closes the rule.
|
||||
For example:
|
||||
For example:</p>
|
||||
<pre>
|
||||
expr ::= expr PLUS expr. { printf("Doing an addition...\n"); }
|
||||
</pre>
|
||||
</p>
|
||||
|
||||
<p>In order to be useful, grammar actions must normally be linked to
|
||||
their associated grammar rules.
|
||||
@@ -391,18 +389,18 @@ rule and say "$7" when you really mean "$8".</p>
|
||||
<p>Lemon avoids the need to count grammar symbols by assigning symbolic
|
||||
names to each symbol in a grammar rule and then using those symbolic
|
||||
names in the action.
|
||||
In yacc or bison, one would write this:
|
||||
In yacc or bison, one would write this:</p>
|
||||
<pre>
|
||||
expr -> expr PLUS expr { $$ = $1 + $3; };
|
||||
</pre>
|
||||
But in Lemon, the same rule becomes the following:
|
||||
<p>But in Lemon, the same rule becomes the following:</p>
|
||||
<pre>
|
||||
expr(A) ::= expr(B) PLUS expr(C). { A = B+C; }
|
||||
</pre>
|
||||
In the Lemon rule, any symbol in parentheses after a grammar rule
|
||||
<p>In the Lemon rule, any symbol in parentheses after a grammar rule
|
||||
symbol becomes a place holder for that symbol in the grammar rule.
|
||||
This place holder can then be used in the associated C action to
|
||||
stand for the value of that symbol.<p>
|
||||
stand for the value of that symbol.</p>
|
||||
|
||||
<p>The Lemon notation for linking a grammar rule with its reduce
|
||||
action is superior to yacc/bison on several counts.
|
||||
@@ -412,11 +410,11 @@ Secondly, if a terminal or nonterminal in a Lemon grammar rule
|
||||
includes a linking symbol in parentheses but that linking symbol
|
||||
is not actually used in the reduce action, then an error message
|
||||
is generated.
|
||||
For example, the rule
|
||||
For example, the rule</p>
|
||||
<pre>
|
||||
expr(A) ::= expr(B) PLUS expr(C). { A = B; }
|
||||
</pre>
|
||||
will generate an error because the linking symbol "C" is used
|
||||
<p>will generate an error because the linking symbol "C" is used
|
||||
in the grammar rule but not in the reduce action.</p>
|
||||
|
||||
<p>The Lemon notation for linking grammar rules to reduce actions
|
||||
@@ -424,7 +422,7 @@ also facilitates the use of destructors for reclaiming memory
|
||||
allocated by the values of terminals and nonterminals on the
|
||||
right-hand side of a rule.</p>
|
||||
|
||||
<a name='precrules'></a>
|
||||
<a id='precrules'></a>
|
||||
<h3>Precedence Rules</h3>
|
||||
|
||||
<p>Lemon resolves parsing ambiguities in exactly the same way as
|
||||
@@ -443,50 +441,50 @@ using the
|
||||
mentioned in earlier directives have a lower precedence than
|
||||
terminal symbols mentioned in later directives. For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%left AND.
|
||||
%left OR.
|
||||
%nonassoc EQ NE GT GE LT LE.
|
||||
%left PLUS MINUS.
|
||||
%left TIMES DIVIDE MOD.
|
||||
%right EXP NOT.
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>In the preceding sequence of directives, the AND operator is
|
||||
defined to have the lowest precedence. The OR operator is one
|
||||
precedence level higher. And so forth. Hence, the grammar would
|
||||
attempt to group the ambiguous expression
|
||||
attempt to group the ambiguous expression</p>
|
||||
<pre>
|
||||
a AND b OR c
|
||||
</pre>
|
||||
like this
|
||||
<p>like this</p>
|
||||
<pre>
|
||||
a AND (b OR c).
|
||||
</pre>
|
||||
The associativity (left, right or nonassoc) is used to determine
|
||||
<p>The associativity (left, right or nonassoc) is used to determine
|
||||
the grouping when the precedence is the same. AND is left-associative
|
||||
in our example, so
|
||||
in our example, so</p>
|
||||
<pre>
|
||||
a AND b AND c
|
||||
</pre>
|
||||
is parsed like this
|
||||
<p>is parsed like this</p>
|
||||
<pre>
|
||||
(a AND b) AND c.
|
||||
</pre>
|
||||
The EXP operator is right-associative, though, so
|
||||
<p>The EXP operator is right-associative, though, so</p>
|
||||
<pre>
|
||||
a EXP b EXP c
|
||||
</pre>
|
||||
is parsed like this
|
||||
<p>is parsed like this</p>
|
||||
<pre>
|
||||
a EXP (b EXP c).
|
||||
</pre>
|
||||
The nonassoc precedence is used for non-associative operators.
|
||||
So
|
||||
<p>The nonassoc precedence is used for non-associative operators.
|
||||
So</p>
|
||||
<pre>
|
||||
a EQ b EQ c
|
||||
</pre>
|
||||
is an error.</p>
|
||||
<p>is an error.</p>
|
||||
|
||||
<p>The precedence of non-terminals is transferred to rules as follows:
|
||||
The precedence of a grammar rule is equal to the precedence of the
|
||||
@@ -497,9 +495,9 @@ you can specify an alternative precedence symbol by putting the
|
||||
symbol in square braces after the period at the end of the rule and
|
||||
before any C-code. For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
expr = MINUS expr. [NOT]
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>This rule has a precedence equal to that of the NOT symbol, not the
|
||||
MINUS symbol as would have been the case by default.</p>
|
||||
@@ -508,7 +506,7 @@ MINUS symbol as would have been the case by default.</p>
|
||||
symbols and individual
|
||||
grammar rules, we can now explain precisely how parsing conflicts
|
||||
are resolved in Lemon. Shift-reduce conflicts are resolved
|
||||
as follows:
|
||||
as follows:</p>
|
||||
<ul>
|
||||
<li> If either the token to be shifted or the rule to be reduced
|
||||
lacks precedence information, then resolve in favor of the
|
||||
@@ -528,7 +526,7 @@ as follows:
|
||||
<li> Otherwise, resolve the conflict by doing the shift, and
|
||||
report a parsing conflict.
|
||||
</ul>
|
||||
Reduce-reduce conflicts are resolved this way:
|
||||
<p>Reduce-reduce conflicts are resolved this way:</p>
|
||||
<ul>
|
||||
<li> If either reduce rule
|
||||
lacks precedence information, then resolve in favor of the
|
||||
@@ -553,7 +551,7 @@ grammar rules. It doesn't matter. The relative order of
|
||||
directives used to assign precedence to terminals is important, but
|
||||
other than that, the order of directives in Lemon is arbitrary.</p>
|
||||
|
||||
<p>Lemon supports the following special directives:
|
||||
<p>Lemon supports the following special directives:</p>
|
||||
<ul>
|
||||
<li><tt><a href='#pcode'>%code</a></tt>
|
||||
<li><tt><a href='#default_destructor'>%default_destructor</a></tt>
|
||||
@@ -584,10 +582,10 @@ other than that, the order of directives in Lemon is arbitrary.</p>
|
||||
<li><tt><a href='#ptype'>%type</a></tt>
|
||||
<li><tt><a href='#pwildcard'>%wildcard</a></tt>
|
||||
</ul>
|
||||
Each of these directives will be described separately in the
|
||||
<p>Each of these directives will be described separately in the
|
||||
following sections:</p>
|
||||
|
||||
<a name='pcode'></a>
|
||||
<a id='pcode'></a>
|
||||
<h4>The <tt>%code</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%code</tt> directive is used to specify additional C code that
|
||||
@@ -599,13 +597,13 @@ the <tt><a href='#pinclude'>%include</a></tt> directive except that
|
||||
a tokenizer or even the "main()" function
|
||||
as part of the output file.</p>
|
||||
|
||||
<a name='default_destructor'></a>
|
||||
<a id='default_destructor'></a>
|
||||
<h4>The <tt>%default_destructor</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%default_destructor</tt> directive specifies a destructor to
|
||||
use for non-terminals that do not have their own destructor
|
||||
specified by a separate <tt>%destructor</tt> directive. See the documentation
|
||||
on the <tt><a name='#destructor'>%destructor</a></tt> directive below for
|
||||
on the <tt><a href='#destructor'>%destructor</a></tt> directive below for
|
||||
additional information.</p>
|
||||
|
||||
<p>In some grammars, many different non-terminal symbols have the
|
||||
@@ -613,14 +611,14 @@ same data type and hence the same destructor. This directive is
|
||||
a convenient way to specify the same destructor for all those
|
||||
non-terminals using a single statement.</p>
|
||||
|
||||
<a name='default_type'></a>
|
||||
<a id='default_type'></a>
|
||||
<h4>The <tt>%default_type</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal
|
||||
symbols that do not have their own data type defined using a separate
|
||||
<tt><a href='#ptype'>%type</a></tt> directive.</p>
|
||||
|
||||
<a name='destructor'></a>
|
||||
<a id='destructor'></a>
|
||||
<h4>The <tt>%destructor</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%destructor</tt> directive is used to specify a destructor for
|
||||
@@ -630,24 +628,24 @@ directive which is used to specify a destructor for terminal symbols.)</p>
|
||||
|
||||
<p>A non-terminal's destructor is called to dispose of the
|
||||
non-terminal's value whenever the non-terminal is popped from
|
||||
the stack. This includes all of the following circumstances:
|
||||
the stack. This includes all of the following circumstances:</p>
|
||||
<ul>
|
||||
<li> When a rule reduces and the value of a non-terminal on
|
||||
the right-hand side is not linked to C code.
|
||||
<li> When the stack is popped during error processing.
|
||||
<li> When the ParseFree() function runs.
|
||||
</ul>
|
||||
The destructor can do whatever it wants with the value of
|
||||
<p>The destructor can do whatever it wants with the value of
|
||||
the non-terminal, but its design is to deallocate memory
|
||||
or other resources held by that non-terminal.</p>
|
||||
|
||||
<p>Consider an example:
|
||||
<p>Consider an example:</p>
|
||||
<pre>
|
||||
%type nt {void*}
|
||||
%destructor nt { free($$); }
|
||||
nt(A) ::= ID NUM. { A = malloc( 100 ); }
|
||||
</pre>
|
||||
This example is a bit contrived, but it serves to illustrate how
|
||||
<p>This example is a bit contrived, but it serves to illustrate how
|
||||
destructors work. The example shows a non-terminal named
|
||||
"nt" that holds values of type "void*". When the rule for
|
||||
an "nt" reduces, it sets the value of the non-terminal to
|
||||
@@ -670,18 +668,18 @@ the destructor is not called in this circumstance.</p>
|
||||
allocated objects when they go out of scope.
|
||||
To do the same using yacc or bison is much more difficult.</p>
|
||||
|
||||
<a name='extraarg'></a>
|
||||
<a id='extraarg'></a>
|
||||
<h4>The <tt>%extra_argument</tt> directive</h4>
|
||||
|
||||
The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
|
||||
<p>The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
|
||||
to the parameter list of the Parse() function it generates. Lemon
|
||||
doesn't do anything itself with this extra argument, but it does
|
||||
make the argument available to C-code action routines, destructors,
|
||||
and so forth. For example, if the grammar file contains:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%extra_argument { MyStruct *pAbc }
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>Then the Parse() function generated will have an 4th parameter
|
||||
of type "MyStruct*" and all action routines will have access to
|
||||
@@ -690,29 +688,29 @@ in the most recent call to Parse().</p>
|
||||
|
||||
<p>The <tt>%extra_context</tt> directive works the same except that it
|
||||
is passed in on the ParseAlloc() or ParseInit() routines instead of
|
||||
on Parse().
|
||||
on Parse().</p>
|
||||
|
||||
<a name='extractx'></a>
|
||||
<a id='extractx'></a>
|
||||
<h4>The <tt>%extra_context</tt> directive</h4>
|
||||
|
||||
The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter
|
||||
to the parameter list of the ParseAlloc() and ParseInif() functions. Lemon
|
||||
<p>The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter
|
||||
to the parameter list of the ParseAlloc() and ParseInit() functions. Lemon
|
||||
doesn't do anything itself with these extra argument, but it does
|
||||
store the value make it available to C-code action routines, destructors,
|
||||
and so forth. For example, if the grammar file contains:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%extra_context { MyStruct *pAbc }
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>Then the ParseAlloc() and ParseInit() functions will have an 2nd parameter
|
||||
of type "MyStruct*" and all action routines will have access to
|
||||
a variable named "pAbc" that is the value of that 2nd parameter.</p>
|
||||
|
||||
<p>The <tt>%extra_argument</tt> directive works the same except that it
|
||||
is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().
|
||||
is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().</p>
|
||||
|
||||
<a name='pfallback'></a>
|
||||
<a id='pfallback'></a>
|
||||
<h4>The <tt>%fallback</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one
|
||||
@@ -729,7 +727,7 @@ obscure language keyword for an identifier. The <tt>%fallback</tt> directive
|
||||
provides a mechanism to tell the parser: "If you are unable to parse
|
||||
this keyword, try treating it as an identifier instead."</p>
|
||||
|
||||
<p>The syntax of <tt>%fallback</tt> is as follows:
|
||||
<p>The syntax of <tt>%fallback</tt> is as follows:</p>
|
||||
|
||||
<blockquote>
|
||||
<tt>%fallback</tt> <i>ID</i> <i>TOKEN...</i> <b>.</b>
|
||||
@@ -742,7 +740,7 @@ token to which all the other tokens fall back to. The second and subsequent
|
||||
arguments are tokens which fall back to the token identified by the first
|
||||
argument.</p>
|
||||
|
||||
<a name='pifdef'></a>
|
||||
<a id='pifdef'></a>
|
||||
<h4>The <tt>%if</tt> directive and its friends</h4>
|
||||
|
||||
<p>The <tt>%if</tt>, <tt>%ifdef</tt>, <tt>%ifndef</tt>, <tt>%else</tt>,
|
||||
@@ -773,7 +771,7 @@ its corresponding <tt>%endif</tt>.</p>
|
||||
intended to be a single preprocessor symbol name, not a general expression.
|
||||
Use the "<tt>%if</tt>" directive for general expressions.</p>
|
||||
|
||||
<a name='pinclude'></a>
|
||||
<a id='pinclude'></a>
|
||||
<h4>The <tt>%include</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%include</tt> directive specifies C code that is included at the
|
||||
@@ -787,9 +785,9 @@ generated parser, in the same order as it appeared in the grammar.</p>
|
||||
preprocessor statements at the beginning of the generated parser.
|
||||
For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%include {#include <unistd.h>}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>This might be needed, for example, if some of the C actions in the
|
||||
grammar call functions that are prototyped in unistd.h.</p>
|
||||
@@ -797,7 +795,7 @@ grammar call functions that are prototyped in unistd.h.</p>
|
||||
<p>Use the <tt><a href="#pcode">%code</a></tt> directive to add code to
|
||||
the end of the generated parser.</p>
|
||||
|
||||
<a name='pleft'></a>
|
||||
<a id='pleft'></a>
|
||||
<h4>The <tt>%left</tt> directive</h4>
|
||||
|
||||
The <tt>%left</tt> directive is used (along with the
|
||||
@@ -809,14 +807,14 @@ a <tt>%left</tt> directive but before the next period (".") is
|
||||
given the same left-associative precedence value. Subsequent
|
||||
<tt>%left</tt> directives have higher precedence. For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%left AND.
|
||||
%left OR.
|
||||
%nonassoc EQ NE GT GE LT LE.
|
||||
%left PLUS MINUS.
|
||||
%left TIMES DIVIDE MOD.
|
||||
%right EXP NOT.
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>Note the period that terminates each <tt>%left</tt>,
|
||||
<tt>%right</tt> or <tt>%nonassoc</tt>
|
||||
@@ -827,29 +825,29 @@ a large amount of stack space if you make heavy use or right-associative
|
||||
operators. For this reason, it is recommended that you use <tt>%left</tt>
|
||||
rather than <tt>%right</tt> whenever possible.</p>
|
||||
|
||||
<a name='pname'></a>
|
||||
<a id='pname'></a>
|
||||
<h4>The <tt>%name</tt> directive</h4>
|
||||
|
||||
<p>By default, the functions generated by Lemon all begin with the
|
||||
five-character string "Parse". You can change this string to something
|
||||
different using the <tt>%name</tt> directive. For instance:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%name Abcde
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>Putting this directive in the grammar file will cause Lemon to generate
|
||||
functions named
|
||||
functions named</p>
|
||||
<ul>
|
||||
<li> AbcdeAlloc(),
|
||||
<li> AbcdeFree(),
|
||||
<li> AbcdeTrace(), and
|
||||
<li> Abcde().
|
||||
</ul>
|
||||
The <tt>%name</tt> directive allows you to generate two or more different
|
||||
</p>The <tt>%name</tt> directive allows you to generate two or more different
|
||||
parsers and link them all into the same executable.</p>
|
||||
|
||||
<a name='pnonassoc'></a>
|
||||
<a id='pnonassoc'></a>
|
||||
<h4>The <tt>%nonassoc</tt> directive</h4>
|
||||
|
||||
<p>This directive is used to assign non-associative precedence to
|
||||
@@ -858,7 +856,7 @@ one or more terminal symbols. See the section on
|
||||
or on the <tt><a href='#pleft'>%left</a></tt> directive
|
||||
for additional information.</p>
|
||||
|
||||
<a name='parse_accept'></a>
|
||||
<a id='parse_accept'></a>
|
||||
<h4>The <tt>%parse_accept</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is
|
||||
@@ -868,13 +866,13 @@ without error.</p>
|
||||
|
||||
<p>For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%parse_accept {
|
||||
printf("parsing complete!\n");
|
||||
}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='parse_failure'></a>
|
||||
<a id='parse_failure'></a>
|
||||
<h4>The <tt>%parse_failure</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%parse_failure</tt> directive specifies a block of C code that
|
||||
@@ -883,13 +881,13 @@ executed until the parser has tried and failed to resolve an input
|
||||
error using is usual error recovery strategy. The routine is
|
||||
only invoked when parsing is unable to continue.</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%parse_failure {
|
||||
fprintf(stderr,"Giving up. Parser is hopelessly lost...\n");
|
||||
}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='pright'></a>
|
||||
<a id='pright'></a>
|
||||
<h4>The <tt>%right</tt> directive</h4>
|
||||
|
||||
<p>This directive is used to assign right-associative precedence to
|
||||
@@ -897,7 +895,7 @@ one or more terminal symbols. See the section on
|
||||
<a href='#precrules'>precedence rules</a>
|
||||
or on the <a href='#pleft'>%left</a> directive for additional information.</p>
|
||||
|
||||
<a name='stack_overflow'></a>
|
||||
<a id='stack_overflow'></a>
|
||||
<h4>The <tt>%stack_overflow</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%stack_overflow</tt> directive specifies a block of C code that
|
||||
@@ -905,28 +903,28 @@ is executed if the parser's internal stack ever overflows. Typically
|
||||
this just prints an error message. After a stack overflow, the parser
|
||||
will be unable to continue and must be reset.</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%stack_overflow {
|
||||
fprintf(stderr,"Giving up. Parser stack overflow\n");
|
||||
}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>You can help prevent parser stack overflows by avoiding the use
|
||||
of right recursion and right-precedence operators in your grammar.
|
||||
Use left recursion and and left-precedence operators instead to
|
||||
encourage rules to reduce sooner and keep the stack size down.
|
||||
For example, do rules like this:
|
||||
For example, do rules like this:</p>
|
||||
<pre>
|
||||
list ::= list element. // left-recursion. Good!
|
||||
list ::= .
|
||||
</pre>
|
||||
Not like this:
|
||||
<p>Not like this:</p>
|
||||
<pre>
|
||||
list ::= element list. // right-recursion. Bad!
|
||||
list ::= .
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='stack_size'></a>
|
||||
<a id='stack_size'></a>
|
||||
<h4>The <tt>%stack_size</tt> directive</h4>
|
||||
|
||||
<p>If stack overflow is a problem and you can't resolve the trouble
|
||||
@@ -935,11 +933,11 @@ of the parser's stack using this directive. Put an positive integer
|
||||
after the <tt>%stack_size</tt> directive and Lemon will generate a parse
|
||||
with a stack of the requested size. The default value is 100.</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%stack_size 2000
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='start_symbol'></a>
|
||||
<a id='start_symbol'></a>
|
||||
<h4>The <tt>%start_symbol</tt> directive</h4>
|
||||
|
||||
<p>By default, the start symbol for the grammar that Lemon generates
|
||||
@@ -947,22 +945,22 @@ is the first non-terminal that appears in the grammar file. But you
|
||||
can choose a different start symbol using the
|
||||
<tt>%start_symbol</tt> directive.</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%start_symbol prog
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='syntax_error'></a>
|
||||
<a id='syntax_error'></a>
|
||||
<h4>The <tt>%syntax_error</tt> directive</h4>
|
||||
|
||||
<p>See <a href='#error_processing'>Error Processing</a>.</p>
|
||||
|
||||
<a name='token_class'></a>
|
||||
<a id='token_class'></a>
|
||||
<h4>The <tt>%token_class</tt> directive</h4>
|
||||
|
||||
<p>Undocumented. Appears to be related to the MULTITERMINAL concept.
|
||||
<a href='http://sqlite.org/src/fdiff?v1=796930d5fc2036c7&v2=624b24c5dc048e09&sbs=0'>Implementation</a>.</p>
|
||||
|
||||
<a name='token_destructor'></a>
|
||||
<a id='token_destructor'></a>
|
||||
<h4>The <tt>%token_destructor</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%destructor</tt> directive assigns a destructor to a non-terminal
|
||||
@@ -971,14 +969,14 @@ symbol. (See the description of the
|
||||
The <tt>%token_destructor</tt> directive does the same thing
|
||||
for all terminal symbols.</p>
|
||||
|
||||
<p>Unlike non-terminal symbols which may each have a different data type
|
||||
<p>Unlike non-terminal symbols, which may each have a different data type
|
||||
for their values, terminals all use the same data type (defined by
|
||||
the <tt><a href='#token_type'>%token_type</a></tt> directive)
|
||||
and so they use a common destructor.
|
||||
Other than that, the token destructor works just like the non-terminal
|
||||
destructors.</p>
|
||||
|
||||
<a name='token_prefix'></a>
|
||||
<a id='token_prefix'></a>
|
||||
<h4>The <tt>%token_prefix</tt> directive</h4>
|
||||
|
||||
<p>Lemon generates #defines that assign small integer constants
|
||||
@@ -986,26 +984,26 @@ to each terminal symbol in the grammar. If desired, Lemon will
|
||||
add a prefix specified by this directive
|
||||
to each of the #defines it generates.</p>
|
||||
|
||||
<p>So if the default output of Lemon looked like this:
|
||||
<p>So if the default output of Lemon looked like this:</p>
|
||||
<pre>
|
||||
#define AND 1
|
||||
#define MINUS 2
|
||||
#define OR 3
|
||||
#define PLUS 4
|
||||
</pre>
|
||||
You can insert a statement into the grammar like this:
|
||||
<p>You can insert a statement into the grammar like this:</p>
|
||||
<pre>
|
||||
%token_prefix TOKEN_
|
||||
</pre>
|
||||
to cause Lemon to produce these symbols instead:
|
||||
<p>to cause Lemon to produce these symbols instead:</p>
|
||||
<pre>
|
||||
#define TOKEN_AND 1
|
||||
#define TOKEN_MINUS 2
|
||||
#define TOKEN_OR 3
|
||||
#define TOKEN_PLUS 4
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<a name='token_type'></a><a name='ptype'></a>
|
||||
<a id='token_type'></a><a id='ptype'></a>
|
||||
<h4>The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
|
||||
|
||||
<p>These directives are used to specify the data types for values
|
||||
@@ -1016,9 +1014,9 @@ to the Parse() function generated by Lemon. Typically, you will
|
||||
make the value of a terminal symbol be a pointer to some kind of
|
||||
token structure. Like this:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%token_type {Token*}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>If the data type of terminals is not specified, the default value
|
||||
is "void*".</p>
|
||||
@@ -1028,9 +1026,9 @@ the data type of a non-terminal is a pointer to the root of a parse tree
|
||||
structure that contains all information about that non-terminal.
|
||||
For example:</p>
|
||||
|
||||
<p><pre>
|
||||
<pre>
|
||||
%type expr {Expr*}
|
||||
</pre></p>
|
||||
</pre>
|
||||
|
||||
<p>Each entry on the parser's stack is actually a union containing
|
||||
instances of all data types for every non-terminal and terminal symbol.
|
||||
@@ -1042,7 +1040,7 @@ non-terminal whose data type requires 1K of storage, then your 100
|
||||
entry parser stack will require 100K of heap space. If you are willing
|
||||
and able to pay that price, fine. You just need to know.</p>
|
||||
|
||||
<a name='pwildcard'></a>
|
||||
<a id='pwildcard'></a>
|
||||
<h4>The <tt>%wildcard</tt> directive</h4>
|
||||
|
||||
<p>The <tt>%wildcard</tt> directive is followed by a single token name and a
|
||||
@@ -1053,7 +1051,7 @@ match any input token.</p>
|
||||
the wildcard token and some other token, the other token is always used.
|
||||
The wildcard token is only matched if there are no alternatives.</p>
|
||||
|
||||
<a name='error_processing'></a>
|
||||
<a id='error_processing'></a>
|
||||
<h3>Error Processing</h3>
|
||||
|
||||
<p>After extensive experimentation over several years, it has been
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
C Fix\sa\scouple\sof\sunreachable\sbranches.
|
||||
D 2020-08-28T12:58:21.030
|
||||
C Update\sLemon\sdocumentation.\s\sPatches\sfrom\ssgbeal.
|
||||
D 2020-08-28T13:10:00.216
|
||||
F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
|
||||
F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
|
||||
F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724
|
||||
@@ -38,7 +38,7 @@ F configure 63af83d31b9fdf304f2dbb1e1638530d4ceff31702d1e19550d1fbf3bdf9471e x
|
||||
F configure.ac 40d01e89cb325c28b33f5957e61fede0bd17da2b5e37d9b223a90c8a318e88d4
|
||||
F contrib/sqlitecon.tcl 210a913ad63f9f991070821e599d600bd913e0ad
|
||||
F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd
|
||||
F doc/lemon.html 1edc0f916e771212792d4d077aedc05168bf13fd65d64d41b2c13e46ac0063a8
|
||||
F doc/lemon.html 5155bf346e59385ac8d14da0c1e895d8dbc5d225a7d93d3f8249cbfb3c938f55
|
||||
F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
|
||||
F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
|
||||
F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a
|
||||
@@ -1879,7 +1879,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
|
||||
F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
|
||||
F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
|
||||
F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
|
||||
P 1a04920998368e56276fd0b100be8343609c6ff8a731cf8e26a0490f9c6dabdf
|
||||
R 226559a22dbe15c2cb43501373224222
|
||||
P f2d26f2b11317abd4f993faa1a4df7afcd1a2d4e448ecc69ca05e9ebf102cd62
|
||||
R 218381265e6cb716367061639a662304
|
||||
U drh
|
||||
Z 999aada41f64ce3323554005b08290d8
|
||||
Z cdee44e4fe786c40d615ee283889a90d
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
f2d26f2b11317abd4f993faa1a4df7afcd1a2d4e448ecc69ca05e9ebf102cd62
|
||||
f5dc83442bf010bc4083e083b3a1acbb9918b7e685ca676dd899a0e09df196bc
|
||||
Reference in New Issue
Block a user