Token sets

Algorithms > Token sets

The decision on the next branching in the grammar can be made dependent in special cases of a look-ahead in the text or of the semantic predicates. In most cases however, which will discussed now, the grammatical alternative is chosen whose first symbol is recognized in the text next. Besides the preference rules already explained this can depend on the set of the tokens which are tested: tokens which aren't tested cannot cause any conflicts either. Therefore there is the possibility in the options of the project and in the local options, only to test expected tokens. However, it can just be desired to recognize conflicts early, too. So e.g. reserved words of a programming language may not be used for variables as names:

double int;  // error

The option is to test expected tokens only in this case has to be deactivated.

However, the token sets have to be discussed at the use of productions outside of the main parser.

Token sets of inclusions, sub-parsers and in a look-ahead

It's conspicuous for the case of comments that conflicts with the tokens of the main parser are unwanted.

CppComment ::= "/*" ( SKIP | STRING )* "*/"

// ! this definition isn't appropriate for nested comments

/* int iCount : Zähler */

If the keyword int were recognized here, the comment couldn't be parsed.

Inclusions can therefore form a new production system in the TextTransformer which is independent of the token set of the main parser.

Exactly the opposite applies to look-ahead productions: it is mostly desired here, that the same tokens as in the main parser are recognized.

The following rules are allowing a flexible adaptation of the token sets to the respective purposes:

1., Any production which isn't called by another production, i.e. every start production, forms the foundation for an independent production system with a token set of its own. The start rule of the main parser forms the first system.

2., The token sets tested in a production are constant. It is permitted to use a production in several systems. However, the tokens tested in it are determined by the system as a whose part it was parsed first.

3. The order, by which the systems are compiled, starts with the main parser. The look-ahead productions of the main parser are following and finally the inclusions and the sup-parsers are compiled. The same sequence applies to the subsystems respectively.

In accordance with these rules, an inclusion in most cases should consist of productions, which aren't used in the main parser (and also in no other system). For a look-ahead, however, is advisable mostly, to use productions of the system in which the look-ahead is carried out.

This page belongs to the TextTransformer Documentation