DOCUMENTATION.en.rdoc

# REX: Ruby Lex for Racc

## About

    Lexical Scanner Generator for Ruby, with Racc.

## Usage

    rex [options] grammarfile

    -o  --output-file  filename   designated output filename.
    -s  --stub                    append stub main for debug.
    -i  --ignorecase              ignore char case
    -C  --check-only              syntax check only.
        --independent             independent mode.
    -d  --debug                   print debug information
    -h  --help                    print usage.
        --version                 print version.
        --copyright               print copyright.

## Default Output Filename

    It destinate from foo.rex  to foo.rex.rb.
    This name is for a follow description.

      require 'foo.rex'

## Grammar File Format

    A definition is given in order of a header part, a rule part,
    and the footer part.  One or more sections are included in a rule part.
    As for each section, the head of the sentence starts by the keyword.

    Summary:

      [Header Part]
      "class" Foo
      ["option"
        [options] ]
      ["inner"
        [methods] ]
      ["macro"
        [macro-name  regular-expression] ]
      "rule"
        [start-state]  pattern  [actions]
      "end"
      [Footer Part]

### Grammar File Example

    class Foo
    macro
      BLANK         \s+
      DIGIT         \d+
    rule
      {BLANK}
      {DIGIT}       { [:NUMBER, text.to_i] }
      .             { [text, text] }
    end

## Header Part ( Optional )

    All the contents described before the definition of a rule part are
    posted to head of the output file.

## Footer Part ( Optional )

    All the contents described after the definition of a rule part are
    posted to tail of the output file.

## Rule Part

    Rule part is from the line which begins from the "class" keyword
    to the line which begins from the "end" keyword.
    The class name outputted after a keyword "class" is specified.
    If embellished with a module name, it will become a class in a module.
    The class which inherited Racc::Parser is generated.

### Rule Header Example

    class Foo
    class Bar::Foo

## Option Section ( Optional )

    "option" is start keyword.

      "ignorecase"  when pattern match, ignore char case.
      "stub"        append stub main for debug.
      "independent" independent mode, for it is not inherited Racc.

## Inner Section ( Optional )

    "inner" is start keyword.
    The contents defined here are defined by the inside of the class
    of the generated scanner.

## Macro Section ( Optional )

    "macro" is start keyword.
    One regular expression is named.
    A blank character (0x20) can be included by escaping by \ .

### Macro Section Example

    DIGIT         \d+
    IDENT         [a-zA-Z_][a-zA-Z0-9_]*
    BLANK         [\ \t]+
    REMIN         \/\*
    REMOUT        \*\/

## Rule Section

    "rule" is start keyword.

    [state]  pattern  [actions]

### state: Start State ( Optional )

    A start state is expressed with the identifier which prefaces ":".
    When the continuing alphabetic character is a capital letter,
    it will be in an exclusive start state.
    When it is a small letter, it will be in an inclusive start state.
    Initial value and default value of a start state is nil.

### pattern: String Pattern

    The regular expression for specifying a character string.
    The macro definition bundled with { } can be used for description
    of a regular expression.  A macro definition is used in order to use
    a regular expression including a blank.

### actions: Processing Actions ( Optional )

    Action is performed when a pattern is suited.
    The processing which creates a suitable token is defined.
    The arrangement whose token has the second clause of classification
    and a value, or nil.
    The following elements can be used in order to create a token.

      lineno    Line number     ( Read Only )
      text      Matched string  ( Read Only )
      state     Start state     ( Read/Write )

    Action is bundled with { }.  It is the block of Ruby.
    Don't use the function to change the flow of control exceeding a block.
    ( return, exit, next, break, ... )
    If action is omitted, the character string which matched will be canceled
    and will progress to the next scan.

### Rule Part Example

    {REMIN}                 { state = :REM ; [:REM_IN, text] }

REM {REMOUT} { state = nil ; [:REM_OUT, text] }
REM (.+)(?={REMOUT}) { [:COMMENT, text] }
  {BLANK} -?{DIGIT} { [:NUMBER, text.to_i] } {WORD} { [:word, text] } . { [text, text] }

## Comment ( Optional )

    From "#" to the end of the line becomes a comment in each line.

## Usage for Generated Class

### scan_setup()

    The event for initializing at the time of the execution start of a scanner.
    It is redefined and used.

### scan_str( str )

    Parse the string described by the defined grammar.
    Token is stored in an inside.

### scan_file( filename )

    Parse the file described by the defined grammar.
    Token is stored in an inside.

### next_token

    One token stored in the inside is taken out.
    The last returns nil.

## Notice

    This specification is provisional and may be changed without a preliminary
    announcement.