foam3

Table of Contents generated with DocToc

FOAM Grammar-Based Parsers for JavaScript

FOAM Grammar-Based Parsers for JavaScript

Introduction

FOAM includes a powerful parser library for JavaScript that uses parser combinators - simple parsers that can be combined to form larger, more complex parsers. This approach offers several key benefits over traditional parsing techniques like regular expressions:

Benefits of FOAM Grammar-Based Parsers

More Powerful Than Regular Expressions: Can handle recursive patterns, nested structures, and context-sensitive grammars that regular expressions cannot parse.
Composable and Modular: Build complex parsers from simple, reusable building blocks - similar to how complex UIs are built from simple components like buttons and text fields.
Readable and Maintainable: Parser definitions are more explicit and easier to understand than cryptic regex patterns, especially for complex parsing tasks.
Object-Oriented: FOAM grammars leverage OO principles like inheritance and composition, enabling code reuse and specialization.
Semantic Actions: Easily transform parsed input into useful data structures (ASTs, computed values, etc.) rather than just matching patterns.
Versatile Applications:
- Extracting data from files
- Understanding network requests
- Validating and reformatting user input
- Creating Domain-Specific Languages (DSLs)
- Building source code linters, refactoring tools, and syntax highlighters

How FOAM Parsers Work

FOAM parsers operate on PStreams (Parser Streams), which are immutable objects with this interface:

pos - Current character position in the input
head - First character at current position
tail - PStream for the next position
value - Result value associated with this PStream
setValue(value) - Creates new PStream at same position with new value

A Parser has a single method:

PStream parse(PStream stream);

It returns either a PStream advanced past consumed input (with the parsed value in .value), or undefined if parsing failed.

Simple Parsers

Simple parsers parse individual characters or literal strings without combining other parsers.

Setting Up

First, create a singleton instance for accessing the parser library:

var P = foam.parse.Parsers.create();

literal(str, opt_value)

Parses an exact string match.

var ps = foam.parse.StringPStream.create({str: 'abcdef'});
var parser = P.literal('abc');
ps = parser.parse(ps);
// ps.value = 'abc', ps.pos = 3

Optional value parameter: Return a different value on successful parse:

var boolParser = P.alt(
  P.literal('true', true), 
  P.literal('false', false)
);
// Returns actual boolean values instead of strings

literalIC(str)

Case-insensitive literal parser.

var parser = P.literalIC('abc');
// Matches: 'abc', 'ABC', 'Abc', 'aBc', etc.

anyChar()

Parses any single character.

var parser = P.anyChar();
// Matches any character except EOF

eof()

Matches end-of-file/end-of-input.

var parser = P.eof();
// Only succeeds at end of input

chars(str)

Parses any single character contained in the provided string.

var parser = P.chars('abc');
// Matches 'a', 'b', or 'c'

notChars(str)

Parses any single character NOT in the provided string (but not EOF).

var parser = P.notChars('abc');
// Matches any character except 'a', 'b', 'c', or EOF

range(start, end)

Parses any character within a range.

var digitParser = P.range('0', '9');
var letterParser = P.range('a', 'z');
// digitParser matches '0' through '9'
// letterParser matches 'a' through 'z'

Parser Combinators

Combinators compose simple parsers into complex ones.

seq(…parsers)

Sequential parser - all sub-parsers must succeed in order. Returns array of parsed values.

var parser = P.seq('a', 'b', 'c');
// Input 'abc' → ['a', 'b', 'c']

var parser = P.seq(P.range('a','z'), P.range('0','9'));
// Input 'r2' → ['r', '2']

Shortcut: String arguments automatically convert to literal() parsers.

seq1(index, …parsers)

Like seq() but returns only the value at specified index instead of an array.

var quote = P.literal('"');
var stringBody = P.str(P.repeat(P.not(quote, P.anyChar())));
var parser = P.seq1(1, quote, stringBody, quote);
// Input '"hello"' → 'hello' (ignores quotes, returns only stringBody)

seq0(…parsers)

Like seq() but returns no value - optimized for performance when you don’t need results.

var comment = P.seq0('/*', P.until0('*/'));
// Parses C-style comments efficiently without capturing content

alt(…parsers)

Alternative parser - succeeds if any sub-parser succeeds. Returns first successful match.

var parser = P.alt('A', 'B', 'C');
// Input 'B' → 'B'

repeat(parser, opt_delim, opt_min, opt_max)

Repeats parser zero or more times. Returns array of values.

var parser = P.str(P.repeat(P.range('0','9')));
// Input '123' → '123'
// Input '' → '' (zero matches is valid)

With delimiter: Parse values separated by delimiter (delimiter not included in result):

var parser = P.str(P.repeat(P.range('a','z'), ','));
// Input 'a,b,c' → 'abc'

With min/max repetitions:

var parser = P.repeat(P.range('a','z'), undefined, 5, 10);
// Requires at least 5 matches, continues up to 10

repeat0(parser, opt_delim, opt_min, opt_max)

Like repeat() but returns no value - optimized for performance.

const whitespace = P.repeat0(P.chars(' \t\n'), null, 1);
// Efficiently skips whitespace without capturing it

plus(parser, opt_delim)

Repeats parser one or more times (fails on zero matches).

var parser = P.str(P.plus(P.range('0','9')));
// Input '123' → '123'
// Input '' → undefined (fails)

optional(parser, opt_value)

Parser that always succeeds - returns parsed value or continues from same position.

var parser = P.seq(P.optional('header'), 'body');
// Input 'headerbody' → ['header', 'body']
// Input 'body' → [null, 'body']

With default value:

var parser = P.optional(P.literal('-'), '-');
// Returns '-' even if not present in input

not(parser, opt_parser)

Succeeds only if delegate parser fails.

var quote = P.literal('"');
var stringBody = P.str(P.repeat(P.not(quote, P.anyChar())));
// Parses everything except quotes

until(parser)

Parses input until the specified parser matches. The terminator is consumed but left out of the value, and the value is an array of characters (wrap in str() for a string).

var parser = P.str(P.until(P.literal('"')));
// Input 'abc"def' → 'abc', leaving the stream positioned after the quote

until0(parser)

Like until() but returns no value - optimized for performance.

var comment = P.seq0('/*', P.until0('*/'));
// Efficiently skips comment content

cut(parser)

Memory optimization parser that destroys the input PStream after a successful parse. This allows the JavaScript garbage collector to reclaim memory, enabling parsing of very large files that would otherwise cause heap exhaustion.

var parser = P.cut(P.seq('a', 'b', 'c'));
// After successful parse, input PStream memory is released

Why cut() is needed:

FOAM’s PStream is immutable - each tail operation creates a new PStream object. For large files (millions of characters), this creates millions of intermediate objects that can exhaust heap memory. Without cut(), a 60MB file might require 7+ GB of heap memory (116x the file size). With cut(), the same file needs only ~400MB (6x the file size).

Memory usage comparison:

Approach	Memory Ratio	60MB File Needs
Without cut	~116x	7+ GB
With cut	~6x	~400 MB

Important usage patterns:

cut() must be used carefully because it destroys the input PStream. There are two valid approaches for collecting results when using cut():

Approach 1: Using repeat with actions that return values

Actions return values directly, and repeat() collects them into an array:

foam.CLASS({
  name: 'LargeFileParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, cut, eof, not, repeat, seq, seq0, sym) {
      return {
        // repeat() collects values returned by actions
        START: repeat(not(eof(), sym('line'))),

        // cut() wraps the entire line alternatives
        line: cut(alt(
          sym('header'),
          sym('dataLine'),
          sym('ignored')
        )),

        // Each rule must consume its own terminator (newline)
        header: seq0(sym('headerContent'), sym('nl')),
        ignored: seq0(sym('ignoredContent'), sym('nl')),
        dataLine: seq(sym('field1'), ',', sym('field2'), sym('nl')),

        nl: alt('\r\n', '\n', '\r')
      };
    },

    // Action returns the record - repeat() collects these
    function dataLineAction(v) {
      return {
        field1: v[0],
        field2: v[2]
      };
    },

    // Header and ignored lines return null (filtered out or kept as null)
    function headerAction(v) { return null; },
    function ignoredAction(v) { return null; }
  ]
});

Approach 2: Using repeat0 with external collection

When you don’t need repeat() to collect values, use repeat0() and push to an external array:

foam.CLASS({
  name: 'LargeFileParser',
  extends: 'foam.parse.Grammar',

  properties: [
    {
      name: 'records_',
      factory: function() { return []; }
    }
  ],

  methods: [
    function grammar(alt, cut, eof, not, repeat0, seq, seq0, sym) {
      return {
        // repeat0() doesn't collect values
        START: repeat0(not(eof(), sym('line'))),

        line: cut(alt(
          sym('header'),
          sym('dataLine'),
          sym('ignored')
        )),

        header: seq0(sym('headerContent'), sym('nl')),
        ignored: seq0(sym('ignoredContent'), sym('nl')),
        dataLine: seq(sym('field1'), ',', sym('field2'), sym('nl')),

        nl: alt('\r\n', '\n', '\r')
      };
    },

    // Action pushes to external array
    function dataLineAction(v) {
      this.records_.push({
        field1: v[0],
        field2: v[2]
      });
      return null;
    },

    function STARTAction(v) {
      return this.records_;
    },

    function parseString(str) {
      this.records_ = []; // Reset for fresh parse
      this.SUPER(str);
      return this.records_;
    }
  ]
});

Key rules for using cut():

Wrap the main alt with cut() - not individual rules inside
Each rule must consume its own terminator - include newline in each line rule
Choose repeat vs repeat0 based on your needs - use repeat() to collect action return values, use repeat0() when you don’t need results collected

When to use cut():

Parsing files larger than 10MB
Processing files with hundreds of thousands of lines
When you encounter “JavaScript heap out of memory” errors
When memory usage is critical

When NOT to use cut():

Small files (<10MB) - overhead not worth it
When you need backtracking after a match
Simple parsers that don’t process large amounts of data

Value Transformation Parsers

These parsers modify how values are returned.

str(parser)

Converts array results into concatenated strings.

var parser = P.seq(P.range('a','z'), P.range('0','9'));
// Without str: ['r', '2']

var parser = P.str(P.seq(P.range('a','z'), P.range('0','9')));
// With str: 'r2'

substring(parser)

Returns the original input substring that was parsed (ignoring transformed values).

var word = P.plus(P.range('a','z'));
var sentence = P.repeat(word, P.plus(' '));

var parser = P.substring(sentence);
// Input 'abc   def   xyz' → 'abc   def   xyz'
// Preserves whitespace that would be lost with repeat()

Grammars

For complex or recursive parsers, use Grammars - collections of named, mutually recursive parsers.

Creating a Grammar

foam.CLASS({
  name: 'MyGrammar',
  extends: 'foam.parse.Grammar',
  
  methods: [
    function grammar(/* parser constructors as params */) {
      return {
        START: /* your top-level parser */,
        rule1: /* named parser */,
        rule2: /* named parser */
      };
    }
  ]
});

sym(name)

References a named parser within a grammar.

function grammar(str, sym, repeat, seq, seq1, eof) {
  return {
    START: seq1(0, sym('parens'), eof()),
    
    // Recursive reference to itself
    parens: str(repeat(str(seq('(', sym('parens'), ')'))))
  };
}

Example: Matched Parenthesis Parser

foam.CLASS({
  name: 'MatchedParensParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(str, sym, eof, repeat, seq, seq1) {
      return {
        START: seq1(0, sym('parens'), eof()),

        parens: str(repeat(
          str(seq('(', sym('parens'), ')'))
        ))
      };
    }
  ]
});

var p = MatchedParensParser.create();
// Matches: '', '()', '(())', '()()', '((())()(()))'
// Fails: ')(', '(()', '())'

Example: Phone Number Grammar

foam.CLASS({
  name: 'PhoneNumberParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, sym, seq1, seq, str, literal, optional, plus, range) {
      return {
        START: str(seq(
          optional(sym('countryCode')),
          alt(
            str(seq(sym('areaCode'), optional(' ', ' '), sym('simple'))),
            sym('simple'))
        )),

        simple: str(seq(sym('prefix'), optional(sym('separator'), '-'), sym('line'))),

        digit: range('0', '9'),

        countryCode: seq1(1, optional('+'), literal('1', '1 '), optional(' ')),

        areaCode: str(seq(
          optional('(', '('),
          sym('digit'), sym('digit'), sym('digit'),
          optional(')', ')'))),

        prefix: str(seq(sym('digit'), sym('digit'), sym('digit'))),

        separator: alt('-', ' '),

        line: str(seq(sym('digit'), sym('digit'), sym('digit'), sym('digit')))
      };
    }
  ]
});

var p = PhoneNumberParser.create();
// Matches: '5551234', '555-1234', '(555) 555-1234', '+1 (555) 555-1234'

Semantic Actions

Transform parsed values by defining action methods in your grammar.

Defining Actions

Create methods named <symbolName>Action:

foam.CLASS({
  name: 'VersionParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, not, sym, seq, str, optional, plus, repeat, range, anyChar) {
      return {
        START: repeat(sym('component'), optional(sym('separator'))),
        component: alt(sym('number'), sym('string')),
        digit: range('0', '9'),
        number: str(plus(sym('digit'))),
        string: str(plus(not(alt(sym('digit'), sym('separator')), anyChar()))),
        separator: alt('.', '-')
      };
    },

    // Semantic action for 'number' symbol
    function numberAction(v) { 
      return parseInt(v); // Convert string to integer
    }
  ]
});

var p = VersionParser.create();
// Input '3.14159-26b' → [3, 14159, 26, 'b']
// (the repeat() delimiter's value is discarded, so separators are not in the result)

Example: Expression Interpreter

foam.CLASS({
  name: 'ExpressionInterpreter',
  extends: 'foam.parse.Grammar',

  properties: ['x', 'y', 'z'],

  methods: [
    function grammar(plus, sym, alt, opt, seq, seq1, lit, range, eof) {
      return {
        START: seq1(0, sym('expr'), eof()),
        expr: seq(sym('expr1'), opt(seq(alt('+', '-'), sym('expr')))),
        expr1: seq(sym('expr2'), opt(seq(alt('*', '/'), sym('expr1')))),
        expr2: seq(sym('expr3'), opt(seq1(1, '^', sym('expr2')))),
        expr3: alt(sym('fun'), sym('variable'), sym('number'), sym('group')),
        variable: alt('x', 'y', 'z'),
        fun: seq(alt('sqrt', 'log2'), '(', sym('expr'), ')'),
        group: seq1(1, '(', sym('expr'), ')'),
        number: /* number parser; must accept a leading '-' for the input below */
      };
    },
    
    function exprAction(v) {
      if (v[1] === null) return v[0];
      if (v[1][0] == '+') return v[0] + v[1][1];
      return v[0] - v[1][1];
    },
    
    function expr1Action(v) {
      if (v[1] === null) return v[0];
      if (v[1][0] == '*') return v[0] * v[1][1];
      return v[0] / v[1][1];
    },
    
    function expr2Action(v) {
      if (v[1] === null) return v[0];
      return Math.pow(v[0], v[1]);
    },
    
    function funAction(v) {
      return v[0] == 'sqrt' ? Math.sqrt(v[2]) : Math.log2(v[2]);
    },
    
    function variableAction(v) { 
      return this[v]; 
    }
  ]
});

const interp = ExpressionInterpreter.create({x: 1, y: 10, z: 100});
// Input '1*(x+y*sqrt(9))*-23/4' → -178.25

Grammar Reuse

FOAM grammars support OO principles for code reuse.

Composition

Embed grammars within other grammars:

foam.CLASS({
  name: 'ExpressionParser',
  extends: 'foam.parse.Grammar',
  requires: ['NumberParser'],

  methods: [
    function grammar(sym) {
      return {
        START: sym('expr'),
        // ... other rules ...
        number: this.NumberParser.create() // Compose NumberParser
      };
    }
  ]
});

Inheritance

Extend grammars to modify or add behavior:

foam.CLASS({
  name: 'BinaryInterpreter',
  extends: 'ExpressionInterpreter',
  requires: ['BinaryNumberParser as NumberParser']
});

const interp = BinaryInterpreter.create();
// Now interprets binary numbers: '101+111+1' → 13

Utility Methods

match(input)

Find first occurrence in input text:

var p = PhoneNumberParser.create();
var input = 'Call me at (555) 123-4556 or email...';
var result = p.match(input);
// Returns '(555) 123-4556'

matchAll(input)

Find all occurrences in input text:

var p = PhoneNumberParser.create();
var input = 'Phone: (555) 123-4556, Fax: 555-1234';
var results = p.matchAll(input);
// Returns ['(555) 123-4556', '555-1234']

getSymParser(symbolName)

Access individual grammar rules for testing:

var p = PhoneNumberParser.create();
var areaCodeParser = p.getSymParser('areaCode');
// Test just the area code rule

stringPStream(str)

Convenience method to create PStream from string:

var ps = P.stringPStream('abc');
// Equivalent to: foam.parse.StringPStream.create({str: 'abc'})

Auto-Complete Suggestions

FOAM parsers support auto-complete suggestions through the Suggest parser decorator and SmartView UI component. This enables intelligent autocomplete for search bars, query builders, and other text inputs.

Suggestion Class

The foam.parse.Suggestion class represents a single suggestion:

foam.CLASS({
  package: 'foam.parse',
  name: 'Suggestion',
  properties: [
    { class: 'String', name: 'text' },       // Text to insert on selection
    { class: 'String', name: 'label' },      // Display label (defaults to text)
    { class: 'String', name: 'tooltip' },    // Hint text (doesn't insert anything)
    { class: 'String', name: 'category' },   // For styling: property, operator, value, format
    { name: 'view', class: 'foam.u2.ViewSpec' }, // Custom suggester view
    { class: 'Boolean', name: 'prependSpaceOnSelect', value: true }
  ]
});

Creating Suggestions with sug()

Use the sug(parser, suggestionSpec) combinator to attach suggestions to parsers:

var P = foam.parse.Parsers.create();

// Basic text suggestion
sug(literal('AND'), {text: 'AND', category: 'operator'})

// Suggestion with different label and text
sug(literal('>='), {text: '>=', label: 'Greater or Equal', category: 'operator'})

// Tooltip-only suggestion (shows hint, doesn't auto-complete)
sug(seq(sym('digits'), '-', sym('digits'), '-', sym('digits')),
    {tooltip: 'YYYY/MM/DD', category: 'format'})

// Suggestion with custom view
sug(nop(), {view: 'foam.parse.auto.DateSuggester'})

Suggestion Categories

Categories control styling in the suggestion dropdown:

Category	Color	Use Case
`property`	Green	Field/property names
`operator`	Orange	Operators like AND, OR, =, >
`value`	Blue	Enumeration values, keywords
`format`	Gray	Format hints (tooltips only)

Example: Query Parser with Suggestions

foam.CLASS({
  name: 'SearchQueryParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, sym, seq, seq1, literal, literalIC, repeat, repeat0, sug) {
      // Helper for operators with suggestions
      let operator = (str) => alt(
        seq1(2, ' ', sym('ws'), sug(literalIC(str), {text: str, category: 'operator'})),
        literalIC(str)
      );

      return {
        START: sym('query'),

        query: sym('or'),

        // OR with suggestion
        or: repeat(sym('and'),
          seq(' ', seq1(1, sym('ws'),
            sug(alt(literalIC('OR'), literal('|')), {text: 'OR', category: 'operator'})
          ))
        ),

        // AND with suggestion
        and: repeat(sym('expr'),
          seq(' ', seq1(1, sym('ws'),
            sug(alt(literalIC('AND'), literal('&')), {text: 'AND', category: 'operator'})
          ))
        ),

        expr: seq(sym('prop'), '=', sym('statusValue')),

        // Property with suggestion
        prop: seq1(1, sym('ws'),
          sug(literal('status'), {text: 'status', label: 'Status', category: 'property'})
        ),

        // Enum values with suggestions
        statusValue: alt(
          sug(literal('ACTIVE'), {text: 'ACTIVE', category: 'value'}),
          sug(literal('INACTIVE'), {text: 'INACTIVE', category: 'value'})
        ),

        ws: repeat0(' ')
      };
    }
  ]
});

Custom Suggester Views

For complex input like dates or colors, create custom suggester views:

foam.CLASS({
  package: 'foam.parse.auto',
  name: 'DateSuggester',
  extends: 'foam.u2.View',

  properties: [
    'suggestText',  // Function to call when suggestion is selected
    { class: 'Date', name: 'date', onKey: true }
  ],

  methods: [
    function render() {
      this.startContext({data: this}).add(this.DATE);
      this.date$.sub(() => {
        // Call suggestText with the formatted date
        this.suggestText(this.date.toISOString().substring(0,10) + ' ');
      });
    }
  ]
});

Use in grammar:

date: seq1(1, sym('ws'),
  alt(
    sug(nop(), {view: 'foam.parse.auto.DateSuggester'}), // Show date picker
    sym('literal date'),
    sym('relative date')
  )
)

SmartView - The Suggestion UI

foam.parse.auto.SmartView is a TextField that provides autocomplete:

foam.CLASS({
  name: 'SearchBar',
  extends: 'foam.u2.View',

  requires: ['foam.parse.auto.SmartView'],

  methods: [
    function render() {
      this.tag(this.SmartView, {
        parser: MyQueryParser.create({of: MyModel}),
        data$: this.query$
      });
    }
  ]
});

How Suggestion Collection Works

SmartView parses input with a custom apply callback
Tracks the furthest parse position (maxPos)
Collects suggestions only at the furthest position
As user types, filters suggestions to those matching partial input
Tab key auto-completes when single match exists

// Simplified from SmartView's 'apply' property (normalization omitted):
return function(p, grammar) {
  // 'this' is the PStream
  if ( p.suggest && this.pos >= self.maxPos ) {
    let s = p.suggest();
    if ( s ) {
      let label = s.tooltip || s.text;
      // A suggestion further along replaces all earlier ones
      if ( this.pos > self.maxPos ) {
        self.suggestions = {};
        self.maxPos      = this.pos;
      }
      if ( ! self.suggestions[label] ) self.suggestions[label] = s;
    }
  }

  let result = p.parse(this, grammar);

  // A successful parse past maxPos makes the collected suggestions stale
  if ( result && result.pos > self.maxPos ) self.suggestions = {};

  return result;
}

Building Dynamic Property Suggestions

For query parsers that work with FOAM models, dynamically generate property suggestions:

// In propertiesGrammar_
let props = cls.getAxiomsByClass(foam.lang.Property);
let propPredicates = [];

for ( let prop of props ) {
  if ( ! prop.searchable ) continue;

  // Create parser with suggestion for each property
  let propParser = seq1(1, sym('ws'),
    sug(literal(prop.name, prop), {
      text: prop.name,
      label: prop.label,
      category: 'property'
    })
  );

  // Add appropriate comparison based on property type
  if ( foam.lang.String.isInstance(prop) ) {
    propPredicates.push(seq(propParser, sym('compareString')));
  } else if ( foam.lang.Int.isInstance(prop) ) {
    propPredicates.push(seq(propParser, sym('compareNumber')));
  }
  // ... handle other types
}

return { propPredicates: alt.apply(null, propPredicates) };

Nested Property Suggestions

For FObjectProperty types, support dot notation:

let innerProperty = (prop, innerProp) => {
  let expr = foam.mlang.expr.Dot.create({arg1: prop, arg2: innerProp});
  return seq1(2,
    sym('ws'),
    // First part: "address."
    sug(seq1(0, literal(prop.name), '.'), {
      text: prop.name + '.',
      label: prop.label,
      category: 'property'
    }),
    // Second part: "city" (no space before)
    sug(literal(innerProp.name, expr), {
      text: innerProp.name,
      label: innerProp.label,
      category: 'property',
      prependSpaceOnSelect: false
    })
  );
};

Parser Messages

FOAM parsers support attaching messages to parsers through the msg() decorator. This enables parsers to communicate status information, warnings, or metadata back to the calling code when specific patterns are matched.

Creating Messages with msg()

Use the msg(parser, messageConfig) combinator to attach a message to a parser:

var P = foam.parse.Parsers.create();

// Basic message for empty file detection
msg(eof(), {type: 'EMPTY', code: 'NO_DATA', text: 'File contains no data'})

// Warning message for deprecated format
msg(sym('oldFormat'), {type: 'WARNING', code: 'DEPRECATED_FORMAT', text: 'This format is deprecated'})

// Message with additional data
msg(sym('emptySection'), {
  type: 'EMPTY',
  code: 'NO_TRANSACTIONS',
  text: 'File contains no rejected transactions for this day',
  data: { section: 'transactions' }
})

Message Properties

The message property in FOAM’s Msg class is untyped - it can store any value (string, object, array, etc.). The structure is determined by your application.

Example convention (for file processing systems):

Property	Type	Description
`type`	String	Message type (e.g., ‘EMPTY’, ‘WARNING’)
`code`	String	Unique code identifier (e.g., ‘NO_DATA’, ‘DEPRECATED_FORMAT’)
`text`	String	Human-readable message text

Alternative uses:

// Simple string message
msg(sym('rule'), 'Parsing completed')

// Custom object structure
msg(sym('rule'), { severity: 'low', category: 'validation', details: {...} })

// Array of messages
msg(sym('rule'), ['Message 1', 'Message 2'])

The consuming code is responsible for interpreting the message format.

Example: Parser with Messages

Messages can be attached to any parser rule to communicate status back to the caller:

foam.CLASS({
  name: 'DataFileParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, sym, seq, msg, eof, literal, repeat) {
      return {
        START: alt(
          sym('dataFile'),
          sym('emptyFile')
        ),

        // Normal file with data
        dataFile: seq(sym('header'), sym('records'), sym('footer')),

        // Empty file - valid structure but no data
        emptyFile: msg(
          seq(sym('header'), sym('emptyMarker'), sym('footer')),
          'File contains no data'  // Simple string message
        ),

        // Deprecated format warning
        oldHeader: msg(
          sym('legacyHeaderFormat'),
          { warning: 'Deprecated header format detected' }
        ),

        // Record with validation note
        specialRecord: msg(
          sym('recordWithFlag'),
          { note: 'Special processing applied', flag: 'SPECIAL' }
        )
      };
    }
  ]
});

Collecting Messages

Messages are collected via the apply callback mechanism, following the same pattern as foam.parse.auto.SmartView uses for suggestions:

var messages = [];

// Create an apply function that collects messages
function createApply() {
  return function(p, grammar) {
    // 'this' is the PStream
    var result = p.parse(this, grammar);

    // If parse succeeded and parser has a message, collect it
    if ( result && p.msg ) {
      var m = p.msg();
      if ( m ) messages.push(m);
    }

    return result;
  };
}

// Pass apply function to parseString
var result = grammar.parseString(inputText, null, createApply());

// messages array now contains collected messages

parseString() calls the START parser directly rather than through apply, so a msg() placed directly on START is never collected; put it on a rule that START references.

Common use cases:

Display status messages to users
Log warnings or notes during processing
Flag special conditions (empty files, deprecated formats)
Collect metadata about parsed content

Checking a Grammar

Some grammar mistakes only show when a parse reaches them, and some never show. foam.parse.lint.GrammarLint finds them without parsing: it creates each grammar class and reads the parser objects its rules are made of.

Running the Check

From the root of a project that includes foam3:

node foam3/tools/lintGrammars.js     # from foam3 itself: node tools/lintGrammars.js

It prints one line per finding, file: class: rule: severity check: message, then a summary. The exit code is 1 when there is an error.

What It Reports

Check	Severity	Meaning
`undefined-symbol`	error	`sym('x')` and the grammar has no rule `x`; the parse asserts when it gets there
`orphan-action`	error	a method `xAction` (or a `grammars:` action) and no rule `x`; actions attach by rule name, so it never runs
`left-recursion`	error	a rule reaches itself before reading a character, e.g. `expr: seq(sym('expr'), '+', ...)`
`empty-repeat`	error	an unbounded `repeat()` or `repeat0()` whose item can match without reading; the loop stays at one position until `maximum`. `repeat0()` also loops at the end of the input (`repeat0(alt('a', eof()))`), and its delimiter does not count as progress
`duplicate-symbol`	error	two rules with one name; the last one wins
`unreachable`	warning	rules `START` never reaches; expected for rules used by name, such as `parseString(text, 'yymmdd')`
`no-start`	warning	rules but no `START`; `parseString()` without a rule name asserts

An action on a base grammar counts as used when any subclass has its rule, and a subclass is not warned about base rules it does not use.

What It Does Not Check

Grammars created inside a method, such as this.Grammar.create({ symbols }). The tool finds classes only.
Grammars built from data (a rule per property or record) when that data is not there. These are listed as not built.
Rules added after create(), such as alternatives filled from a DAO.
Anything under cut(), which hides its parser in a closure. Grammars using it skip the unreachable check.
Java grammars.
Loops through a parser class an app defines: the loop checks assume such a class always reads a character.

Checking One Grammar From a Test

For a grammar the tool cannot find, call the checker directly. GrammarLint is flagged js&test|grammarlint, so it loads in test builds and in the tool, not in an app’s production bundle:

var findings = foam.parse.lint.GrammarLint.create().lintGrammar(grammar);
x.test(findings.every(f => f.severity !== 'error'), 'grammar has no lint errors');

Best Practices

Start Simple: Test individual parsers before combining them
Use Grammars for Complexity: Move to Grammar classes when parsers become recursive or large
Test Each Symbol: Use getSymParser() to test grammar rules individually
Choose Appropriate Optimizations: Use repeat0(), seq0(), until0() when you don’t need return values
Leverage Composition and Inheritance: Reuse parser components across projects
Add Semantic Actions: Transform parsed data into useful structures early
Use substring() to Preserve Input: When you need the exact input text rather than transformed values
Use Suggestions for UX: Add sug() wrappers to improve discoverability in search/query interfaces
Categorize Suggestions: Use consistent categories (property, operator, value, format) for styling
Custom Suggesters for Complex Input: Create custom views for dates, colors, and other complex types

Complete Example: Email Address Parser

foam.CLASS({
  name: 'EmailAddressParser',
  extends: 'foam.parse.Grammar',

  methods: [
    function grammar(alt, sym, seq, str, repeat, notChars) {
      return {
        START: sym('address'),

        addressList: repeat(sym('address'), seq(',', repeat(' '))),

        address: alt(sym('labelledAddress'), sym('simpleAddress')),

        labelledAddress: seq(
          repeat(notChars('<,')), 
          '<', 
          sym('simpleAddress'), 
          '>'
        ),

        simpleAddress: seq(
          repeat(notChars('@')), 
          '@', 
          repeat(notChars('\r>,'))
        )
      };
    },

    function labelledAddressAction(v) { 
      return v[0].join('') + v[1] + v[2] + v[3]; 
    },

    function simpleAddressAction(v) { 
      return v[0].join('') + v[1] + v[2].join(''); 
    }
  ]
});

var p = EmailAddressParser.create();
// Parses: 'john@email.com'
// Parses: 'John Doe <john@email.com>'
// Parses: 'john@email.com, jane@email.com' (using 'addressList')

With FOAM’s grammar-based parsers, you can tackle parsing challenges that would be impossible or unwieldy with regular expressions, while maintaining readable, testable, and reusable code.

This site is open source. Improve this page.