This document explains how the FOAM parser callback system works, including error handling, with visual diagrams.
┌─────────────────────────────────────────────────────────────────┐
│ PARSING FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Input String ──► StringPStream ──► Parser ──► Result/undefined │
│ │ │
│ ▼ │
│ apply(parser, grammar) │
│ (intercepts every sub-parser call) │
│ │
└─────────────────────────────────────────────────────────────────┘
| Component | Description | File Location |
|---|---|---|
| StringPStream | Immutable stream of characters with position tracking | foam/parse/StringPStream.js |
| Parser | Object with parse(ps, grammar) method |
foam/parse/parse.js |
| Grammar | Collection of named parsers (symbols) with actions | foam/parse/parse.js |
| apply | Callback that intercepts every sub-parser call | StringPStream.js:101 |
apply Callback (Low-Level Interception)When a combinator (seq, alt, repeat, sym, …) runs one of its sub-parsers, it doesn’t call parser.parse() directly. Instead, it calls:
ps.apply(parser, grammar)
Two calls bypass apply: the grammar’s START parser, which parseString() and parse() invoke with start.parse(ps, grammar), and the parser wrapped by cut().
From StringPStream.js:101 (inside setString()):
this.apply = function(p, obj) {
return p.parse(this, obj); // Just runs the parser normally
};
┌────────────────────────────────────────────────────────────────┐
│ apply() INTERCEPTION │
├────────────────────────────────────────────────────────────────┤
│ │
│ Parser A │
│ │ │
│ ▼ │
│ ps.apply(parserB, grammar) ◄─── YOUR CALLBACK HERE │
│ │ │
│ │ ┌─────────────────────────────────┐ │
│ │ │ function(p, grammar) { │ │
│ │ │ // 'this' = PStream │ │
│ │ │ // 'p' = parser being called │ │
│ │ │ // Do your interception... │ │
│ │ │ return p.parse(this, grammar);│ │
│ │ │ } │ │
│ │ └─────────────────────────────────┘ │
│ ▼ │
│ Result (PStream or undefined) │
│ │
└────────────────────────────────────────────────────────────────┘
var myApply = function(p, grammar) {
// 'this' = the PStream (current position in input)
// 'p' = the parser being called
// 'grammar' = the grammar object
console.log('Parsing at position:', this.pos);
console.log('Parser:', p.toString());
var result = p.parse(this, grammar); // Actually run it
console.log('Result:', result ? 'SUCCESS' : 'FAILED');
return result;
};
// Use it:
var ps = foam.parse.StringPStream.create({
str: 'hello world',
apply: myApply
});
grammar.parse(ps);
apply| Use Case | Description |
|---|---|
| Debugging | Log every parser invocation |
| Error Tracking | Track maximum position reached |
| Suggestions | Collect autocomplete suggestions (SmartView) |
| Normalization | Rewrite input to preferred syntax |
| Performance | Count parser invocations |
After a grammar symbol parses successfully, you can transform the result using Action methods.
Symbol Name: "number"
Action Method: "numberAction"
┌────────────────────────────────────────────────────────────────┐
│ SEMANTIC ACTIONS │
├────────────────────────────────────────────────────────────────┤
│ │
│ Grammar Symbol: 'number' │
│ │ │
│ ▼ │
│ Parser matches "123" │
│ │ │
│ ▼ │
│ value = "123" (string) │
│ │ │
│ ▼ │
│ numberAction(value) called ◄─── YOUR TRANSFORMATION │
│ │ │
│ ▼ │
│ return parseInt(value) │
│ │ │
│ ▼ │
│ Final value = 123 (integer) │
│ │
└────────────────────────────────────────────────────────────────┘
foam.CLASS({
name: 'MyGrammar',
extends: 'foam.parse.Grammar',
methods: [
function grammar(seq, sym, range, plus, str) {
return {
START: sym('number'),
number: str(plus(range('0', '9')))
};
},
// This runs AFTER 'number' parses successfully
function numberAction(v) {
// v = "123" (string from parser)
return parseInt(v); // Return 123 (integer)
}
]
});
var g = MyGrammar.create();
var result = g.parseString('456');
console.log(result); // 456
console.log(typeof result); // "number"
From parse.js:1605-1615:
function init() {
this.SUPER();
// Add semantic actions for any symbol where there is a method named: <sym>Action
this.symbols.forEach(s => {
var m = this[s.name + 'Action'];
if ( m ) {
this.addAction(s.name, m.bind(this));
}
});
}
Key Insight: FOAM parsers don’t throw exceptions. They return
undefinedon failure.
┌────────────────────────────────────────────────────────────────┐
│ ERROR DETECTION │
├────────────────────────────────────────────────────────────────┤
│ │
│ parser.parse(ps, grammar) │
│ │ │
│ ├──► Returns PStream ──► SUCCESS (has .value) │
│ │ │
│ └──► Returns undefined ──► FAILURE │
│ │
└────────────────────────────────────────────────────────────────┘
var grammar = MyGrammar.create();
var result = grammar.parseString('invalid input');
if ( result === undefined ) {
console.log('Parse failed!');
} else {
console.log('Parsed:', result);
}
The trick is to track the maximum position reached before failure:
var maxPos = 0;
var inputStr = 'hello 123 ???';
var trackingApply = function(p, grammar) {
// Track furthest position we've reached
maxPos = Math.max(maxPos, this.pos);
return p.parse(this, grammar);
};
var ps = foam.parse.StringPStream.create({
str: inputStr,
apply: trackingApply
});
var result = myParser.parse(ps, myGrammar);
if ( ! result ) {
console.log('Error at position:', maxPos);
console.log('Error near:', inputStr.substring(maxPos, maxPos + 10));
}
The Grammar class has error reporting built in (parse.js:1644-1702). It takes the input string and re-parses it:
var grammar = MyGrammar.create();
var result = grammar.parseString('bad input');
if ( ! result ) {
console.log(grammar.getLastError('bad input'));
// Output: "Unexpected character 'x' at index: 5, Expected one of: [a,b,c],
// Text leading up to this was: ..."
}
┌────────────────────────────────────────────────────────────────┐
│ getLastError() INTERNALS │
├────────────────────────────────────────────────────────────────┤
│ │
│ 1. Creates ErrorReportingPStream │
│ 2. Re-parses with error tracking │
│ 3. Records [position, parser, grammar] at each failure │
│ 4. Keeps the FURTHEST failure point │
│ 5. Tests code points 0-255 to find "valid" next characters │
│ 6. Builds human-readable error message │
│ │
└────────────────────────────────────────────────────────────────┘
parseString("abc123")
│
▼
┌──────────────────────────────────────────────────────────────┐
│ StringPStream created │
│ str: "abc123" │
│ pos: 0 │
│ apply: [your callback or default] │
└──────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ START symbol looked up from grammar │
└──────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ startParser.parse(ps, grammar) (called directly, no apply) │
│ │
│ For each sub-parser it runs, ps.apply(p, grammar): │
│ 1. Your code runs (logging, tracking, etc.) ◄── CALLBACK │
│ 2. p.parse(this, grammar) called │
│ 3. Parser may call ps.apply() on its own sub-parsers │
│ 4. Result returned (PStream or undefined) │
└──────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Actions already ran inside step 3: init() wrapped each symbol │
│ that has an Action method in a ParserWithAction, which returns│
│ ps.setValue(symbolAction(value)) wherever the symbol matches │
└──────────────────────────────────────────────────────────────┘
│
▼
Final result.value returned
┌─────────┐ ┌──────────┐ ┌────────┐ ┌─────────┐
│ User │ │ Grammar │ │ PStream│ │ Parser │
└────┬────┘ └────┬─────┘ └────┬───┘ └────┬────┘
│ │ │ │
│ parseString() │ │ │
│──────────────►│ │ │
│ │ │ │
│ │ create() │ │
│ │───────────────►│ │
│ │ │ │
│ │ parse() │ │
│ │───────────────►│ │
│ │ │ │
│ │ │ apply(p,g) │
│ │ │─────────────►│
│ │ │ │
│ │ │ [callback │
│ │ │ runs] │
│ │ │ │
│ │ │ parse(ps,g) │
│ │ │◄─────────────│
│ │ │ │
│ │ │ result │
│ │ │◄─────────────│
│ │ │ (actions run │
│ │ │ inside parse│
│ │ │ via Parser- │
│ │ │ WithAction) │
│ │ │ │
│ result.value │ │ │
│◄──────────────│ │ │
│ │ │ │
foam.CLASS({
name: 'FileParser',
extends: 'foam.parse.Grammar',
properties: [
{ name: 'errorMessage', value: '' },
{ name: 'errorPos', value: 0 }
],
methods: [
function grammar(seq, sym, eof, plus, anyChar, alt, not) {
return {
START: alt(
sym('emptyFile'),
sym('content')
),
emptyFile: eof(), // Matches if file is empty
content: seq(
plus(sym('line'), '\n'),
eof()
),
line: plus(not('\n', anyChar()))
};
},
// Action for empty file - returns error object
function emptyFileAction(v) {
return {
error: 'FILE_EMPTY',
message: 'The file is empty'
};
},
// Main parsing method with error tracking
function parseWithErrors(str) {
var self = this;
self.errorPos = 0;
var apply = function(p, grammar) {
self.errorPos = Math.max(self.errorPos, this.pos);
return p.parse(this, grammar);
};
var ps = foam.parse.StringPStream.create({
str: str,
apply: apply
});
var result = this.parse(ps);
if ( ! result ) {
return {
error: 'PARSE_ERROR',
message: 'Parse failed at position ' + self.errorPos,
position: self.errorPos,
context: str.substring(self.errorPos, self.errorPos + 20)
};
}
return result.value;
}
]
});
// Usage:
var parser = FileParser.create();
var result1 = parser.parseWithErrors('');
// { error: 'FILE_EMPTY', message: 'The file is empty' }
var result2 = parser.parseWithErrors('hello\nworld');
// Parsed content
var result3 = parser.parseWithErrors('hello\n');
// Trailing newline with no line after it:
// { error: 'PARSE_ERROR', message: 'Parse failed at position 6', ... }
foam.CLASS({
name: 'CSVParser',
extends: 'foam.parse.Grammar',
properties: [
'maxPos',
'lastParser',
'inputStr'
],
methods: [
function grammar(seq, sym, alt, repeat, plus, not, anyChar, eof, literal) {
return {
START: seq(sym('rows'), eof()),
rows: repeat(sym('row'), sym('newline')),
row: repeat(sym('field'), literal(',')),
field: alt(
sym('quotedField'),
sym('unquotedField')
),
quotedField: seq('"', sym('quotedContent'), '"'),
quotedContent: repeat(alt(
literal('""', '"'), // Escaped quote
not('"', anyChar())
)),
unquotedField: repeat(not(alt(',', '\n', '\r', '"', eof()), anyChar())),
newline: alt('\r\n', '\n', '\r')
};
},
function parseCSV(str) {
var self = this;
self.inputStr = str;
self.maxPos = 0;
self.lastParser = null;
var trackingApply = function(p, grammar) {
if ( this.pos > self.maxPos ) {
self.maxPos = this.pos;
self.lastParser = p;
}
return p.parse(this, grammar);
};
var ps = foam.parse.StringPStream.create({
str: str,
apply: trackingApply
});
var result = this.parse(ps);
if ( ! result ) {
return this.buildErrorReport();
}
return { success: true, data: result.value };
},
function buildErrorReport() {
var line = 1;
var col = 1;
// Calculate line and column
for ( var i = 0; i < this.maxPos; i++ ) {
if ( this.inputStr[i] === '\n' ) {
line++;
col = 1;
} else {
col++;
}
}
// Get context around error
var start = Math.max(0, this.maxPos - 20);
var end = Math.min(this.inputStr.length, this.maxPos + 20);
var context = this.inputStr.substring(start, end);
var pointer = ' '.repeat(this.maxPos - start) + '^';
return {
success: false,
error: {
message: 'CSV parse error',
line: line,
column: col,
position: this.maxPos,
expected: this.lastParser ? this.lastParser.toString() : 'unknown',
context: context,
pointer: pointer
}
};
}
]
});
// Usage:
var parser = CSVParser.create();
var result = parser.parseCSV('name,age\nJohn,30\nJane,"bad quote');
if ( ! result.success ) {
console.log('Error at line', result.error.line, 'column', result.error.column);
console.log('Context:', result.error.context);
console.log(' ', result.error.pointer);
console.log('Expected:', result.error.expected);
}
foam.CLASS({
name: 'DebugGrammar',
extends: 'foam.parse.Grammar',
properties: [
{
name: 'debugLog',
factory: function() { return []; }
},
{
class: 'Int',
name: 'depth',
value: 0
}
],
methods: [
function grammar(seq, sym, literal, plus, range) {
return {
START: sym('expression'),
expression: seq(sym('term'), plus(seq('+', sym('term')))),
term: sym('number'),
number: plus(range('0', '9'))
};
},
function parseWithDebug(str) {
var self = this;
self.debugLog = [];
self.depth = 0;
var debugApply = function(p, grammar) {
var indent = ' '.repeat(self.depth);
var parserName = p.toString().substring(0, 30);
var inputPreview = this.str[0].substring(this.pos, this.pos + 10);
self.debugLog.push(indent + '► ' + parserName + ' @ "' + inputPreview + '..."');
self.depth++;
var result = p.parse(this, grammar);
self.depth--;
if ( result ) {
self.debugLog.push(indent + '✓ ' + parserName + ' = ' + JSON.stringify(result.value));
} else {
self.debugLog.push(indent + '✗ ' + parserName + ' FAILED');
}
return result;
};
var ps = foam.parse.StringPStream.create({
str: str,
apply: debugApply
});
var result = this.parse(ps);
return {
result: result ? result.value : null,
log: self.debugLog
};
}
]
});
// Usage:
var parser = DebugGrammar.create();
var debug = parser.parseWithDebug('1+2+3');
console.log('Result:', debug.result);
console.log('Parse trace:');
debug.log.forEach(line => console.log(line));
/* Output:
Result: [["1"], [["+", ["2"]], ["+", ["3"]]]]
Parse trace:
► seq(sym("term"), plus(seq("+", @ "1+2+3..."
► sym("term") @ "1+2+3..."
► sym("number") @ "1+2+3..."
► plus(range("0", "9")) @ "1+2+3..."
► range("0", "9") @ "1+2+3..."
✓ range("0", "9") = "1"
► range("0", "9") @ "+2+3..."
✗ range("0", "9") FAILED
✓ plus(range("0", "9")) = ["1"]
✓ sym("number") = ["1"]
✓ sym("term") = ["1"]
...
The START symbol, sym("expression"), does not appear: parse() calls it
directly, so the first logged call is the parser it refers to.
*/
| File | Description |
|---|---|
foam/parse/StringPStream.js |
PStream implementation with apply callback |
foam/parse/parse.js |
Core parser classes and Grammar |
foam/parse/auto/SmartView.js |
Example of using apply for suggestions |
parse.js| Lines | Content |
|---|---|
| 74-136 | Suggestion class |
| 270-326 | Literal parser |
| 394-431 | Alternate parser |
| 434-469 | Sequence parser |
| 472-494 | Suggest parser decorator |
| 1158-1185 | ParserWithAction (wraps actions) |
| 1201-1248 | Symbol parser |
| 1547-1738 | Grammar class |
| 1605-1615 | Action registration in init() |
| 1644-1702 | getLastError() implementation |
StringPStream.js| Lines | Content |
|---|---|
| 29-44 | tail getter (propagates apply) |
| 83-96 | setValue() (propagates apply) |
| 98-103 | setString() (default apply defined) |
┌─────────────────────────────────────────────────────────────────┐
│ CALLBACK SYSTEM SUMMARY │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. apply(parser, grammar) │
│ └─► Intercepts every sub-parser invocation │
│ └─► Use for: debugging, error tracking, suggestions │
│ │
│ 2. <symbol>Action(value) │
│ └─► Transforms result AFTER successful parse │
│ └─► Use for: type conversion, AST building │
│ │
│ 3. Error Handling │
│ └─► undefined = parse failed │
│ └─► Track maxPos in apply() for error location │
│ └─► Use getLastError() for detailed messages │
│ │
└─────────────────────────────────────────────────────────────────┘