Documentation Comments in FLIN

Code that works but cannot be understood is a liability. Every function, every entity, every type declaration carries intent that exists only in the developer's mind -- until it is written down. Documentation comments bridge the gap between what code does and why it exists.

FLIN's documentation comment system draws from the best of Rust's /// syntax and JSDoc's structured annotations. Triple-slash comments are not just text alongside code -- they are first-class elements of the AST, captured during parsing, preserved during formatting, and available for automatic documentation generation.

The Syntax

Documentation comments in FLIN use the /// prefix. They attach to the declaration that immediately follows them:

/// Calculates the total price including tax.
/// Returns the price multiplied by (1 + taxRate).
fn calculateTotal(price: float, taxRate: float) -> float {
    return price * (1 + taxRate)
}

/// Represents a user in the system. /// Users have different roles and permissions. struct User { name: text email: text }

/// A product in the inventory. entity Product { name: text price: float }

/// Status of an order. enum OrderStatus { Pending, Shipped, Delivered }

/// A user identifier type. type UserId = int ```

Regular // comments are not captured. This distinction is important: regular comments are developer notes that may be temporary, contextual, or informal. Doc comments are permanent documentation that should survive code generation, formatting, and tooling transformations.

Lexer Changes

The distinction between regular and documentation comments is made at the lexer level. When the scanner encounters //, it checks whether a third slash follows:

fn scan_comment(&mut self) {
    let is_doc_comment = self.peek() == Some('/');
    if is_doc_comment {
        self.advance(); // consume third /
    }
    let text = self.consume_until_newline();
    if is_doc_comment {
        self.add_token(TokenKind::DocComment(text));
    } else {
        self.add_token(TokenKind::Comment(text));
    }
}

This produces two distinct token types: TokenKind::Comment(String) for regular comments and TokenKind::DocComment(String) for documentation comments. The parser can then handle them differently -- skipping regular comments and collecting doc comments for attachment to declarations.

Parser Integration

The parser collects documentation comments and attaches them to the next declaration. The mechanism uses a pending slot:

1. When the parser encounters a DocComment token, it stores the text in pending_doc_comment. 2. When it parses a declaration (function, struct, entity, enum, type), it calls take_doc_comment() to retrieve and attach any pending doc comment. 3. Multiple consecutive /// lines are concatenated with newlines.

The AST nodes gain a doc_comment field:

// In ast.rs
Stmt::FnDecl {
    name: String,
    params: Vec<Param>,
    return_type: Option<Type>,
    body: Vec<Stmt>,
    doc_comment: Option<String>,  // NEW
    span: Span,
}

Stmt::EntityDecl { name: String, fields: Vec, doc_comment: Option, // NEW span: Span, }

Stmt::EnumDecl { name: String, variants: Vec, doc_comment: Option, // NEW span: Span, }

Stmt::StructDecl { name: String, fields: Vec, doc_comment: Option, // NEW span: Span, }

Stmt::TypeDecl { name: String, value: Type, doc_comment: Option, // NEW span: Span, } ```

The doc_comment field is Option -- most declarations will not have doc comments, and we should not force developers to write them. But when they do, the comments are preserved through every stage of the compilation pipeline.

Multi-Line Doc Comments

Consecutive /// lines are automatically joined into a single doc comment string:

/// Calculates the total price including tax.
/// Applies the given tax rate to the base price
/// and returns the result.
fn calculateTotal(price: float, taxRate: float) -> float {
    return price * (1 + taxRate)
}

The parser concatenates these three lines into a single string:

"Calculates the total price including tax.\nApplies the given tax rate to the base price\nand returns the result."

This concatenation happens in the parser's comment collection logic. Each time it encounters a DocComment token while already holding a pending doc comment, it appends the new text with a newline separator.

Formatter Preservation

The formatter must output doc comments correctly. When formatting a declaration that has a doc_comment, the formatter emits each line with the /// prefix before the declaration:

fn format_fn_decl(
    &self,
    name: &str,
    params: &[Param],
    return_type: &Option<Type>,
    body: &[Stmt],
    doc_comment: &Option<String>,
    indent: usize,
) -> String {
    let mut output = String::new();

// Emit doc comment lines if let Some(doc) = doc_comment { for line in doc.lines() { output.push_str(&" ".repeat(indent)); output.push_str("/// "); output.push_str(line); output.push('\n'); } }

// Emit function declaration output.push_str(&" ".repeat(indent)); output.push_str(&format!("fn {}(", name)); // ... rest of function formatting output } ```

This ensures that formatting a file does not lose documentation. The round-trip property holds: parse a file, format the AST, and the doc comments appear in the output exactly as they were in the input.

Supported Declaration Types

Doc comments attach to seven declaration types:

Field-level doc comments are supported but less commonly used. They document individual fields within a struct or entity, which is useful for complex data models where the field name alone does not convey the full meaning.

What Doc Comments Enable

Having doc comments in the AST opens several possibilities for tooling:

Automatic API documentation. A flin docs command (or future flin doc-gen) can extract all doc comments from a project and generate browsable HTML documentation, similar to rustdoc or javadoc.

IDE hover information. When a developer hovers over a function call in their editor, the LSP server can retrieve the function's doc comment from the AST and display it as a tooltip.

Type signature display. Combined with the type checker's output, doc comments can be displayed alongside inferred types:

calculateTotal(price: float, taxRate: float) -> float

Calculates the total price including tax. Returns the price multiplied by (1 + taxRate). ```

Inline documentation in flin docs. The built-in documentation command can show doc comments for standard library functions, giving developers help without leaving the terminal.

The Distinction That Matters

The separation between // and /// is not syntactic sugar. It represents a fundamental difference in intent:

// TODO: optimize this loop later
// NOTE: this is a temporary workaround for issue #42

/// Processes all pending orders in the queue. /// Returns the number of orders successfully processed. fn processOrders() -> int { // Implementation details... } ```

The regular comments are notes to the current developer. They may be temporary, contextual, or even incorrect. They should not appear in generated documentation.

The doc comment is a contract with all future developers. It describes what the function does, what it returns, and how it should be used. It should appear in generated documentation, IDE tooltips, and API references.

By making this distinction at the language level rather than through naming conventions or annotations, FLIN ensures that documentation tooling can reliably separate developer notes from public documentation without heuristics or guessing.

Testing

Seven tests verify the documentation comment system:

#[test]
fn test_parse_doc_comment_on_function() {
    let program = parse("/// A function.\nfn foo() {}");
    let fn_decl = &program.statements[0];
    assert_eq!(fn_decl.doc_comment(), Some("A function."));
}

#[test] fn test_parse_doc_comment_multiline() { let program = parse("/// Line 1.\n/// Line 2.\nfn foo() {}"); let fn_decl = &program.statements[0]; assert!(fn_decl.doc_comment().unwrap().contains("Line 1.")); assert!(fn_decl.doc_comment().unwrap().contains("Line 2.")); }

#[test] fn test_parse_regular_comment_not_captured() { let program = parse("// Not a doc comment.\nfn foo() {}"); let fn_decl = &program.statements[0]; assert_eq!(fn_decl.doc_comment(), None); } ```

The tests verify that doc comments are captured on functions, structs, entities, and enums; that multi-line doc comments are concatenated; that regular comments are not captured; and that the type alias declaration also supports doc comments. Each test is minimal and focused, testing exactly one behavior.

Files Modified

The doc comment implementation touched seven files across the compiler:

1. src/lexer/token.rs -- Added DocComment(String) token type. 2. src/lexer/scanner.rs -- Detection of /// and emission of DocComment tokens. 3. src/parser/ast.rs -- Added doc_comment: Option to all declaration nodes. 4. src/parser/parser.rs -- Collection and attachment of doc comments to declarations. 5. src/codegen/emitter.rs -- Pattern updates for new fields (doc comments are not emitted to bytecode). 6. src/typechecker/checker.rs -- Pattern updates for new fields (doc comments are not type-checked). 7. src/fmt/formatter.rs -- Output doc comments with /// prefix during formatting.

The code generator and type checker changes are minimal -- they simply add doc_comment: _ to their pattern matches, since doc comments have no semantic effect on compilation or type checking. They exist purely for tooling and documentation purposes.

Documentation comments are a small feature with outsized impact. They cost almost nothing to implement -- a token type, a parser field, a formatter rule -- but they enable an entire ecosystem of documentation tooling. The investment pays dividends every time a developer writes /// instead of leaving a function undocumented.

---

This is Part 175 of the "How We Built FLIN" series, documenting how a CEO in Abidjan and an AI CTO designed and built a programming language from scratch.

Series Navigation: - [174] Testing, Benchmarks, and Fuzzing - [175] Documentation Comments in FLIN (you are here) - [176] Embedded Demo and Templates