2.1 General
A Buoy program consists of one or more source files containing Unicode text.
During compilation, each source file is transformed into a sequence of lexical elements known as tokens. Tokens are the smallest meaningful elements of the language and include identifiers, keywords, literals, operators, punctuation, and comments.
Except where explicitly stated, whitespace and comments serve only to separate tokens and have no semantic meaning.
2.2 Character Set
Buoy source files are encoded as Unicode text.
The compiler shall recognize the Unicode characters required to represent identifiers, string literals, comments, and source code punctuation.
Implementations should accept UTF-8 encoded source files.
2.3 Source Lines
A source file is composed of a sequence of source lines.
A source line may contain:
- declarations
- executable statements
- compiler directives
- comments
- whitespace
The significance of line endings is determined by the grammar of the enclosing construct.
2.4 Whitespace
Whitespace separates adjacent lexical elements.
Whitespace may appear between any two tokens unless prohibited by the grammar.
The following characters are considered whitespace:
- Space
- Horizontal tab
- Carriage return
- Line feed
Multiple whitespace characters are equivalent to a single separator unless they occur within a string literal.
2.5 Comments
Comments are ignored by the compiler except where required for documentation generation.
Comments may appear wherever whitespace is permitted.
The syntax of comments is described in Section 2.11.
2.6 Identifiers
Identifiers name language entities including variables, constants, functions, subroutines, types, modules, properties, parameters, and classes.
An identifier consists of a sequence of characters beginning with a valid identifier-start character followed by zero or more identifier characters.
Unless otherwise specified, identifiers are case-sensitive.
The following identifiers are distinct:
Value
value
VALUE
Identifiers shall not be identical to reserved words.
The complete list of reserved words is provided in Appendix A.
2.7 Keywords
Keywords are reserved identifiers having predefined meaning within the language.
Examples include:
If
Then
Else
For
While
Function
Sub
Class
Structure
Interface
Module
Return
Const
Var
Dim
Keywords shall be used only according to the grammar of the language.
2.8 Literals
A literal denotes a constant value written directly in source code.
Buoy defines literal forms for:
- integer values
- floating-point values
- Boolean values
- string values
- enumeration values
- collection literals (where supported)
The syntax and semantics of each literal form are described in later chapters.
2.9 Operators and Punctuation
Operators perform computations upon one or more operands.
Examples include arithmetic, comparison, logical, assignment, and member-access operators.
Punctuation characters delimit grammatical constructs such as parameter lists, array indexing expressions, generic type arguments, and statement blocks.
Operator precedence and associativity are defined in Appendix B.
2.10 Case Sensitivity
Keyword spelling is fixed.
Identifier comparison is case-sensitive.
String comparison semantics are determined by the operations performing the comparison and are independent of identifier matching.
2.11 Documentation Comments
Buoy supports documentation comments for generating API documentation.
Documentation comments are associated with the declaration immediately following the comment.
The syntax and processing rules for documentation comments are described in Chapter 27.
2.12 Lexical Errors
A lexical error occurs when the compiler encounters source text that cannot be tokenized.
Examples include:
- malformed literals
- invalid characters
- unterminated string literals
- unterminated comments
- illegal escape sequences
A conforming compiler shall report the location of the offending source text before terminating compilation.
2.13 Summary
The lexical structure defined by this chapter establishes the sequence of tokens consumed by the parser.
Subsequent chapters describe how those tokens combine to form declarations, expressions, statements, and complete programs.