Lexer

Lexer

The lexer converts raw source code into tokens.

This process is called lexical analysis.

For:

let x = 10

the input characters are grouped into:

LET
IDENTIFIER("x")
EQUAL
NUMBER(10)

Lexer responsibilities

Keywords

let const fn if else for while return true false null and or not

Operators

+ - * / % = == != > >= < <=

Delimiters

( ) { } [ ] , . :

Strings

"Hello"
'Tanmoy'

Numbers

123
42.5

Comments

// This is a comment

Identifiers

name
age
my_variable

Tokens preserve source information such as the token type, lexeme, literal value, line, and column.