Struct Token 

pub struct Token {
    pub token_type: TokenType,
    pub value: String,
    pub literal: Literal,
}

A single lexical token. value is the raw source text; literal is the parsed payload (or None). No position field — the 4-language token contract excludes positions, matching the shared fixture’s token shape.

Fields

token_type: TokenType

The token kind.

value: String

Raw source text for this token.

literal: Literal

Parsed literal payload, if any.

Implementations

impl Token

pub fn new( token_type: TokenType, value: impl Into<String>, literal: Literal, ) -> Self

Build a token from its type, raw text, and literal payload.

pub fn to_value(&self) -> Value

Serialize to the shared-fixture token shape {type,value,literal}.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token