The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes the syntax of those tokens. The grammar is designed to be suitable for high speed scanning and to make it easy to write a correct scanner for it. It has a minimum of special case rules and there is only one phase of translation.
Source text can be in one of the following formats:
One of the following UTF BOMs (Byte Order Marks) can be present at the beginning of the source text:
| Format | BOM |
|---|---|
| UTF-8 | EF BB BF |
| UTF-16BE | FE FF |
| UTF-16LE | FF FE |
| UTF-32BE | 00 00 FE FF |
| UTF-32LE | FF FE 00 00 |
| ASCII | no BOM |
If the source file does not start with a BOM, then the first character must be less than or equal to U+0000007F.
The source text is decoded from its source representation into Unicode Characters. The Characters are further divided into: WhiteSpace, EndOfLine, Comments, SpecialTokenSequences, Tokens, all followed by EndOfFile.
The source text is split into tokens using the maximal munch technique, i.e., the lexical analyzer makes the longest token it can. For example >> is a right shift token, not two greater than tokens. There are two exceptions to this rule:
.. embedded inside what looks like two floating point literals, as in 1..2, is interpreted as if the .. was separated by a space from the first integer.1.a is interpreted as the three tokens 1, ., and a, whereas 1. a is interpreted as the two tokens 1. and a.Character:
any Unicode character
EndOfFile:
physical end of the file
\u0000
\u001A The source text is terminated by whichever comes first.
EndOfLine:
\u000D
\u000A
\u000D \u000A
\u2028
\u2029
EndOfFile WhiteSpace:
Space
Space WhiteSpace
Space:
\u0020
\u0009
\u000B
\u000C
Comment:
BlockComment
LineComment
NestingBlockComment
BlockComment:
/* Characters */
LineComment:
// Characters EndOfLine
NestingBlockComment:
/+ NestingBlockCommentCharacters +/
NestingBlockCommentCharacters:
NestingBlockCommentCharacter
NestingBlockCommentCharacter NestingBlockCommentCharacters
NestingBlockCommentCharacter:
Character
NestingBlockComment
Characters:
Character
Character Characters There are three kinds of comments:
The contents of strings and comments are not tokenized. Consequently, comment openings occurring within a string do not begin a comment, and string delimiters within a comment do not affect the recognition of comment closings and nested /+ comment openings. With the exception of /+ occurring within a /+ comment, comment openings within a comment are ignored.
a = /+ // +/ 1; // parses as if 'a = 1;' a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";' a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'
Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.
Token:
Identifier
StringLiteral
CharacterLiteral
IntegerLiteral
FloatLiteral
Keyword
/
/=
.
..
...
&
&=
&&
|
|=
||
-
-=
--
+
+=
++
<
<=
<<
<<=
>
>=
>>=
>>>=
>>
>>>
!
!=
(
)
[
]
{
}
?
,
;
:
$
=
==
*
*=
%
%=
^
^=
^^
^^=
~
~=
@
=>
#
Identifier:
IdentifierStart
IdentifierStart IdentifierChars
IdentifierChars:
IdentifierChar
IdentifierChar IdentifierChars
IdentifierStart:
_
Letter
UniversalAlpha
IdentifierChar:
IdentifierStart
0
NonZeroDigit
Identifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D of the C99 Standard. Identifiers can be arbitrarily long, and are case sensitive.
__ (two underscores) are reserved. StringLiteral:
WysiwygString
AlternateWysiwygString
DoubleQuotedString
HexString
DelimitedString
TokenString
WysiwygString:
r" WysiwygCharacters " StringPostfixopt
AlternateWysiwygString:
` WysiwygCharacters ` StringPostfixopt
WysiwygCharacters:
WysiwygCharacter
WysiwygCharacter WysiwygCharacters
WysiwygCharacter:
Character
EndOfLine
DoubleQuotedString:
" DoubleQuotedCharacters " StringPostfixopt
DoubleQuotedCharacters:
DoubleQuotedCharacter
DoubleQuotedCharacter DoubleQuotedCharacters
DoubleQuotedCharacter:
Character
EscapeSequence
EndOfLine
EscapeSequence:
\'
\"
\?
\\
\0
\a
\b
\f
\n
\r
\t
\v
\x HexDigit HexDigit
\ OctalDigit
\ OctalDigit OctalDigit
\ OctalDigit OctalDigit OctalDigit
\u HexDigit HexDigit HexDigit HexDigit
\U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit
\ NamedCharacterEntity
HexString:
x" HexStringChars " StringPostfixopt
HexStringChars:
HexStringChar
HexStringChar HexStringChars
HexStringChar:
HexDigit
WhiteSpace
EndOfLine
StringPostfix:
c
w
d
DelimitedString:
q" Delimiter WysiwygCharacters MatchingDelimiter "
TokenString:
q{ Tokens }
A string literal is either a double quoted string, a wysiwyg quoted string, a delimited string, a token string, or a hex string.
In all string literal forms, an EndOfLine is regarded as a single \n character.
String literals are read only.
Undefined Behavior: Writes to string literals cannot always be detected, but cause undefined behavior. Wysiwyg ("what you see is what you get") quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string. There are no escape sequences inside wysiwyg strings.
r"I am Oz"
r"c:\games\Sudoku.exe"
r"ab\n" // string is 4 characters,
// 'a', 'b', '\', 'n'
An alternate form of wysiwyg strings are enclosed by backquotes, the ` character.
`the Great and Powerful.`
`c:\games\Empire.exe`
`The "lazy" dog`
`a"b\n` // string is 5 characters,
// 'a', '"', 'b', '\', 'n'
Double quoted strings are enclosed by "". EscapeSequences can be embedded into them.
"Who are you?"
"c:\\games\\Doom.exe"
"ab\n" // string is 3 characters,
// 'a', 'b', and a linefeed
"ab
" // string is 3 characters,
// 'a', 'b', and a linefeed
Hex strings allow string literals to be created using hex data. The hex data need not form valid UTF characters.
x"0A" // same as "\x0A"
x"00 FBCD 32FD 0A" // same as
// "\x00\xFB\xCD\x32\xFD\x0A"
Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.
Note: Hex Strings are deprecated. Please use std.conv.hexString instead.
Delimited strings use various forms of delimiters. The delimiter, whether a character or identifer, must immediately follow the " without any intervening whitespace. The terminating delimiter must immediately precede the closing " without any intervening whitespace. A nesting delimiter nests, and is one of the following characters:
| Delimiter | Matching Delimiter |
|---|---|
[ |
] |
| ( | ) |
< |
> |
{ |
} |
q"(foo(xxx))" // "foo(xxx)"
q"[foo{]" // "foo{"
If the delimiter is an identifier, the identifier must be immediately followed by a newline, and the matching delimiter is the same identifier starting at the beginning of the line:
writeln(q"EOS This is a multi-line heredoc string EOS" );
The newline following the opening identifier is not part of the string, but the last newline before the closing identifier is part of the string. The closing identifier must be placed on its own line at the leftmost column.
Otherwise, the matching delimiter is the same as the delimiter character:
q"/foo]/" // "foo]" // q"/abc/def/" // error
Token strings open with the characters q{ and close with the token }. In between must be valid D tokens. The { and } tokens nest. The string is formed of all the characters between the opening and closing of the token string, including comments.
q{this is the voice of} // "this is the voice of"
q{/*}*/ } // "/*}*/ "
q{ world(q{control}); } // " world(q{control}); "
q{ __TIME__ } // " __TIME__ "
// i.e. it is not replaced with the time
// q{ __EOF__ } // error
// __EOF__ is not a token, it's end of file
The optional StringPostfix character gives a specific type to the string, rather than it being inferred from the context. The types corresponding to the postfix characters are:
| Postfix | Type | Aka |
|---|---|---|
| c | immutable(char)[] |
string |
| w | immutable(wchar)[] |
wstring |
| d | immutable(dchar)[] |
dstring |
"hello"c // string "hello"w // wstring "hello"d // dstring
The string literals are assembled as UTF-8 char arrays, and the postfix is applied to convert to wchar or dchar as necessary as a final step.
The escape sequences listed in EscapeSequence are:
| Sequence | Meaning |
|---|---|
\' |
Literal single-quote: '
|
\" |
Literal double-quote: "
|
\? |
Literal question mark: ?
|
\\ |
Literal backslash: \
|
\0 |
Binary zero (NUL, U+0000). |
\a |
BEL (alarm) character (U+0007). |
\b |
Backspace (U+0008). |
\f |
Form feed (FF) (U+000C). |
\n |
End-of-line (U+000A). |
\r |
Carriage return (U+000D). |
\t |
Horizontal tab (U+0009). |
\v |
Vertical tab (U+000B). |
\xnn
|
Byte value in hexadecimal, where nn is specified as two hexadecimal digits. For example: \xFF represents the character with the value 255. |
\n\nn\nnn
|
Byte value in octal. For example: \101 represents the character with the value 65 ('A'). Analogous to hexadecimal characters, the largest byte value is \377 (= \xFF in hexadecimal or 255 in decimal) |
\unnnn
|
Unicode character U+nnnn, where nnnn are four hexadecimal digits. For example, \u03B3 represents the Unicode character γ (U+03B3 - GREEK SMALL LETTER GAMMA). |
\Unnnnnnnn
|
Unicode character U+nnnnnnnn, where nnnnnnnn are 8 hexadecimal digits. For example, \U0001F603 represents the Unicode character U+1F603 (SMILING FACE WITH OPEN MOUTH). |
\name
|
Named character entity from the HTML5 specification. See NamedCharacterEntity. |
CharacterLiteral:
' SingleQuotedCharacter '
SingleQuotedCharacter:
Character
EscapeSequence
Character literals are a single character or escape sequence enclosed by single quotes.
'h' // the letter h '\n' // newline '\\' // the backslash character
IntegerLiteral:
Integer
Integer IntegerSuffix
Integer:
DecimalInteger
BinaryInteger
HexadecimalInteger
IntegerSuffix:
L
u
U
Lu
LU
uL
UL
DecimalInteger:
0
NonZeroDigit
NonZeroDigit DecimalDigitsUS
BinaryInteger:
BinPrefix BinaryDigitsNoSingleUS
BinPrefix:
0b
0B
HexadecimalInteger:
HexPrefix HexDigitsNoSingleUS
NonZeroDigit:
1
2
3
4
5
6
7
8
9
DecimalDigits:
DecimalDigit
DecimalDigit DecimalDigits
DecimalDigitsUS:
DecimalDigitUS
DecimalDigitUS DecimalDigitsUS
DecimalDigitsNoSingleUS:
DecimalDigit
DecimalDigit DecimalDigitsUS
DecimalDigitsUS DecimalDigit
DecimalDigitsNoStartingUS:
DecimalDigit
DecimalDigit DecimalDigitsUS
DecimalDigit:
0
NonZeroDigit
DecimalDigitUS:
DecimalDigit
_
BinaryDigitsNoSingleUS:
BinaryDigit
BinaryDigit BinaryDigitsUS
BinaryDigitsUS BinaryDigit
BinaryDigitsUS BinaryDigit BinaryDigitsUS
BinaryDigitsUS:
BinaryDigitUS
BinaryDigitUS BinaryDigitsUS
BinaryDigit:
0
1
BinaryDigitUS:
BinaryDigit
_
OctalDigit:
0
1
2
3
4
5
6
7
HexDigits:
HexDigit
HexDigit HexDigits
HexDigitsUS:
HexDigitUS
HexDigitUS HexDigitsUS
HexDigitsNoSingleUS:
HexDigit
HexDigit HexDigitsUS
HexDigitsUS HexDigit
HexDigitsNoStartingUS:
HexDigit
HexDigit HexDigitsUS
HexDigit:
DecimalDigit
HexLetter
HexDigitUS:
HexDigit
_
HexLetter:
a
b
c
d
e
f
A
B
C
D
E
F
Integers can be specified in decimal, binary, or hexadecimal.
Decimal integers are a sequence of decimal digits.
Binary integers are a sequence of binary digits preceded by a ‘0b’ or ‘0B’.
C-style octal integer notation was deemed too easy to mix up with decimal notation; it is only fully supported in string literals. D still supports octal integer literals interpreted at compile time through the std.conv.octal template, as in octal!167.
Hexadecimal integers are a sequence of hexadecimal digits preceded by a ‘0x’ or ‘0X’.
Integers can have embedded ‘_’ characters, which are ignored.
20_000 // leagues under the sea 867_5309 // number on the wall 1_522_000 // thrust of F1 engine (lbf sea level)
Integers can be immediately followed by one ‘L’ or one of ‘u’ or ‘U’ or both. Note that there is no ‘l’ suffix.
The type of the integer is resolved as follows:
| Literal | Type | Usual decimal notation |
|---|---|
0 .. 2_147_483_647 |
int |
2_147_483_648 .. 9_223_372_036_854_775_807 |
long |
Explicit suffixes |
0L .. 9_223_372_036_854_775_807L |
long |
0U .. 4_294_967_295U |
uint |
4_294_967_296U .. 18_446_744_073_709_551_615U |
ulong |
0UL .. 18_446_744_073_709_551_615UL |
ulong |
Hexadecimal notation |
0x0 .. 0x7FFF_FFFF |
int |
0x8000_0000 .. 0xFFFF_FFFF |
uint |
0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFF |
long |
0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFF |
ulong |
Hexadecimal notation with explicit suffixes |
0x0L .. 0x7FFF_FFFF_FFFF_FFFFL |
long |
0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFL |
ulong |
0x0U .. 0xFFFF_FFFFU |
uint |
0x1_0000_0000U .. 0xFFFF_FFFF_FFFF_FFFFU |
ulong |
0x0UL .. 0xFFFF_FFFF_FFFF_FFFFUL |
ulong |
An integer literal may not exceed those values.
Best Practices: Octal integer notation is not supported in integer literals. However, octal integer literals can be interpreted at compile time through thestd.conv.octal template, as in octal!167. FloatLiteral:
Float
Float Suffix
Integer FloatSuffix
Integer ImaginarySuffix
Integer FloatSuffix ImaginarySuffix
Integer RealSuffix ImaginarySuffix
Float:
DecimalFloat
HexFloat
DecimalFloat:
LeadingDecimal .
LeadingDecimal . DecimalDigits
DecimalDigits . DecimalDigitsNoStartingUS DecimalExponent
. DecimalInteger
. DecimalInteger DecimalExponent
LeadingDecimal DecimalExponent
DecimalExponent
DecimalExponentStart DecimalDigitsNoSingleUS
DecimalExponentStart
e
E
e+
E+
e-
E-
HexFloat:
HexPrefix HexDigitsNoSingleUS . HexDigitsNoStartingUS HexExponent
HexPrefix . HexDigitsNoStartingUS HexExponent
HexPrefix HexDigitsNoSingleUS HexExponent
HexPrefix:
0x
0X
HexExponent:
HexExponentStart DecimalDigitsNoSingleUS
HexExponentStart:
p
P
p+
P+
p-
P-
Suffix:
FloatSuffix
RealSuffix
ImaginarySuffix
FloatSuffix ImaginarySuffix
RealSuffix ImaginarySuffix
FloatSuffix:
f
F
RealSuffix:
L
ImaginarySuffix:
i
LeadingDecimal:
DecimalInteger
0 DecimalDigitsNoSingleUS
Floats can be in decimal or hexadecimal format.
Hexadecimal floats are preceded by a 0x or 0X and the exponent is a p or P followed by a decimal number serving as the exponent of 2.
Floating literals can have embedded ‘_’ characters, which are ignored.
2.645_751 6.022140857E+23 6_022.140857E+20 6_022_.140_857E+20_
Floating literals with no suffix are of type double. They can be followed by one f, F, or L suffix. The f or F suffix types it is a float, and L types it is a real.
If a floating literal is followed by i, then it is an ireal (imaginary) type.
Examples:
0x1.FFFFFFFFFFFFFp1023 // double.max 0x1p-52 // double.epsilon 1.175494351e-38F // float.min 6.3i // idouble 6.3 6.3fi // ifloat 6.3 6.3Li // ireal 6.3
The literal may not exceed the range of the type. The literal is rounded to fit into the significant digits of the type.
If a floating literal has a . and a type suffix, at least one digit must be in-between:
1f; // OK, float 1.f; // error 1.; // OK, double
Keywords are reserved identifiers.
Keyword:
abstract
alias
align
asm
assert
auto
body
bool
break
byte
case
cast
catch
cdouble
cent
cfloat
char
class
const
continue
creal
dchar
debug
default
delegate
delete (deprecated)
deprecated
do
double
else
enum
export
extern
false
final
finally
float
for
foreach
foreach_reverse
function
goto
idouble
if
ifloat
immutable
import
in
inout
int
interface
invariant
ireal
is
lazy
long
macro (reserved)
mixin
module
new
nothrow
null
out
override
package
pragma
private
protected
public
pure
real
ref
return
scope
shared
short
static
struct
super
switch
synchronized
template
this
throw
true
try
typeid
typeof
ubyte
ucent
uint
ulong
union
unittest
ushort
version
void
wchar
while
with
__FILE__
__FILE_FULL_PATH__
__MODULE__
__LINE__
__FUNCTION__
__PRETTY_FUNCTION__
__gshared
__traits
__vector
__parameters
These tokens are replaced with other tokens according to the following table:
| Special Token | Replaced with |
|---|---|
__DATE__ |
string literal of the date of compilation "mmm dd yyyy" |
__EOF__ |
sets the scanner to the end of the file |
__TIME__ |
string literal of the time of compilation "hh:mm:ss" |
__TIMESTAMP__ |
string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy" |
__VENDOR__ |
Compiler vendor string |
__VERSION__ |
Compiler version as an integer |
__VENDOR__ and the replacement integer value for __VERSION__. SpecialTokenSequence:
# line IntegerLiteral EndOfLine
# line IntegerLiteral Filespec EndOfLine
Filespec:
" Characters "
Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.
There is currently only one special token sequence, #line.
This sets the current source line number to IntegerLiteral, and optionally the current source file name to Filespec, beginning with the next line of source text.
The backslash character is not treated specially inside Filespec strings.
For example:
int #line 6 "foo\bar" x; // this is now line 6 of file foo\barImplementation Defined: The source file and line number is typically used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.
© 1999–2019 The D Language Foundation
Licensed under the Boost License 1.0.
https://dlang.org/spec/lex.html