Metamorphing Machine I rather be this walking metamorphosis
than having that old formed opinion about everything!

Let's build a transpiler! Part 18

This is the eighteenth post in a series of building a transpiler.
You can find the previous ones here.

Last time I said we would parse Consts and keep all keywords in a single place.
We'll create a module called Vocabulary and a Property Get for each keyword or contextual keyword we have.
As it is mostly a boring thing to do, I'll just show a snippet instead of the whole module:

Module Vocabulary
Public Property Get vAccess() As String
vAccess = "Access"
End Property

(...)

Public Property Get vXor() As String
vXor = "Xor"
End Property
End Module

Now we will replace every keyword string literal (like "New") for its Vocabulary counterpart (vNew).
You may be wondering why we made this change. Not only we keep track of every keyword or contextual in a single place, but it will allow us to do a crazy thing later.
Also, I could easily be going with a class instead of a module. Maybe I change that later.

Moving on to parsing Consts. Its syntax is [Public | Private] Const name1 [As datatype1] = expression1 [, name2 [As datatype2] = expression2, ...]
The problem is we are not ready yet to parse expressions. For now, we will create a function to return whatever the next token is and nothing beyond that.
It is good enough to scan things like "Const PI = 3.14159" but not something like "Const SomeBitPattern = BitMask And Not FirstBitPattern."

Private Function GetExpression() As Token
Set GetExpression = NextToken
End Function

Note that we are not validating what comes from it. We'll change that when we deal properly with expressions.
Next, we will create a class to represent Consts and a place to store them inside Entity:

Class ConstConstruct
Public Access As Accessibility
Public Name As Token
Public DataType As Token
Public Value As Token
End Class

Class Entity
Private Consts_ As Dictionary

Private Sub Class_Initialize()
Set Consts_ = New Dictionary
Consts_.CompareMode = vbTextCompare
End Sub

Public IsClass As Boolean
Public Accessibility As Accessibility
Public Name As Token
Public OptionBase As Integer
Public OptionCompare As VbCompareMethod
Public OptionExplicit As Boolean

Public Property Get Consts() As Dictionary
Set Consts = Consts_
End Property
End Class

Let's adapt Parse to deal with Consts (new code is highlighted:)

Private Function ParseDeclarationArea(ByVal Entity As Entity) As Token
Dim HadOptionBase As Boolean
Dim HadOptionCompare As Boolean
Dim Token As Token

Do
(...)

ElseIf IsKw(Token, vConst) Then
If Access = AccessLocal Then Access = AccessPublic
ParseConsts Access, Entity
Access = AccessLocal

ElseIf IsKw(Token, vEnd) Then
Exit Do

Else
Fail Token, "Expected: Option or Deftype or Public or Private or Const or Enum or Declare or Type"
End If
Loop

Set ParseDeclarationArea = Token
End Function

Let's create some utility function to check if a token is an identifier.
We will also create a function to instantiate a synthetized Token based on the type-declaration character we have in the Sufix property:

Private Function IsId(ByVal Token As Token) As Boolean
IsId = Token.Kind = tkIdentifier Or Token.Kind = tkEscapedIdentifier
End Function


Private Function IsOp(ByVal Token As Token, ByVal Name As String) As Boolean
If Token.Suffix <> vbNullChar Then Exit Function
If Token.Kind <> tkOperator Then Exit Function
IsOp = StrComp(Token.Text, Name, vbTextCompare) = 0
End Function


Private Function FromChar(ByVal TypeDeclarationChar As String) As Token
Dim Token As Token

Set Token = New Token
Token.Kind = tkKeyword

Select Case TypeDeclarationChar
Case "%"
Token.Text = vInteger

Case "&"
Token.Text = vLong

Case "^"
Token.Text = vLongLong

Case "@"
Token.Text = vCurrency

Case "!"
Token.Text = vSingle

Case "#"
Token.Text = vDouble

Case "$"
Token.Text = vString
End Select

Set FromChar = Token
End Function

Finally, let's implement ParseConsts:

Private Sub ParseConsts(ByVal Access As Accessibility, ByVal Entity As Entity)
Const RULE = "[Public | Private] Const identifier [As datatype] = expression (list separator | line break)"
Dim Token As Token
Dim Cnt As ConstConstruct

Do
Rem Get Const's name
Set Token = SkipLineBreaks
If Not IsId(Token) Then Fail Token, RULE, "identifier"

Set Cnt = New ConstConstruct
Cnt.Access = Access
Set Cnt.Name = Token

Set Token = NextToken

Rem Do we have an As clause?
If IsKw(Token, vAs) Then
If Token.Suffix <> vbNullChar Then Fail Token, "Identifier cannot be assigned a new data type"

Rem Get Const's data type name
Set Token = NextToken
If Token.Kind <> tkKeyword Then Fail Token, RULE, "data type"

Select Case Token.Text
Case vBoolean, vByte, vInteger, vLong, vLongLong, vLongPtr, _
vCurrency, vDecimal, vSingle, vDouble, vDate, vString
Rem OK

Case Else
Fail Token, RULE, "data type"
End Select

Set Cnt.DataType = Token
Set Token = NextToken

ElseIf Cnt.Name.Suffix <> vbNullChar Then
Rem Assign DataType property based on type sufix
Set Cnt.DataType = FromChar(Cnt.Name.Suffix)
End If

Rem Discard "="
If Not IsOp(Token, "=") Then Fail Token, RULE, "="

Rem Get Const's value
Set Token = GetExpression
Set Cnt.Value = Token

Rem Ensure it's not a duplicated Const
If Entity.Consts.Exists(Cnt.Name.Text) Then Fail Token, "Ambiguous name detected: " & Cnt.Name.Text

If Cnt.DataType Is Nothing Then
Rem TODO: Infer its data type
End If

Rem Save it
Entity.Consts.Add Cnt.Name.Text, Cnt

Rem Move on
Set Token = NextToken

If IsBreak(Token) Then Exit Do
If Token.Kind <> tkListSeparator Then Fail Token, RULE, "list separator or line break"
Loop
End Sub

For now we can have names that are Tokens. Soon enough we will need to convert them to proper Identifiers.
Also, we left a TODO there in ParseConsts. We are too far away from being able to infer the const's data type yet.

Next week we'll parse Enums.

Andrej Biasic
2020-11-18