token engine?

[Agamemnus]

Anyone know a nice and fast "tokenizer" engine? Something that lets me input QB's syntax rules and converts code into unambiguous tokens?

[TheBigBasicQ]

nope, but i have made my own tokenizer. dunno if its any use =P.

[Agamemnus]

Only if it's easily modifiable...

[TheBigBasicQ]

I may send it to u tommorow. I'm pretty tired and cant find it atm =(

[Agamemnus]

I want to just make my own BASIC language. It'll be called AgaBasic..

Here's an example of what it will be able to do:

Code:

TYPE flexivar (valueAmount AS INTEGER, segmentAmount AS INTEGER, recordAmount AS INTEGER, typeValue AS LITERAL)
value(0 TO valueAmount - 1) AS typeValue
segStart(0 TO segmentAmount - 1) AS INTEGER
segEnd(0 TO segmentAmount - 1) AS INTEGER
recStart(0 TO recordAmount - 1) AS INTEGER
recEnd(0 TO recordAmount - 1) AS INTEGER
END TYPE

TYPE flexivar2
DERIVES FROM flexivar
extraBlah AS INTEGER
END TYPE

DIM var1 AS flexivar2(10000, 2000, 100, BYTE)

var1.extraBlah = 1
var1.value(10000) = 255

[LooseCaboose]

You could use flex http://www.gnu.org/software/flex/ and Bison http://www.gnu.org/software/bison/bison.html.

Dont know how easy your langauge will be to actually implement, but good luck.

[Agamemnus]

Thanks.

Looks difficult though.

EDIT:

I found a nice explanation here:

http://accent.compilertools.net/algorithms.html

(...which I found from http://catalog.compilertools.net/lexparse.html

...which I found from http://dmoz.org/Computers/Programming/Co...tion_Kits/)

ONE problem is that most of the stuff I want to do is not parlor trick stuff.. it involves heavy usage of memory: loading in function names, variable names, etc.. and I can't find it in there.. Still looking though.

[LooseCaboose]

Thanks.

Looks difficult though.

Well, it is difficult. Creating a compiler, especially a complicated compiler like the one you have in mind is a very difficult task. I would suggest that you write the grammar down before you do anything (including deciding on your full token set) and verifying that it is unambigous (this can all be done with a pen and paper), once you have the grammar done it should be easy to build a lexer and parser for it using flex and bison. The difficult bit then is the actual compiling, I really have no idea how you would implement some of your ideas.

[Agamemnus]

It's quite ambiguous in the case of one-line parsers, though. =(

QB is not a context free grammar at all..

[na_th_an]

It's quite ambiguous in the case of one-line parsers, though. =(

QB is not a context free grammar at all..

Well, you can make it a context free grammar and deal with non-context stuff in later steps or with extra treatment.

The easiest way of doing this would be using LL, not LR. Flex/Bison use LR. Let me ask somebody I know and I'll point you to a *very easy to use* replacement to flex/bison which uses LL(1).