Haxe got (awesome) macros!

Hi guys, It has been a while since I posted the last time.
However I felt like it was the right moment to go back on the blog, to talk you about the new awesome Haxe macro system.

As you can read on Nicolas’s blog, they’re only present in the latest SVN update, but they will be included in the next build as soon as they will be perfect.
I must say I’m extremely happy about that addition: one of the most important reasons I’m so happy is the fact that instead of implementing an old-and-bording Macro system ala C/C++, Nicolas decided to go for a much more powerful macro system, which allows typesafe macros and real runtime code generation.

You are not limited to simple stuff like replace a macro call with a block of source code, but you can generate any kind of valid AST expressions that will be evaluated by the macro preprocessor at compile time.
Moreover you have access to almost the whole neko library (io, net, etc) which means you can write extremely powerful code generators and DSL! That’s an awesome feature – really.

Give it a look!

Top Down or Bottom Up for Expression Evaluation ?

Last time I wrote about Top Down parsing, a technique for parsing that can be easilly implemented manually using function recursion.

Today it is time to talk about one of the other ways of parsing, called Bottom Up parsing. This technique try to find the most important units first and then, based on a language grammar and a set of rules, try to infer higher order structures starting from them. Bottom Up parsers are the common output of Parser Generators (you can find a good comparison of parser generators here) as they are easier to generate automatically then recursive parser because they can be implemented using sets of tables and simple state based machines.

Writing down manually a bottom up parser is quite a tedious task and require quite a lot of time; moreover, this kind of parsers are difficult to maintain and are usually not really expandable or portable. This is why for my tests I decided to port bYacc (a parser generator that usually generates C code) and edit it so it generates ActionScript code starting from Yacc-compatible input grammars. Having this kind of tool makes things a lot easier, because maintaining a grammar (that usually is composed by a few lines) is less time expensive than working with the generated code (that usually is many lines long).
I will not release today the port because actually I had not time to make sure it is bugfree and I’ve only a working version for Mac, but I plan to release it shortly if you will ever need it for your own tasks. My goal for today was to compare the speed of the parser I wrote with an automatically generated bottom up parser, to see which is the faster approach.

I created a bottom up parser which is able to execute the same expressions accepted by the expression evaluator I wrote last time. There are anyways some differences that – as you will probably and hopefully understand in the future – that make those parsers really different. Some will be disussed shortly here.

To do that I created a Yacc grammar and some support classes.
The parser grammar is really simple and really readable:
%{
%}
%token NUMBER SYMBOL
%left '+' '-'
%left '*' '/'
%left NEG
%%
program
: expr { Vars.result = $1; }
;
expr
: expr '+' expr { $$ = $1 + $3; }
| expr '-' expr { $$ = $1 - $3; }
| expr '*' expr { $$ = $1 * $3; }
| expr '/' expr { $$ = $1 / $3; }
| '-' expr %prec NEG { $$ = -$2; }
| '(' expr ')' { $$ = $2; }
| SYMBOL '(' expr ')' {
if( Vars.SYMBOL_TABLE[ $1 ] )
{
$$ = Vars.SYMBOL_TABLE[ $1 ]( $3 );
} else
{
trace( "can't find function" );
}
}
| SYMBOL {
if( Vars.SYMBOL_TABLE[ $1 ] )
{
$$ = Vars.SYMBOL_TABLE[ $1 ];
} else
{
trace( "can't find symbol" );
}
}
| NUMBER { $$ = yyval; }
;
%%

Continue reading the extended entry to see the results. Continue reading

Runtime expression evaluation in ActionScript

Finally I have a bit of free time to write down this post.
Some days ago a guy on the forum asked about how to evaluate at runtime simple mathematical expressions with support for symbols and function calls.

Now that the eval function have been removed from the language for reasons I’ve not enought time to talk about, we need to parse and execute those expressions manually.

I built a simple Actionscript library that can be used to parse mathematical expressions: it has support for function calls, for variables and it is able to convert the expression into a postfix rappresentation if you may need it. The expression parser is quite simple and have been built manually following some concepts related to programming language compilers; it includes a Scanner (or lexer – however you want to call it) to tokenize the expression, a Parser that convert the expression into an Abstract Syntax Tree and a simple routine that evaluates that AST using a Symbol Table as evaluation context.

There are no comments inside the code right now; I hope to find a little bit of time to write an in depth discussion about this topic. In the mean time you can continue reading the entry for a general explanation about how does the code works and for some examples that may be useful.

Here is a simple example that shows how does the code works:

import it.sephiroth.expr.CompiledExpression;
import it.sephiroth.expr.Parser;
import it.sephiroth.expr.Scanner;
public class Example
{
public static function run(): void
{
var expression: String = "sin( x / ( 8 / 2 + (-0.12 + 2.3) * x / x ) ) * 100";
var scanner: Scanner = new Scanner( expression );
var parser: Parser = new Parser( scanner );
var compiled: CompiledExpression = parser.parse();
var context: Object = {
x: 100,
sin: Math.sin,
cos: Math.cos
};
trace( 'Postfix:', compiled.toString() );
trace( 'Result:', compiled.execute( context ) );
}
}

Before I forgot, you can download the source code here with a simple example included.

Continue reading

Actionscript parsing experiences: PyBison & PLY

My experiments with text-parsing continue..

PyBison
Last day I founded a python library (pybison) which runs the generated python parser at near the speed of C-based parsers, due to direct hooks into bison-generated C code.
Cool, unfortunately I couldn’t compiled it for Windows and so I made my test on Ubuntu only. What I did was just to export the already written lexer/grammar using bison2py (boundled with pybison) and run it.

If you want to take a look at the python parser try it by downloading the source code here.
The run.py file accept these parameters:
usage: run.py [options]
options:
-h, --help show this help message and exit
-v, --verbose Turn on verbose mode
-o OUTPUT_FILE, --output=OUTPUT_FILE
Don't print to stdout but to the output file
-i INPUT_FILE, --input=INPUT_FILE
Input file to be parsed
-x, --to-xml Returns a human-readable xml representation of the
parse tree
-b, --bison-debug Print the Bison/Flex debug

You can download the source code here

PLY
The second test I did was using PLY, an implementation of lex and yacc parsing tools for Python. Being implemented entirely in python it should be much more slower that pybison, but I didn’t find any difference with the pybison parser version. In fact PLY , like the traditional bison, creates tables starting from the grammar syntax.
Ok, Both of the implementations are slower that the pure C parser, but extremely faster that antlr!
(They took more or less 0.02 to 0.5 secs for parsing and generating the AST.)
Unlike pybison PLY is still mantained and offers more features and a better error handling.. even if the whole grammar has to be rewritten in python, and it can be compiled in Windows too.

To run the test just write:
python run.py {filename}
P.S. Unfortunately the yacc parser isn’t yet complete because I still need to find a way for parsing correctly E4X and XML syntax..