<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>parser &#8211; Alessandro Crugnola &#8211; sephiroth.it</title>
	<atom:link href="https://blog.sephiroth.it/tag/parser/feed/" rel="self" type="application/rss+xml" />
	<link>https://blog.sephiroth.it</link>
	<description>programmer by pure chance</description>
	<lastBuildDate>Mon, 21 Jun 2010 20:33:43 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	

<image>
	<url>https://blog.sephiroth.it/wp-content/uploads/2019/03/cropped-sepy-big-32x32.png</url>
	<title>parser &#8211; Alessandro Crugnola &#8211; sephiroth.it</title>
	<link>https://blog.sephiroth.it</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AsYacc &#8211; first alpha release</title>
		<link>https://blog.sephiroth.it/2009/01/15/asyacc-first-alpha-release/</link>
					<comments>https://blog.sephiroth.it/2009/01/15/asyacc-first-alpha-release/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Thu, 15 Jan 2009 15:35:48 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[asyacc]]></category>
		<category><![CDATA[compiler]]></category>
		<category><![CDATA[gnu flex]]></category>
		<category><![CDATA[lexer]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[regular expression]]></category>
		<category><![CDATA[yacc]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=271</guid>

					<description><![CDATA[Before going further with discussing how to add runtime scripting support to a Flash applications, I&#8217;d like to share with you the source code of a porting of Yacc I did a while ago. I did some simple modification to make Yacc able to generate ActionScript 3.0 source code instead of C. It works quite &#8230; <a href="https://blog.sephiroth.it/2009/01/15/asyacc-first-alpha-release/" class="more-link">Continue reading <span class="screen-reader-text">AsYacc &#8211; first alpha release</span></a>]]></description>
										<content:encoded><![CDATA[<p>Before going further with discussing how to add <a title="Runtime expression evaluation in ActionScript" href="https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/">runtime scripting support to a Flash applications</a>, I&#8217;d like to share with you the source code of a porting of <a href="http://en.wikipedia.org/wiki/Yacc">Yacc</a> I did a while ago. I did some simple modification to make Yacc able to <strong>generate ActionScript 3.0 source code</strong> instead of C. It works quite well and supports a lot standard Yacc features. There might be some issues &#8211; report them to me and I&#8217;ll try to fix them 🙂</p>
<p>You can easilly find documentation about Yacc searching Google (you can start <a href="http://dinosaur.compilertools.net/">here</a> for example).</p>
<p>Usually Yacc is used in conjunction with a <strong>Scanner generator</strong> (like <a href="http://en.wikipedia.org/wiki/Lex_programming_tool">Lex</a>/<a href="http://en.wikipedia.org/wiki/Flex_lexical_analyserx">Flex</a>), but I didn&#8217;t to any porting of commonly used Scanner generators yet.</p>
<p><a href="/wp-content/uploads/2009/01/AsYacc.zip">Here you can download the sources</a>. The <strong>source code should be portable</strong> and compilable on all the most common platform, but I didn&#8217;t tested it on Windows yet. To compile the source code on Mac or Linux, <em>cd</em> to the source code directory and type use:</p>
<pre>gcc *.c -o AsYacc</pre>
<p>Once you have the compiled binary file, you can run the Parser generator using:</p>
<pre>./AsYacc -Hpackage=it.sephiroth.test grammar.y</pre>
<p>Where <em>grammar.y</em> is a text file that contains the grammar of a language defined using the proper syntax (see the docs online for all the detailed information you may need).</p>
<p><a href="https://blog.sephiroth.it/wp-content/uploads/2009/01/TestAsYacc.zip">Here you can download a simple calculator example</a> that uses the RegexLexer <a title="Scripting Flash apps: scanning the input file" href="https://blog.sephiroth.it/2008/12/16/scripting-flash-apps-scanning-the-input-file/">described previously</a> to implement the Scanner. For the ones who might be interested, here is the grammar used:<br />
<code>/* Infix notation calculator. */<br />
%{<br />
%}<br />
%token NUM<br />
%left '-' '+'<br />
%left '*' '/'<br />
%left NEG<br />
%right '^' /* exponentiation */<br />
%%<br />
input:<br />
exp			{ trace( $1 ); }<br />
;<br />
exp:<br />
NUM 			{ $$ = $1; }<br />
| exp '+' exp 		{ $$ = $1 + $3; }<br />
| exp '-' exp 		{ $$ = $1 - $3; }<br />
| exp '*' exp 		{ $$ = $1 * $3; }<br />
| exp '/' exp 		{ $$ = $1 / $3; }<br />
| '-' exp %prec NEG 	{ $$ = -$2; }<br />
| exp '^' exp 		{ $$ = Math.pow( $1, $3 ); }<br />
| '(' exp ')' 		{ $$ = $2; }<br />
;<br />
%%</code></p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2009/01/15/asyacc-first-alpha-release/feed/</wfw:commentRss>
			<slash:comments>3</slash:comments>
		
		
			</item>
		<item>
		<title>Scripting Flash apps: scanning the input file</title>
		<link>https://blog.sephiroth.it/2008/12/16/scripting-flash-apps-scanning-the-input-file/</link>
					<comments>https://blog.sephiroth.it/2008/12/16/scripting-flash-apps-scanning-the-input-file/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Tue, 16 Dec 2008 17:06:54 +0000</pubDate>
				<category><![CDATA[scripting languages]]></category>
		<category><![CDATA[aif]]></category>
		<category><![CDATA[lexer]]></category>
		<category><![CDATA[namespace]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[regular expression]]></category>
		<category><![CDATA[scanner]]></category>
		<category><![CDATA[scripting]]></category>
		<category><![CDATA[token]]></category>
		<category><![CDATA[yacc]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=269</guid>

					<description><![CDATA[Here we go. I know that probably I should have started this new topic talking about the grammar of the language we are going to implement, but as long as I see grammar strictly related to parsing, I&#8217;ve preferred to talk about scanning first. We will go back to the grammar the next time, when &#8230; <a href="https://blog.sephiroth.it/2008/12/16/scripting-flash-apps-scanning-the-input-file/" class="more-link">Continue reading <span class="screen-reader-text">Scripting Flash apps: scanning the input file</span></a>]]></description>
										<content:encoded><![CDATA[<p>Here we go. I know that probably I should have started this new topic talking about the grammar of the language we are going to implement, but as long as I see grammar strictly related to parsing, I&#8217;ve preferred to talk about <strong>scanning</strong> first.</p>
<p>We will go back to the grammar the next time, when talking about how to parse an input file.</p>
<p>I wrote about scanning (or <strong>lexing</strong> if you prefer) a while ago, when blogging about expression evalution in ActionScript. Scanning an input file is quite always the same (although some languages might require unusual features), and what I wrote about expressions works also for a general language.</p>
<p>The goal of the scanning process is to group some characters together, skipping the ones that have no meaning for the language like spaces. Each group of characters is usually called <strong>token</strong>. So a <strong>Scanner</strong> converts a textual input into a stream of tokens, each one rappresenting a possible valid word for our language. While scanning you don&#8217;t take care about the meaning of what you are grouping or about the fact that a sequence of tokens is meaningful. This is a task for the <strong>Parser</strong> as we will see.</p>
<p>When talking about expressions, I showed a manual implementation for a Lexer. Now I want to take a different approach and show you a possible implementation for a simple dynamic scanner. This scanner will be based on <em>regular expressions</em>: each regular expression will rappresent a given token, and we will be able to assign callbacks to the scanner that will be executed each time a given token is extracted from the input.<br />
Writing a general and reusable scanner is usually a good practice. A common approach is to use some <strong>scanner generators</strong>, that are usually regular expression based too but are able to generate the code for a scanner at compile time. Our approach is different (and generates slower scanners) because regular expressions are evaluated at runtime; but it is fine for a test project.</p>
<p>You can download the source code for the regular expression based lexer (<em>RegexLexer</em>) <strong><a href="https://blog.sephiroth.it/wp-content/uploads/2008/12/scriptingflash.zip">here</a></strong>, as long as a simple usage example. Let&#8217;s see this example together so we can briefly discuss it (<em>PBLexer.as</em>):</p>
<p><span id="more-269"></span><br />
<code>package test<br />
{<br />
import it.sephiroth.scripting.utils.RegexLexer;<br />
import it.sephiroth.scripting.utils.RegexLexerContext;<br />
import it.sephiroth.scripting.utils.Token;<br />
public class PBLexer extends RegexLexer<br />
{<br />
private static const keyword_map: Object = {<br />
languageVersion:  TokensTable.LANGUAGE_VERSION,<br />
kernel:        TokensTable.KERNEL,<br />
parameter:      TokensTable.PARAMETER,<br />
input:        TokensTable.INPUT,<br />
image1:        TokensTable.IMAGE1,<br />
image2:        TokensTable.IMAGE2,<br />
image3:        TokensTable.IMAGE3,<br />
image4:        TokensTable.IMAGE4,<br />
"void":        TokensTable.VOID,<br />
output:        TokensTable.OUTPUT,<br />
evaluatePixel:    TokensTable.EVALUATE_PIXEL,<br />
"if":        TokensTable.IF,<br />
"else":        TokensTable.ELSE,<br />
"const":      TokensTable.CONST,<br />
"int":        TokensTable.INT,<br />
int2:        TokensTable.INT2,<br />
int3:        TokensTable.INT3,<br />
int4:        TokensTable.INT4,<br />
float:        TokensTable.FLOAT,<br />
float2:        TokensTable.FLOAT2,<br />
float3:        TokensTable.FLOAT3,<br />
float4:        TokensTable.FLOAT4,<br />
float2x2:      TokensTable.FLOAT2X2,<br />
float3x3:      TokensTable.FLOAT3X3,<br />
float4x4:      TokensTable.FLOAT4X4,<br />
pixel1:        TokensTable.PIXEL1,<br />
pixel2:        TokensTable.PIXEL2,<br />
pixel3:        TokensTable.PIXEL3,<br />
pixel4:        TokensTable.PIXEL4  };<br />
public function PBLexer( source_name: String, input: String )<br />
{<br />
super( source_name, input );<br />
addRule( "nl",      "\n",            -1,          RegexLexer.NL );<br />
addRule( "space",    "[ \t]+",           -1,          RegexLexer.SKIP );<br />
addRule( "comment",    "//.*",           -1,          RegexLexer.SKIP );<br />
addRule( "start_cmt",  "/\\*",           -1,          RegexLexer.START_EAT );<br />
addRule( "stop_cmt",  "\\*/",           -1,          RegexLexer.STOP_EAT );<br />
addRule( "ident",    "[a-zA-Z_][a-zA-Z_0-9]*",  TokensTable.IDENT,  onIdent );<br />
addRule( "number",    "0|([1-9][0-9]*(\\.[0-9]*)?)",  TokensTable.NUMBER, onNumber );<br />
addRule( "string",    "\"(\\\\\"|[^\"])+\"",    TokensTable.STRING, onString );<br />
addRule( "symbol",    "\\+\\+|\\-\\-|< =|>=|==|" +<br />
"!=|&amp;&amp;|\\|\\||\\*=|/=|" +<br />
"\\+=|\\-=|\\(|\\)|\\*|" +<br />
"\\+|,|\\-|\\.|/|:|;|#|" +<br />
"\\< |=|\\>|\\{|\\}|\\[|" +<br />
"\\]|\\?|!",          -1,          onSymbol );<br />
}<br />
private function onSymbol( context: RegexLexerContext ): void<br />
{<br />
var token: Token;<br />
token = new Token(<br />
context.token.value.charCodeAt( 0 ),<br />
context.token.value,<br />
context.token.sourceName,<br />
context.token.charno,<br />
context.token.lineno,<br />
context.token.columnno );<br />
context.token = token;<br />
}<br />
private function onIdent( context: RegexLexerContext ): void<br />
{<br />
var type: int;<br />
var token: Token;<br />
type = context.token.type;<br />
if( keyword_map[ context.token.value ] )<br />
{<br />
type = keyword_map[ context.token.value ];<br />
}<br />
token = new Token(<br />
type,<br />
context.token.value,<br />
context.token.sourceName,<br />
context.token.charno,<br />
context.token.lineno,<br />
context.token.columnno );<br />
context.token = token;<br />
}<br />
private function onNumber( context: RegexLexerContext ): void<br />
{<br />
var token: Token;<br />
var value: Number;<br />
value = parseFloat( context.token.value );<br />
token = new Token(<br />
context.token.type,<br />
value,<br />
context.token.sourceName,<br />
context.token.charno,<br />
context.token.lineno,<br />
context.token.columnno );<br />
context.token = token;<br />
}<br />
private function onString( context: RegexLexerContext ): void<br />
{<br />
var token: Token;<br />
var value: String;<br />
value = context.token.value.substr( 1, context.token.value.length - 2 );<br />
token = new Token(<br />
context.token.type,<br />
value,<br />
context.token.sourceName,<br />
context.token.charno,<br />
context.token.lineno,<br />
context.token.columnno );<br />
context.token = token;<br />
}<br />
}<br />
}</code><br />
As you may imagine, this example is a simple Lexer that is able to scan Pixel Bender input files. The Lexer is composed by a serie of rules registered using the function <em>addRule( name: String, regex: String, id: int, callback: Function )</em>. The callback function can be a user defined callback or a standard callback.<br />
There are some standard callbacks:</p>
<ul>
<li><strong>RegexLexer.NL</strong>: tells the Lexer that a new line token have been matched. This action increase an internal counter that keeps track of the line number;</li>
<li><strong>RegexLexer.SKIP</strong>: skip the current token;</li>
<li><strong>RegexLexer.START_EAT</strong>: start eating the tokens;</li>
<li><strong>RegexLexer.STOP_EAT</strong>: stop eating the tokens;</li>
</ul>
<p>Custom actions instead can be used to change the state of the Lexer or to generate a custom Token; take for instance the <em>onIdent</em> calback: each time an identifier is matched, we check if it is a keyword and if so, we generate the proper.</p>
<p>Here is an simple usage example:<br />
<code>package {<br />
import flash.display.Sprite;<br />
import it.sephiroth.scripting.utils.Token;<br />
import test.PBLexer;<br />
public class test extends Sprite<br />
{<br />
private static var INPUT: XML = <input />< ![CDATA[
/*****************************************************************************
*
* Copyright (C) 2008 Adobe Systems Incorporated
* All Rights Reserved.
*
* NOTICE:  All information contained  herein is,  and remains the property of
* Adobe Systems Incorporated and its suppliers, if any.  The intellectual and
* technical  concepts  contained  herein  are  proprietary  to  Adobe Systems
* Incorporated  and  its suppliers  and may  be covered  by U.S.  and Foreign
* Patents, patents in process, and are protected by trade secret or copyright
* law.  Dissemination of this information or reproduction of this material is
* strictly forbidden  unless prior  written permission is obtained from Adobe
* Systems Incorporated.
*
*****************************************************************************/
<languageVersion: 1.0;><br />
// basicBoxBlur: A simple example to demonstrate how to implement a 1<br />
//                iteration box blur.  The kernel is a good example to<br />
//                demonstrate how to implement a convolution using for loops.<br />
kernel basicBoxBlur<br />
< namespace : "AIF";
vendor : "Adobe Systems";
version : 2;
description : "variable Radius Box Blur"; ><br />
{<br />
// A parameter that represents the radius of the blur in both the X and Y<br />
// directions.<br />
parameter float blurRadius<br />
< minValue:float(0.0);
maxValue:float(25.0);
defaultValue:float(5.0);
>;<br />
// A dependent value.  These are calculated in the evaluateDependents()<br />
// function and save us from having to perform repetitive calculations<br />
// for each pixel.<br />
dependent int radiusAsInt;<br />
// evaluateDependents(): Built-in function to do any pre-processing of the<br />
//                       parameters.  Usage of this function helps avoid<br />
//                       repeating any necessary instructions per pixel.<br />
//                       In a filter as simple as this, the extra cycles<br />
//                       are probably not necessary, but it serves as a<br />
//                       good example.<br />
void<br />
evaluateDependents()<br />
{<br />
// Convert the dimension into a integer value for use in the iteration<br />
// over the kernel window.  All type conversions are required to be<br />
// explicit.<br />
radiusAsInt = int(ceil(blurRadius));<br />
}<br />
input image4 src;<br />
output float4 dst;<br />
// Region functions are not available in Flash targets, so we only define<br />
// the functions if we are executing on a different backend.<br />
#if !AIF_FLASH_TARGET<br />
// needed(): Indicates what area of the input is needed to fulfill the<br />
//           requested output region.  In this case, we consider a<br />
//           radius sized area of the input for each output pixel, so we<br />
//           need to outset the area by one.<br />
region needed(region outputRegion, imageRef inputRef)<br />
{<br />
float2 singlePixel = pixelSize(src);<br />
return outset(outputRegion, float2(singlePixel.x * ceil(blurRadius), singlePixel.y * ceil(blurRadius)));<br />
}<br />
// changed(): Indicates what area of the output is affected by the<br />
//            specified input.  In this case, we will blur out the image<br />
//            by the size of the radius, so the output is the input outset by<br />
//            one.<br />
region changed(region inputRegion, imageRef inputRef)<br />
{<br />
float2 singlePixel = pixelSize(src);<br />
return outset(inputRegion, float2(singlePixel.x * ceil(blurRadius), singlePixel.y * ceil(blurRadius)));<br />
}<br />
#endif<br />
// evaluatePixel(): The function of the filter that actually does the<br />
//                  processing of the image.  This function is called once<br />
//                  for each pixel of the output image.<br />
void<br />
evaluatePixel()<br />
{<br />
// Containers for both the accumulated color as well as the number of<br />
// pixels touched.<br />
float denominator = 0.0;<br />
float4 colorAccumulator = float4(0.0, 0.0, 0.0, 0.0);<br />
float2 singlePixel = pixelSize(src);<br />
// Iterate through the kernel window, adding up the samples in<br />
// that location.<br />
for(int i = -radiusAsInt; i < = radiusAsInt; i++)
{
for(int j = -radiusAsInt; j <= radiusAsInt; j++)
{
colorAccumulator += sampleNearest(src,
outCoord() + float2(float(i) * singlePixel.x, float(j) * singlePixel.y));
denominator++;
}
}
// calculate the resultant pixel value which is the accumulated color
// divided by the total number of pixels.
// In this case, the denominator is 9.
dst = colorAccumulator / denominator;
}
}
]]&gt;<br />
public function test()<br />
{<br />
super();<br />
var token: Token;<br />
var lexer: PBLexer;<br />
lexer = new PBLexer( "<file>", INPUT.text() );<br />
while( token = lexer.next() )<br />
{<br />
if( token.type == Token.EOF )<br />
{<br />
break;<br />
}<br />
trace( token );<br />
}<br />
}<br />
}<br />
}</file></code><br />
Next week we will move further and start discussing the grammar, implementing a simple Parser using a custom version of <a href="http://en.wikipedia.org/wiki/Yacc">Yacc</a> that generates ActionScript code.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2008/12/16/scripting-flash-apps-scanning-the-input-file/feed/</wfw:commentRss>
			<slash:comments>2</slash:comments>
		
		
			</item>
		<item>
		<title>Top Down or Bottom Up for Expression Evaluation ?</title>
		<link>https://blog.sephiroth.it/2008/05/03/top-down-or-bottom-up-for-expression-evaluation/</link>
					<comments>https://blog.sephiroth.it/2008/05/03/top-down-or-bottom-up-for-expression-evaluation/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Sun, 04 May 2008 01:00:35 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[scripting languages]]></category>
		<category><![CDATA[abstract syntax tree]]></category>
		<category><![CDATA[ast]]></category>
		<category><![CDATA[byacc]]></category>
		<category><![CDATA[gnu flex]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[yacc]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=252</guid>

					<description><![CDATA[Last time I wrote about Top Down parsing, a technique for parsing that can be easilly implemented manually using function recursion. Today it is time to talk about one of the other ways of parsing, called Bottom Up parsing. This technique try to find the most important units first and then, based on a language &#8230; <a href="https://blog.sephiroth.it/2008/05/03/top-down-or-bottom-up-for-expression-evaluation/" class="more-link">Continue reading <span class="screen-reader-text">Top Down or Bottom Up for Expression Evaluation ?</span></a>]]></description>
										<content:encoded><![CDATA[<p><a title="Runtime expression evaluation in ActionScript" href="https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/">Last time</a> I wrote about <a href="http://en.wikipedia.org/wiki/Top-down_parsing">Top Down</a> parsing, a technique for parsing that can be easilly implemented manually using function recursion.</p>
<p>Today it is time to talk about one of the other ways of parsing, called <a href="http://en.wikipedia.org/wiki/Bottom-up_parsing">Bottom Up</a> parsing. This technique try to find the most important units first and then, based on a language grammar and a set of rules, try to infer higher order structures starting from them. Bottom Up parsers are the common output of <a href="http://en.wikipedia.org/wiki/Parser_generator">Parser Generators</a> (you can find a good comparison of parser generators <a href="http://en.wikipedia.org/wiki/Comparison_of_parser_generators">here</a>) as they are easier to generate automatically then recursive parser because they can be implemented using sets of tables and simple state based machines.</p>
<p>Writing down manually a bottom up parser is quite a tedious task and require quite a lot of time; moreover, this kind of parsers are difficult to maintain and are usually not really expandable or portable. This is why for my tests I decided to port <a href="http://invisible-island.net/byacc/byacc.html">bYacc</a> (a parser generator that usually generates C code) and edit it so it generates ActionScript code starting from Yacc-compatible input grammars. Having this kind of tool makes things a lot easier, because maintaining a grammar (that usually is composed by a few lines) is less time expensive than working with the generated code (that usually is many lines long).<br />
I will not release today the port because actually I had not time to make sure it is bugfree and I&#8217;ve only a working version for Mac, but I plan to release it shortly if you will ever need it for your own tasks. My goal for today was to compare the speed of the parser I wrote with an automatically generated bottom up parser, to see which is the faster approach.</p>
<p>I created a bottom up parser which is able to execute the same expressions accepted by the expression evaluator I wrote last time. There are anyways some differences that &#8211; as you will probably and hopefully understand in the future &#8211; that make those parsers really different. Some will be disussed shortly here.</p>
<p>To do that I created a Yacc grammar and some support classes.<br />
The parser grammar is really simple and really readable:<br />
<code>%{<br />
%}<br />
%token NUMBER SYMBOL<br />
%left '+' '-'<br />
%left '*' '/'<br />
%left NEG<br />
%%<br />
program<br />
: expr                { Vars.result = $1; }<br />
;<br />
expr<br />
: expr '+' expr       { $$ = $1 + $3; }<br />
| expr '-' expr       { $$ = $1 - $3; }<br />
| expr '*' expr       { $$ = $1 * $3; }<br />
| expr '/' expr       { $$ = $1 / $3; }<br />
| '-' expr %prec NEG  { $$ = -$2; }<br />
| '(' expr ')'        { $$ = $2; }<br />
| SYMBOL '(' expr ')' {<br />
if( Vars.SYMBOL_TABLE[ $1 ] )<br />
{<br />
$$ = Vars.SYMBOL_TABLE[ $1 ]( $3 );<br />
} else<br />
{<br />
trace( "can't find function" );<br />
}<br />
}<br />
| SYMBOL              {<br />
if( Vars.SYMBOL_TABLE[ $1 ] )<br />
{<br />
$$ = Vars.SYMBOL_TABLE[ $1 ];<br />
} else<br />
{<br />
trace( "can't find symbol" );<br />
}<br />
}<br />
| NUMBER              { $$ = yyval; }<br />
;<br />
%%</code><br />
<strong>Continue reading the extended entry to see the results.<span id="more-252"></span></strong></p>
<p>It defined a simple expression grammar which includes addition, subtraction, multiplication, negation, division, variables, function calls and sub expressions. The tokens accepted are numbers (integers or simple floating point numbers), symbols and all the punctuation and operands needed.</p>
<p>The grammar uses a simple support class (called <em>Vars</em>) which is used to store a global symbol table and the result of the evaluation.<br />
As you might have noted there is a really big difference about how the expression will be evaluated here and how do I evaluate it in my original expression parser: the expression is executed as soon as a rule is matched, and it is not compiled into an intermediate <a href="http://en.wikipedia.org/wiki/Abstract_syntax_tree">abstract syntax tree</a>.<br />
Usually those kind of tools are used in conjunction with a lexer generator (<a href="http://en.wikipedia.org/wiki/Flex_lexical_analyser">Flex</a> is a good example of lexer generator &#8211; don&#8217;t confuse it with Adobe Flex!! :D), that is able to automatically generate the code for a scanner based in a set of rules usually defined by regular expressions. I didn&#8217;t ported this kind of tool because I was already having a working scanner for my needs that is a porting of the scanner used in the expression evaluator library:</p>
<p><code lang="actionscript"><br />
package it.sephiroth.expr2<br />
{<br />
	public class Lexer<br />
	{<br />
		private var _pos: int;<br />
		private var _pool: Array;<br />
		private var _source: String;<br />
		public function Lexer( source: String )<br />
		{<br />
			_pos = 0;<br />
			_source = source;<br />
			_pool = new Array();<br />
		}<br />
		public function scan(): Object<br />
		{<br />
			if( _pool.length > 0 )<br />
			{<br />
				return _pool.shift();<br />
			}<br />
			skipWhites();<br />
			if( isEOF() )<br />
			{<br />
				return null;<br />
			}<br />
			var c: String = _source.charAt( _pos++ );<br />
			switch( c )<br />
			{<br />
				case '+':<br />
				case '-':<br />
				case '*':<br />
				case '/':<br />
				case '(':<br />
				case ')':<br />
				case ',':  return  { type: c.charCodeAt( 0 ), val: c };<br />
				default:<br />
					var buf: String = "";<br />
					var code: int = c.charCodeAt( 0 );<br />
					if( isNumber( code ) )<br />
					{<br />
						var num: Number;<br />
						while( isNumber( code ) )<br />
						{<br />
							buf += c;<br />
							if( isEOF() )<br />
							{<br />
								++_pos;<br />
								break;<br />
							}<br />
							c  = _source.charAt( _pos++ );<br />
							code = c.charCodeAt( 0 );<br />
						}<br />
						if( c == '.' )<br />
						{<br />
							buf += c;<br />
							c  = _source.charAt( _pos++ );<br />
							code = c.charCodeAt( 0 );<br />
							while( isNumber( code ) )<br />
							{<br />
								buf += c;<br />
								if( isEOF() )<br />
								{<br />
									++_pos;<br />
									break;<br />
								}<br />
								c  = _source.charAt( _pos++ );<br />
								code = c.charCodeAt( 0 );<br />
							}<br />
							num = parseFloat( buf );<br />
						}else<br />
						{<br />
							num = parseInt( buf );<br />
						}<br />
						--_pos;<br />
						return  { type: Token.NUMBER, val: num };<br />
					}<br />
					if( isAlpha( code ) || ( c == '_' ) )<br />
					{<br />
						while( isAlpha( code ) || ( c == '_' ) || isNumber( code ) )<br />
						{<br />
							buf += c;<br />
							if( isEOF() )<br />
							{<br />
								++_pos;<br />
								break;<br />
							}<br />
							c  = _source.charAt( _pos++ );<br />
							code = c.charCodeAt( 0 );<br />
						}<br />
						--_pos;<br />
						return  { type: Token.SYMBOL, val: buf };<br />
					}<br />
					break;<br />
			}<br />
			return null;<br />
		}<br />
		public function pushBack( token: Token ): void<br />
		{<br />
			_pool.push( token );<br />
		}<br />
		protected function isNumber( c: int ): Boolean<br />
		{<br />
			return ( c >= 48 ) && ( c < = 57 );
		}
		protected function isAlpha( c: int ): Boolean
		{
			return ( ( c >= 97 ) && ( c < = 122 ) ) || ( ( c >= 65 ) && ( c < = 90 ) );
		}
		protected function isEOF(): Boolean
		{
			return ( _pos >= _source.length );<br />
		}<br />
		protected function skipWhites(): void<br />
		{<br />
			while( !isEOF() )<br />
			{<br />
				var c: String = _source.charAt( _pos++ );<br />
				if( ( c != " " ) && ( c != "\t" ) )<br />
				{<br />
					--_pos;<br />
					break;<br />
				}<br />
			}<br />
		}<br />
	}<br />
}<br />
</code></p>
<p>After compiling the grammar with my parser generator and having written some support code we can use to do our tests, I finally had my small comparison test suite.  A quick note about the parser: It should be used only for testing and if you want to have a fast and safe expressions evaluator I suggest you to build your own or &#8211; if you can wait &#8211; wait a bit more 😉  The test can be easilly run from the command line without the need fo having Flex Builder (but you need the Flex SDK that you can freely download from the <a href="http://opensource.adobe.com/wiki/display/flexsdk/">Adobe Open Source Website</a>); I choose a quite long expression to make the execution and parsing time a bit high.  Here is the test I ran:</p>
<p><code><br />
package {<br />
	import flash.display.Sprite;<br />
	import flash.utils.getTimer;<br />
	import it.sephiroth.expr.CompiledExpression;<br />
	import it.sephiroth.expr.Parser;<br />
	import it.sephiroth.expr.Scanner;<br />
	import it.sephiroth.expr2.Vars;<br />
	public class TestExprEval extends Sprite<br />
	{<br />
		public static const ITERATIONS: int = 10000;<br />
		public static const EXPRESSION: String = "sin( x / ( 4 / 2.0 * 2 + (- 2.12345 + 2.12345) * x / x ) ) * 100 - ( sin( x / ( 4 / 2.0 * 2 + (- 2.12345 + 2.12345) * x / x ) ) * 100 )";<br />
		public function TestExprEval()<br />
		{<br />
			runTests();<br />
		}<br />
		private function runTests(): void<br />
		{<br />
			trace( "Running tests (this may take a while)" );<br />
			runTopDown();<br />
			runBottomUp();<br />
		}<br />
		private function runTopDown(): void<br />
		{<br />
			var sum: Number = 0;<br />
			for( var i: int = 0; i < ITERATIONS; ++i )
			{
				var t: Number = getTimer();
				var scanner: Scanner = new Scanner( EXPRESSION );
				var parser: Parser = new Parser( scanner );
				var compiled: CompiledExpression = parser.parse();
				compiled.execute( {
					'x'    : Math.random(),
					'sin'  : Math.sin,
					'cos'  : Math.cos
				} );
				sum += ( getTimer() - t );
			}
			trace( "Top Down parser:", sum / ITERATIONS );
		}
		private function runBottomUp(): void
		{
			var sum: Number = 0;
			Vars.symbolTable.sin = Math.sin;
			for( var i: int = 0; i < ITERATIONS; ++i )
			{
				Vars.symbolTable.x = Math.random();
				var t: Number = getTimer();
				var myparser: TestBottomUpParser = new TestBottomUpParser();
				myparser.yyparse()
				sum += ( getTimer() - t );
			}
			trace( "Bottom Up parser:", sum / ITERATIONS );
		}
	}
}
import it.sephiroth.expr2.BottomUpParser;
import it.sephiroth.expr2.Lexer;
class TestBottomUpParser extends BottomUpParser
{
	private var _lexer: Lexer;
	public function TestBottomUpParser()
	{
		super();
		_lexer = new Lexer( TestExprEval.EXPRESSION );
	}
	override public function yylex(): int
	{
		var value: Object = _lexer.scan();
		if( value == null )
		{
			return 0;
		}
		set_yylval( value.val );
		return value.type;
	}
}
</code></p>
<p>As you can see the test includes also the instantiation of all the objects needed. Running that from the command line, I obtained this result:<br />
</code><code>$ mxmlc -debug=true TestExprEval.as<br />
Loading configuration file /Applications/Adobe Flex Builder 3 Plug-in/sdks/3.0.0/frameworks/flex-config.xml<br />
/Users/gabriele/Desktop/TestExprParsers/src/TestExprEval.swf (11647 bytes)<br />
$ fdb TestExprEval.swf<br />
Adobe fdb (Flash Player Debugger) [build 3.0.0.477]<br />
Copyright (c) 2004-2007 Adobe, Inc. All rights reserved.<br />
Attempting to launch and connect to Player using URL<br />
TestExprEval.swf<br />
Player connected; session starting.<br />
Set breakpoints and then type 'continue' to resume the session.<br />
[SWF] Users:gabriele:Desktop:TestExprParsers:src:TestExprEval.swf - 25.942 bytes after decompression<br />
(fdb) continue<br />
[trace] Running tests (this may take a while)<br />
[trace] Top Down parser: 0.6979<br />
[trace] Bottom Up parser: 0.9063<br />
(fdb) quit</code><br />
This is not totally unexpected for such a simple grammar, but the execution time is much more better (around 20%) using a manually written top down parser than using an autogenerated bottom up parser; I say it is not totally unexpected because the performances should change as the grammar becames more complex, but in our situation this is what we get. And we can also achive a better result if we precompile the expression into an AST and the execute it. Add this code to your test and add the function call inside the <em>runTests</em> method:<br />
<code><br />
private function runTopDownCompiled(): void<br />
{<br />
	var sum: Number = 0;<br />
	var scanner: Scanner = new Scanner( EXPRESSION );<br />
	var parser: Parser = new Parser( scanner );<br />
	var compiled: CompiledExpression = parser.parse();<br />
	for( var i: int = 0; i < ITERATIONS; ++i )
	{
		var t: Number = getTimer();
		compiled.execute( {
			'x':     Math.random(),
			'sin'  : Math.sin,
			'cos'  : Math.cos
		} );
		sum += ( getTimer() - t );
	}
	trace( "Top Down parser (precompiled):", sum / ITERATIONS );
}
</code><br />
And then run again to see the result:<br />
</code><code>$ mxmlc -debug=true TestExprEval.as<br />
Loading configuration file /Applications/Adobe Flex Builder 3 Plug-in/sdks/3.0.0/frameworks/flex-config.xml<br />
/Users/gabriele/Desktop/TestExprParsers/src/TestExprEval.swf (11794 bytes)<br />
$ fdb TestExprEval.swf<br />
Adobe fdb (Flash Player Debugger) [build 3.0.0.477]<br />
Copyright (c) 2004-2007 Adobe, Inc. All rights reserved.<br />
Attempting to launch and connect to Player using URL<br />
TestExprEval.swf<br />
Player connected; session starting.<br />
Set breakpoints and then type 'continue' to resume the session.<br />
[SWF] Users:gabriele:Desktop:TestExprParsers:src:TestExprEval.swf - 26.272 bytes after decompression<br />
(fdb) continue<br />
[trace] Running tests (this may take a while)<br />
[trace] Top Down parser: 0.698<br />
[trace] Bottom Up parser: 0.9001<br />
[trace] Top Down parser (precompiled): 0.0315</code><br />
This is really a good thing: as we can see, the most of the time is taken by the parser and the scanner, and if we remove their execution from the loop and just evaluate the generated AST, we get a great speed improvment. As far as I know this is quite good if you have a set of expressions that does not vary and do you want to execute them many times.</p>
<p>And this is why quite all the languages precompile their source code into bytecode before execution.</p>
<p>Unfortunately it is time to go now, but I&#8217;ve really a lot of stuff to show you next week (the parser generator, a way to &#8211; try to &#8211; write an expression evaluator that evaluate expressions at near-native speed &#8230;) so stay tuned as usual!</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2008/05/03/top-down-or-bottom-up-for-expression-evaluation/feed/</wfw:commentRss>
			<slash:comments>2</slash:comments>
		
		
			</item>
		<item>
		<title>Runtime expression evaluation in ActionScript</title>
		<link>https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/</link>
					<comments>https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Sat, 19 Apr 2008 15:00:00 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[scripting languages]]></category>
		<category><![CDATA[abstract syntax tree]]></category>
		<category><![CDATA[ast]]></category>
		<category><![CDATA[evaluation]]></category>
		<category><![CDATA[expression]]></category>
		<category><![CDATA[lexer]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[postfix]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=250</guid>

					<description><![CDATA[Finally I have a bit of free time to write down this post. Some days ago a guy on the forum asked about how to evaluate at runtime simple mathematical expressions with support for symbols and function calls. Now that the eval function have been removed from the language for reasons I&#8217;ve not enought time &#8230; <a href="https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/" class="more-link">Continue reading <span class="screen-reader-text">Runtime expression evaluation in ActionScript</span></a>]]></description>
										<content:encoded><![CDATA[<p>Finally I have a bit of free time to write down this post.<br />
Some days ago a guy on the <a href="http://sephiroth.it/phpBB/">forum</a> asked about how to <strong>evaluate at runtime simple mathematical expressions</strong> with support for symbols and function calls.</p>
<p>Now that the <em>eval</em> function have been removed from the language for reasons I&#8217;ve not enought time to talk about, we need to parse and execute those expressions manually.</p>
<p>I built a simple Actionscript library that can be used to parse mathematical expressions: it has support for function calls, for variables and it is able to convert the expression into a postfix rappresentation if you may need it. The expression parser is quite simple and have been built manually following some concepts related to programming language compilers; it includes a <a href="http://en.wikipedia.org/wiki/Lexical_analysis">Scanner</a> (or lexer &#8211; however you want to call it) to tokenize the expression, a <a href="http://en.wikipedia.org/wiki/Parser">Parser</a> that convert the expression into an <a href="http://en.wikipedia.org/wiki/Abstract_Syntax_Tree">Abstract Syntax Tree</a> and a simple routine that evaluates that AST using a <a href="http://en.wikipedia.org/wiki/Symbol_table">Symbol Table</a> as evaluation context.</p>
<p>There are no comments inside the code right now; I hope to find a little bit of time to write an in depth discussion about this topic. In the mean time you can continue reading the entry for a general explanation about how does the code works and for some examples that may be useful.</p>
<p>Here is a simple example that shows how does the code works:<br />
<code lang="actionscript"><br />
import it.sephiroth.expr.CompiledExpression;<br />
import it.sephiroth.expr.Parser;<br />
import it.sephiroth.expr.Scanner;<br />
public class Example<br />
{<br />
public static function run(): void<br />
{<br />
var expression: String = "sin( x / ( 8 / 2 + (-0.12 + 2.3) * x / x ) ) * 100";<br />
var scanner: Scanner = new Scanner( expression );<br />
var parser: Parser = new Parser( scanner );<br />
var compiled: CompiledExpression = parser.parse();<br />
var context: Object = {<br />
x:    100,<br />
sin:   Math.sin,<br />
cos:  Math.cos<br />
};<br />
trace( 'Postfix:', compiled.toString() );<br />
trace( 'Result:', compiled.execute( context ) );<br />
}<br />
}<br />
</code><br />
Before I forgot, you can download the source code here with a simple example included.</p>
<p><span id="more-250"></span></p>
<p>Building an expression parser and evaluator is quite a simple task if you don&#8217;t plan to write a superfast library or a superpowerful engine. The steps are something like <em>predefined</em> and can be subdivided into tree groups:</p>
<ul>
<li><strong>Lexical analysis</strong></li>
<li><strong>Parsing</strong></li>
<li><strong>Evaluation</strong></li>
</ul>
<p>My code simply follow those steps to make you able to evaluate an expression. Before starting with the code anyways it is really important to define a set of rules and functionalities our expression evaluator will include. This is foundamental because based on the features the operations performed during the tree steps above may vary.</p>
<p>In my code I simply want to accept integer and float numbers (float numbers defined using the simple <em>dot syntax</em>), negative numbers, variables, function calls, basic operations (subtract, add, multiply, divide and negate) and parethesis for grouping.</p>
<h3>Lexical Analysis</h3>
<p>During Lexical analysis the <strong>input string is analyzed</strong> to provide the parser a set of chunks, called <strong>tokens</strong>, that can be <strong>interpreted easilly by our code</strong>. The main goal of the lexical analysis is just to group togheter some characters and give them a meaning (for instance: 1 is a number, / the divide operation and so on), discarding the unuseful ones (like spaces). At this stage there is <strong>no validation of the input</strong>, because the lexical analyzer (<em>Scanner.as</em>) doesn&#8217;t know anything about how does the input will be interpreted.<br />
The lexical analyzer generates a stream of tokens (<em>Token.as</em>) of different type (<em>TokenType.as</em>) which are then feeded to the Parser.</p>
<h3>Parsing</h3>
<p>The Parsing phase is one of the most important and is when the program starts to take a streams of token and give a meaning to a set of them. During this step the Parser (<em>Parser.as</em>) <strong>reads from the Scanner the tokens</strong> as needed and, given a defined Grammar, is able to group them togheter and <strong>understand if the input is grammatically correct</strong> or not.<br />
In our sitatuation the grammar is implicit: the expression must be formatted correctly, every open parenthesis must match a closed one and function calls allows one or more arguments. When working on more complex parsers, the grammar may became much more complex (think about a whatever language parser for instance), and so the technique I used (called <a href="http://en.wikipedia.org/wiki/Recursive_descent_parser">Recursive Descent Parsing</a>) risk to become too difficult to use, too time and memory consuming. At this stage may be helpful to use <strong>parser generators</strong>, but I&#8217;ll talk about that the next time because I&#8217;ve something interestingto show you.<br />
During this phase the input is validated but also other two really important tasks are performed: the first one is that <strong>all the identifiers found inside the input are recorded inside a Symbol Table</strong> (<em>SymbolTable.as</em>), than later will be filled at runtime with valid values for the indetifiers to execute the expression correctly. The second one is that <strong>the input is translated into an intermediate form</strong> that is <strong>easier for a computer language to deal with</strong>: the Abstract Syntax Tree (<em>it.sephiroth.expr.ast.*</em> package). The <strong>AST</strong> is a tree structure where each node rappresents a portion of code that then can be analyzed easilly using a tree walker or other means. When evaluating expressions an AST is not always useful because <strong>expressions evaluation may be performed using a stack based direct execution</strong>.</p>
<h3>Evaluation</h3>
<p>When the AST is ready, we are sure that the expression is syntactically valid and can be executed providing the right <strong>context</strong>. A context is simply the variables and functions that are needed by the expression to return a value, and are assigned to the identified included inside the Symbol Table. When the expression is executing, it requests from the symbol table the values needed. In my code the evaluation is performed by each AST node itself, but we may use also a tree walker to walk each node and execute it based on its type.</p>
<p>This is a super general introduction about how does the code works, hope to have a bit of free time to go more in depth about this. Just to show you something working, here is a simple Flex examples that plots an expression. The potting is not accurate, but the example can be used to give you an idea about how does the code may be used. With the separation provided, we will be able to replace the single parts of the code with improved versions, obtaining anyway a correct result.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2008/04/19/runtime-expression-evaluation-in-actionscript/feed/</wfw:commentRss>
			<slash:comments>25</slash:comments>
		
		
			</item>
		<item>
		<title>Actionscript parsing experiences: PyBison &#038; PLY</title>
		<link>https://blog.sephiroth.it/2007/01/16/actionscript-parsing-experiences-pybison-ply/</link>
					<comments>https://blog.sephiroth.it/2007/01/16/actionscript-parsing-experiences-pybison-ply/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Tue, 16 Jan 2007 21:35:45 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[antlr]]></category>
		<category><![CDATA[ast]]></category>
		<category><![CDATA[bison]]></category>
		<category><![CDATA[e4x]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[ply]]></category>
		<category><![CDATA[pybison]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=195</guid>

					<description><![CDATA[My experiments with actionscript text-parsing continue, This time I tested two bison python implementations: pybison and ply.<br />
]]></description>
										<content:encoded><![CDATA[<p>My experiments with text-parsing continue..</p>
<p><span style="font-weight: bold;">PyBison</span><br />
Last day I founded a python library <a title="pybison" href="http://www.freenet.org.nz/python/pybison/" target="_blank">(pybison</a>) which runs the generated python parser at near the speed of C-based parsers, due to direct hooks into bison-generated C code.<br />
Cool, unfortunately I couldn&#8217;t compiled it for Windows and so I made my test on Ubuntu only. What I did was just to export the already written lexer/grammar using bison2py (boundled with pybison) and run it.</p>
<p>If you want to take a look at the python parser try it by downloading the source code here.<br />
The run.py file accept these parameters:<br />
<code>usage: run.py [options]<br />
options:<br />
-h, --help            show this help message and exit<br />
-v, --verbose         Turn on verbose mode<br />
-o OUTPUT_FILE, --output=OUTPUT_FILE<br />
Don't print to stdout but to the output file<br />
-i INPUT_FILE, --input=INPUT_FILE<br />
Input file to be parsed<br />
-x, --to-xml          Returns a human-readable xml representation of the<br />
parse tree<br />
-b, --bison-debug     Print the Bison/Flex debug</code><br />
You can download the source code <a title="pybison code source code" href="https://blog.sephiroth.it/wp-content/uploads/2007/01/pybison.tar.gz">here</a></p>
<p><span style="font-weight: bold;">PLY</span><br />
The second test I did was using <a href="http://www.dabeaz.com/ply/" target="_blank">PLY</a>, an implementation of  lex and yacc parsing tools for Python. Being implemented entirely in python it should be much more slower that pybison, but I didn&#8217;t find any difference with the pybison parser version. In fact PLY , like the traditional bison, creates tables starting from the grammar syntax.<br />
Ok, Both of the implementations are slower that the pure C parser, but extremely faster that <span style="font-weight: bold;">antlr</span>!<br />
(They took more or less 0.02 to 0.5 secs for parsing and generating the AST.)<br />
Unlike pybison PLY is still mantained and offers more features and a better error handling.. even if the whole grammar has to be rewritten in python, and it can be compiled in Windows too.</p>
<p>To run the test just write:<br />
<code>python run.py {filename}</code><br />
<span style="font-weight: bold;">P.S.</span> Unfortunately the yacc parser isn&#8217;t yet complete because I still need to find a way for parsing correctly E4X and XML syntax..</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2007/01/16/actionscript-parsing-experiences-pybison-ply/feed/</wfw:commentRss>
			<slash:comments>2</slash:comments>
		
		
			</item>
		<item>
		<title>ActionScript Parsing, the YACC revenge :)</title>
		<link>https://blog.sephiroth.it/2006/12/10/actionscript-parsing-the-yacc-revenge/</link>
					<comments>https://blog.sephiroth.it/2006/12/10/actionscript-parsing-the-yacc-revenge/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Mon, 11 Dec 2006 00:47:30 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[antlr]]></category>
		<category><![CDATA[bison]]></category>
		<category><![CDATA[ecma]]></category>
		<category><![CDATA[ecmascript]]></category>
		<category><![CDATA[elastic]]></category>
		<category><![CDATA[gnu flex]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[pyrex]]></category>
		<category><![CDATA[yacc]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=188</guid>

					<description><![CDATA[After my first <a target="_blank" href="http://www.sephiroth.it/weblog/archives/2006/12/an_experience_with_antlr_with_java_an.php">attempts with ANTLR</a> scanners in python/java I decided to start back with Bison/Flex again to see the difference in performances...
]]></description>
										<content:encoded><![CDATA[<p>After my first <a title="An experience with antlr, java and python" href="https://blog.sephiroth.it/2006/12/01/an-experience-with-antlr-java-and-python/" target="_blank">attempts with ANTLR</a> scanners in python/java I decided to start back with Bison/Flex again to see the difference in performances.<br />
So first I need to wrote from scratch the grammar/lexer files using only the <a href="http://www.mozilla.org/js/language/es4/" target="_blank">ECMAScript 4</a> specifications and much patience (the <a href="http://svn.berlios.de/wsvn/elastic/trunk/elastiC/src/elastic_p.y?op=file&amp;rev=0&amp;sc=0">elastic grammar file</a> help me a lot too).</p>
<p>After finishing a first version of the parser I tested it on the same file (75Kb actionscript file) which both java and python parsed in more than 1 second.<br />
The result was unbelievable: <span style="font-weight: bold;">0.02</span> seconds for that file!</p>
<p>Then I tested it on multiple files, and for about 320 files of the whole adobe corelib library it took 220ms</p>
<p>Ok, the parser it&#8217;s not yet complete and doesn&#8217;t care about regexp and xml syntax, but its performance convinced me enough&#8230;<br />
Now, the next step is to finish and test the parser and finally create a python library using <a href="http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/" target="_blank">pyrex</a>, then a benchmark test again.</p>
<p>If someone is interested in testing the parser, <a href="https://blog.sephiroth.it/wp-content/uploads/2006/12/parser.exe">download it</a> (use &#8220;<span style="font-style: italic;">parser &#8211;help</span>&#8221; form the command line for usage help), but remember this is only a first test.. not really helpful right now (I just wanted to share my text/parsing experiences).</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2006/12/10/actionscript-parsing-the-yacc-revenge/feed/</wfw:commentRss>
			<slash:comments>5</slash:comments>
		
		
			</item>
		<item>
		<title>An experience with antlr, java and python</title>
		<link>https://blog.sephiroth.it/2006/12/01/an-experience-with-antlr-java-and-python/</link>
					<comments>https://blog.sephiroth.it/2006/12/01/an-experience-with-antlr-java-and-python/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Fri, 01 Dec 2006 19:09:11 +0000</pubDate>
				<category><![CDATA[general]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[antlr]]></category>
		<category><![CDATA[asdt]]></category>
		<category><![CDATA[bison]]></category>
		<category><![CDATA[eclipse]]></category>
		<category><![CDATA[ecma]]></category>
		<category><![CDATA[gnu flex]]></category>
		<category><![CDATA[grammar]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[profiler]]></category>
		<category><![CDATA[psyco]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=187</guid>

					<description><![CDATA[I just wanted to share a little experience with generating an AS3 parser using antlr and python.I was trying first to create the parser using...
]]></description>
										<content:encoded><![CDATA[<p>I just wanted to share a little experience with generating an AS3 parser using <a href="http://www.antlr.org">antlr</a> and <a href="http://www.python.org">python</a>.<br />
I was trying first to create the parser using GNU <a title="gnu flex" href="http://www.gnu.org/software/flex/" target="_blank">Flex</a> and <a title="gnu bison" href="http://www.gnu.org/software/bison/" target="_blank">Bison</a> in C, probably the best way for a very performancing parser.<br />
Yeah, that&#8217;s right.. but looking at the antlr syntax I realized that&#8217;s easier and easier.<br />
Moreover I start using this very useful <a title="antlrstudio" href="http://www.placidsystems.com/antlrstudio.aspx" target="_blank">eclipse plugin</a> for antlr debugging which made my life easier!</p>
<p>The grammar file I created is a compromise between the <a title="as3.g" href="http://www.google.com/codesearch?hl=it&amp;q=+file:%5C.g%24+as3.g+show:gFeBx72kxIk:XKFG9i76Jiw:mSlmdOLa2XQ&amp;sa=N&amp;cd=1&amp;ct=rc&amp;cs_p=:pserver:anonymous%40aseclipseplugin.cvs.sourceforge.net:/cvsroot/aseclipseplugin+org.asdt.astview&amp;cs_f=src/org/asdt/core/internal/antlr/as3.g#a0" target="_blank">asdt grammar</a> file and the <a title="ECMA-262 specifications" href="http://www.ecma-international.org/publications/standards/Ecma-262.htm" target="_blank">ECMA-262 grammar</a> specification.</p>
<p>Once finished working on my eclipse project I&#8217;ve managed to parse succesfully all the adobe corelibs files using this java test file:<br />
<code lang="java">package org.sepy.core.parsers.as3;<br />
import java.io.File;<br />
import java.io.FileInputStream;<br />
import java.io.FileNotFoundException;<br />
import antlr.CommonAST;<br />
import antlr.RecognitionException;<br />
import antlr.TokenStreamException;<br />
public class Application {<br />
public static void main(String argv[])<br />
{<br />
if(argv.length &gt; 0)<br />
{<br />
File file = new File(argv[0]);<br />
if(file.exists())<br />
{<br />
FileInputStream is = null;<br />
try {<br />
is = new FileInputStream(file);<br />
} catch (FileNotFoundException e) {<br />
// TODO Auto-generated catch block<br />
e.printStackTrace();<br />
}<br />
AS3Lexer L = new AS3Lexer(is);<br />
AS3Parser P = new AS3Parser(L);<br />
try {<br />
P.compilationUnit();<br />
} catch (RecognitionException e) {<br />
// TODO Auto-generated catch block<br />
System.out.println(" line=" + e.line + ", column="+ e.column);<br />
System.out.println(e.getMessage());<br />
e.printStackTrace(System.err);<br />
} catch (TokenStreamException e) {<br />
// TODO Auto-generated catch block<br />
System.out.println(" line=" + L.getLine() + ", column="+ L.getColumn());<br />
System.out.println(L.getGuessInfo());<br />
System.out.println(e.getMessage());<br />
e.printStackTrace(System.err);<br />
}<br />
CommonAST.setVerboseStringConversion(false, P.getTokenNames());<br />
CommonAST ast = (CommonAST) P.getAST();<br />
System.out.println("Tree:");<br />
System.out.println(ast.toStringTree());<br />
}<br />
}<br />
}<br />
}</code><br />
Ok, done that I decided to export the grammar file for python (thanks to antrl python export feature).<br />
Everything works fine also for python, but I realized that the python script were so much slower than the java one!<br />
<code lang="python">import sys<br />
import antlr<br />
import AS3Parser<br />
import AS3Lexer<br />
L = AS3Lexer.Lexer(filename);<br />
P = AS3Parser.Parser(L);<br />
P.setFilename(filename)<br />
try:<br />
   P.compilationUnit();<br />
   ast = P.getAST();<br />
except:<br />
   pass<br />
</code><br />
On a 75Kb actionscript file the python script took about 7 seconds to run, while the java application only 2 seconds. I know python interpreter caould be slower than many other languages, but I never thought so much slower.<br />
So I run the python <a href="http://docs.python.org/lib/module-hotshot.html" target="_blank">hotshot profiler</a> to see which could be the bottleneck in the python script and I found most of the problems were due to unuseless antlr (the python module) method&#8217;s calls.<br />
After making corrections to the antlr.py file the same script took exactly half of the time. Now 3 seconds. Wow 🙂<br />
But not fast enough.<br />
So I enabled for the antlr python script <a href="http://psyco.sourceforge.net/" target="_blank">psyco</a> module and this time the same script took <strong>only 1.6 seconds.</strong><br />
Now the python script is fast enough, even if I&#8217;m sure I can make more optimizations in the antlr module&#8230;</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2006/12/01/an-experience-with-antlr-java-and-python/feed/</wfw:commentRss>
			<slash:comments>3</slash:comments>
		
		
			</item>
		<item>
		<title>E4X, ActionScript 3 new &#8220;sublanguage&#8221;</title>
		<link>https://blog.sephiroth.it/2006/07/12/e4x-actionscript-3-new-sublanguage/</link>
					<comments>https://blog.sephiroth.it/2006/07/12/e4x-actionscript-3-new-sublanguage/#comments</comments>
		
		<dc:creator><![CDATA[Alessandro Crugnola]]></dc:creator>
		<pubDate>Wed, 12 Jul 2006 23:56:34 +0000</pubDate>
				<category><![CDATA[actionscript]]></category>
		<category><![CDATA[e4x]]></category>
		<category><![CDATA[ecma]]></category>
		<category><![CDATA[geko]]></category>
		<category><![CDATA[parser]]></category>
		<category><![CDATA[xml]]></category>
		<guid isPermaLink="false">http://blog.sephiroth.it/?p=159</guid>

					<description><![CDATA[actionscript3 e4x<br />
]]></description>
										<content:encoded><![CDATA[<p>Since I&#8217;ve heard the voices about the upcoming <a href="http://www.ecma-international.org/publications/standards/Ecma-357.htm" target="_blank">E4x</a> in actionscript 3 I awaited this moment&#8230;<br />
Finally <a href="http://livedocs.macromedia.com/flex/2/langref/XML.html" target="_blank">ActionScript supports E4X</a> and we can kick away the old XML object (always supported for backward compatibility)! we can stop using all the various xml2object classes created in these years, we finally have a <strong>validating parser</strong>, so we will see finally well formatted XML files from flash users 🙂<br />
But finally we have a powerful &#8220;language&#8221; with a very strong syntax.</p>
<p>I&#8217;ve written a small tutorial with a quick guide  about actionscript3 and e4x basic syntax (without including Namespaces and QNames for the moment)</p>
<p><a href="http://www.sephiroth.it/tutorials/flashPHP/E4X/index.php">http://www.sephiroth.it/tutorials/flashPHP/E4X/index.php </a></p>
<p>P.s. I forgot to say that E4X is also available in the Geko1.8 browser (such as Firefox 1.5). For more info see this article: <a href="http://developer.mozilla.org/en/docs/E4X" target="_blank">http://developer.mozilla.org/en/docs/E4X</a>. Unfortunately E4X ins&#8217;t supported by python, neither there&#8217;s a module for it 🙁</p>
]]></content:encoded>
					
					<wfw:commentRss>https://blog.sephiroth.it/2006/07/12/e4x-actionscript-3-new-sublanguage/feed/</wfw:commentRss>
			<slash:comments>3</slash:comments>
		
		
			</item>
	</channel>
</rss>
