A brand new way of Perl product, and it’s future

Masaaki Goshima (@goccy54)mixi Inc.

Me

• Name : Masaaki Goshima (@goccy54)• Job : mixi (Tanpopo Group)– Development for Developers– Developing a platform for refining legacy

software– Managing Jenkins

• Personally, interested in Perl and developing yet another Perl, gperl– (YAPC::Asia 2012) Perl と出会い Perl を作る

gperl

• Fastest perl-like language– Aiming perl5 compatible syntax–Written with C++

• The last commit for repository was 11 months ago...– “Is this already departed !?”

NO! ITS STILL ALIVE!!!

• However, gperl itself has departed• Modules(lexical analyzer, parser,

code generator, interpreter) are useful– Lexical analyzer: implementing perl5

syntax highlighter– Parser: static analysis tool (used as

refinement of legacy perl5 code)

Input -> Lexer -> Parser -> CodeGenerator -> VM (JIT) -> output

This presentation’s Goal

• Today, we’re going to present “Spinout projects” for each modules

gperl

Compiler::Lexer Compiler::Parser Compiler::CodeGenerator::LLVM

LexerParser CodeGenerator

???::???::??? ?????Next Module

Agenda

• Section 1– Introduction of Compiler::* Modules

• Section 2 – (Application1) Running on multi

platforms

• Section 3 – (Application2) Static analysis tool

Section1

• Introduction of Compiler::* Modules1. Compiler::Lexer – Lexical Analyzer for Perl5

2. Compiler::Parser – Create Abstract Syntax Tree for Perl5

3. Compiler::CodeGenerator::LLVM– Create LLVM IR for Perl5

Compiler::Lexer

• Lexical Analyzer for Perl5• Features

– Simple (return Array Reference of Token Object)– Fast (faster 10 ~ 100 times than PPI::Tokenize)

• Wirtten with C++• Perfect hashing for reserved keywords• memory pool for token objects

– Readable Code• Nothing use parser generator like yacc/bison

Example1: my $v = sprintf(<<'TEMPLATE', 'yapc');2: %s::Asia3: TEMPLATE4: $v =~ s#yapc#YAPC#;

%s::Asia : HereDocumentTEMPLATE : HereDocumentEnd$v : Var=~ : RegOKs : RegReplace# : RegDelimyapc : RegReplaceFrom# : RegMiddleDelimYAPC : RegReplaceTo# : RegDelim; : SemiColon

my : VarDecl$v : LocalVar= : Assignsprintf : BuiltinFunc( : LeftParenthesis<< : LeftShiftTEMPLATE : HereDocumentRawTag, : Commayapc : RawString) : RightParenthesis; : SemiColon

tokenize

Tips

• Cannot tokenize pattern1. func * v • 「 * 」 is Glob or Mul

2. func / ..• 「 / 」 is RegexDelimiter or Div

3. func <<FLAG• 「 FLAG 」 is HereDocumentTag or Constant

It may make a mistake

Future plan

• Supporting recursively tokenizing–More wide range of parsing application– Recursive parsing will take time,

optimizing will be next future work

func * v => func(*v) or func() * vfunc / … => func(/../) or func() / vfunc <<FLAG => func(<<FLAG) or func() << FLAG

sub func($) {} or sub func() {}

Compiler::Parser

• Create Abstract Syntax Tree for Perl5• Features– Fast (faster than PPI)– Readable Code• Simple design• Nothing use generator

Can generate Virtual Machine code as walking tree by post order

->

$a->{b}->[0]->c(@args)

->

->

$a {}

b

[]

c

0

@args

left

left

left

right

argsright

right

data

data

Example

my $v = sin $v + $v * $v / $v - $v && $v

my ($v = (sin((($v + (($v * $v) / $v)) - $v)) && $v))

=

$v &&

left right

rightleft

The most difficult part of Perl5 parser:

hidden(optional) parenthesis

Example2) Can parse BlackPerl

BEFOREHAND: close door, each window & exit;wait until time;open spell book; study;read (spell, $scan, select); tell us;write it, print(the hex) while each watches, reverse length, write again;kill spiders, pop them, chop,split, kill them. unlink arms, shift,wait and listen (listening, wait).sort the flock (then,warn "the goats", kill "the sheep");kill them, dump qualms,shift moralities, values aside, each one;die sheep; die (to, reverse the => systemyou accept (reject, respect));next step, kill next sacrifice,each sacrifice, wait, redo ritualuntil "all the spirits are pleased";do it ("as they say").do it(*everyone***must***participate***in***forbidden**s*e*x*).return last victim; package body;

exit crypt(time, times & “half a time”)& close it.select (quickly) andwarn next victim;AFTERWARDS: tell nobody.wait, wait until time;wait until next year, next decade;sleep, sleep, die yourself,die @last

BlackPerl for Perl5

Future plan

• Replace PPI !!!– Supporting some expressions• ThreeTermOperator, Glob, given/when, goto

etc..

– Supporting compatible methods PPI provides• PPI::Document::find• PPI::Document::prune

Compiler::CodeGenerator::LLVM

• Create LLVM IR for Perl5

->

$a->{b}->[0]->c(@args)

->

->

$a {}

b

[]

c

0

@args

left

left

left

right

argsright

right

data

data1 2

3

4 5

6

7 8

9

10Compiler::Parser Compiler::CodeGenerator::LLVM

LLVM IR

Running with JIT

Native Code

Other Language

LLVM(Low Level Virtual Machine)

• Compiler Infrastructure• Better than GNU GCC

LLVM

X86

ARM

Power

C/C++/Objective-C

JavaScriptpy2llvm

MacRuby

clang

?? ?

How to use

• Dependency: clang/llvm version 3.2 or 3.3

use Compiler::Lexer;use Compiler::Parser;use Compiler::CodeGenerator::LLVM;

my $tokens = Compiler::Lexer->new('')->tokenize($code);my $ast = Compiler::Parser->new->parse($tokens);my $generator = Compiler::CodeGenerator::LLVM->new();my $llvm_ir = $generator->generate($ast); # generate LLVM IR$generator->debug_run($ast); # run with JIT

Benchmark(Fibonacci)

Languagetime(real

)[sec] X

gperl 0.10 X81.2

LuaJIT(2.0.0-beta10) 0.12 X67.6

v8(0.8.0) 0.18 X45.1

Compiler::CodeGenerator::LLVM 0.72 X11.2

Perl(5.16.0) 8.12 X1.0

Environment

MacOSX(10.8.4)2.2GHz Intel Core

i7Memory: 8GB

N=35

Section 2

• Application1)– Running on multi platforms

!?

1. Running on Web Browser

• Recently, way of writing code that runs on web browser is not only JavaScript– e.g.) CoffeScript, JSX, TypeScript, Dart

• I wrote module for Perl5!

SEE ALSO : perl.js (@gfx)

Compiler::Tools::Transpiler

• Translate Perl5 codes to JavaScript codes– (Step1) translate Perl5 codes to LLVM-IR with

Compiler::* modules– (Step2) translate LLVM-IR to JavaScript codes

with emscripten

Com

pile

r::Le

xer

Com

pile

r::Pa

rser

Com

pile

r::Co

deGe

nera

tor::

LLVM

Perl5 LLVM emscripten

How to use

use Compiler::Tools::Transpiler;

my $transpiler = Compiler::Tools::Transpiler->new({ engine => 'JavaScript’, library_path => [‘lib’]});

open my $fh, '<', 'target_application.pl';my $perl5_code = do { local $/; <$fh> };my $javascript_code = $transpiler->transpile($perl5_code);

open $fh, '>', 'target_application.js';print $fh $javascript_code;close $fh;

※ Needs clang-3.2 and LLVM-3.2 (Not ver. 3.3 or later) because emscripten has still not support LLVM 3.3 or later

ROADMAP/Repository

• Fix some emscripten’s bugs• Supporting accessors for HTML objects• Supporting Canvas API• etc..

• Repository– https://github.com/goccy/p5-Compiler-Tools-Transpiler.git

2. Running on iOS and OSX

• Recently, software that can develop iOS or OSX applications using light weight language has being released– RubyMotion– mocl– Titanium

• I wrote module for Perl5!

PerlMotioniOS and OS X development

using the Perl5 programming language

PerlMotion’s Architecture

UIKit

Cocoa Touch FrameworksCore Animation Core Audio Core Data

Perl5 x Cocoa Touch FrameworksBindings Library

LLVM IR(Links Perl5 codes and Cocoa Touch Frameworks)

Perl5 Codes

Compiler::Lexer

Compiler::Parser

Compiler::CodeGenerator::LLVM

iOS Simulator iOS DeviceMacOSX

Generates Native Code for each target Architecture

DEMO

• HelloWorld by PerlMotion

Current Status

Still not support functions

Symbol Management System : GlobGarbage Collection

RegexpThree Term Operator

etc..

Not support functionsDynamic evaluation like eval,

s/../../e, require

Supported functions

Primitive Types : Int, double, String, Array, Hash, ArrayRefence,

HashReference,IOHandler, BlessedObject …

Operators : binary operator, single term operator …

BuiltinFunction : print, say, shift, push, sin, open …

Variable Definition, Function Definition

OOP System : package, bless, @ISA

ROADMAP

• Supporting deployment on device• Preparing debugging environment– (stdout/stderr/gdb..etc)

• Supporting existing framework (e.g. UIKit)• Supporting CPAN modules written by Pure

Perl

Support CPAN modules

Support some Frameworks

Support deployment on device

Support debugging environment

2014/4

Will release at April, 2014!

Welcome your Contribution!

• I want discuss design or objective– Especially, I need advice on “What Perl

community likes?” (Naming rule, etc…)

https://github.com/goccy/perl-motion.git

Conclusion at Section 1 and 2

• Perl5 codes Run Anywhere!!!– iOS,OSX,Web Browser and others

• Welcome your Contribution– Compiler::Lexer

• https://github.com/goccy/p5-Compiler-Lexer.git

– Compiler::Parser• https://github.com/goccy/p5-Compiler-Parser.git

– Compiler::CodeGenerator::LLVM• https://github.com/goccy/p5-Compiler-CodeGenerator.git

– Compiler::Tools::Transpiler• https://github.com/goccy/p5-Compiler-Tools-Transpiler.git

– PerlMotion• https://github.com/goccy/perl-motion.git

• Application2)– Static analysis tool

Section3

Compiler::Tools::CopyPasteDetector

• Detect Copy and Paste for Perl5• Features– Fast (detecting Engine written by C++

with POSIX Thread)– Accuracy (based on Compiler::Lexer

and B::Deparse)–High functionality Visualizer

(some metrics or scattergram etc…)

DEMO

• Detect Copy & Paste of Catalyst and Mojolicious

Another Modules

• Perl::MinimumVersion::Fast(@tokuhirom)– Find a minimum required version of perl for

Perl code

• Test::LocalFunctions::Fast(@papix)– Detect unused local function

Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI

Future plan

• Perl::Metrics::Simple::Fast–Will release faster module than

Perl::Metrics::Simple by using Compiler::Lexer and Compiler::Parser

• Next future, I will CPANize of static alnalysis module that has been used at mixi– It includes rich Visualizer

Conclusion at Section 3

• Introduction of copy and paste detecting tool for Perl5– Compiler::Tools::CopyPasteDetector

• Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI

Finally

Compiler::Lexer Compiler::Parser Compiler::CodeGenerator::LLVM

gperl will reborn

Type Inference Engine

Static Typing

- Coming soon -gperl will assemble these brand new modules...