vsi's open source strategy - · pdf filelet’s watch other big project to understand...

VSI'sOpenSourceStrategyPlansandschemesforOpenSourceso9wareonOpenVMS

Bre%Cameron/CamielVanderhoevenApril2016

AGENDA• Programminglanguages•  Integra;ontechnologies• Databases• Web• Libraries/u;li;es• SoDwaredevelopmenttools

• Cloud• UNIXcompa;bility• Analy;cs• Add-ons• Otherconsidera;ons• Summary/conclusions• Ques;ons

3

Programminglanguages•  Scrip;nglanguages

–  Lua

–  Perl(probablyinreasonableshape)

–  Tcl

–  Python

–  Ruby

–  PHP

–  JavaScript(Node.jsandfriends)

–  Alsoneedtoconsidertoolsandpackagescommonlyusedwiththeselanguages

•  Interpretedlanguages–  Scala(JVM)

–  Clojure(JVM)

–  Erlang(poten;allyagoodfitwithOpenVMS;cangetgoodsupportfromESL)

–  Alltheaboveareseeingincreasedadop;on

4

Programminglanguages•  Compiledlanguages

–  Go(seeingrapidadop;on)

–  Rust(rela;velynew)–  AppleSwiD

•  Prerequisites(notallarerequiredinallcases)–  LLVMbackend

–  TweakstoOpenVMSCandC++compilers–  Supportforlatestlanguagestandards(C++)

–  SupportforsomeGNUC/C++extensions–  UpdatestoOpenVMSCRTLandthreadslibrary

5

Programminglanguages

Seeh%p://redmonk.com/sogrady/2015/07/01/language-rankings-6-15/

1.  JavaScript 2.  Java

3.  PHP 4.  Python

5.  C#

6.  C++ 7.  Ruby

8.  CSS 9.  C

10.  Objective-C

11.  Perl 12.  Shell

13.  R 14.  Scala

15.  Go

16.  Haskell 17.  Matlab

18.  Swift 19.  Clojure

20.  Groovy

21.  Visual Basic

6

ProgramminglanguagesGrowingprogramminglanguages,June2015SteveO’Gradypublishedanotheredi;onofhisgreatpopularitystudyonprogramminglanguages:RedMonkProgrammingLanguageRankings:June2015.

Asusual,itisaveryvaluablepiece.Therearemanytake-awayfromthisresearch.IwillnotgooverSteveO’Gradyfindings,butwhatIfoundinteres;ngis:

•  OpenSourceandlicensema%ers.Fortwoofthehotlanguages,ErlangandSwiD,wehaveseenimportantchangesinlicensingthatmayhaveanimpactinthecomingmonths.

o  ErlangchangeditslicensefromitsErlangPublicLicensetoamorewidelyacceptedApacheV2.SteveO’Gradynotesthatitwillnotchangelanguagepopularitybutwillremovefric;onforadop;onandwillmakethelanguagemorea%rac;veforlargecontributors.Itisworthno;ngthatErlangiss;llgrowingandisnowinthetop25,thankstotheamazingprojectsbuildwithit.

o  AppleannouncedthatSwiDwillbeopensourcebyendoftheyear,withaLinuxversioncomingatthesame;me.Thiswasamuchneededchangethatwillexpandthecommunityandacceleratetheadop;onofoneofthefastestgrowingprogramminglanguages.

•  Thereare4boomingprogramminglanguages:Go,SwiD,RustandJulia.GoandRustarecompe;ngforthesametypeofprojectsanddevelopers.Inasense, with an open sourced SwiD, it could reach the same target of system programming, even if it will be difficult to overshadow its mobiledevelopmentroots.Itwillbeinteres;ngtoseehowthosethreelanguagesevolvescompara;velyinthenextresearch.Juliaisascien;ficlanguageandevolvesinitsownnichespace.

•  Mycurrentfavorite,Elixir,isnotprogressingasfastasRust,despitereachingversion1andbeingdevelopedatanincrediblepageunderJoseValim’svision.My feeling is that it is s;ll in developer projet incep;on phase and that it is winning the heart of Erlang developers first, andmany Rubydevelopers.Iexpectittogrowslowlyinthecomingmonths,asPhoenixWebFrameworkmatures.

Theprogramminglanguagespaceiss;llextremelyinteres;ngtoseeevolveandIamlookingforwardseeingwhatthedevelopercommunityisdoingwiththem.Actualprojectsarethelanguagekingmakers.Forexampleejabberdhasbeencri;calforErlangpopularity.Dockerprojectisboos;ngGoadop;on.Let’swatchotherbigprojecttounderstandprogramminglanguagesfuture.

h%ps://blog.process-one.net/growing-programming-languages-june-2015/

7

Programminglanguages

The15mostpopularlanguagesinGitHubsince2012h%ps://www.loggly.com/blog/the-most-popular-programming-languages-in-to-github-since-2012/

AGENDA• Programminglanguages•  IntegraHontechnologies• Databases• Web• Libraries/u;li;es• SoDwaredevelopmenttools


9

IntegraHontechnologies•  Data-levelintegra;on

–  FreeTDS(SQLServer,Sybase)–  UnixODBC–  NotOpenSource,butdon’tforgetaboutpartnerslikeA%unity–  Variousothers

10

IntegraHontechnologies–messagequeuing•  Protocols:

–  AMQP(0.9.1and1.0)

–  MQTT–  STOMP

–  CoAP(maybe)

•  Products:–  Mosqui%obrokerandPahoclient–  RabbitMQ(requiresErlang)andlibRabbitMQ–  PossiblyQpidorOpenAMQ(needsalotofwork)

–  ZeroMQ–  Kama(requiresScala)

–  Ac;veMQ(Java;runswellwithcurrentversionsofJavaandOpenVMS)–  …

11

IntegraHontechnologies•  Webservices

–  gSOAP–  AXIS2–  RESTfulservices(libcURLisusefulhere)–  …

•  API-levelintegra;on–  VariousWSITenhancements(seelater)–  …

•  CIFS–  Notexactlyanintegra;ontechnology,butmightaswells;ckitinhere–  Needsupda;ng

13

Databases•  SQL/rela;onal:

–  PostgreSQL–  MySQLand/orMariaDB

–  Firebird(ex-InterBase)–  ...

•  NoSQL:–  Riak(requiresErlang;poten;allyagoodmatchwithOpenVMSandclustering)

–  CouchDB(requiresErlang)–  MongoDB(needsC++atcurrentstandard)–  Cassandra(Java)

–  Manyotherpossibili;esinthisspace(HBase,…)

•  Caching:–  Redis(certainlyclientAPIs)–  MemcacheD

Inaddi;ontothedatabasesandcachesthemselves,itisalsoimportanttoprovideclientAPI’s

15

Web(clientsandservers)•  Webservers:

–  ApacheHTTPD•  Newversionsareafairlyhighpriority•  Moremodules

–  Mongoose–  Tomcat

•  Newversions(needJava1.7,1.8)–  Nginx

•  HTTPclients:–  cURL,libcURL–  Other

•  AlsoneedtobethinkingaboutHTTP/2

AGENDA• Programminglanguages•  Integra;ontechnologies• Databases• Web• Libraries/uHliHes• SoDwaredevelopmenttools


17

Libraries/uHliHes–buildingblocks•  YAML,JSON,XML,…•  GoogleProtocolBuffers(andseveralsimilarsuchtechnologies)•  OAuth•  OpenSSL

–  Needstobekeptcurrent–  Shipobjectlibrariesaswellasshareableimages(donewithBolton)

•  Gearman•  Others(libraries,API’s,…)

•  libffi(seeh%ps://sourceware.org/libffi/)–  Hasbeenportedbefore–  UsedwiththelikesofPythontointerfacewith3GLcode–  FFI==ForeignFunc;onInterface

AGENDA• Programminglanguages•  Integra;ontechnologies• Databases• Web• Libraries/u;li;es• So9waredevelopmenttools


19

So9waredevelopmenttools•  IDE’s

–  eCube(Eclipse-based)–  Otherop;ons

•  PossiblydomorewithNetBeansandDistributedNetBeans•  SomethinglikeCode::BlocksandUniwinperhaps

•  Sourcecodecontrol–  Git

–  Subversion–  Mercurial–  CVS(old,butsimpletouse,s;llaviableop;on)

•  Tes;ngtools•  Con;nuousintegra;on

–  NXTwareRemoteforJenkinsfromeCube•  OpenSourcepackagemanagement(alonglinesoftoolsprovidedbyCygwinorUbuntu)

•  …

Seemytalk“OpenVMSmeetsGitHub–anoverviewofmodernsourcecodecontrolsystemsforOpenVMS”formoreinforma;onaboutcurrentlyavailablesourcecode

controlop;ons.

21

Cloud•  ClientAPIsandCLIstofacilitateinterac;onwithcloud-basedservices

–  AmazonEC2–  Google–  OpenStack(HPHelion,Rackspace,…)

•  APIsupportforservicessuchas:–  IronMQ–  AmazonSQS,SNS–  Dweet.io–  Xively–  …

•  Containers–  Longer-term(x86)maybe

ItisnotreallyOpenSource,butbeingabletohookintocloud-basedservicesissomethingthatisofconsiderableinterest,andtechnicallyitis

nottoodifficulttodo.


• Cloud• UNIXcompaHbility• Analy;cs• Add-ons• Otherconsidera;ons• Summary/conclusions• Ques;ons

23

UNIXcompaHbility(GNV)•  Needsupda;ngandexpanding

–  Canleveragegoodworkdonebythecommunity–  Needstobeproperlysupported

•  Probablyaseparatediscussion…


• Cloud• UNIXcompa;bility• AnalyHcs• Add-ons• Otherconsidera;ons• Summary/conclusions• Ques;ons

25

AnalyHcs•  R(extensibleprogramminglanguageforsta;s;calcompu;ng)•  …

•  ApacheSpark•  ApacheFlink•  Kama•  …

•  ThelikesofBigDataandInternetofThingsneedtobekeyfocusareas

ManyoftheseJava-basedtechnologiessuchasSpark,Flink,andKaParequirehigherversionsoftheJVM.

h%p://www.dotne;t.co.uk/blog/ar;cle/the-internet-of-things-the-changing-world-of-manufacturing

27

Add-ons(value-adds)•  MoreOpenVMS-friendlyAPIsforsomeOpenSourceproducts

ManyOpenVMSusersdonothavedevelopersthatareabletoincorporateC-basedAPI’sintotheirlegacyapplica;oncode,whichmaybewri%eninlanguagessuchasCOBOL,Fortran,orBASIC.ItisthereforeimportanttoprovidewrapperAPIsthatcanbemorereadilyusedwiththeselanguagesinamoreOpenVMS-likeway.

28

Add-ons(value-adds)•  OpenVMS-specificextensionsforlanguagessuchasPython,Ruby,Lua,PHP,Erlang,…

•  Integra;onwithotherOpenVMS-specifictechnologies–  ACMS

•  Be%eraccesstoaudittraildata•  Mechanism(s)forpassingobjectsgreaterthan64K

–  WSIT•  SupportforprotocolsotherthanICCo  AMQPo  ...

•  SupportforotherlanguagessuchasC#.NET(requiresaddi;onalprotocolsupport)–  RTR(maybe)–  UAF-basedauthen;ca;on(Mosqui%o,RabbitMQ,…)–  …

•  Monitoringasaservice


• Cloud• UNIXcompa;bility• Analy;cs• Add-ons• OtherconsideraHons• Summary/conclusions• Ques;ons

30

OtherconsideraHons•  Priori;za;on•  Valuetocustomers(usefulness)•  Resources•  Prerequisites(updatestotheCRTL,C/C++compilers,…)•  Communityinvolvement

–  WhichOpenSourcepackagesdoVSIport,maintain,andsupport;whichpackagesareleDtothecommunity;collabora;on?

•  Inputfromcommunityandcustomers•  IPconsidera;ons•  OpenSourcelicensingconsidera;ons•  Support•  Consul;ngservicesandtraining•  Documenta;on•  ...

31

ValueconsideraHonsWhatOpenSourcesoDwareislikelytobeofgreatestusetocurrentOpenVMSusers?•  Shortterm•  Mediumterm•  Longterm

Needtotakeastrategicapproachandconsiderindustrytrends•  InternetofThings•  BigData•  Containers•  Cloud•  …•  WheredoesOpenVMSprovideadvantages?

ManyOpenVMSusershaveveryoldapplica;onenvironments•  NeedtoprovidesoDware(andpoten;allyservices)thatwillhelptheseusers

32

ValueconsideraHonsWhatOpenSourcesoDwarewillhelptosolidifyOpenVMS'posi;onwithexis;ngusers?•  Integra;ontechnologies

–  Someuserswithlegacyapplica;onshaveveryli%lepercep;onofwhatcanbedonefromanintegra;onperspec;ve

•  Technologiesthatpresentopportuni;estoreduce3rdpartylicenseand/orsupportcosts–  OpenSourcereplacementsforexpensive,poorlysupported,orunsupportedsoDware–  …

•  UsersonAlphalookingtomovetoIntegritywhoneedreplacementop;onsfortechnologiesnotavailableonIntegrity

33

ValueconsideraHonsWhatOpenSourcesoDwareislikelytoa%ractdeveloperstoOpenVMS?•  ModernlanguageenvironmentssuchasRuby,Python,Go,Erlang,Rust,Node.js,Scala,Clojure,latestJava

versions,…Moderntoolsets•  IDE’sandrelateddevelopmenttools•  Sourcecodemanagement•  Tes;ngtools•  …WhatOpenSourcesoDwareislikelytoencouragedeveloperstoportapplica;onstoOpenVMS?•  Aspertheabove•  Theop;onofagoodUNIXshellandrelatedu;li;es(enhancementstoGNV)

34

DependenciesandrelatedmaWers•  ManyOpenSourceproductsdependonotherOpenSourceproducts

–  Libraries/API’s–  TheseareoDenfundamentalbuildingblocks

•  CRTLissues–  Missingfunc;ons–  Differencesinheaderfiles–  Behaviouraldifferences

•  C/C++languagestandards•  …

Butdon’tforget,OpenVMSisOpenVMS;itisnotLinux,andtherewillalwaysbedifferencesthat

needtobecontendedwith,justastherearewhenpor;ngOpenSourcecodetoWindows,oreven

betweenLinux/UNIXvariants.

35

DependenciesandrelatedmaWers•  Someposi;oningforthenextslide…

–  Illustratessome(butnotall)dependenciesandissuesassociatedwithbuildingErlangonOpenVMS

–  Blueboxesaredependencies–  Redboxesareproblema;calbutop;onalaspects–  Blackboxesarejusttheusualsortsofthingsthathavetobedealtwithwhenpor;ngcomplexOpenSourceapplica;ontoOpenVMS

36

PCRE(Perlregularexpressionslibrary).EasilyportedtoOpenVMS;

nocodechangesrequired.

zlibcompressionlibrary.EasilyportedtoOpenVMS(hasbeen

donenumerous;mes)

libreadlineforcommandlineedi;ngandhistory(op;onal).PortedtoOpenVMSaspartoftheGNVproject;

func;onalityeasilyreplicatedusingSMGrou;nes.

OpenSSLlibraries.HPSSLforOpenVMSisatahighenoughversion.Otherwisecanuseh%p://www.polarhome.com/openssl/.

UseofsyslogAPI.Poten;allyneedtoimplementsomethingcomparable,

butcanlivewithoutit.

fork()func;onnotavailableonOpenVMS.Onlyusedinafewplaces,andcodecanbecompiledtonotuseit.Cangetawaywithusingvfork()

inmostcases.

poll()andselect()func;onsonlyworkwithsocketsonOpenVMS.Itisthereforenecessarytoimplementcustomversionsofpoll()andselect()thatworkwithothertypesofdescriptor.Thesecustomversionsareunlikelytobe

par;cularlyefficient(scopeforimprovement).Theselect()func;onalsobehavesdifferentlyonOpenVMS

underothercertaincircumstances(likewhentimeoutissetto0).

fcntl() CRTLfunc;onbehavesdifferentlyonOpenVMSincertainsitua;ons.Cannotbeusedtotogglesocketsblocking/non-blocking;

necessarytouseioctl()instead.Someotherplaxormsbehavethiswayalso,soErlangcodeisawareofthisandjustneedtoensurecertain

macroscorrectlydefinedwhencompiling.

SometerminalI/Ocoderequiresuseofappropriateioctl()callsinplaceof

tcsetattr().

Threads(Erlangscheduler).ThedefaultthreadstacksizeonOpenVMSistoosmall.Thisisalsoanissueonotherplaxorms,andcodetriestoaccommodatesuchma%ers(needtoensurethatcertainmacrosarecorrectlydefined).TheOpenVMS

POSIXthreadslibraryismissingsomefunc;ons(pthread_sigmask() forexample);howeverthisis

generallynotamajorproblemtocontendwith.

Behaviourofgetcwd().OpenVMSextensioncontrolshowtheresultisreturned(UNIX-styleorOpenVMS-

style).

Defaultbehaviourofpipe()isnottotallyconsistentwithLinux.Canneedtospecifyop;onalargumenttosetnon-blockingandusedecc$set_child_standard_streams()toensurethat

thingsaresetupcorrectlyforcommunica;onbetweenparentandchildprocesses.

Sundryheaderfiledifferencesandheaderfilesthatdon’texistonOpenVMS(<sys/param.h>would

beoneexample)

Na;vecompila;onofErlangcode(HiPE).CurrentlynosupportfortheIA64processor;needtoconsiderwhetherimplemen;ngsupportforIA64makessense(givenfuture

plansaroundx86)

Assortedotherop;onalcomponentsthatdependonotherOpenSourcepackages(unixODBC,

wxWidgets,…)

Essen;altorunwithUNIX-stylepathandfilenamingotherwiseextensivecodechangeswouldberequired.

NecessitatesODS-5.

CertainOpenVMSCRTLfunc;onswillseterrnoto EWOULDBLOCKinsteadofEINTR.Otherplaxorms

(suchasWindows)behaveinasimilarmanner,andtheErlangcodehas(inmostplaces)takenthisma%erinto

considera;on.

64-bitversionsofsomefunc;onsmissingfrom

OpenVMSCRTL.

Erlang/OTPonOpenVMS

38

Summary•  There’salottodo!

–  Newstuffaswellasupdatestoexis;ngpackages•  Languages,databases,andintegra;ontechnologiesarearguablysomeofthebig-;cketitems

–  Bothforexis;ngusersandtoa%ractnewusers–  Developmenttoolscanbeaddedtothislist

•  Sourcecodecontrol•  Tes;ng•  …

–  Packagemanagementisanotherimportantconsidera;on•  TherearesomesignificantdependenciesonJava•  Needtotakeamul;-prongedapproach,involvingsome“quickwins”(smallbutstrategicitems)andlargerpiecesofworkbeingdoneinparallel

•  Iden;fy,priori;se,andsystema;callyaddressCRTLandcompiler-relatedissues•  Modelforcommunityandpartnerengagementneedstobedefined•  …


• Cloud• UNIXcompa;bility• Analy;cs• Add-ons• Otherconsidera;ons• Summary/conclusions• QuesHons

QuesHons

vsi's open source strategy - · pdf filelet’s watch other big project to understand...

Documents