Thursday, October 13, 2011

Why "Polyglot Programming" or "Do It Yourself Programming Languages" or "Language Oriented Programming" sucks?

Last year we saw the launch of a new Web programming language Dart - Structured Web Programming from Google. A very interesting approach to support web application development. Not so long after Go, Groovy, Ruby, Scala, << name your DSL here >>; we see Dart. Is it a good thing to have at least one programming language to solve one problem? The answer is, like we already know, it depends.

Some important backgrounds you should know about the multi programming language paradigm are following:

1. You can read Martin Fowler article about language oriented programming with language workbenches which enables you to write small programming languages easily. In this article I see everyone writing their small languages, everywhere. In this concept we see DSL (Domain Specific Language) as the future of our programming activities. Source: http://martinfowler.com/articles/languageWorkbench.html

2. Neal Ford talked about Polyglot Programming, combining multiple programming languages in application development. Later Mr. Fowler add this paradigm with Polyglot Persistence, using different type of databases within one application. Source: http://memeagora.blogspot.com/2006/12/polyglot-programming.html and http://martinfowler.com/bliki/PolyglotPersistence.html

Since 2006 I already discussed and collected some experiences in multi programming language paradigm:

1. I remember a “hot” discussion in year 2006 with Sebastian Meyen, chief in editor of JavaMagazin Germany, also the biggest organisator of Java Conference JAX. We agreed to see the future of programming in multi language paradigm concept. I also said that all those languages will be based on Java VM. I also told him that in one day SAP will move ABAP as a language which can be run within the Java VM, so just another language within the Java environment, no two different personalities any more. Today we see the beginning of this in the project called Caffeine ABAP. Source: https://cw.sdn.sap.com/cw/groups/caffeine

2. Also in year 2006 I had a project in which we also used different kind of languages and also creating our own DSL:

  • Java for the most implementation stuffs
  • UML for design of the business objects. We generate a lot of things using the concept of MDA (Model Driven Architecture)
  • Groovy for a lot of things, especially for writing unit tests
  • Based on ANTLR we create our own DSL for some aspects of the application
It was really exciting and we had some very good programmers in the project. The result was a very nice and flexible product, just as what we expected in the beginning of the project. Please read this article in German language for more information: http://www.sigs.de/publications/os/2009/02/dewanto_egger_OS_02_09.pdf

So after all those nice things about multi language paradigm I told you, why this sucks at the end? Here are some reasons from my point of view:

1. As typical in application development the problem comes first in the maintenance phase, after all the capable programmers leave the project. Did you, programming language creators ever try to teach a new programming language to a “normal”, 9 till 5 programmers? I’m not talking about 9 (am) till 9 (pm) programmers who love to learn every new languages. It is definitely tough to be proficient in one programming language. This is comparable with the languages we speak everyday. I’m not a native English speaker, so I’m quite sure that I made a lot of syntax and grammar errors in this article. It is possible to be able to speak three or four languages perfectly but this is an exception.

2. Did you ever try to maintain a big Struts web application with AJAX? Just try to add a functionality and you will end up with creating and editing a lot of files: Action and Form files, Struts XML configuration files, JavaScript files with JSON and also HTML or JSP files. Can you imagine to add Groovy, Scala, Dart additionally into that web app? The complexity of such a project is very high.

3. Creating a new programming language means that you also have to build the environment for it. Good IDE, good documentation, good community support, clear roadmap, backward compatibility are some points to be done. Groovy is a bad example of this. In the early version of this language the editor for Eclipse was really bad. After a while they improved the editor but they make a lot of basic changes in the language so your old groovy applications do not work anymore. You are punished to update to the new version. This never happens to Java. You still can compile Java 1.1 applications with Java 6 compiler.

4. Before you are creating your own DSL with e.g. ANTLR ask those language Gurus first, how hard it is to maintain a programming language for a long term. Before you discuss with them don’t ever create your own DSL. Especially if you are working for SME (Small and Medium sized Enterprise). With a small team and small budget you will never ever maintain your own language decently.

So in year 2012, six years after my support to Polyglot Programming, I hope to see following things happen:

1. One language for all aspects in one application is the best concept ever. I name this as “One for All Programming Language paradigm”. Just like we don’t use English for technical language, German as literate language and Indonesian as community language, to be able to communicate internationally with each other we just use English pragmatically for all aspects of our life. In Germany you need to speak German in all aspects to be able to communicate with others. My best solution sofar is Java + XML, that’s it, no more, no less. No mixing with Groovy, Dart, Ruby, Scala, << name your DSL here >> in one application. Especially if you are working as contractor, please don’t try to use all those languages just for a small Java web application. I don’t say that you should not use the other languages at all. The only thing which is important is not to mix those languages in one application. In SME you maybe also want to use just one programming language for all your applications.

2. Concept like GWT (Java to JavaScript compiler) or XMLC (XML compiler which compiles XML, HTML to Java classes) is great. You can work just in plain Java. Guice with all Java and no XML is also a great solution (I know that SpringFramework is also doing this with Annotations). Android is great because it uses Java as its application programming language.

As a conclusion I can only hope to see more such pure and plain Java solutions in year 2012!
OpenID and OAuth: Step2 Protocol

My new article at heise.de about OpenID and OAuth: Step2 Protocol or also known as Hybrid Protocol OpenID and OAuth: OpenID and OAuth: Step2 Protocol.

Enjoy!
Lofi.

Thursday, June 16, 2011

Moving from Ant to Maven: Best Practices with Maven Plugins

I would like to share my experience of moving from "our reinventing wheel" of Ant build scripts to Maven standard.

I reworked some apps which have following characteristics:

(1) Java Version:
- The apps should be compiled with Java 6 version.

(2) Basic technologies:
- Some of the apps are using Model Driven Architecture/Model Driven Software Development things, it means we have some MDA/MDSD generators inside our Ant scripts, like AndroMDA or oAW.
- Almost all the apps use Hibernate, so we need to generate the whole Database scheme to be able to create the whole database automatically.
- Some of the apps use older version of ANTLR. So we have some of ANTLR source codes which have to be compiled.
- Some of the apps use older version of Groovy. So we have some of Groovy source codes which have to be compiled.
- A webapp uses GWT as presentation layer.

(3) Packaging:
- Some of the webapps have somekind of information about the version. For example the webapps have version.html file which should be automatically generated with some information like build timestamp, the person who build the app, the description of the app, etc.
- The apps or modules should be packaged as war or jar. Additionally they also include some additional information like installation documentation, SQL scripts, etc. which have to be package as a zip file separately.
- All of the apps or modules should also package the source codes additionally to the packaged artifacts, so we can debug the apps easily.
- The webapps need to package the class files additionally as a separate jar file, so that this can be added as a dependency into another Maven project or module.

(4) Test:
- The apps have some unit tests in JUnit.
- The apps have some integration test for SpringFramework context and database or service methods test in SpringFramework.

(5) Optimization:
- JavaScript, CSS of some webapps should be optimized using YUICompressor.
- JSP files should be precompiled with JSPC from Weblogic.

(6) Quality check:
- Quality check: the apps should be quality checked using Emma (code coverage) and CheckStyle (quality of the codes).

(7) Deployment:
- The webapps should be automatically deployed to Weblogic.
- All configurations should be automatically uploaded to the server.
- The database scheme should be automatically upgraded and downgraded.

(8) Development support:
- JRebel rebel.xml file should be automatically generated for chosen projects or modules.

Yes, our Ant scripts can do the (almost) whole things above. You can imagine how complex they are? Yes, everything are in the Ant scripts and you need to turn on/off the features you need. Remember how SAP works? Yes, we call this customizing! This is where I see the power of Maven. The plugin concept is just a better concept. Instead of having everything, we only have a small "core" and we add plugins on the top to fullfil our need. In case of the Ant scripts, you have everything (although you don't need all of them) and then you use switcher to turn them on and off.

So how did I solve the requirements above with Maven? Here they are, the plugins we need! In this article I just want to show you the plugins I used:

(1) Java Version:
This is quite simple, just add maven-compiler-plugin (2.3.2).

(2) Basic technologies:
- MDA/MDSD: In this case I have two main technologies: AndroMDA and oAW cartridges. For oAW 4.3.x cartridges we have fornax-oaw-m2-plugin (3.0.1). No problem at all. For AndroMDA 3.2 we also have some Maven plugins. My problem was that we use AndroMDA 3.1. In this version there are only some Maven 1.x available. So in this case I have to use maven-antrun-plugin (1.2). Using maven-antrun-plugin I can execute the AndroMDA task to generate the codes without any problems. I need to use the older version of this plugin because of the dependency to the older Ant version of the AndroMDA 3.1 task. One thing you should not forget: if you are generating codes you need to include the generated codes into your compile classpath. Using build-helper-maven-plugin (1.5) you can achieve this. Great!
- To be able to generate the SQL DDL scripts from Hibernate you just need to add hibernate3-maven-plugin (2.2). Working like a charm!
- Because we are using older version of ANTLR we also have to use maven-antrun-plugin (1.2) to be able to execute the ANTLR compiler.
- The same thing also happens with our older version of Groovy. Just use maven-antrun-plugin (1.2).
- For some GWT projects just use gwt-maven-plugin (2.2.0). This is a great Maven plugin (and the one and only) for GWT. This plugin has a very good documentation, so integrating it is quite easy.

(3) Packaging:
- The first step is to delete some of the target directories (generated directories). For this purpose we have maven-clean-plugin (2.4.1).
- To be able to generate a version.html file I use maven-resources-plugin (2.5). With this plugin you will be able to copy some resources (like version.html), parse the file and exchange all the variables with the content you like. In our case we need to exchange some variables with Maven pom.xml information (${maven.build.timestamp}, ${pom.version}, ${pom.description}, etc.). To be able to read these information you need to use at least the latest Maven 2.2.1. In the future we want to create our own Maven plugin to do this stuff, so that we can encapsulate the function better.
- To package additional information and files we have maven-assembly-plugin (2.1). This plugin is very flexible. So I can capture all the requirements of packaging additional zip files with it.
- To be able to deploy the source codes into Maven repository just use maven-source-plugin (2.1.2).
- In some of the webapps I need to package the "classes" into a separate jar file, so that this file can be added into a Maven dependency by other modules. In this case you can split the projects into two: one project for the "classes" and one project for the "webapp". But I did not want this because it will add the complexity of my projects (two projects instead of one). So I use maven-war-plugin (2.1.1) and add a configuration: attachClasses = true. That's it. You will get a new artifact for the classes with additional classifier: classes.
- In some business modules (Jar files) I need to exclude the model files which are the foundation of MDA/MDSD development style. For this purpose just use maven-jar-plugin (2.3.1) and exclude all the model files you don't want to be packaged in your Jar file. The cool thing is that your maven-source-plugin won't be disturbed by this action. So you will have all those model files still included in your source Jar file.

(4) Test:
- For unit test just use maven-surefire-plugin (2.8.1).
- For integration test just use maven-failsafe-plugin (2.8.1). It is important to separate unit and integration test because the integration test can take sometime. Using both plugins will make your project build more flexible, trust me!

(5) Optimization:
- For CSS, JavaScript optimization with YUICompressor just use yuicompressor-maven-plugin (1.1). It works without any problems. One thing you should aware of: if you are build a webapp you also need to configure maven-war-plugin so that it can copy the result of the optimization into the war file. This is also one thing you maybe want to use Maven profile concept, since you don't want to have optimized files within your development lifecycle.
- Precompiling JSP files with JSPC from Weblogic: this is still an open point. I can use maven-antrun-plugin, but I think that there should be a better way.

(6) Quality check:
- For Emma just use emma-maven-plugin (1.0-alpha-2).
- For CheckStyle just use maven-checkstyle-plugin (2.3). You also need to use surefire-report-maven-plugin (2.8.1) to be able to get a report on CheckStyle.

(7) Deployment:
- Automatic deployment war file with Weblogic can now be done easily with this plugin: weblogic-maven-plugin (10.3.4) from Oracle itself. This is great since this Maven support comes from Oracle. Here is the link to the Oracle documentation of this Maven plugin: Maven plugin from Oracle. To be able to define the server location without changing the pom.xml I use this plugin properties-maven-plugin (1.0-alpha-2). With this plugin you can read any properties file and you will have those properties accessible from your Maven build. Great stuff!
- If you want to be able to make automatic deployment you also need to support upload of configuration files to your server. For this purpose you can use this plugin: wagon-maven-plugin (1.0-beta-3). You can download and upload all kind of files with different kind of protocols (http, scp, ftp, etc.).
- The last important thing is to be able to automatically upgrade or downgrade your database scheme. This is an open point for me. There are some Maven plugins out there, but I had no time to analyse them. See my buzz for the list of them.

(8) Development support:
- To generate rebel.xml you can use jrebel-maven-plugin (1.0.7).

As a conclusion I just can say, Maven and its plugin ecosystem are great. No matter how complex your project you can still handle it easily with Maven plugins. My lessons learned is that DO NOT REINVENTING WHEEL! There are a lot of good Maven plugins out there! In case you cannot find one, just use maven-antrun-plugin. You can almost do everything with it. Just don't forget that you don't want that chaos back to Ant, so use it with precautions.

... and to all other developers who still doing their own build processes... try Maven, you won't regret!

Cheers,
Lofi.

Monday, April 11, 2011

Inside-Out and Outside-In Integration of Webapps: Services and Extensions

In terms of integrating your webapp with other webapps and another way around I see two different integration types:

1. Inside-Out Integration
This is the standard way and almost every webapps available use this possibility. For other webapps to be able to use some functionalities of your webapp you often offer Web services (mostly in RESTful manner) which can be called from outside or other webapps easily.

In this area you offen need following strategies and standards:
  • Webservice protocol: REST for allowing to define the functionalities you want to export to other webapps.
  • OAuth (two and three legged) for allowing other webapps to calls your webapp services, authorization.
  • Sometimes OpenID for allowing other webapps to use the same login, authentification.

2. Outside-In Integration
This is still not commonly used in webapps development. In some desktop apps we know this as a concept of Extensions and/or Plugins (I use these terms exchangeable). This integration lets other webapps run in the context of your webapp.

Mostly this is the second step after you implement the first step above (Inside-Out integration) because you need all the things from the first step. All in all you'll need following strategies and standards:
  • Webservice protocol: REST for allowing to define the functionalities you want to export to other webapps.
  • OAuth (two and three legged) for allowing other webapps to calls your webapp services, authorization.
  • Sometimes OpenID for allowing other webapps to use the same login, authentification.
Additionally you also need following strategies and standards:
  • OpenSocial Gadgets to allowing other webapps integrating their Plugins or Extensions into your webapp. Users of your webapp will be able to install such Plugins or Extensions for their own needs.
  • AppStore or Market Place or WebStore (I use these terms exchangeable) to let your webapp users' easily browse, select, buy, install and review the Plugins or Extensions from a central repository.
  • Sometimes OSGi to let people extends your webapp with some new functionalities. This is however a tight integration of Plugins or Extensions with your webapp which is not easily done and needs more mature concept of your webapp (security, dependencies). Such an integration will not be available per user basis, instead it will be an extension per webapp. The system administrator of your webapp will likely install such Plugins or Extensions.

The main idea of doing these two integration concepts is to build a big ecosystem for your webapp so that a lot of people will use your webapp. By doing these two steps you will be able to open your webapp, not only that other webapps can integrate some functionalities or services of your webapp in theirs but also they will be able to extend the functionalites or services within the context of your webapp.

Let's take a short look of some successful webapps and how they support the two types of integration concepts I mentioned above:

1. Facebook: No doubt, both integrations have been done here. You have Facebook API (e.g.: Like Button) for the first and Facebook Apps (Canvas app) for the second.

2. Twitter: This webapp only supports the first integration as a collection of Twitter API (e.g.: Tweet Button, REST API). They still do not support the concept of Extensions for the second integration type.

3. Google Mail: This webapp supports both integration types. For the first type they offer e.g.: Email Settings API and OAuth Access to IMAP and SMTP API. For the second type you can use Sidebar Gadgets for visual Plugins and Contextual Gadgets for non-visual Plugins. For more information please take a look at this API documentation.


In my opinion, if you want your webapp to be successful, you need to think about both integration concepts directly in the beginning of your development.
Don't create a webapp without a concept of ecosystem anymore!

Next time I'll take a look at how you can plan and implement both integration steps in a simple webapp project. What are the things you need to take care of.

Cheers,
Lofi.