Thoughts on Scripting Languages vs. Real Programming Languages

I spent several years working with VBScript in TestComplete, Python in Robot Framework, and couple years using a home grown proprietary framework based on TestNG, and now doing test framework design/implementation using IBM's Rational Functional Tester. I'd like to blog my thoughts and observations of the differences moving between Scripting Languages and Real Programming Languages. There are many more advantages / disadvantages out there, this is my general observation working between the 2 realms.

This probably wouldn't influence your decision by much when picking a automation tool, as you probably should pick one that recognizes your AUT's object the best.

Scripting

Advantages :

* Easier to learn - Scripting languages tend to be easier to learn for non-programmers. Given they are far more forgiving about object types, it's generally easier for non-programmers to start writing useful scripting code fast. For example, in scripting, you generally do not have to worry about converting between numbers and strings, which is a common thing within automated tests. You may want to test inpputing 2 numbers and verifying the total, but when reading the value in a resulting text field, you'll get the result number will be in a String form. In scripting, the conversion is automatic, where programming languages there tends to be the need of utility parsing methods to do the conversion.

* Generally more example test code posted on the Web - Most of the popular testing tool are based on Scripting languages. You can generally find more fully written test examples in Scripting languages than you will with tools based on real programming languages. QTP being one of the most proficient tools in the market, it'll be easier to adapt QTP examples to VBScript based tools. Programming code examples tend to be outside of the realm of testing, so some adaptation of code needs to happen.

* Easy to test code using interactive real time interpreters - Many scripting languages are simple top down interpreted. This allows many scripting languages such as Ruby and Python to have interactive interpreter. This makes debugging / modifying the code on the spot much easier as you can say test different variations in a line of code without the compile/rebuild cycle.

Disadvantages :

* Lack of compile time checking - A lot of times, you won't find out you did something wrong until you actually run the script. This occurs a lot for simple things like name collisions, namespace conflicts, and passing in the wrong type for a given method. These can all be easily check as you type.

* Lack of fixed types - This leads to situations where 2 wrongs make a right that goes unnoticed. Imagine the following situation, you want to verify after an operation produces a result that should not be equal to a given value. But say an error in the script causes the object you're comparing it against give you a null value. In this case, this test might pass as null != your value.

* Exception handling usually more primitive - Since exceptions are generally all the same in most scripting languages, you tend to have branching logic that has to examine to error codes, messages, etc... then try to set things to the right state. The most painful example of this is VBScript. Exception handling is a set of painful On Error Resume Next, then right after each possibly offending line, you'll need to add an if statement that checks the Err.Number, then Err.Clear the error before the next line. Although many more modern scripting languages have the concept of try/catch blocks. The lack of strongly typed exception and being able to extend those types to convey and pass data around beyond a simple error code and message makes it more annoying to write proper error handling routines, especially at a framework level.

*Projects are generally less self contained - The idea of 3rd party library is more annoying in scripting languages. 3rd party libraries usually have to be installed at some expected location which is out side of a project. When creating self contained projects where someone can just checkout a project from source control and run tests immediately, there usually needs to be some sort of setup script that ships with the project to copy all the necessary 3rd party libs to their destinations.

Real Programming Languages

Advantages:

*Compile time checking of syntax and object compatibility errors - Since modern programming environments will do real time type checking as you type, it's easier to avoid silly mismatching mistakes such as mismatching types or using the wrong variable when there are 2 similarly named variables before the program compiles. It'll also give you better feedback to what's wrong.

*Exception and error handling is more robust - This is especially helpful when debugging. When printing out a stack trace, you can narrow down the search for the offending code. Also when handling expected exceptions, you can handle certain types and throw others back up the stack for the parent calling code to handle.

*Projects are generally more self contained - Real programming languages tend to have this strong concept of stand alone modules and APIs. When needing to use 3rd party libraries, it's usually easier as you can just include the library in the project, or use something like maven or ant to manage the dependency. The result is one can generally grab a project from source control and be up and running without complicated test tool setup.

Disadvantages:

* Fewer test tools use compiled languages, so test code examples are generally harder to find. You'll find yourself having to translate scripting code into programming code and have fewer cut and paste examples.