Introducing IronPython

IronPython is easy to learn yet surprisingly powerful language for .NET development.

In this article, I’ll introduce you to IronPython and demonstrate it differs from C# and Visual Basic while still allowing you to leverage your existing .NET knowledge.

A Little History

Back before version 1.0 of the CLR shipped, Microsoft engaged a variety of commercial and academic organizations to produce languages that ran on .NET, an effort that was code-named “Project 7”. One of those languages was Python for .NET, which was developed by ActiveState. While ActiveState’s Python for .NET implementation worked, they found “[t]he speed of the current system is so low as to render the current implementation useless for anything beyond demonstration purposes.”1 Furthermore, while they blamed some of the performance problems on “the simple implementation of the Python for .NET compiler”, they also claimed that “[s]ome of the blame for this slow performance lies in the domain of .NET internals and Reflection::Emit”.

Largely due to ActiveState’s experience, it became conventional wisdom that “[t]he CLI is, by design, not friendly to dynamic languages.”2 This caught the eye of Jim Huginin, the original creator of Jython-an implementation of Python for the Java VM. Given that Jython runs reasonably well on the JVM, Jim wondered why CLR ran Python so badly. He decided to take a couple of weeks and build a simple implementation of Python on .NET in order to determine what Microsoft had done wrong and write a short paper titled “Why .NET is a Terrible Platform for Dynamic Languages”.

Jim was shocked to discover that his two-weeks of effort resulted in a Python implementation that ran faster than Jython. He kept working on IronPython, eventually joining Microsoft in the summer of 2004. The 1.0 version of IronPython shipped two years later, in September 2006. Today, Jim is an architect in the Dynamic Language Runtime team while the IronPython team is driving towards a fall 2008 release of IronPython v2.0.

Getting IronPython

The IronPython project lives on CodePlex at http://www.codeplex.com/ironpython. Unlike most other Microsoft language products, IronPython runs as a transparent, open source project. The IronPython team releases a new version approximately every six weeks. Furthermore, Microsoft makes all the source code available under the OSI approved Microsoft Public License. On the CoDePlex site you’ll find a mailing list for community discussion as well as a public issue tracker where anyone can submit bugs they find.

The current released version of IronPython is v1.1.1, however, the IronPython team is hard at work on v2.0. As I write this, we have just released Beta 3 of IronPython v2.0. IronPython v2.0 implements the 2.5 version of the Python language (IronPython v1.x implemented Python v2.4). Furthermore, IronPython v2.0 is built on the new Dynamic Language Runtime, which I’ll discuss further later in the article.

Regardless of which release you download, IronPython releases are distributed as a simple zip file with the compiled binaries, as well as some tutorials, documentation, and the usual readmes and license files. You can simply unzip that archive wherever you want to on your hard drive. Inside the unzipped files you’ll find the main IronPython executable, ipy.exe. If you run that with no arguments, you get an interactive console window where you can start typing Python code directly. Alternatively, you can pass the name of a python file as the first argument, and IronPython will execute that code.

Currently, there isn’t a production quality development experience for IronPython in Visual Studio. The VS Extensibility SDK includes a sample IronPython language service, project system and console window. This sample has been packaged up as IronPython Studio and is available as a free download from CodePlex (http:// www.codeplex.com/IronPythonStudio). Improving the Visual Studio experience for the IronPython developer is one of the areas of investment for the IronPython development team going forward.

Significant Whitespace

Probably the most notable difference when you first start working with Python is the use of significant whitespace. In C#, whitespace is entirely insignificant - statements end with a semicolon and scope blocks are delineated with curly braces. VB, on the other hand, uses the end of line to indicate the end of a statement. However, scope blocks are still explicitly terminated in VB, typically with the word “End” (i.e. End Sub, End Function, End Class) though sometimes with a word more relevant to the block being closed (Do/Loop, For/Next, etc).

In contrast, Python uses significant whitespace both for statement termination as well as delineating scope blocks. Like VB, statements are terminated by the end of the line. The start of a scope block is explicitly marked with a colon in Python, similar to how C# uses the open curly brace. However, unlike VB or C#, Python uses indentation to implicitly determine the end of a code block. All code with the same level of indentation is part of the same code block. To demonstrate, here’s an implementation of the bubble sort algorithm written in Python:

def bubble_sort(ar):
    l = len(ar) - 1
  
    for i in range(l):
        for j in range(l-i):
            if ar[j] > ar[j+1]:
                temp = ar[j]
                ar[j] = ar[j+1]
                ar[j+1] = temp
        
    return len(ar)

This code has four scope blocks: the bubble_sort function itself, the two for loops (Python’s for loop is the equivalent of C#’s foreach loop and the range function is the equivalent of Enumerable.Range), and the if block. Each of those blocks is at a different level of indentation. You can see that the return statement is at the same level of indentation that the first for loop is at, indicating that it’s not part of either the for or if scope blocks, but it is a part of the function scope block. In this example, I’m using four spaces for indentation, but the specific amount of indentation is irrelevant, as long as it’s consistent.

Significant whitespace enables both consistency and readability. In C-syntax languages like C#, everyone has their own opinion as to where braces go, tabs vs. spaces, when to indent and how much, etc. Since C# doesn’t care about whitespace, there are numerous ways to format your code-but a given person typically has their single preference. In Python, there’s only one legal way to format the code. This means that if you walk up to a random bit of Python code written by some person you’ve never met, it’ll be formatted exactly in the same style the Python code you write is. This typically makes Python code more readable. Furthermore, readability is also improved because you don’t need extraneous syntax (semi colons and curly braces) to indicate the ends of statements and scope blocks.

Python’s significant whitespace is one of those things developers tend to either love or hate. However, the use of significant whitespace does appear to becoming more popular. Both Boo and Cobra, two open source .NET languages, use a Python-inspired syntax, including significant whitespace. Furthermore, Microsoft’s new F# language includes an optional lightweight syntax mode that makes indentation significant, just like Python.

Dynamic Typing

While significant whitespace is the most obvious difference between Python and C#/VB, the biggest difference between them lies in the type system. Python shares many high-level concepts with these .NET languages, such as imperative code, functions, classes and objects. However, C# and VB are statically typed, which means that the capabilities of individual types are fixed at compile time. Once you decide on the fields, properties and methods of your type and you run the compiler, those capabilities can never change, ever. If you need to change them, your only option is to throw away your original compiled binary and recompile a new one.

Dynamic typing, on the other hand, does not fix type capabilities at compile time. Rather, types become completely mutable and can be manipulated at run time in much the same way that type instances are. Creating new types, adding fields or methods to a type, removing fields or methods from a type, adding fields or methods to a specific instance of a type, even changing the inheritance hierarchy of a type are all allowed at run-time in Python.

If you come from a static type language background, the idea of being able to manipulate types at run-time may seem strange, or even dangerous. Static advocates typically highlight type safety as one of the primary benefits of using static typing. When all the capabilities of a type are fixed at compile time, the compiler can do quite a bit of validation to ensure that types you create and methods you call all exist, fields and parameters are of the right types, etc. We’ve all had to fix compiler errors because we misspelled a method or type name. In Python, those compiler errors become run-time exceptions.

However, I’d like to put this type safety benefit in perspective. In any system of significant scale, there are essentially an infinite number of ways the application can be wrong. Type safe languages eliminate one of this infinite number of ways. The overwhelming majority of errors and bugs you might make when developing your system won’t be caught by a static type language compiler. If they could, then any application that compiled would automatically be bug free! You need some other mechanism to catch these other errors, typically automated unit tests. So while it’s true that static types do provide a safety net, it’s just not a very big one.

This isn’t to say type safe languages are bad. Rather, I look at it as a tradeoff of static type safety vs. dynamic type flexibility. There is no universally right decision-only the right decision for you and your project. The good news is that with the addition of dynamic languages like IronPython to Microsoft’s language stable, you can make that tradeoff decision while still staying in the .NET realm.

Refreshing Simplicity

One of the somewhat frustrating aspects of C# is the amount of structure that gets forced on the developer. Some of the structure is related to static typing, such as specifying variable types and casting operators. However, there are also no stand-alone functions in C#, much less just a bunch of code in a simple script. All functions have to be attached to an object, meaning the simplest implementation of “hello, world!” in C# looks like this:

class Program
{
  static void Main()
  {
      System.Console.WriteLine("Hello, World");
  }
}

Contrast that with the Python version:

print "Hello, World”

Obviously, you don’t judge a language completely by its implementation of “hello, world”. But the point is that Python scales well with complexity. If you just have a simple script, you don’t have to add unneeded class and function constructs for the code to live inside. If you want to make a function, it can stand alone; you don’t need to find some class to attach it to (like the bubble_sort example above). If you really need classes and objects, you can build them, but you don’t have to take on that complexity unless you really need it.

Polymorphic Mallards

Polymorphism-the ability to treat different types the same way-is a staple of object-oriented programming. Statically typed languages have a variety of mechanisms designed to enable polymorphism, including inheritance, interfaces and generics. Since Python is dynamically typed, it doesn’t need any of these mechanisms to provide polymorphism. (Python does support inheritance, but not to enable polymorphism.) Type compatibility is determined at run time based on the type’s capabilities rather than on the declared types. This is commonly known as “duck typing”. This enables code to be more flexible and reusable in a wider variety of situations.

To see this in action, let’s look back at the bubble sort function I wrote before. It takes a single parameter “ar” which represents the collection being sorted. Looking at the code, there are three capabilities that the object instance “ar” must have in order to be compatible with the bubble sort function:

You must be able to ask the ar object for its length.
You must be able to get items in the ar object by numeric index.
You must be able to set items in the ar object by numeric index.

If I had written the bubble sort in C#, I would have had to declare the ar parameter as a specific type-either an interface or a base class-that implements the above capabilities. The C# compiler would validate that any code that called bubble sort would pass a parameter that implements that parameter type. Passing any other type as a parameter would result in a compile error, even if the type in question implemented all the capabilities above.

Python, as I discussed above, does all the type checking at run time. Python will throw a TypeError when it executes the bubble sort code if the type instance passed in as the ar parameter is missing an implementation of one of the three methods listed above. In the case of the bubble sort code, those three methods have special names. Python defines quite a few special names for a variety of circumstances. __len__ returns the length of the collection, __getitem__ and __setitem__ get and set values by index, respectively.

Listing 1 shows an example of a custom linked list class that implements these methods and is thus compatible with the bubble sort code above. Of course, linked lists don’t usually provide indexed access to their members. But the point is that because this custom class that I whipped up implements __len__, __getitem__ and __setitem__ I can pass it to the bubble_sort function I wrote earlier without any changes at all.

.NET Interop

One of the challenges in designing IronPython has been satisfying the two separate audiences it’s intended to serve. On the one hand, it needs to work as much like the standard C-based Python implementation as possible. On the other, it has to have high fidelity interop with the .NET Framework. Sometimes, the needs of those two priorities clash. For example, what’s should be the result of this code?

s = 'hello, world!'
s.ToUpper()

In Python, the string type does not have a ToUpper method, so this code should throw an exception. However, in .NET, calling the ToUpper function on a string should return ‘HELLO, WORLD!’. These are obviously contradictory requirements.

IronPython handles this by being a good Python implementation by default, but allowing the developer to indicate they want high fidelity .NET interop if they want it. In Python, code is organized into modules and namespaces, similar to .NET. IronPython includes a special module called ‘clr’. By importing that module, developers indicate they want to use .NET interop. Here’s an interactive IronPython session that demonstrates the use of import clr:

>>> s = 'hello, world'
>>> s.ToUpper()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no
                attribute 'ToUpper'
>>> import clr
>>> s.ToUpper()
'HELLO, WORLD'

Before the call to import clr, you can’t call the ToUpper method on a string. String objects in Python don’t have a ToUpper method so IronPython hides that method, among others. However, after the developer imports the clr module, all of String’s methods-both native .NET as well as Python compatible-are available.

The clr module is also used to load external assemblies. For example, if you want use .NET XML processing classes like XmlReader and XmlDocument, you need to add a reference to System.Xml.dll. In C#, the reference is added declaratively at compile time. In IronPython, there is no compile time, so you need to add the reference imperatively via code. Then you can import classes into your current scope and use them just like any other Python object.

import clr
clr.AddReference('System.Xml')
    
from System.Xml import XmlDocument
    
xml = XmlDocument()
xml.Load('http://devhawk.net/rss.aspx')

It should be pretty obvious what this code does, but there are a few things I want to note. First off, there is no “new” statement in Python, type instances are created by calling the type like a function. Second, in C# importing classes from namespaces (via the using statement) is optional, designed to save typing. In Python, it’s mandatory. In other words, writing the code this way wouldn’t work:

import clr
clr.AddReference('System.Xml')
    
# the following line doesn’t work
xml = System.Xml.XmlDocument()

The from … import form of the statement imports the short name. If I want the fully namespace scoped name, I can type ‘import System.Xml.XmlDocument’. Import also supports renaming types to avoid collisions by using import <real name> as <new name>.

Nearly the entire .NET Framework is available to IronPython simply by adding references to the relevant assemblies and importing the needed types. In addition to being able to create instances of .NET types, you can also consume .NET events, implement .NET interfaces and inherit from .NET types. For example, here’s some code from the IronPython tutorial that uses Windows Forms.

# the winforms module in the tutorial directory
import winforms
    
from System.Windows.Forms import *
from System.Drawing import *
    
def click(*args):
    print args
    
f = Form()
f.Text = "My First Interactive Application"
f.Click += click
    
f.Show()

One thing you can’t do from IronPython is adorn your code with attributes. Any part of the .NET framework that requires attributes, such as WCF contracts or XML serialization, won’t work with IronPython. Those libraries depend on custom attributes that act like custom metadata that extends the existing static type. Python objects don’t have a static type, so there’s nothing to attach the custom attribute to.

The other thing about .NET interop is that it’s essentially one way. It’s easy for Python to call into .NET classes written in statically typed languages. However, there’s no easy way for statically typed languages to call into dynamically typed objects. Static languages depend on that compile-time type metadata in order to dispatch method calls, but that metadata just doesn’t exist in dynamically typed languages like Python. Obviously, given the multi-language nature of the CLR, enabling statically typed languages to call into dynamically typed code is a scenario we would like to enable in the future3.

Embedding IronPython

One of the areas that Python has significant traction in the industry is as an easily embeddable language. Likewise, IronPython can be easily embedded inside your .NET applications to provide a scripting or macro development experience.

In the 1.x version of IronPython, embedding was handled entirely thru the PythonEngine type. Here’s an example accessing the PythonEngine from the interactive console.

>>> import clr
>>> clr.AddReference("IronPython.dll")
>>> from IronPython.Hosting import PythonEngine
>>> pe = PythonEngine()
>>> pe.Evaluate('2+2')
4

In this example, we haven’t provided the hosted Python environment any hooks into the host, so the code it can execute is fairly limited. However, PythonEngine provides a Globals collection that you can use to expose your application object model to the Python environment.

In IronPython 2.0, the hosting code gets a bit more complicated:

import clr
clr.AddReference("IronPython.dll")
from IronPython.Hosting import PythonEngine
pe = PythonEngine.CurrentEngine
scope = pe.CreateScope()
source = pe.CreateScriptSourceFromString('2+2')
result = source.Execute(scope)

One of the reasons the code is more complicated is because of the introduction of the Dynamic Language Runtime, or DLR for short. The DLR is an extension to the CLR to provide common capabilities needed in dynamic languages. It also provides a common hosting API to allow any application that hosts the DLR to support any language that targets the DLR. That means if your application uses the DLR hosting API, it can support not only IronPython but IronRuby and Managed JavaScript as well as any third-party language that gets built on the DLR. You can find details on the hosting API at http://compilerlab.members.winisp.net/

Conclusion

Between significant whitespace and dynamic typing, there’s no question Python is a wholly different development experience from C# or VB. I’m a recent convert to Python from C#, so I know exactly how strange it can feel. But once you get past that feeling of unfamiliarity, you start to see just how productive Python can be.

Footnotes

1 “Python for .NET: Lessons learned”, Mark Hammond of ActiveState, November 2000. http://web.archive.org/web/20060925091459/http:/starship.python.net/crew/mhammond/dotnet/PythonForDotNetPaper.doc Note, the whitepaper is no longer generally available on Mr. Hammond’s or ActiveState’s site. The author retrieved a copy via the September 25, 2006 snapshot of Mr. Hammond’s site in the Internet Archive.

2 “Dynamic languages and virtual machines”, Jon Udell, Infoworld, August 22, 2003, http://www.infoworld.com/article/03/08/22/33OPstrategic_1.html

3 Charlie Calvert of the Visual Studio Languages team blogged about possible inclusion of Dynamic Lookup in a future version of C# earlier this year. http://blogs.msdn.com/charlie/archive/2008/01/25/future-focus.aspx

Introducing IronPython

Published in:

Filed under: