Doug Hodges is interviewed by Ken Levy discussing the history of the Visual Studio IDE (Integrated Development Environment).
Ken: Who are you, what do you work on, where are we?
Doug: Well, hello, I'm Doug Hodges. I'm the original architect of the Visual Studio IDE.
Ken: Is that also referred to as the Shell?
Doug: Yeah, the main shell for the Visual Studio product. All the language products that plug in are all integrated in the IDE (integrated development environment).
Ken: What's your exact title?
Doug: I'm Software Architect for the Visual Studio IDE. At least that's what I call myself.
Ken: So we're in building 41, which is kind of the home of Visual Studio.
Doug: Yep. Building 41 in Redmond, Washington.
Ken: So Jason Weber tells me that he refers to you as the grandfather of the Visual Studio shell.
Doug: I prefer to consider myself as the father, not the grandfather.
Ken: So he said that you consider yourself the cool uncle, is that right? Have you said that before?
Doug: Yeah, well. So I've been on this product and this vision to create this integrated extensible IDE from starting back in the Visual Studio 5.0 timeframe.
Ken: What year was that, do you recall? What version of Visual Studio?
Doug: Visual Studio 97. And back then I was the architect of Visual InterDev (VI). And actually we started the product back in �95 when Microsoft changed its focused on addressing the Web. And we took my product, I was actually part of the Microsoft Access team, and we were working on a future version of Microsoft Access that was based off of the component technology. We had a forms-based technology package that ultimately became Internet Explorer as you know it today. We had the database design tools which were a component that ultimately became the database design tools in Visual Studio today.
I was on a team that was suppose to take these components and repackage and rebuild a product that would be Microsoft Access, the follow-up and successor to Microsoft Access 2.0 at the time. I was an original member of the COM (OLE) team, and so we started with a COM-based strategy for how you could integrate components. When we [got] a new charter to build a tool to address developing for the Web, we created this Visual InterDev product. We took my team and another team that was doing development tools for Blackbird, -it was for MSN-and we formed a team to create Visual InterDev. At that time we had one tool for VB (Visual Basic), one tool for VC (Visual C++), and at that time we thought that the VC codebase was going to be the codebase that was going to evolve into an integrated shell. We built our first version of that Visual InterDev product in the VC codebase, and I re-hosted those same components and forms package for HTML, design, and database tools in the VC 5.0 shell.
That was the start of our component (COM) interfaces that you see still existing today. We had a vision in the division to create one common IDE for all language products and all tools. We had a separate IDE for VB at that time, then we had VC, VJ, and VI in this dev shell, the VC-based shell. And it was a derived MFC-based classes implementation of a C++-based framework. We tried to have a third-party extensibility program like VSIP (Visual Studio Industry Partner) program today. We found that wasn't going to work very well unless everyone gave us their source code that we would use to build in our own build lab, because any time you change something in a base class you had to recompile the world. So that wasn't going to work to have a third-party extensibility program.
So then in the Visual Studio 6.0 timeframe, I started this new shell and the whole strategy was based on a COM-based architecture that could allow independently built things that were independent versions of each other to be plugged together. In our first version, we took the VJ++ product and the VI product out of the MS Dev shell, and they were the first two products that we integrated into this new IDE that we created. So at that time it looked like we took a step backwards, because we went from two shells, one for VB and one for VJ/VC, and then in the VS 6.0 timeframe we moved to three shells with one for VB, one for VJ++, and one for VC++. But that was a stepping stone to get us to the goal of our division which was a single shell which we achieved in the Visual Studio 2002 (VS 7.0) timeframe.
Ken: So at this stage of the game when you're building this third but new one, how are you thinking about extensibility? Is it mostly around certain partners in the VSIP program, or was it about anyone doing anything they want at any time?
Doug: The Shell was designed to be fully extensible from the ground up and designed to have independent pieces that independently plugged together. We had all of our existing products internally, and we wanted to be at least as useable as the VB product and the VC product combined. We picked the best features from those two products, and we wanted to meet the extensibility requirements of those independent teams. The VB team, the VC team, and the VI team were all customers of mine, plugged into this combined IDE as well as planning for the requirements of third parties. Yeah, it was designed from the beginning to be a fully extensible shell and we felt that if we could meet the requirements of VB and VC, that together we could probably meet the requirements of most language partners. And then of course you had to anticipate the kind of other products that could be built, like what you see today with Team System or Server Explorer for database development, and other things. Yeah, the goal from the beginning was to be extensible.
Ken: So you are targeting both languages to be integrated as well as UI tools and components. And sometimes they are separate-you may have one company that wants to build a new language, like Cobol or something, and another company that has a toolset that is for a specific vertical market and they want to resell that integrated into Visual Studio, is that right?
Doug: Well, yeah, and trying to cover the full lifecycle of development tools to assist in not just code development but in testing, deployment, servicing, source control, and code analysis.
Ken: The companies that did stuff in the beginning were basically partners-they got access to the team-and it didn't scale to the masses.
Doug: Originally, these were the same APIs that we used to build our own products, and we opened it up to a select program that you could pay to get into. It used to cost $10,000. Those where the high-end ISVs at that time. Very often it was a high end C++ programmer who worked on it, and priority was given to powerful features but not necessarily on making it easy to use. At that time we had no VS SDK team and I had to beg and borrow to get resources to get a sample here and there or documentation. Originally when someone needed to write a feature for the shell, I would help them with a little planning of their product and they would write a sample and send it to me. Some of the samples we had at the beginning came from different sources and we didn't have a team to create or organize an SDK. It was a very small program at that time. Another aspect of extensibility was add-ins and macros, and those were in the product in the beginning, and every box of Visual Studio had wizards so if you wanted to write add-ins you could do extensibility right in the box. That was more targeted more at VB-level languages using an automation-based model. But you were limited in the kind of features that you could implement there. You could add commands or tool windows, or hook to certain events and react to them. But you couldn't implement a new language, or a new debug engine with a custom runtime. That's when you needed to access the lower level APIs, and originally, like I said, it cost $10,000.
Ken: Did they get code as well as help? Was there kind of an SDK for VSIP folks?
Doug: There was a little bit of an SDK. Originally there was a set of samples I scrounged together by begging and borrowing. We had someone in PSS come and help us write some samples. They helped to build the tools you needed-the IDL and .H files, and other related tools that you needed. There wasn't a nicely packaged SDK at that time.
Ken: Did you end up working in some capacity with most VSIP folks, and you knew them in some way from e-mail or a demo?
Doug: Yeah, I worked with them. We started way back having these developer labs where we brought partners in house for week long developer labs. Those started up way back in the 2002 timeframe. Originally that first extension of the shell was in the VS 6.0 timeframe and was not yet open. We did not expose the extensibility APIs. We had a couple of internal teams write some things, and I think someone from the SQL division wrote something for the actual language query or something. In the 2002 timeframe we basically had developer labs were we could bring the partners in house, and they got a lot of hand holding.
Ken: How many times have you given the overview presentation for Visual Studio extensibility?
Doug: <laugh> Yeah, I can't count how many times. Practically every developer lab started out that way with my overview presentation. Yeah, gosh, I'm not sure. A hundred?
Ken: Where are we today? How did we get to where we are today?
Doug: So with that first version we achieved our goal of a single integrated IDE in 2002. We shipped the 2003 and 2005 versions and are now working on the Visual Studio 2008 version. Now we can enhance and refine the product with new scenarios of Visual Studio extensibility, and it's a very mature platform now.
Ken: So a few things happened over the last few years. There is now a VSIP level that is less than $10,000, and then in 2005 the VS SDK became free, so that basically anyone can download the SDK and extend Visual Studio, although they don't have the same co-marketing or complete technical support access, but yet it's free so anyone can do it.
Doug: Yeah, yeah, that's really cool. So from a technology point of view there is a free level with access to all of the technology, then two pay levels above that which have additional marketing and support benefits. When we went public with the SDK, there were many many downloads. Then that raised the priority to not just worry about C++ programmers that have direct access to dev labs and hand holding with Microsoft. We needed to start focusing on the accessibility of the SDK and the accessibility of how easy it is to do that development. So that's been a major goal over the recent years, to try to improve the quality of our SDK. We developed an actual team to own the release of the SDK (VS Ecosystem, or VSX team).
Ken: We kind of had two different groups out there. First we have VSIP, the partners, which are part of the program, who are extending Visual Studio usually to a product that they can sell or use in the enterprise. And now this new group that we're cultivating called VSX, which stands for Visual Studio Extensibility, which are essentially non-partners who are using the free VS SDK, and that's a fairly new thing. Is this something that you had envisioned a long time ago, where you see .NET developers, Visual Studio developers, coming in and building all kinds of stuff for CodePlex and other places?
Doug: The original goal of focusing on the high end ISVs shaped the style of APIs we have. It was targeted at making things possible. Now that we have released the technology to everyone, we need to greatly improve our level of abstraction and ease of development for some of these scenarios. I can't say I originally thought that we would have that available. We had myself and a program manager trying to run a program that doesn't scale well. But now that we have a full product unit around managing and growing this program, it's really exciting and I hope it goes and develops really well.
Ken: A moment ago you touched on ease of use. In addition to more functionality and capability for VS extensibility and the VS SDK, it's also about making what's there easier to use and access, whether it is simply better docs, or better sample classes, or better wrapper classes that get you in, or wizard-type functionality, right?
Doug: Right. The first thing that made it easier was finally delivering the ability to use managed code to write these extensions. When we first shipped the extensibility via the SDK, you could only use C++ with it. We shipped both Visual Studio 2002 and 2003 that way. But we knew lots of developers would want to write in managed code like C# and VB. So we did what we called the Everett extras release. Everett was the codename for Visual Studio 2003. So we had an Everett extras SDK release which was an add-on to the previously released VS 2003 SDK release that delivered the minimal pieces you needed to be able to write in manage code. It gave you the COM interfaces in managed code, a redistributable MSI that you could use to install those assemblies, and then the beginning of a helper framework that you could use as some helper classes to begin to write something. That was really thin and we put a lot of effort and massively grew those helper classes into what we call now the Managed Packaged Framework (MFP) in the Visual Studio 2005 release.
So we continued to build and always provide the interop assemblies, and one of the things that has always been important to me was designing the APIs to take into account different extensions that different companies will work with at different times and different schedules. Addressing versioning has been a particular focus, so we spend a lot of effort maintaining compatibility with old versions of Visual Studio. It's always been our goal to maintain compatibility, and the API is designed in a fashion that you could write a single extension that runs in Visual Studio 2003 and Visual Studio 2005 that runs with a single binary. More realistically between Visual Studio 2005 and Visual Studio 2008; that's going to be popular, that you can create a single binary that runs in both versions. So part of what we do is to make a big effort to not change or break compatibility with any previously shipped interfaces.
We create new APIs in a way that is defined in a new assembly and file, so you can see what is new in a version. If you need to, you can provide the new distributable that lets you install the interop assemblies for the new interfaces on a down-level machine. If you write a package that depends on new features you can run the new version and can take advantage of the new features.
But that same binary can run a down-level version, so that query interface or query service for an API can allow you to dummy down and write code to work without it. That's particularly interesting to high-level product development, like partners who release VSIP products that may address more than one version of Visual Studio. Someone writing a tool for themselves probably won't be worrying about that and will only be worrying about one version of Visual Studio.
Ken: What are the key prerequisites for someone who wants to download the VS SDK? Say someone has Visual Studio 2005. Do they need to know a bit about COM interop or a little bit about anything else? Because there are parts of the shell that are pure COM. The Managed Packaged Framework (MPF) helps a .NET developer access them.
Doug: So we have this Managed Packaged Framework that covers and abstracts, let me say, 30% or 40% of our APIs. And so if you are staying within that subset, you can see what looks like a .NET experience when writing your extensions. But if you are doing something like writing a language or interacting with, oh gosh, there are so many features. You are going to come upon areas that are not yet abstracted in the Managed Package Framework, and then yes, you will need an understanding of COM interfaces. Unfortunately in COM interop there are several places you are going to encounter an end pointer and then will have to deal with the marshal class, the marshalling, and deal with an end pointer into an object or a string. We have samples for those, and hopefully when you hit one of those areas you can find some sample that does something like it so you can copy from it. And that's one of those areas we hope to improve the coverage of the MPF and other techniques to eliminate those places where you have to directly go to a COM interface that was originally designed for a C++ programmer that is rather awkward to deal with from a managed language.
Ken: One of the things that we've seen recently is the community doing a bunch of cool stuff building on top of the SDK. For example, the VSSDK Assist on CodePlex.com. You've seen a demo of that recently.
Doug: Yeah, that looks really cool. That looks really awesome.
Ken: How would you describe it? Is that something that assists people that use the SDK?
Doug: It's leveraging the Guidance Automation Toolkit that was developed in combination with Visual Studio people and the PAG (Patterns & Practices Group). And it's really cool. It's a factory. It's a wizard, on steroids. You can start with a new project wizard that starts you off with writing something for Visual Studio extensibility, but it doesn't drop you out of the wizard when it's done. It's like fine-grained points to launch wizards to add additional features to extend your project and give you guidance on what's next. It gives you help. Like if this is what you're looking to do, it can give you help on how you can do it. It's a direction that's fantastic. It's a direction that I'd like to see continue to grow.
Ken: One thing that I noticed in the demo is at runtime after you run it, it does not require any components at runtime. It just generates code in the right places that just use the VS SDK and extensibility APIs. So there's nothing at runtime that you need; it's just helping you and assisting you learn the SDK. So if you want a tool window and a button, here's how to glue those together.
Doug: And it knows about editing the five places you have to edit, you know, this file over here and this file over there, with one UI sequence. It takes care of making all those edits. But you're right, it's not a new runtime you install to use it; it actually generates the code for you.
Ken: What are some things that developers might be capable of doing if they were to download the SDK? Things like creating services, tool windows, menu buttons. You can even just have a simple thing that is on a menu that launches your own application. What kind of things can they hook into? What would you like to see the community build?
Doug: The first simple thing is adding a command on a menu or a toolbar. That command can do some action on selection, and maybe the next thing is showing some UI in a tool window. And let's say you have your own kind of code review tools and you have your own tool window that shows the results of a code review, and your command launches your code review based on the selected code file and presents the results in a window. Now incidentally, that level of extensibility, where you are adding command and tool windows, that's something you can also do using add-ins. And the extensibility that's been in the product for free since the beginning. In many ways it's easier to use add-ins, so I don't want to discourage people from using add-ins. There are some subtle features that you can't do in a tool window, for example through an add-in, that you can do when you are coding at a package level. For example, if the user shows a tool window and then shuts down, the next time they launch Visual Studio the tool window automatically reappears.
Ken: Something that becomes native to their development experience.
Doug: But if you do it as an add-in, you don't get that feature. So in some ways it feels second class in that sense because it doesn't behave identically like every built-in feature. Whereas if you do it as a package, and we do our own features as packages, you get that functionality. Add-ins are controlled through the Add-in Manager where the user can load and unload; Turn their add-ins on and off. In some cases you don't want the user to turn parts of your product off. That's one difference with a package.
Ken: In some key areas people can extend project types and create tool windows.
Doug: There are interesting cases where you have your own project type. In many cases those will be custom XML file types. You have your own XML format type that you would like to create your own UI design experience for. That's a great scenario. So I call that in a general category creating your own designer or editor.
Ken: A couple of years ago I worked on sponsoring the XML Editor in Visual Studio 2005. Chris Lovett created that.
Doug: I'm glad you worked on that. It's a great feature.
Ken: That's one major example since it's actually built around the stuff we are talking about-an external team extending Visual Studio.
Doug: It (XML Editor) is designed with extensibility in mind so that you can write an XSD schema and you get IntelliSense for your XML. And then, in addition, you might want a custom UI experience. Think of it as the View design for your code. That's where you would drop down and write with the SDK.
Other scenarios might extend projects in other ways. There's a rather advanced point of extensibility that we have which we call project sub types, or project flavors. We use this ourselves. There is a project type that lets you run assemblies on a smart device, a pocket PC or such. You don't just have a regular VB project or a CE project. You have a project that knows it's building code for a device-it adds special features that are unique to building for a device. It gives you an emulator and it knows that after you build your assembly it has to deploy to the device. Then it knows how to launch that application under a debugger appropriate for running the device at runtime.
Ken: Would you say in that case, rather than creating a whole new custom project type, it was kind of like inheriting from a VB or C# class, adding what it needs for that custom behavior, in addition to the basic VB or C# project?
Doug: Exactly. It's the COM equivalent of sublcassing. Another scenario is what we've done in SQL Server development. One of the new features in SQL Server is that you can run managed code inside SQL Server for data types and stored procedures. There's a project subtype that knows how to build an assembly on the user's desktop but then install and deploy it into SQL Server and launch it under the debugger for SQL Server. So you can write you own kind of project subtype using the SDK. You might use this if you have some place where you use application code like your own application server or some kind of environment where you load managed code in some kind of assemblies in a managed code environment. It's a great scenario to create your own project subtype that knows about the extra steps that automates it for the user for the experience of getting that code into the right place and then debugging it.
Ken: You get everything you normally would, with custom behaviors on custom projects.
Doug: This is a case where you are not writing your own language. You want to use VB or C# or one of the other third-party languages, and yet you want to write this extension to extend the experience that is perfect for your server or your runtime. Then you get into scenarios where you have your own language. That's a great scenario for VSIP, where you have your own custom language and you want to get custom features that you have for your language, and possibly forms, WinForms hosing or Web development-that's a classic scenario. Then look at non-traditional projects that can be added to Visual Studio. You know, imagine the team that invented Team System, and imagine you are the one who invented Team Explorer, and you have features for a team project. It's not a VB or C# project–it's in a whole new space of something where you want to show this hierarchy in the tool window and from there to launch these documents that let you edit information or view information. So you could invent your own thing like that.
Ken: It might not even be for compiling to a .NET solution. It could be for modeling or something.
Doug: Yeah, or modeling a runtime deployment scenario where your tools help you monitor a deployed application and it has nothing to do with deploying that application-it has something to do with monitoring or servicing or something. There are infinite possibilities.
Ken: You can control the properties of the project as well as the file nodes?
Doug: You're writing an object that is expressing in a series of nodes. You see icons and captions; the drag and drop behavior and contents menus; the context that you give to the designers and editors that come out of that hierarchy. So if you are a language-like thing, you have a programmatic namespace that you have built up by examining all the files in your project and you build up a namespace that gives an IntelliSense experience. Or perhaps you have some other kind of notion of project references and you can access project data that you make available to your designer experience. You can invent lots of interesting things. Some of these project hierarchies might not be based on a file system. You might have your own custom repository or custom system that you want to monitor, and you have a hierarchy that gives you access to documents that you want to edit, and they don't save to the file system and are not part of a project that builds. Think of lifecycle tools that go beyond development.
Ken: Other things you can do with the VS SDK are add to the Options pages.
Doug: You can have your own custom Tools option pages. You can add to the Error List, the Toolbox, and you can use the Document outline window. If you are a project or hierarchy implementer, you can create your own property pages. If you want to add your own property pages to add to someone else's project, that's when you would create your own project subtype. You can add properties to the Properties window which is called automation extenders. You can add a command almost anywhere.
Ken: So a lot of your past, besides working on the core shell, was working with customers that usually work in a vertical market. Now we are in a space where any .NET developer can extend Visual Studio in any way for themselves, for other developers, for free, for CodePlex. Do you expect to see some cool stuff?
Doug: We are already seeing cool stuff. VSSDK Assist was created by the community. There's going to be an explosion, I think, of really cool things. One I'd love to see, and I can't believe we don't have this feature in the product already, is diff tools.
Ken: Other editors can have certain file types, like web.config or other XML types that can have a property editor.
Doug: I think that's going to be a huge scenario. There are so many XML files that people have created that could have a much better editing experience than having to use or look at the XML, and you can write those and do what I call a custom designer view for an XML file. You can create those designers and attach them to the schema so that whenever an XML file type is open with your schema attached, then your designer can come into play. Most of the time you have to make a new file extension. Consider the XML Editor that you helped encourage to happen that Chris Lovett developed. You can create those designers and attach them to a certain schema.
Ken: Let me ask you a question I hear a lot. We have VS SDK 4.0, which is the last SDK for VS 2005, and we have the VS SDK for VS 2008, which is backwards compatible. Do you see an easy transition for someone who built something with VS 2005 like a tools window or whatever, and then they get the VS SDK for VS 2008 and want to rebuild it to take advantage of new things? Do you see that whole experience as painless?
Doug: I hope the experience is painless. You may have to open up your project file and fix a path to the SDK install location. We are making the transition from the old file format called CTC files for creating menus to an XML-based file format called VSCT, and we will have tools to make that transition easy for you. From the API point of view, we're highly compatible. We want to make it easy to develop for VS while logged on as a normal user rather than only as an administrator. That's something I hope to see in the next version of the VS SDK (for VS 2008)-not running as administrator.
Ken: Let me ask you another question that I hear in reference to the history of the Shell. In the IDE, in the message looping, what's the thing called the Thunder Message Loop? This is a very old reference, correct?
Doug:
Ken: In terms of the VSX community, what would you like to evangelize the community to do with the SDK? What are you going to do to push Microsoft to do internally to improve the SDK and the experience developers have when they are building extensions?
Doug: I would love to see an explosion of tools that increase productivity for small teams, large teams, and individuals for the things they do that is repetitive. When one says “Boy, wouldn't it be great to have a designer for this?” For us, I'd like to see us make extensibility easier, and that's why I'm so excited about things like the VSSDK Assist and leveraging things like the guidance automation Toolkit and DSLs (Domain Specific Languages). I think we need to push to make things easier, more accessible, there should be more samples. I'd like to see a lot more things like VSSDK Assist that help to write code for you and a lot more frameworks of great base classes that you can derive from that help you do scenarios. For example, we're coming out with a new tool to help you work with the menus. I'd love to see an explosion of tools to help VSX developers like VSSDK Assist.
Ken: Another thing is we'll increase transparency, as in telling the community what we are working on before we do it so they are more in parallel with our efforts and they can contribute.
Doug: Yeah, that's all great stuff.
Ken: Well thanks for talking the time to talk to me Doug.
Doug: Thanks for this opportunity.
Ken Levy is the community program manager on the Visual Studio Ecosystem tam focusing on developer community for VSX (Visual Studio Extensibility). The VSX community includes developers who build extensions (tools, editors, designers, languages, and more) for Visual Studio using the VS SDK found at the VSX Developer Center (http://msdn.com/vsx). Ken was previously a product planner on Microsoft's Windows Live Platform team working on developer community and future product planning. Before working in the Windows Live Division, Ken was a product manager in the Visual Studio data team responsible for Visual FoxPro product management, the VFP developer Web site, as well a sponsorship of the new XML tools in Visual Studio 2005 created by the WebData XML team. Ken is a long time recognized member of the FoxPro community and has developed many high profile applications and tools in all versions of FoxBase/FoxPro since 1986. Ken spent over four years as a software engineer consulting for Microsoft on the Visual FoxPro team from version 3.0 through 7.0 and is the author of many components of Visual FoxPro including the Class Browser and Component Gallery. While working as a consultant as NASA's Jet Propulsion Laboratory (JPL) in the 1990s, Ken developed many public domain open source programs including GenScrnX and other developer tools used worldwide in crating in-house and commercial applications. Ken is a former technical contributing writer and editor to many software magazines, and has been a frequent speaker at industry conferences worldwide since 1992. You can read Ken's blog at http://blogs.msdn.com/klevy.
