Anatomy of a "Small" Software Design Change

File this one away for the next time your boss comes in and asks,

Yeaaah, I’m going to need you to make that little change to the code. It’ll only take you a couple hours, right?

Software has this deceptive property in which some changes that seem quite big and challenging to the layman end up being quite trivial, while other changes that seem quite trivial, end up requiring a lot of thought, care, and work.

Often, little changes add up to a lot.

I’m going to walk through a change we made that seemed like a no-brainer but ended up having a lot of interesting consequences that were invisible to most onlookers.

The Setup

I’ll provide just enough context to understand the change. The project I work on, ASP.NET MVC allows you to call methods on a class via the URL.

For example, a request for /product/list might call a method on a class named ProductController with a method named List. Likewise a request for /product/index would call the method named Index. We call these “web callable” methods, Actions. Nothing new here.

There were a couple rules in place for this to happen:

• The controller class must inherit from the Controller class.
• The action method must be annotated with the [ControllerAction] attribute applied to it.

We received a lot of feedback on the requirement for having that attribute. There were a lot of good reasons to have it there, but there were also a lot of good reasons to remove it.

Removing that requirement should be pretty simple, right? Just find the line of code that checks for the existence of the attribute and change it to not check at all.

The Consequences

Ahhh, if only it were that easy my friend. There were many consequences to that change. The solutions to these consequences were mostly easy. The hard part was making sure we caught all of them. After all, you don’t know what you don’t know.

Base Classes

So we removed the check for the attribute and the first thing we noticed is “Hey! Now I can make a request for /product/gethashcode, Cool!”

Not so cool. Since every object ultimately inherits from System.Object, every object has several public methods: ToString(), GetHashCode(), GetType(), and Equals(), and so on... In fact, our Controller class itself has a few public methods.

The solution here is conceptually easy, we only look at public methods on classes that derive from our Controller class. In other words, we ignore methods on Controller and on Object.

Controller Inheritance

One of the rationales for removing the attribute is that in general, there isn’t much of a reason to have a public method on a controller class that isn’t available from the web. But that isn’t always true. Let’s look at one situation. Suppose you have the following abstract base controller.

public abstract class CoolController : Controller
{
  public virtual void Smokes() {...}

  public virtual void Gambles() {...}

  public virtual void Drinks() {...}
}

It is soooo cool!

You might want to write a controller that uses the CoolController as its base class rather than Controller because CoolController does some cool stuff.

However, you don’t think smoking is cool at all. Too bad, Smokes() is a public method and thus an action, so it is callable. At this point, we realized we need a [NonAction] attribute we can apply to an action to say that even though it is public, it is not an action.

With this attribute, I can do this:

public class MyReallyCoolController : CoolController
{
  [NonAction]
  public override void Smokes()
  {
    throws new NotImplementedException();
  }
}

Now MyReallyCoolController doesn’t smoke, which is really cool.

Interfaces

Another issue that came up is interfaces. Suppose I implement an interface with public methods.  Should those methods by default be callable? A good example is IDisposable. If I implement that interface, suddenly I can call Dispose() via a request for /product/dispose.

Since we already implemented the [NonAction] attribute, we decided that yes, they are callable if you implicitly implement them because they are public methods on your class and you have a means to make them not callable.

We also decided that if you explicitly implement an interface, those methods would not be callable. That would be one way to implement an interface without making every method an action and would not require you to annotate every interface method.

Special Methods

Keep in mind that in C# and VB.NET, property setters and getters are nothing more than syntactic sugar. When compiled to IL, they end up being methods named get_PropertyName() and set_PropertyName(). The constructor is implemented as a method named .ctor(). When you have an indexer on a class, that gets compiled to get_Item().

I’m not saying it was hard to deal with this, but we did have to remember to consider this. We needed to get a list of methods on the controller that are methods in the “typical” sense and not in some funky compiler-generated or specially named sense.

Edge Cases

Now we started to get into various edge cases. For example, what if you inherit a base controller class, but use the new keyword on your action of the same name as an action on the base class? What if you have multiple overloads of the same method? And so on. I won’t bore you with all the details. The point is, it was interesting to see all these consequences bubble up for such a simple change.

Is This Really Going To Help You With Your Boss?

Who am I kidding here? Of course not. :) Well..., maybe.

If your boss is technical, it may be a nice reminder that software is often like an iceberg. It is easy to see the 10% of work necessary, but the other 90% of work doesn’t become apparent till you dig deeper.

If your boss is not technical, we may well be speaking a different language here. I need to find an analogy that a business manager would understand. A situation in their world in which something that seems simple on the surface ends up being a lot of work in actuality. If you have such examples, be a pal and post them in the comments. The goal here is to find common ground and a shared language for describing the realities of software to non-software developers.

Defining ASP.NET MVC Routes and Views in IronRuby

In a recent post I expressed a few thoughts on using a DSL instead of an XML config file. I followed that up with a technical look at monkey patching CLR objects using IronRuby, which explores a tiny bit of interop.

These posts were precursors to this post in which I apply these ideas to an implementation that allows me to define ASP.NET MVC Routes using IronRuby. Also included in this download is an incomplete implementation of an IronRuby view engine. I haven't yet implemented layouts.

IronRubyMvcDemo.zip Download (4.93 MB)

This implementation works with the latest CodePlex drop of MVC.

To use routes written in Ruby, reference the IronRubyMvcLibrary from your MVC Web Application and import the IronRubyMvcLibrary.Routing namespace into your Global.asax code behind file. From there, you can just call an extension method on RouteCollection like so...

public class GlobalApplication : System.Web.HttpApplication
{
  protected void Application_Start()
  {
    RouteTable.Routes.LoadFromRuby();
  }
}

This will look for a Routes.rb file within the webroot and use that file to load routes. Here's a look at mine:

$routes.map "products/{action}/{id}"
  , {:controller => 'products', :action => 'categories', :id => ''}
$routes.map "{controller}/{action}/{id}", {:id => ''}
$routes.map "{controller}", {:action => 'index'}, {:controller => '[^\.]*'}
$routes.map "default.aspx", {:controller => 'home', :action => 'index'}

That’s it. No other cruft in there. I tried experimenting with lining up each segment using tabs so it looks like an actual table of data, rather than simply code definitions.

Also included in this download is a sample web app that makes use of the IronRubyViewEngine. You can see how I applied Monkey Patching to make referencing view data cleaner. Within an IronRuby view, you can access the view data via a global variable, $model. The nice part is, whether you pass strongly typed data or not to the view, you can always reference view data via $model.property_name.

In the case where the view data is a view data dictionary, this will perform a dictionary lookup using the property name as the key.

Be sure to check out the unit tests which provide over 95% code coverage of my code if you want to understand this code and improve on it. Next stop, Controllers in IronRuby...

Monkey Patching CLR Objects

In my last post I set the stage for this post by discussing some of my personal opinions around integrating a dynamic language into a .NET application. Using a DSL written in a dynamic language, such as IronRuby, to set up configuration for a .NET application is an interesting approach to application configuration.

With that in mind, I was playing around with some IronRuby interop with the CLR recently. Ruby has this concept called Monkey Patching. You can read the definition in the Wikipedia link I provided, but in short, it is a way to modify the behavior of a class or instance of a class at runtime without changing the source of that class or instance. Kind of like extension methods in C#, but more powerful. Let me provide a demonstration.

I want to pass a C# object instance that happens to have an indexer to a Ruby script via IronRuby. In C#, you can access an indexer property using square brackets like so:

object value = indexer["key"];

Being able to use braces to access this property is merely syntactic sugar by the C# language. Under the hood, this gets compiled to IL as a method named get_Item.

So when passing this object to IronRuby, I need to do the following:

value = $indexer.get_Item("key");

That’s not soooo bad (ok, maybe it is), but we’re not taking advantage of any of the power of Ruby. So what I did was monkey patch the method_missing method onto my object and used the method name as the key to the dictionary. This method allows you to handle unknown method calls on an object instance. You can read this post for a nice brief explanation.

So this allows me now to access the indexer from within Ruby as if it were a simple property access like so:

value = $indexerObject.key

The code for doing this is the following, based on the latest IronRuby code in RubyForge.

ScriptRuntime runtime = IronRuby.CreateRuntime();
ScriptEngine rubyengine = IronRuby.GetEngine(runtime);
RubyExecutionContext ctx = IronRuby.GetExecutionContext(runtime);

ctx.DefineGlobalVariable("indexer", new Indexer());
string requires = 
@"require 'My.NameSpace, Version=1.0.0.0, Culture=neutral, PublicKeyToken=...'

def $indexer.method_missing(methodname)
  $indexer.get_Item(methodname.to_s)
end
";

//pretend we got the ruby script I really want to run from somewhere else
string rubyScript = GetRubyCode();

string script = requires + rubyScript;
ScriptSource source = rubyengine.CreateScriptSourceFromString(script);
runtime.ExecuteSourceUnit(source);

What’s going on here is that we instantiate the IronRuby runtime and script engine and context (I still need to learn exactly what each of these things are responsible for apart from each other). I then set a global variable and set it to an instance of a CLR object written in C#.

After that, I start constructing a string that contains the beginning of the Ruby script I want to execute. I will pre-append this beginning section with the actual script I want to run.

The beginning of the Ruby script imports the .NET namespace that contains my CLR type to IronRuby (I believe that by default you don’t need to import mscorlib and System).

I then added a missing_method method to that CLR instance within the Ruby code via this snippet.

def $indexer.method_missing(methodname);
  $indexer.get_Item(methodname.to_s)
end

At that point now, when I execute the rest of the ruby script, any calls from within Ruby to this CLR object can take advantage of this new method we patched onto the instance.

Pretty nifty, eh?

In my next post, I will show you the concrete instance of using this and supply source code.

Dynamic Language DSL vs Xml Configuration

Disclaimer: My opinions only, not anyone else’s. Nothing official here. I shouldn’t have to say this, but past history suggests I should. P.S. I’m not an expert on DSLs and Dynamic Languages ;)

This week I attended a talk by John Lam on IronRuby in which he trotted out the Uncle Ben line, with great power comes great responsibility. He was of course referring to the power in a dynamic language like Ruby.

Another quip he made stuck with me. He talked about how his brain sometimes gets twisted in a knot reading Ruby code written using metaprogramming techniques for hours at a time. It takes great concentration comprehending code on a meta and meta-meta level in which the code is manipulating and even rewriting code at runtime. Perhaps this is why C# will remain my primary language in the near term while I try and expand my brain to work on a higher level. ;)

However, the type of code I think he is referring to is the code for implementing a DSL itself. Once a DSL is written though, the code on top of that DSL ought to be quite readable. This is the nook where I see myself adopting IronRuby prior to using it as my primary language.I can see myself creating and using mini-DSLs (Domain Specific Languages) here and there as replacement for configuration.

Ahhh... configuration. I sometimes think this is a misnomer. At least in the way that the Java and .NET community have approached config in practice. We’ve had this trend in which we started jamming everything into XML configuration.

So much so, we often get asked to provide XML to configure features I think ought to be set in code along with unit tests. We’ve turned XML into a programming language, and a crappy one at that. Ayende talks about one issue with sweeping piles of XML configuration under a tool. This is not an intractable problem, but it highlights the fact that XML is code, but it is code with a lot of ceremony compared to the amount of essence. To understand what I mean by ceremony vs essence read Ending Legacy Code In Our Lifetime.

With the ASP.NET MVC project, we’ve taken the approach of Code First, Config Second. You can see this with our URL Routing feature. You define routes in code, and we might provide configuration for this feature in a future version.

With this approach, you can write unit tests for your route definitions which is a good thing! Routes basically turn the URL into a method invocation, why wouldn’t you want to have tests for that?

The reason I write about this now is that I’ve been playing around with IronRuby lately and want to post on some of the interesting stuff I’ve been doing in my own time. This post sets the context for why I am looking into this, apart from it just being plain fun and coming from a haacker ethic of wanting to see how things work.

ASP.NET MVC Preview of a Preview

UPDATE: Just to prove that this is a preview of a preview, we had a signing problem with the initial pre-built VSI download. If you tried building from source, everything should’ve been ok. We apologize for that. Even though this is meant to be a rough preview, we do want to have a high quality bar in that you should be able to try out the code. So if you run into that problem, please do download the VSI again.

It’s no secret that Microsoft can get better at naming non RTM (Release to Manufacturing) releases. We have terms like CTP, Preview, Alpha, Beta, RC (Release Candidate), and so on. On the other hand, at least Microsoft does try to move things along to RTM rather than keeping products in perpetual Beta.

With ASP.NET MVC, we also need to add yet another type of release. For now I’ve been calling this a CodePlex Source Release meaning it’s simply sharing source code that is in progress. ScottGu called it a “Preview of a Preview” in my office one day and that name stuck with me. This is really a preview of an upcoming preview. ;)

Speaking of ScottGu, he posted a detailed description of this release on his blog last night. I recommend taking a look at it because he covers most of the changes in good detail.

One aspect of this latest source release that I’m particularly happy about is that we released our unit tests. As Scott mentioned, we are using MoQ as a mock framework within our tests. Note that this is not some official endorsement of any particular mock framework. Originally we started out trying to port our tests to Rhino Mocks (which I’ve written a lot about). MoQ just happened to have a programming model that was closer to the way our internal mock framework works, so we switched over to MoQ.

I will write more about this release later. But for now, I will leave you with an updated Northwind Demo based on this release because there ain’t no party like a Northwind party.

You've Been Haacked In Chinese

If ever someone was undeserving of having others spend their valuable time translating his blog, it would be me. But hey, some people from the http://blog.joycode.com/ site went ahead and did it anyway. I must admit that I’m very flattered that anyone would put the effort in.

Before this, I learned that Subtext powers MySpace China's blogs, and now my blog is translated to Chinese. As David Hasselhoff says, “I’m big in China”. (To my Chinese audience, that is a joke. I am quite small.)

Upcoming Changes In Routing

Made a few corrections on having default.aspx in the root due to a minor bug we just found. Isn’t preview code so much fun?

We’ve been making some changes to routing to make it more powerful and useful. But as Uncle Ben says, with more power comes more responsibility. I’ll list out the changes first and then discuss some of the implication of the changes.

• Routes no longer treat the . character as a separator. Currently, routes treat the . and / characters as special. They are separator characters. The upcoming release of routing will only treat the / as separator.
• Routes may have multiple (non-adjacent) url parameters in a segment. Currently, URL parameters in a route must fill up the space between separators. For example, {param1}/{param2}.{param3}. With the upcoming release, a URL segment may have more than one parameter as long as they are separated by a literal. For example {param1}.{ext}/{param3}-{param4} is now valid.

Passing Parameter Values With Dots

With this change, the dot character becomes just another literal. It’s no longer “special”. What does this buy us? Suppose you are building a site that can present information about other sites. For example, you might want to support URLs like this:

• http://site-info.example.com/site/www.haacked.com/rss
• http://site-info.example.com/site/www.haacked.com/stats

That first URL will display information about the RSS feed at www.haacked.com while the second one will show general site stats. To make this happen, you might define a route like this.

routes.Add(new Route("site/{domain}/{action}" 
  , new MvcRouteHandler()) 
{ 
  Defaults=new RouteValueDictionary(new {controller="site"})  
});

Which routes to the following controller and action method.

public class SiteController : Controller
{
  public void Rss(string domain)
  {
    RssData rss = GetRssData(domain);
    RenderView("Rss", rss);
  }

  public void Stats(string domain)
  {
    SiteStatistics stats = GetSiteStatistics(domain);
    RenderView("Stats", stats);
  }
}

The basic idea here is that the domain (such as www.haacked.com in the example URLs above) would get passed to the domain parameter of the action methods. The only problem is, it does not work with the previous routing system because routing considered the dot character in the URL as a separator. You would have had to define routes with URLs like site/{sub}.{domain}.{toplevel}/{action} but then that doesn't work for URLs with two sub-domains or no sub-domain.

Since we no longer treat the dot as special, this scenario is now possible.

Multiple URL Segments

What does adding multiple URL segments buy us? Well it continues to allow using routing with URLs that do have extensions. For example, suppose you want to route a request to the following action method: public void List(string category, string format)

With both the previous and new routing, you can match the request for the URL...

/products/list/beverages.xml

with the route

{controller}/{action}/{category}.{format}

To call that action method. But suppose you don’t want to use file extensions, but still want to specify the format in a special way. With the new routing, you can use any character as a separator. For example, maybe you want to use the dash character to separate the category from the format. You could then match the URL

/products/list/beverages-xml

with the route

{controller}/{action}/{category}-{format}

and still call that action method.

Note that we now allow any character (allowed in the URL and that is not a dash) to pretty much be a separator. So if you really wanted to, though not sure why you would, you could use BLAH to separate out the format. Thus you could match the route

/products/list/beveragesBLAHxml

with the route

{controller}/{action}/{category}BLAH{format}

and it would still route to the same List method above.

Consequences

This makes routing more powerful, but there are consequences to be aware of.

For example, using the default routes as defined in the ASP.NET MVC Preview 2 project template, a request for “/Default.aspx” fails because it can’t find a controller with the name “Default.aspx”. Huh? Well “/Default.aspx” now matches the route {controller}/{action}/{id} (because of the defaults for {id} and {action}) because we don’t treat the dot as special.

Not only that, what about a request for /images/jpegs/foo.jpg? Wouldn’t routing try to route that to controller="images", action="jpegs", id="foo.jpg" now?

The decision we made in this case was that by default, routing should not apply to files on disk. That is, routing now checks to see if the file is on disk before attempting to route (via the Virtual Path Provider).

If the file is on disk, we pop out and don’t route and let the web server handle the request normally. If the file doesn’t exist, we attempt to apply routing. This makes sure we don’t screw around with requests for static resources on disk. Of course, this default can be changed by setting the property RouteTable.Routes.RouteExistingFiles to be true.

Why Blog This Now?

The Dynamic Data team is scooping the MVC team on these routing changes. ;) They are releasing a preview of their latest changes to Dynamic Data which includes using our new routing dll.

Check out ScottGu’s post on the subject. I really feel Dynamic Data is the most underrated new technology coming out from the ASP.NET team. When you dig into it, it is really cool. The “scaffolding” part is only the tip of the iceberg.

Installing the Dynamic Data preview requires installing the routing assembly into the GAC. If you install this, it may break existing MVC Preview 2 sites because the assembly loader favors the GAC when the assembly is the same version. And the routing assembly is the same version as the one in Preview 2.

The Dynamic Data Preview readme has the steps to update your MVC Preview 2 project to work with the new routing. You’ll notice that the readme recommends having a Default.aspx file in the root which redirects to /Home. Technically, the Default.aspx file in the root won’t be necessary in the final release because of the suggested routing changes (it is necessary now due to a minor bug). Unfortunately, Cassini doesn’t work correctly when you make a request for "/" and there is no default document. It doesn’t run any of the ASP.NET Modules. So we kept the file in the root, but you can will be able to remove it when deploying to IIS 7. So to recap this last point, the Default.aspx in the project root is to make sure that pre-SP1 Cassini works correctly as well as IIS 6 without star mapping. It's will not be needed for IIS 7 in the future, but is needed for the time being.

We will have a new CodePlex source code push in a couple of weeks with an updated version of MVC that supports the new routing engine.

My First IronRuby Unit Test Spec For ASP.NET MVC

Way down the road, it would be nice to be able to build ASP.NET MVC applications using a DLR language such as IronRuby. However, enabling DLR language support isn’t free.

There are going to be places in our design that are specific to statically typed languages (such as Attribute based filters) that just wouldn’t work (or would be too unnatural) with a dynamic language.

Ideally we can minimize those cases, and for the ones we can’t, we need to make sure the extensibility of the framework allows for extending the system in such a way that we can provide a DLR friendly version of that feature.

How do we identify and minimize such hot spots? Design reviews help, but only goes so far. There is nothing like executing code to highlight issues. So in collaboration with some of the DLR team members, I’ve been exploring the minispec framework used to test IronRuby and wrote my first testspec tonight. Check it out. (NOTE: line breaks added in the require statements so it fits within the width of my blog)

require File.dirname(__FILE__) + '/../../spec_helper'
require 'System.Web.Abstractions
, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
require 'System.Web.Routing
, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
require 'System.Web.Mvc
, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

describe "Route#<<" do
 
  it "can create RouteCollection which is empty" do
    rc = System::Web::Routing::RouteCollection.new
    rc.count.should == 0
  end
  
  it "can add route to RouteCollection" do
    rc = System::Web::Routing::RouteCollection.new
    r = System::Web::Routing::Route.new "", nil
    rc.add "route-name", r
    
    rc.count.should == 1
  end

end

And here is the result so far.

Yay! Two passing tests.

Yeah, the tests are really really simple so far, but hey, this is just my first step. I need to get familiar with the minispec framework. Not only that, but I haven’t written any Ruby code in a long while. Fortunately I do have a copy of The Ruby Way on my shelf, which should help.

I probably have much higher priority items on my plate that I could be working on, but sometimes you have to treat yourself to a little fun. Besides, I am doing this on my own time right now. :)

Unit Test Project Structure Poll

When I build applications, I personally like to have my unit tests in a separate class library project than the application I am testing. That’s just how I roll.

I just assumed this is how everyone structures their unit tests, but I’ve talked with some people who have a deep history in TDD who put their code in the same project.

So I wanted to create a simple poll to find out how people are actually structuring their unit tests. I’ve been involved in various internal discussions looking at how Microsoft design, tools, code can better support unit testing across the board.

Note that this is not an opinion question. I don’t think one way is “more right” than another. I just want to make sure we have an accurate view of what the real practice is out there.

So please do answer this poll and encourage others to do so. I’d hate for internal teams to make choices based on a wrong assumption. Thanks!

Interview With Brad Wilson On Microsoft And Open Source

There’s a great interview on the How Software is Built blog with Brad Wilson, a developer in Microsoft’s OfficeLabs team, but probably better known for his work on xUnit.net, CodePlex, and ObjectBuilder.

What I particularly liked about this post was the insight Brad provides on the diverse views of open source outside and inside of Microsoft as well as his own personal experience contributing to many OSS projects. It’s hard for some to believe, but there are developers internal to Microsoft who like and contribute to various open source projects.

Another reason that Brad gets a thumbs up in my book (along with Jim Newkirk) is that xUnit.net recently updated their installer to include test project integration with ASP.NET MVC Preview 2.

Subtext Awakens From Its Slumber

It’s been all quiet on the Subtext front for a while. While I think many open source projects face the occasional lull, Subtext was hit by a Perfect Storm of inactivity.

This was mostly because several of the key developers all ended up having job changes (and moves) around the same time. For me, the move to Microsoft and up to the Seattle area took up a lot of my time and energy.

I finally feel settled in so I fired up the old TortoiseSVN client and got latest from the tree excited to see what new goodness people checked in during my absence.

Ok, that’s not exactly true, but captures the spirit of the truth.

In any case, now that I’m mostly settled into lovely Bellevue, WA, I decided to spend a bit of time last week working on Subtext. I’m now swamped again, but I got a few key fixes in already which I’m happy about.

We decided to scale back the 2.0 release a bit. We were a bit too ambitious with the feature set and supporting two branches became way to time consuming. So we replaced the trunk with the 1.9 branch and all new development is in the trunk where it should be.

The next version will still target ASP.NET 2.0, but after that, we want to have fun with this so we’ll start to target ASP.NET 3.5. I won’t go into the feature list for 2.0 right now. The bulk of the work is on bug fixes, small tweaks and improvements, and infrastructure improvements. I feel like one of the best parts of Subtext is our build process and continuous integration server.

Since we decided to label the next version 2.0, we will be adding a few new hotly requested features. It should be a nice release. Sorry it’s been so quiet for so long, but the engine is back up and running full speed ahead.