October 2012 Blog Posts
In my last post, I talked about the MonkeySpace conference conference and how it reflects positive trends in the future of open source in .NET. But getting to a better future is going to take some work on our part. And a key component of that is making NuGet better.
Several discussions at MonkeySpace made it clear to me that there is some pervasive confusion and misconceptions about NuGet. It also made it clear that there are some dramatic changes needed for NuGet to continue to grow into a great open source project. In this post, I’ll cover some of these misconceptions and draw an outline of what I hope to see NuGet grow into.
Myth: NuGet is tied to Visual Studio and Windows
This is only partially true. The most popular NuGet client is clearly the one that ships in Visual Studio. Also, NuGet packages may contain PowerShell scripts. PowerShell is not currently available on any other operating system other than Windows.
However, the architecture of NuGet is such that there’s a core assembly, NuGet.Core.dll, that has no specific ties to Visual Studio. The proof of this is in the fact that ASP.NET Web Pages and Web Matrix both have NuGet clients. In these cases, the PowerShell scripts are ignored. Most packages do not contain PowerShell scripts, and those that do, the changes the scripts make are often optional or easily done manually.
In fact, there’s a NuGet.exe which is a wrapper of NuGet.Core.dll that runs on Mono. Well sometimes it does; and this is where we need your help! So far, Mono support for NuGet.exe has been low priority for the NuGet team. But as I see the growth of Mono, I think this is something we want to improve. My co-worker, Drew Miller (also a former member of the NuGet and ASP.NET MVC team) is keen to make better Mono support a reality. Initially, it could be as simple as adding a Mono CI server to make sure NuGet.exe builds and runs on Mono. Ultimately, we would like to build a MonoDevelop plugin.
Initially, it will probably simply ignore PowerShell scripts. There’s an existing CodePlex work item to provide .NET equivalents to Install.ps1 and the other scripts.
I created a personal fork of the NuGet project under my GitHub account at http://github.com/haacked/nuget. This’ll be our playground for experimenting with these new features with the clear goal of getting these changes back into the official NuGet repository.
Myth: NuGet isn’t truly Open Source
This is an easy myth to dispel. Here’s the license file for NuGet. NuGet is licensed under the Apache version 2 license, and meets the Open Source Definition defined by the Open Source Initiative. The NuGet team accepts external contributions as well, so it’s not just open source, but it’s an open and collaborative project.
But maybe it’s not as collaborative as it could be. I’ll address that in a moment.
Myth: NuGet is a Microsoft Project
On paper, NuGet is a fully independent project of the Outercurve Foundation. If you look at the COPYRIGHT.txt file in the NuGet source tree, you’ll see this:
Copyright 2010 Outercurve Foundation
Which makes me realize, we need to update that file with the current year, but I digress! That’s right, Microsoft assigned the copyright over to the Outercurve foundation. Contributors are asked to assign copyright for their contribution to the foundation as well. So clearly this is not a Microsoft project, right?
Well if you look at the entry in the Visual Studio Extension Manager (or the gallery), you’ll see this:
Huh? What gives? Well, it’s time for some REAL TALK™.
There’s nothing shady going on here. In the same way that Google Chrome is a Google product with its own EULA that incorporates the open source Chromium project, and Safari is an Apple product with its own EULA that incorporates the open source WebKit project, the version of NuGet included in Visual Studio 2012 is officially named the NuGet-Based Microsoft Package Manager and is a Microsoft product with its own EULA that incorporates the open source NuGet project. This is a common practice among companies well known for shipping “open source” and all complies with the terms of the license. You are always free to build and install the Outercurve version of NuGet into Visual Studio should you choose.
Of course, unlike the other two examples, NuGet is a bit confusing because both the proprietary version and the open source version contain the word “NuGet.” This is because we liked the name so much and because it had established its identity that we felt not including “NuGet” in the name of the Microsoft product would cause even more confusion. I almost wish we had named the open source version “NuGetium” following the Chromium/Chrome example.
This explains why NuGet is included in the Visual Studio Express editions when it’s well known that third party extensions are not allowed. It’s because NuGet is not included, it’s NuGet-Based Microsoft Package Manager that’s included.
NuGet is not a Community Project
Ok, this claim is a toss-up. As I pointed out before, NuGet is truly an open source project that accepts external community contributions. But is it really a “community project”
As the originator of the project, the sole provider of full time contributors, and a huge benefactor of the Outercurve Foundation; Microsoft clearly wields enormous influence on the NuGet project. Also, more and more parts of Microsoft are realizing the enormous potential of NuGet and getting on board with shipping packages. NuGet is integrated into Visual Studio 2012. These are all great developments! But it also means lessens the incentive for Microsoft to give up any control of the project to the community at large.
So while I still maintain it is a community project, in its current state the community’s influence is marginalized. But this isn’t entirely Microsoft’s intention or fault. Some of it has to do with the lack of outside contributors. Especially from those who have built products and even businesses on top of NuGet.
My Role With NuGet
Before I talk about what I hope to see in NuGet’s future, let me give you a brief rundown of my role. From the Outercurve perspective, I’m still the project lead of NuGet, the open source project. Microsoft of course has a developer lead, Jeff Handley, and a Program Manager, Howard Dierking, who run the day to day operations of NuGet and manage Microsoft’s extensive contributions to NuGet.
Of course, since NuGet is no longer a large part of my day job, it’s been challenging to stay involved. I recently met with Howard and Jeff to figure out how my role fits in with theirs and we all agreed that I should stay involved, but focus on the high level aspects of the project. So while they run the day to day operations such as triage, feature planning, code reviews, etc. I’ll still be involved in the direction of NuGet as an independent open source project. I recently sat in on the feature review for the next couple of versions of NuGet and will periodically visit my old stomping grounds for these product reviews.
The Future of NuGet
Over time, I would like to see NuGet grow into a true community driven project. This will require buy-in from Microsoft at many levels as well as participation from the NuGet community.
In this regard, I think the governance model of the Orchard Project is a great example of the direction that NuGet could head in. In September of 2011, Microsoft transferred control of the Orchard project to the community. As Bertrand Le Roy writes:
Back in September, we did something with Orchard that is kind of a big deal: we transferred control over the Orchard project to the community.
Most Open Source projects that were initiated by corporations such as Microsoft are nowadays still governed by that corporation. They may have an open license, they may take patches and contributions, they may have given the copyright to some non-profit foundation, but for all practical purposes, it’s still that corporation that controls the project and makes the big decisions.
That wasn’t what we wanted for Orchard. We wanted to trust the community completely to do what’s best for the project.
Why didn’t NuGet follow this model already? It’s complicated.
With something so integrated into so many of areas of Microsoft now, I think this is a pretty bold step for Microsoft to take. It’ll take time to reach this goal and it’ll take us, the community, demonstrating to Microsoft and others who are invested in NuGet’s future that we’re fit and ready to take on this responsibility.
As part of that, I would love to see more corporate sponsors of NuGet supplying contributors. Especially those that profit from NuGet. For example, while GitHub doesn’t directly profit from NuGet, we feel anything that encourages open source is valuable to us. So Drew and I will spend some of our time on NuGet in the upcoming months. The reason I don’t spend more time on NuGet today is really a personal choice and prioritization, not because I’m not given work time to do it since I pretty much define my own schedule.
If you are a company that benefits from NuGet, consider allotting time for some of your developers to contribute back (or become a sponsor of the Outercurve Foundation). Consider it an investment in having more of a say in the future of NuGet should Microsoft transfer control over to the community. NuGet belongs to us all, but we have to do our part to own it.
At the end of last year, I wrote a blurb about the Open Source Fest event at Mix 2011. Imagine the typical exhibition hall, but filled with around 50 open source projects. Each project had a station in a large room where project members presented what they were working on to others. You could see the gleam of inspiration in the smiles of developers as they exchanged ideas and suggestions. I left this event completely fired up.
This is the spirit we tried to capture with the MonkeySpace conference this year. And at least for me, it succeeded. I’m fired up! There was much sharing of ideas, some of which I’ll talk about in subsequent blog posts. One such idea is I hope we add something like the open source exhibition hall in a future MonkeySpace.
MonkeySpace speakers dinner. Lots of idea sharing going on.
MonkeySpace is a rebranded and refocused Monospace conference. While MonoSpace dealt mostly with Mono, the goal of Monkeyspace is to put the spotlight on .NET open source everywhere, not just on Mono. Obviously Mono is a big part of that. But so is Microsoft. But most of all, the many small “labor of love” projects from those in the .NET OSS community are a big part of this.
MonkeySquare
Before I go further, I want to briefly describe the relationship of MonkeySpace to me, Mono, and MonkeySquare (the website is, as the cliché goes, under construction).
As I mentioned already, MonkeySpace is a rebranded Monospace conference, but with a new focus. Dale Ragan and others created a non-profit called MonkeySquare with the goal to “evangelize cross platform and open-source development in .NET.”
I got involved when Dale asked me to be on the board of directors for the organization. I admit, I was a bit hesitant at first as I tend to overcommit and I need my afternoon naps, but the mission resonated with me. I suggested he invite Scott Hanselman because he’s a force to be reckoned with and a fountain of ideas. And he has a big forehead which has to count for something.
As we started to have board meetings, it became clear that we wanted the next MonoSpace to become MonkeySpace. Due to the herculean efforts of Dale Ragan and others like Joseph Hill of Xamarin, this became a reality. We kept the first MonkeySpace small, but we hope to grow the conference as a premier place for .NET open source developers to share ideas and grow the ecosystem.
Pessimism turns to optimism
In recent times, there’s been some pessimism around .NET open source. There’s the occasional rustle of blog posts declaring that someone is “leaving .NET”. There’s also this perception that with Windows 8, the Windows team is trying its best to relegate .NET into the dustbin of legacy platforms. I don’t necessarily believe that to be the case (intentionally), but I do know that many .NET developers feel disillusioned.
But here’s the thing. The .NET ecosystem is becoming less and less solely dependent on Microsoft and this is a good thing.
OSS Fills the Gaps
An example I like to point to is when WPF was first released, support for presentation separation patterns (such as MVP or MVC) was non-existent and there was much gnashing of teeth. It didn’t take long before small open source projects such as Caliburn Micro sprung into existence to fill in the gaps. This by no means excuses some of the design choices that still plague developers who want to write testable WPF applications, but it certainly makes a bad situation much better.
In his keynote at MonkeySpace, Miguel de Icaza told a story that is another dramatic illustration of this phenomena. Microsoft XNA is a toolkit for building PC and X-Box games using .NET. But for whatever reasons, Windows RT does not support running XNA applications. You can still write an XNA game for Windows 8, but it won’t run on the Windows RT devices.
Once again, the open source community comes to the rescue with MonoGame. Here’s a blurb from their project page, emphasis mine:
MonoGame is an Open Source implementation of the Microsoft XNA 4 Framework. Our goal is to allow XNA developers on Xbox 360, Windows & Windows Phone to port their games to the iOS, Android, Mac OS X, Linux and Windows 8 Metro. PlayStation Mobile development is currently in progress.
Interesting! MonoGame makes it possible to take your existing XNA based X-Box game and with a tiny bit of effort, port it on Windows RT. Think of the implications.
A cornerstone of the Windows 8 strategy is to entice a new developer audience, web developers, to build client Windows applications with a development experience that allows them to leverage their existing JavaScript and HTML skills. Nevermind the fact that they ignore the culture and idioms of these communities, replacing such idioms with Windows specific conventions that are awkward at best.
Ironically, something like MonoGame may be better positioned to realize this strategy for Microsoft in the short term than Microsoft’s own efforts!
As an example, Miguel talked about the developers of Bastion, an X-Box and PC game, and how they used MonoGame to port the game to the iPad. Now that same developer could easily port the game to Windows RT should they choose to. Before MonoGame, they might not have considered this option.
Miguel suggested that the majority of applications in the Windows 8 app store are C# applications. This only makes sense because the attempt to bring over web developers is a long lead strategy. Meanwhile, C# developers have been Microsoft’s bread and butter for a while now and are Microsoft’s to lose. They should not take them lightly and one would hope, if these numbers are true, that this has the attention of the Windows folks.
It certainly makes me excited about the potential for C# to become, as Miguel calls it, the lingua franca for cross device applications.
So despite the pessimism, my discussions at MonkeySpace leave me optimistic that .NET and .NET OSS is here to stay. There’s a lot of good OSS stuff coming from Xamarin, independents, and various teams at Microsoft such as the efforts from the MS Open Tech initiative and all the stuff that the Windows Azure team is churning out. But even if Microsoft started to deemphasize .NET, I believe .NET would endure because the community will continue to fill in the gaps so that the ecosystem abides.
With Reactive Extensions you sometimes need one observable sequence to run after another observable sequence completes. Perhaps the first one has side effects the second one depends on. Egads! I know, side effects are evil in this functional world, but it happens.
Let’s make this more concrete with some contrived sample code.
public static class BlogDemo
{
public static IObservable<int> Extract()
{
return new[] { 10, 20, 70, 100 }.ToObservable();
}
public static IObservable<string> GetStuffs()
{
return new[] { "Family Guy", "Cheetos", "Rainbows" }.ToObservable();
}
}
Here we have a class with two methods. The first method extracts something and returns a sequence of progress indicators. The second method returns an observable sequence of strings (good stuffs I hope). With me so far?
Now suppose I need to write a third method, let’s call it GetStuffsAfterExtract. This method needs to run Extract , and only when that is complete, return the sequence from GetStuffs. How would we approach this?
Well, in the Task based world, Extract would probably return a Task<T> instead of a observable. Task<T> represents a single future value as opposed to a sequence of future values. If we did that, we could use the ContinueWith method (or simply use the await keyword it in .NET 4.5).
But I don’t live in that world. I live in shiny happy RxLand where we always deal with sequences. Future sequences! It’s an awesome zany world.
Note that this method I want to write doesn’t care about the actual sequence of values from Extract. All I care to know is when it’s complete and then it will return the sequence of values from GetStuff.
Here’s one way to do it with Observable.Start:
public static IObservable<string> GetStuffsAfterExtract()
{
return Observable.Start(() =>
{
Extract().Wait();
return GetStuffs();
}).Merge();
}
This works, but it’s not optimal. The use of Observable.Start guarantees we’ll have a context switch to the TaskPool rather than doing the operation on the current thread.
Also, it’s ugly.
Let’s try again:
public static IObservable<string> GetStuffsAfterExtract()
{
return Extract().TakeLast(1).SelectMany(_ => GetStuffs());
}
A little better. This works pretty well in most situations. If you’re wondering what the underscore character is in the SelectMany statement, that’s the name for lambda parameters I use to indicate that the parameter is not needed. It’s a convention I learned from someone a while back on the ASP.NET team. It makes my intention to not use it in the expression clear..
But what happens if there’s a case where Extract legitimately returns an empty observable, aka Observable.Empty<int>(). In this case, I could just change it to not do that since I wrote it. But maybe I’m calling a method written by someone else and we don’t trust that person to do everything perfect like me.
Well GetStuffs will never get called because SelectMany projects each element of the second sequence onto the first. If there are no elements in the first, there’s nothing for it to do. Hey, maybe that’s exactly what you want!
But that’s not what I want in this case. So with the help of my co-worker Paul “Rx Master of Disaster” Betts, we went back and forth through several iterations.
I figured the first step is to simply write a method that represents the completion of an observable sequence whether it’s empty or not. I’ll call this method AsCompletion and it’ll return a new sequence with a single Unit.Default when the original sequence is complete. It turns out that the Aggregate method is great for this (just like in standard Linq!):
public static IObservable<Unit> AsCompletion<T>(this IObservable<T> observable)
{
return observable.Aggregate(Unit.Default, (accumulation, _) => accumulation);
}
Aggregate is typically used to aggregate a sequence into a single value. It’s also known as an accumulation function. But in this case, I don’t care about any of the individual values. I simply keep returning the accumulation unchanged.
The first parameter to Aggregate is a seed value. Since I seeded that accumulation with a Unit.Default, it’ll keep returning that same value.
In other words, this method will return a sequence of exactly one Unit.Default when the sequence it’s called upon is complete whether it’s empty or not. Cool.
Now I can use this to build my ContinueAfter method (I didn’t name it ContinueWith because we don’t actually do anything with the previous values and I want to make sure it’s clear we’re talking about doing work after the sequence is complete and not as things come in).
public static IObservable<TRet> ContinueAfter<T, TRet>(
this IObservable<T> observable, Func<IObservable<TRet>> continuation)
{
return observable.AsCompletion().SelectMany(_ => continuation());
}
You’ll notice that the body of this method looks pretty similar to my first attempt, but instead of TakeLast(1) I’m just using the AsCompletion method.
With this method in place, I can rewrite the code I set out to write as:
public static IObservable<string> GetStuffsAfterExtract()
{
return Extract().ContinueAfter(GetStuffs);
}
That is much more lickable code. One nice thing I like about this method is it takes in a parameterless Func. That makes it very clear that it won’t pass in a value to your expression that would then want to ignore and in this case allows me to pass in a method group.
I write this full well knowing someone who’s a much better Rx master than myself will point out an even better approach. I welcome it! For now, this is working pretty well for me.
Oh, I almost forgot. I posted the unit tests I have so far for this method as a gist.
UPDATE 10/9/2012 Just as I expected, folks chimed in with better ideas. Some asked why didn’t I just use Concat since it’s perfect for this. The funny thing is I did think about using it, but I dismissed it because it requires both sequences to be of the same type, as someone pointed out in my comments.
But then it occurred to me, I’m using Rx. I can transform sequences! So here’s my new ContinueAfter implementation.
public static IObservable<TRet> ContinueAfter<T, TRet>(
this IObservable<T> observable
, Func<IObservable<TRet>> continuation)
{
return observable.Select(_ => default(TRet))
.IgnoreElements()
.Concat(continuation());
}
I also updated the AsCompletion method since I use that in other places.
public static IObservable<Unit> AsCompletion<T>(this IObservable<T> observable)
{
return observable.Select(_ => Unit.Default)
.IgnoreElements()
.Concat(Observable.Return(Unit.Default));
}
Please note that I didn’t have to change ContinueAfter. I could have just changed AsCompletion and I would have been ok. I just changed it here to show I could have written this cleanly with existing Rx operators. Also, and I should test this later, it’s probably more efficient to have one Concat call than two.
I added another unit test to the gist I mentioned that makes sure that the second observable doesn’t run if the first one has an exception. If you still want it to run, you can catch the exception and do the right thing.
UPDATE 10/10/212 Ok, after some real world testing, I’m finding issues with the Concat approach. Another commenter, Benjamin, came forward with the most straightforward approach. It’s one I originally had, to be honest, but wanted to try it in a more “functional” approach. But what I’m doing is definitely not functional as I’m dealing with side effects.
Here’s my final (hopefully) implementation.
public static IObservable<Unit> AsCompletion<T>(this IObservable<T> observable)
{
return Observable.Create<Unit>(observer =>
{
Action onCompleted = () =>
{
observer.OnNext(Unit.Default);
observer.OnCompleted();
};
return observable.Subscribe(_ => {}, observer.OnError, onCompleted);
});
}
public static IObservable<TRet> ContinueAfter<T, TRet>(
this IObservable<T> observable, Func<IObservable<TRet>> selector)
{
return observable.AsCompletion().SelectMany(_ => selector());
}
When someone says they want to write a technical book, I take a hammer and slam it on the aspiring author’s thumb and ask “How do you like that?” If the answer is, “Very much! May I have another.” This person is ready to write a technical book.
Sure, writing a book always starts off full of exciting possibilities. But it soon devolves into drudgery and pain with the constant question of whether the time spent is worth it when weighed against all your other obligations and opportunities in life.
But no matter how much I sometimes hate the process of writing a book, I do love this moment. The moment the fruits of your labor are delivered in a cardboard box and you open it up to see your name on a glossy cover with a speedy looking bobsled!
But soon after the big question sets in, “what the hell am I going to do with all these books?!” (I’ll bring some to MonkeySpace in October! And maybe I’ll give a few away via the internet if I can think of a criteria for giving them out.)
At long last, the Wrox Professional ASP.NET MVC 4 book written by me and my esteemed coauthors, Jon Galloway, K. Scott Allen, and Brad Wilson, is available (in Kindle and Print formats) on Amazon.
As Jon points out in his blog post, the Kindle version looks nice and is in color (on devices that support color of course).
Real World Development
This book includes a new chapter that I really did enjoy writing. The focus on the chapter is building a real world ASP.NET MVC application using the NuGet Gallery as a case study. The NuGet Gallery is the code behind http://nuget.org/, the online public host for thousands of NuGet packages with millions of package installs.
This is a great case study because it’s easy to visit the site while you read the chapter and you can easily grab the source code. The chapter covers the packages used to build the gallery as well as some of the trade-offs the NuGet gallery team made. For example, there are some places where we rolled our own features rather than using the built in defaults.
Ironically, because we haven’t upgraded the NuGet Gallery yet, the content of the chapter describes an ASP.NET MVC 3 project. But all of the concepts and packages apply equally well to an ASP.NET MVC 4 project.
If you’re building real web applications with ASP.NET MVC, I think you’ll find at least one thing useful in this chapter. Hopefully more!
Credits
I really must give a lot of credit to my coauthors who really knocked it out of the park. These are some smart folks I have the pleasure of working with. Extra kudos to Jon Galloway who was more or less the project manager and chief cheerleader in organizing the rest of us to get this book out the door.
I also want to give a big thanks to Eilon Lipton, our technical reviewer. He’s the developer lead still responsible for the developer team that works on ASP.NET MVC and Web API and is a meticulous reviewer.
What’s Next?
ASP.NET MVC 4 is the last version of ASP.NET MVC that I had any involvement with. I don’t plan to be a coauthor on the ASP.NET MVC 5/Web API 2 book at this point. I’m just not likely to be involved enough to have the level of insight that I’ve had in the past. Also, it’s really hard work.
So you should buy this book because it’ll be a collector’s item! Or, you can take the advice of another former coauthor of mine who said Don’t Buy This Book right after we finished one together. Either way, I won’t hold it against you. (But do buy it anyways just to spite Atwood.)
Will I write another book again? Maybe, but if I do it won’t be like any of the books I’ve worked on in the past. I have some ideas to write a fiction piece about a shimmery vampire wizard who goes to a school named PigPimples under the control of a spoiled brat king named Jeffrey who hates his midget uncle Teeryon Bannister. I even have a catchy phrase, a Bannister always pays his dates.
If it somehow turns out that writing fiction doesn’t work out, I might be interested in writing a book that explores more of the people aspects of software development or related to core principles. Something that doesn’t change and get outdated every year.
Or perhaps I’ll just keep randomly slamming keys on my keyboard and publish the result on my blog and call it a blog post. At least I still have fun doing that.