Dealing with Multiplatform Project Files

Nov 7, 2013 oss xplat octokit suggest edit

Octokit.net targets multiple platforms. This involves a large risk to my sanity. You can see the general approach here in the Octokit directory of our project:

Mono gets a project! MonoAndroid gets a project file! Monotuch gets a project file! Everybody gets a project file!

Each of these projects references the same set of .cs files. When I add a file to Octokit.csproj, I have to remember to add that file to the other four project files. As you can imagine, this is easy to forget.

It’s a real pain. So I opened up a feature request on FAKE, the tool we use for our build (more on that later) and asked them for a task that would fail the build if another project file in the same directory was missing a file from the “source” project file. I figured this would be something easy for F# to handle.

The initial response from the maintainer of FAKE, Steffen Forkman, was this:

What you need is a better project system ;-)

Touché!

This problem (along with so many project file merge conflicts) would almost completely go away with file patterns in project files. I’ve been asking for this for a long time (I asked the Visual Studio team for this the day I joined Microsoft, or maybe it was the first month, I don’t recall). There’s a User Voice item requesting this,go vote it up! (Also, go vote up thisplatform restriction issuethat’s affecting Octokit.net as well)

In any case, sorry to say unlimited chocolate fountains don’t exist and I don’t have the project system I want. So let’s deal with it.

A few days later, I get this PR to octokit.net. When I ran the build, I see the following snippet in the build output.

Running build failed.
Error:
System.Exception: Missing files in  D:\Octokit\Octokit-MonoAndroid.csproj:
Clients\OrganizationMembersClient.cs

That’s telling me that somebody forgot to add the class OrganizationMembersClient.cs to the Octokit-MonoAndroid.csproj. Wow! Isn’t open source grand?

A big thanks to Steffen and other members of the FAKE community who pitched in to build a small but very useful feature. In a follow-up post, I’ll write a little bit about why we moved to using FAKE to build Octokit.net.

Update

I opened an issue to take this to the next step. Rather than just verify the project files, I want some way to automatically modify or generate them.

Update 2

FAKE just got even better with the new FixProjects task! For now, we’ve added this as an explicit command.

.\build FixProjects

Over time, we may just integrate this into the Octokit.net build directly.

Found a typo or mistake in the post? suggest edit

Comments

24 responses

khalidabuhakmeh • November 7th, 2013
Why not just generate the csproj files from a manifest or pattern file? That way you manage a parent manifest then just create a script file called "generate_projects". The cool thing is, you could do this using Ruby or ScriptCs quiet easily. You could also integrate this generation of csproj files into your build process so that on each check in you could have your build server generate them.

Just a thought, I could be totally off base here.
jdhardy • November 7th, 2013
Is there a reason you use a project-per-platform rather than one project with different configurations?
ozzymcduff • November 7th, 2013
https://github.com/Lundalog... to add files automatically or use a branch of albacore
haacked • November 7th, 2013
That's probably the next step. I just learned about gyp (https://code.google.com/p/g... and may try that out. The key thing is, I need to make it easy for contributors and they're going to want to add a file using VS. I don't want them having to bother with manifests etc.
khalidabuhakmeh • November 7th, 2013
I saw your conversation on twitter about it. I think you could use the current notification process and gyp. A two pronged approach. Good luck!
Yves Reynhout • November 7th, 2013
Dude, csproj is just xml. Xslt all the things, easy.
haacked • November 7th, 2013
They create their own world of pain.

As @paulcbetts tells me,

"Nobody understands them, they're hard to edit, WinRT and WP8 love to fuck with them, it's hard to build multiple configurations at once..."

"VS always magically gets stuck on the wrong one, so then people are like "Why didn't X platform build??""
Mike McG • November 7th, 2013
(Minor note: First link in the article is broken.)
Nik Molnar • November 7th, 2013
Couldn't you just use wild card includes in the .csproj file?

Thats what Glimpse does with XML that looks like this:

<compile include="..\**\*.cs" exclude="Properties\AssemblyInfo.cs;">
      <link/>
    </compile>
    <compile include="Properties\AssemblyInfo.cs"/>

New files are automatically picked up no problem.

(We also have project specific AssemblyInfo.cs files so we don't include those...)
mythz • November 7th, 2013
ServiceStack generates all its Custom builds and signed projects with a single node.js script that merges an empty project template with the files references in the master .csproj to generate the custom project .csprojs we use to build from:
https://github.com/ServiceS...
Stuart Hillary • November 7th, 2013
An approach I use is to place all your common project properties and code files in a separate msbuild project that is imported by each project.

For example, you could have an msbuild project called Common.props in the same directory as your other project files that contained something like:

<project defaulttargets="Build" xmlns="http://schemas.microsoft.co...">
<propertygroup>
<rootnamespace>Octokit</rootnamespace>
<assemblyname>Octokit</assemblyname> <defineconstants>TRACE;CODE_ANALYSIS;SIMPLE_JSON_OBJARRAYINTERNAL;SIMPLE_JSON_INTERNAL;NET_45</defineconstants\>
...
</propertygroup>
<itemgroup>
<externalcompile include="..\SolutionInfo.cs">
<link>Properties\SolutionInfo.cs</link>
</externalcompile>
<externalcompile include="Clients\ActivitiesClient.cs"/>
<externalcompile include="Clients\AssigneesClient.cs"/>
<externalcompile include="Clients\CommitStatusClient.cs"/>
...
</itemgroup>
<choose>
<when condition=" '$(Configuration)' == 'Debug' ">
<propertygroup>
<defineconstants>$(DefineConstants);DEBUG</defineconstants>
...
</propertygroup>
</when>
</choose>

Instead of having entries in this script for each .cs file you could make a list using a wildcard instead.

To use this msbuild file in your projects use an import at the start of each csproj

<import project=".\Common.props"/>

and an item group further down in the file

<itemgroup>
<compile include="@(ExternalCompile)"/>
</itemgroup>

Any other properties and items that are common to each project could be moved to this Common.props file.

The downside to this approach is that if you add a new file in Visual Studio the reference is placed inside the csproj file rather than the Common.props file (if you are running the FAKE task it should still pick up the problem though, you just have to move the reference into just one msbuild script rather than several csproj files).

EDIT. The xml seems to be a bit screwed above.
jdhardy • November 7th, 2013
Agree on the pain. Single project/multiple configs is how IronPython is built. It has over a dozen projects, so one per platform isn't really workable. Still trying to come up with a better solution.
haacked • November 7th, 2013
Wait, that works already? Damn!
haacked • November 7th, 2013
Ah, so I asked @paulcbetts about that and he notes:

So, Wildcards work, but VS gets really slow since it's scanning directories constantly, *and* any time someone adds a file in the normal way, it gets fucked

Or rather, "Add new File..." will break your csproj.

We want contributors to be able to add files to the solution using Visual Studio in the normal way. Does that work?
Manuel Grundner • November 8th, 2013
Sry was not ment to be a reply
Manuel Grundner • November 8th, 2013
What about http://visualstudiogallery....
Syncronizes the changes in multiple project files.
Matt Davey • November 8th, 2013
This is the approach that I take in my projects, I've never noticed any particular performance degradation.

Adding files manually in visual studio does cause issues - seems like VS only has half-hearted support for this scenario, definitely room for improvement!
Steffen Forkmann • November 8th, 2013
You've Got Mail

See https://github.com/octokit/...

Docs at: http://fsharp.github.io/FAK...
Nik Molnar • November 8th, 2013
Like @disqus_asZ0mfCnmj:disqus , I've not noticed any particular performance issues - but I also don't doubt that it could happen. (Particularly at Paul's word.)

VS CERTAINLY will screw up the wildcards if "Add new file" is used. We solve that by having the "main" project that people use being a "normal" project file. The wildcard project files are then only used during the build.

We separate out our wildcard projects into a separate solution folder, but you could also put them in a *.Build.sln or not have them referenced by any solution at all - just by your build script.
Eric Falsken • November 8th, 2013
you can also do this by editing the project file manually to include all of the files from another project file. MSBuild has the tools that you need. It's just Visual Studio that doesn't present all possible options. Creating a PS script to create an msbuild file listing of all of the CS files in your directory is simple simple simple.
Eric Falsken • November 8th, 2013
Use a PS script to "extract" the files from a single "source" project and create the manifest automatically and import them into all the other projects with a simple edit to your .csproj file.
haacked • November 8th, 2013
Ah, I like the source project idea! The FAKE team just added a FixProjects task that solves this problem for us as well. :)
Nik Molnar • November 8th, 2013
Awesome! Fake looks pretty good. I'm looking forward to your next post about it. (Wasn't a fan of your last post. :P)
nho biet yeu • November 12th, 2013
Is there a reason you use a project-per-platform rather than one project with different configurations? - tai opera mini mien phi