What Does My Testing MailServer Test That A Mock Would Not?

Greg Young takes my Testing Mail Server to task and asks the question, what does it test that a mock provider doesn’t?

It is a very good question and as he points out in his blog post on the subject, it seems like a lot of overhead for very little benefit. For the most part, he is right.

To my defense, and as Greg points out, I would not start with such a test when writing email functionality. I would start with the mock email provider and follow the typical Red-Green TDD pattern of development. However there are cases where this approach does not test enough and this testing server was necessitated by some real world scenarios I ran into.

For example, in some situations, it is very important to understand the exact format of the raw SMTP message that is sent. Some systems actually use email from server to server to kick off automated tasks. In that situation, it helps to know that the SMTP message is formatted as expected by the receiving server. For example, you may want to make sure that the appropriate headers are sent and that the message is not a multi-part message. This approach lets you get at the raw SMTP message in a way that the mock provider approach cannot.

A more common issue is when sending mass mailings such as newsletters to subscribers. At one former employer, we had real difficulty getting our emails to land in our user’s mailbox despite adhering to appropriate SPAM laws and only mailing to subscribed users.

It turns out that actually landing a mass mailing even to users who want the email is very tough when dealing with Hotmail, yahoo, and AOL accounts. Something as seemingly innocuous as the X-Mailer header value can trigger the spam filters.

In this case, this very much falls under the rubrik of an integration test, as I am testing the actual mailing component in use. But I am not only testing the particular mailing component. I am also testing that my code uses the mailing component in a correct manner.

So in answer to the question, Where’s the red bar? The red bar comes into play when I write my unit test and asser that the X-Mailer header is missing. The green bar comes into play when I make sure to remove the header. I could probably test this with a mock object as well, but I have been burnt by mailing components that did not remove the X-Mailer header but simply set the value to blank, when I really intended it to be removed. That is not something a mock object would have told me.

Comments

5 responses

Greg Young • May 31st, 2006
As I had said, it seems like it could have some great uses in integration testing.
In the specific case you mention though I find it is in fact TDD showing you a design problem. X-Mailer is not part of the SMTP protocol, it is part of MIME http://www.bath.ac.uk/bucs/... .. the MIME message is sent directly over SMTP. With a good abstraction you should easily be able to get the MIME data for your email object. This is what I was referring to when I discussed passing an object to the routine as opposed to the fields. A properly encapsulated MIME object would in fact make this a non-issue.
Getting back to System.Net.Mail, if you were to pass a MailMessage object (or your own object) as opposed to the indivual parameters you could in fact have your mock simply check that the header was set in the headers collection of the object you were given. It should not be the job of this code to test that the SMTP subsystem can properly take a MailMessage object and convert it to MIME data. This is obviously well outside the scope of a single test.

Cheers,
Greg

Haacked • May 31st, 2006
A couple points:
1. One benefit of using an EmailProvider is to abstract the details of sending an email. In that way, if the consumer of the provider doesn't need explicit control over the headers (or other complex property), then why expose all that in the EmailProvider interface? Instead, just ensure that the concrete providers do the right thing. These integration tests are to test the concrete providers.
2. One counter-example is that in one case, I was using a mail component that had the property named "XMailer". My unit test checked to make sure it was null. However the component still sent the header, but with an empty value. The mock testing didn't capture that. Only my integration test did.

Greg Young • May 31st, 2006
I agree with 2, but this is a very slippery slope.
With 1, I would still pass a MIME encapsulation of some type (be it my own or another). You have already defined the contract as being an email (and are passing multiple email related fields its only a matter of time until these items balloon to need more fields (like attachments or headers))
By implementing it by passing the a mime abstraction it allows you to have a single email sender for all cases.
As you have said, it is only at times that you need to worry about the header ... so the provider is probably not the right place to put it.
You could then have a message validation strategy (or as I would prefer to call it a specification) objects. This would be much easier to test and could be shared accross multiple providers if they needed the same validation.

Haacked • June 1st, 2006
Well in one instance, my EmailProvider is as simple as a method Send(to, from, subject, body) because that was all that was needed.
Of course, I could complicate the provider and create a facade to the provider. That would probably be the best solution. But for now, I'm following the YAGNI principle and doing the least amount of work that works.
And perhaps I was wrong to lump this into TDD, because my main point is that at some point, this is useful for integration testing with 3rd party email components.
Perhaps the best scenario for its usage is the following. In TDD (as I see it), when you find a bug in your software, you write a test to expose the bug (the RED factor in writing test first). Then you write the code that fixes the bug and re-test (GREEN factor).
So this utility comes in handy in that case. It also comes in handy for those who want to test, but don't feel like refactoring their code to use the provider model. I'm trying to deal with real-world developers, not our idealized vision of developers. Some are just plain lazy (like me).
Writing an email provider may be a case of premature generalization.

Steve Harman • June 5th, 2006
For anyone just stumbling across this post, you may be interested in Haacked's rather funny muzings on Premature Generallization.

If you need to instantiate a factory, implement an adapter class and use a bridge to the toilet just to take a dump...

Oh, that is truely classic!