Unit Test Boundaries

Jul 22, 2008 tdd suggest edit

One principle to follow when writing a unit test is that a unit test should ideally not cross boundaries.

Michael Feathers takes a harder stance in saying…

A test is not a unit test if:

It talks to the database

It communicates across the network

It touches the file system

It can’t run at the same time as any of your other unit tests

You have to do special things to your environment (such as editing config files) to run it

Tests that do these things aren’t bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.

Speed isn’t the only benefit of following these rules. In order to make sure your tests don’t reach across boundaries, you have to make sure the unit under test is easily decoupled from code across its boundary, which provides benefits for the code being tested.

Suppose you have a function that pulls a list of coordinates from the database and calculated the best fit line for those coordinates. Your unit test for this method should ideally not make an actual database call, as that is reaching across a boundary and coupling your method to a specific data access layer.

Reaching across a boundary is not the only sin of this method. Data access is an orthogonal concern to calculating the best-fit line of a series of points. In The Pragmatic Programmer, Andrew Hunt and Dave Thomas tout Orthogonality as a key trait of well written code. In an interview on artima.com, Andy describes orthogonality like so:

The basic idea of orthogonality is that things that are not related conceptually should not be related in the system. Parts of the architecture that really have nothing to do with the other, such as the database and the UI, should not need to be changed together. A change to one should not cause a change to the other. Unfortunately, we’ve seen systems throughout our careers where that’s not the case.

Ideally, you would refactor the method so that the data the method needs is provided to it via some other means (another method passing the data via arguments, dependency injection, whatever). That other means, whatever it is, can perform the necessary data access: that’s not your concern at this moment. You aren’t testing that other means (right now at least, you might later), you’re focused on testing this unit.

This isolation enforced by unit test can be challenging, as it’s easy to get distracted by these other orthogonal concerns. For example, if this method doesn’t do data access, which one does? However, having the discipline to focus on the unit being tested can help shape your code so that it follows the single responsibility principle (SRP for short). If your test needs to access an external resource, it might just be violating SRP.

This provides several key benefits.

Your function is no longer tightly coupled to the current system. You could easily move it to another system that happened to have a different data access layer.
Your unit test of this function no longer needs to access the database, helping to keep execution of your unit tests extremely fast.
It keeps your unit test from being too fragile. Changes to the data access layer will not affect this function, and therefore the unit test of this function.

All this decoupling will provide long term benefits for the maintainability of your code.

Found a typo or mistake in the post? suggest edit

Comments

28 responses

superjason • July 22nd, 2008
Yep, just like I mentioned today, write your code for someone with blinders on. Of course it applies to your unit tests as well.
Jim • July 22nd, 2008
Good post. I find TDD interesting, though I still don't understand everything.
For example, "tests shouldn't hit the database." What if I have logic in my stored procedures? That logic has to be tested. In some of our systems, we prefer to keep substantial logic in the stored procedures, because if we find a bug, we can patch the stored procedure much quicker than redistributing a new executable to our clients. So, when I read something like this, it sounds like TDD isn't possible (or at least easy to do) when logic is stored in the database.
Does TDD work if the database is used only for storage?
superjason • July 22nd, 2008
Jim, I've been in that situation. At our company, we ended up writing unit tests specifically for the stored procedures, but they were difficult to maintain, slow, and generally not well maintained.
You have to make the decision if it's really worth having the logic in your stored procedures. I'm 99% confident that you're in a situation that you can't easily change overnight. You might have no choice but to fall back on traditional integration testing. That's typically how cross-boundary problems are tested, and why you try to minimize the cross boundary coupling.
Think of SQL as a different language that doesn't have a unit testing framework. You're trying to test it from .NET, so it's never going to be ideal.
Phil, I feel like I'm posting too many links now, but I think it relates to Jim's question:
Stored procedure reporting scalability. In that post, I talk about the situation at the old company I used to work for, which sounds exactly like what Jim is talking about. I've heard the save logic before.
Francois Ward • July 22nd, 2008
Having tests that hit the database isn't a bad thing, so tests to check the stored procedures aren't an issue.
I feel the important part is that these tests should conceptually be held in a separate category, and not mixed with the "real" unit tests.
They require different consideration to be run (for example, you can use VS Team for Database developer, but thats a far cry from NUnit running with TestDriven.Net, every check-in, or through the Resharper test runner), they may need significant setup data, etc. When you start testing the database, you need to make new test data and whatsnot, things you won't have to do to run a unit test (it will be built in the test).
I feel simply having a team that makes the distinction between a unit test (following the above rules is almost perfect), and an integration test (more or less...everything else...) solves 90% of the problem right there.
This is definately a case where the terminology makes a difference.
haacked • July 22nd, 2008
Like Francois, I'm not saying automated tests that check the logic in a stored proc are not valuable. They just aren't considered *unit* tests.
TDD for example, is a design activity focused on the design of your code, test coverage is a byproduct. Testing your stored procs sounds more like a *testing* activity, and you can put those tests either in your database itself, or in a separate test suite for integration tests.
You really really really want your *unit* tests to run blazing fast. But there's a lot more to testing your app than unit tests.
Colin Jack • July 22nd, 2008
Good post, agree with it all.
Lou • July 22nd, 2008
Agreed and I bumped into this the other day on a source of data component.
It sourceofdata is mocked for other testing on other components, but to test the source itself (ftp pull of zip/csv) a unit test invoked the component. It's non destructive after all. But we quickly added an Ignore on that test for continuous-integration purposes after one or two false-red builds when someone had pulled or reworked.
And yes, I know. ftp? csv? What year is it? Whatcha gonna do.
Gabriel • July 22nd, 2008
I agree. The main problem is that if your unit tests take to long to execute the developers will soon skip one test here and there just to get things done faster and pretty soon you'll be doing no tests at all.
It's not that you can't run tests against the database, it's just that this tests will probably be too slow to run as often as unit tests should be run.
Russell • July 22nd, 2008
>Suppose you have a function that pulls a list of coordinates from >the database and calculated the best fit line for those >coordinates. Your unit test for this method should ideally not >make an actual database call, as that is reaching across a >boundary and coupling your method to a specific data access layer.
But I would argue that this function may belong in the data access code. For example, look at:
download.oracle.com/.../analysis.htm#sthref1775
SELECT s.channel_id, REGR_SLOPE(s.quantity_sold, p.prod_list_price) SLOPE,
REGR_INTERCEPT(s.quantity_sold, p.prod_list_price) INTCPT,
REGR_R2(s.quantity_sold, p.prod_list_price) RSQR,
REGR_COUNT(s.quantity_sold, p.prod_list_price) COUNT,
REGR_AVGX(s.quantity_sold, p.prod_list_price) AVGLISTP,
REGR_AVGY(s.quantity_sold, p.prod_list_price) AVGQSOLD
FROM sales s, products p WHERE s.prod_id=p.prod_id
AND p.prod_category='Electronics' AND s.time_id=to_DATE('10-OCT-2000')
GROUP BY s.channel_id;
And this solution scales. You're doing the work right next to the data, not pulling potentially billions or rows of data across the network and then doing the curve fitting.
I'm fairly new to TDD, but why isn't it unit testing if I encapsulate the above in a stored procedure, and write other stored procedures (the unit tests) to test it?
haacked • July 22nd, 2008
@Russell If you write code and test in your database, then I probably would call that test a unit test. You haven't crossed boundaries. But if you wrote your test in C# as part of your business object tests, then you're reaching across from one system (C# app) into another (database) and that would be an integration test.
Rob G • July 23rd, 2008
Hey Phil, great post!
And while I wish I could say I agree with you completely, I do so only in theory - in practice, I have found very different terminologies coming from people in the teams I've worked with - and therein lies the rub: people (in real world teams I've worked on) can't agree on terminology, and I'm sure it's no different with you. You can point to Martin Fowler's work or definitions from eXtreme Central all you like, but not everyone is going to take it to heart. Everyone knows what a PDF or WWW is, and few would disagree on the general use of the term - seems almost as if it needs to be wrapped in an ISO standard before people actually believe that's what it truly is.
For example - "unit test" to you (in basic terms) means testing a piece of code without crossing any boundaries. Others call it testing a piece of code that performs a single unit of work. Trying to argue with a guy that says: "I'm testing if the columns still exists in the DB Table! That way, if someone stoooopidly decides to delete or rename a column, Cruise Control will pick it up and show a broken build.".
Now take note of a few things here:
1. He knows what Cruise Control is, and therefore what CI is all about.
2. The test code written for this example is extremely short.
3. He insists that the test is valid because it performs a single unit of work.
But according to your definition, he's broken the rule of crossing boundaries - and I tend to agree. You tell him that it's actually an integration test, but he says NOOOOO, it's only testing one unit of work - therefore, by his definition is still a unit test. Oh, and he also says it's blazin' fast too :-)
This really just boiled down to a simple case of semantics really, because in the wide world out there, not everyone calls a spade a spade - I think you guys call it a shovel :-)
Apologies for the long post, but there's one more question I'd like to ask. At which point do you actually test your LINQ to SQL queries - for real I mean - not mocking up the DataContext because you want it snappy. Afterall, what you really want to know is if the database can comply with your requests - no? I'm only talking here about a simple unit (don't shoot) test - the MemberRepository would like to add a Member to a Role - nothing fancy, just one class hitting the DB and getting a positive Assert out of it. That call hits the mock otherwise and gives me no real value because I can just tell the mock to "return true".
Is there not some middle ground where we can still call these "unit tests", but put it into a different testing project - I dunno, call it: "slower unit tests" or something?
Looking for some diretion here...
Cheers,
Rob
Justin - Web Development Blog • July 23rd, 2008
Great post. I have been on a DDD kick since reading Karl's Foundations of Programming. It gave me a good kick in the pants to start adopting these principles faster.
Daniel • July 24th, 2008
I get the concept (I think), but I'm not sure I agree about the data-access part. In my experience, the most fragile piece of a typical forms-on-data business app is the database. In almost every company I've worked with, "The Database" is a) not in source control and b) not versioned or tested. Developers routinely develop a fear of change to the point they would just as soon create a separate table/view/proc than touch existing schema because "we don't know what else uses that".
So, I think a better approach would be to develop, test, and in general "think" about data schema & access code just like you do other code. Sure, separate tests if you need, but in the end try to make it all tested...
Sonu • July 24th, 2008
Sorry but I do not agree. Unit tests should be *legally* allowed to test a unit of an application even if it crosses boundaries. Up to some extent, it also depends on how the team is organized - although some might argue that it should not. But practically speaking, if team is organized in such a way that one single developer is responsible for end-to-end development of a single unit of work - a small piece of functionality - he/she would like to test that as a single unit to make sure it works fine before passing moving them further for other types of testing. Assume an application has 3 layers and unit tests are forbidden to cross boundaries strictly. The developer will need to create 3 sets of unit test cases, two sets of integration test cases to test the two boundaries amongst the layers and another integration test to test these three layers of the same logical unit together. The developer would also be required to create stubs and drivers and imaginary data for each set of tests. I am not able to understand how all this arrangement would work faster! IMHO, any criteria for determining as to what a proper unit test is should consider a number of things including practicality. While we may agree in theory with any definitions, it is practicality that prevails. It doesn't matter if we try and define there are various *types* of proper unit tests to take care of varied contexts and practical scenarios.
Jeroen Mineur • July 24th, 2008
Does this mean that the classic three (or n-) tier model should not be implemented having the business layer call the data layer, and the GUI/Process layer call the business layer? Having a process layer that first calls the data layer, and then call the business layer with any fetched data, would allow us to create true unit tests for the business layer. In the classic model, calling the business layer in a test would be an 'integration test', according to your definition. Right?
R. A. Raboud • July 24th, 2008
For those of you that think unit test can cross boundaries, I would like to pose a question.
If the Database, Web or FTP server is down, do you fail the unit test?
My answer is the test would be inconclusive. I am a huge believer that unit test should not cross boundaries only integration tests should. Just my humble opinion.
Graeme • July 24th, 2008
Jim,
For testing the database logic in stored procedures have a look a DBFit which is a test fixture for fitness and will allow you to test your DB code in a transaction so that the tests do not become brittle over time when the data changes. http://fitnesse.info/dbfit
With TDD the unit tests act as a specification for your application and is one of the many reasons why your logic should live in your applicaion code and not in the Database.
I do understand your concerns regarding deploying changes but if you were to automate the deployment process using tools such as nant deployment of code should be as easy as deploying DB code.
Chris Smith • July 24th, 2008
Great post.
The presented ideas and principals as related to this article should be follows as much as possible in my opinion. Although, I do have some fundamental problems with the principals of unit tests in general.
Depending on the context and scope of your “test script”, “unit test”, issues such as crossing boundaries become acceptable or even a requirement.
For example;
1.Developing a database driver
2.QA engineer’s “test script” is to qa a store locator portion of a web site
We do see applications such as nunit targeting developers, also other testing frameworks geared toward QA engineers. So I question if a QA’s “test script” is actually no different than a developer unit test. After all the goal is the same, and if the done well, the developer and QA worked to write their respective tests from the same requirements document.
Just thoughts,
Chris Smith
Chris B. Behrens • July 24th, 2008
Alright...clearly there are a lot of sharp people here, so I'd like to ask everyone's opinion for the utterly perfect testing scenario for a couple of simple database operations.
I need to test whether a customer which exists loads successfully from the database, loaded by a unique id. Do I preload the database in a setup script for the test harness, or do I have an entire db filled with test data from a backup, or what?
I need to test whether a customer is saved successfully to the database. When I'm done, what's the best way to clean up? Do I abort a transaction context, or throw away the entire db (I've done that before), or what? What's the best way to verify that the customer is saved succesfully? Direct queries, or what?
I'm amenable to the theoretical argument that this article makes, but it seems that pragmatically the kind test we're saying you shouldn't write is exactly the test that is most likely to reveal problems (in my experience). I'd love to hear some other ideas.
Bryan Call • July 27th, 2008
This is some great advice. I've been doing TDD for about 5 year now. Early on I had a lot of tests that crossed boundaries and I paid for it in terms of maintainability and difficulty in implementing TDD. When you don't cross boundaries it becomes easier to write your tests, teach others to write them and your tests are less fragile. It will lead to better design as well as better tests. Mocks are an invaluable tool in accomplishing this.
Regarding some of the comments on stored procedure logic etc, one thing I would add is that while a unit test shouldn't cross boundaries I believe both sides of the boundary should be tested where practical. So in the case of logic in your database (which I personally avoid) I would consider writing unit tests specifically for your database. You may isolate them in a seperate assembly to avoid having them run with your other unit tests as they will most likely take longer.
Now how do you test code that calls your database. I personally have all database code go through what I call a DatabaseProvider class. It has methods like ExecuteNonQuery, ExecuteDataTable etc that allow both dynamic sql or stored procedures to be executed and the results to be returned. In my unit tests I use a mock version of the DatabaseProvider and test that my business logic code properly executed all the commands against the database that I expected. I have it set up so I can simluate transactions and test when a transaction is committed or rolled back and whether a given command is executed as part of the transaction or not. This layer of isolation from the actual database allows me to test that the database is properly called without depending on its existence. At that point the only question is whether the sql commands I send the database are actually valid.
I handle testing that seperately from testing all the business logic that calls the database. I tend to use stored procedures most of the time but keep them very simple, without any real logic. I have a seperate test assembly that I also write using NUnit. I've written some custom asserts I use in it to test for the existence of the stored procedures my business logic calls. I test that they take in the parameters I expect so I can catch any problems between what the business layer sends the database and what the database requires. I do this by querying the informational schema views in sql server so no commands are actually executed in the database. I've started actually executing select only stored procs to also test they return the columns I expect. I leave the testing of the sql statements within the stored procedures to other testing techniques. Because I keep my stored procedures very simple I rarely find problems in them and other testing techniques help me catch them.
Rob G • July 27th, 2008
@Chris B, I tried to post a reply to your question here - but I think I hit the comment character limit. My reply was too long it seems, so if you're interested, and since I don't have my own blog, I've posted the advice here with a reference back to this article.
Hope that helps.
Cheers,
Rob
Portella • July 28th, 2008
There few definition for unit for you. Choose one and find out if your unit test is a "real unit" Test.
- unit of measurement: any division of quantity accepted as a standard of measurement or exchange; "the dollar is the United States unit of currency ...
- an individual or group or structure or other entity regarded as a structural or functional constituent of a whole; "the reduced the number of units and installations"; "the word is a basic linguistic unit"
- an organization regarded as part of a larger social group; "the coach said the offensive unit did a good job"; "after the battle the soldier had trouble rejoining his unit"
- a single undivided whole; "an idea is not a unit that can be moved from one brain to another"
a single undivided natural thing occurring in the composition of something else; "units of nucleic acids"
- whole: an assemblage of parts that is regarded as a single entity; "how big is that part compared to the whole?"; "the team is a unit"
Myself i think you cant categorize what is and what is not a real unit test. You are trying to force your ideal of a unit test when in reality a "unit" has no single definition.
So is short.
In your world your unit tests are only unit tests if they dont cross boundaries, you are capping the tested percentage of your application by leaving other parts untested. leaving connectors untested.
are you trying to imply that should be a framework to test the code integration? let say class A using Class B?
Slashene • July 28th, 2008
I agree with the fact that when we do unit testing on domain object, we mustn't check if the database is really updated.
But how can you do when you want to test that your config file, is well read ?
My response is simple, and use unit test which are already written :
Customer data = new Customer()
data.Name = "John Smith"
data.Address = "123 Main St." //new address

IDataLayer obj = TestFactory.Create<idatalayer>();
obj.UpdateCustomer(db, data);
Assert(obj.RetrieveByName(data.Name).Address == "123 Main ST")...
The TestFactory will be in the config Test, and this test will be run for each IDataLayer possible.
Example of test config :
<testfactory>
<unit name="myTestMethod">
<register type="DataLayerObjectMock" builder="BuilderOfObjectMock"/>
<register type="DataLayerObjectDBAccess" builder="BuilderOfDBAccess"/>
</unit>
</testfactory>
Sound like ioc...
In this case, first the test just use a mock, and when DataLayerObjectDBAccess must be tested add just one line to the config file.
When you run your unit test, the business logic will be tested with the mock, and your real data layer object will be tested too.
What do you think about this solution ?
(sorry if my english is not perfect !)
Slashene • July 29th, 2008
Sorry about my previous post my example, in XML don't display
I agree with the fact that when we do unit testing on domain object, we mustn't check if the database is really updated.
But how can you do when you want to test that your config file, is well read ?
My response is simple, and use unit test which are already written :
Customer data = new Customer()
data.Name = "John Smith"
data.Address = "123 Main St." //new address

IDataLayer obj = TestFactory.Create<idatalayer>();
obj.UpdateCustomer(db, data);
Assert(obj.RetrieveByName(data.Name).Address == "123 Main ST")...
The TestFactory will be in the config Test, and this test will be run for each IDataLayer possible.
Example of test config :
<TestFactory>
<unit name="myTestMethod">
<register type="DataLayerObjectMock" builder="BuilderOfObjectMock" />
<register type="DataLayerObjectDBAccess" builder="BuilderOfDBAccess" />
</unit>
</TestFactory>
(I have replaced braces

Sound like ioc...
In this case, first the test just use a mock, and when DataLayerObjectDBAccess must be tested add just one line to the config file.
When you run your unit test, the business logic will be tested with the mock, and your real data layer object will be tested too.
What do you think about this solution ?
(sorry if my english is not perfect !)
Sebastian • July 31st, 2008
This is a good advice... at least in theory. I have found cases where this is not possible (or at least I was unable to find a perfect solution).
One example is WCF services. Having a service-oriented solution, you often find yourself having service A making calls to service B.
Now, of course you could test the service class directly (skipping the contract and the WCF mechanisms), but sometimes this is not possible (service behaves differently if used as a service instead of plain instantiation). Also if you look at WCF versus remoting or any other communication mechanisms, one of the basic rules is that it treats any service as a remote service. So standard instantiation would break this assumption.
My only solution to this issue was to create mock services (in the example above - mock service B), but still you have the WCF communication mechanisms in place, so this can be considered crossing the border.
Do you have any better solution for this scenario?
Sorry if the English may not be the best, it's not my native language.
Sid Savara- Personal Developme • August 11th, 2008
A lot of the work I have done in the past involves heavy database processing, and while I agree to some extent a unit test shouldn't require database access, the tests that specifically test a DAO definitely should. I like using DBUnit myself - it's not perfect, but it does make me more confident in my code.
David • August 14th, 2008
@Rob,
A spade and shovel are two different tools in the gardening world :D (spade is for digging, a shovel is for scooping up stuff!).
Our team is a real world team and we implement the theory! We have four levels of testing, unit testing at the lowest level, Unit testing. Unit testing should be (in my opinion) isolated (does not hit any databases or requires any external influencing factors – including other logic and configuration files).
Second we have an integration test that tests the application as a whole and how it integrates with the database, external configuration etc.
Thirdly we have an end to end test that tests interaction between different applications ensuring they communicate with each other correctly.
Fourth test we perform is sending our applications off to a different team to install and then test the front end interaction (GUI) and ensure it all appears to work.
VS • December 24th, 2008
VS already has support for database unit tests. By setting the datasource and the table name, unit tests can be made data driven. To retrieve the data, use TestContext object along with DataRow support. The unit test runs for every row in the resultset that is returned. I think unit test that has support for database should not be over-used but at the same time, it is a required feature.