Art Of Test WebAii Looks Promising For Testing The Web

A coworker of mine ran into some problems using WATIN to test our website. Specifically issues with Javascript and AJAX. Yes, I know there’s Selenium out there, but I hoped for something we could run from within NUnit/MbUnit, so that it’s nicely and easily integrated into our build process and development environment.

He did a bit of digging and found this free web automation infrastructure product by Art Of Test Inc. called WebAii (they also have a blog). It seems to do everything we want and more. Check out some of these features (the full feature list is much longer).

  • Supports IE and Firefox
  • Supports DOM actions and pure UI Mouse/Keyboard actions for Ajax Testing
  • For ASP.NET testing, you can use an In-Process Host for fast browser-less testing without a webserver
  • Use ASP.NET Development Server (aka Cassini) to run tests without IIS
  • Use name, id, xpath identifications. (Need to see if they support Regex too)
  • Unit test your JavaScript by calling JS methods from your .NET code.
  • Extensions for NUnit and VS Team Test (I’ll ping them to build one for MbUnit)

These are just a small sampling of the many features. It sounds like this would be a killer web testing app. Does anyone have experience with it?

We’re going to evaluate it and I’ll keep you posted, but I thought I’d mention it here because it sounds great.

Features I’d Like To See In A Web Unit Testing Tool

  • Ability to set the UserAgent and Referrer
  • Ability to set request headers
  • Ability to hook into and verify the status of a redirect. For example, if a page redirects, I want to be able to assert that the HTTP Status code for the redirect

What others have said

Requesting Gravatar... Kevin Dente Aug 02, 2007 10:42 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Looks intriguing. I look forward to your report.
Requesting Gravatar... Joe Brinkman Aug 02, 2007 11:47 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Myself and a few others are looking at this for DotNetNuke testing. One downside on this initial release is that there is no script recorder. For UI testing I think that a recorder is very beneficial so that non-developers and general community members can create scripts without fully understanding the page framework or the testing tool. Even if the recorder did 80% of the work and then let me edit the script to add in some assertions or tweak the script to make it more resilient to change.
Requesting Gravatar... Dave G Aug 02, 2007 11:52 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
We've been using WebAii now for a week, with a view to getting a lot of UI tests up very quickly. We chose it purely for it's ability to abstract the id's of controls outside of the tests, and for test regions, that should allow us to make our component tests reuseable for testing functionality when integrated into a site.

I wouldn't worry too much about the MbUnit testing extensions. I did have a crack at trying to write my own MbUnit extension, but found most of the hooks for doing this are internal methods. Apart from the new attributes for test cases (Dialog and another one), all the other shortcuts stuff works fine in MbUnit, if you just use the NUnitTest template. If you want to use the Initialize() call with some non default/config settings though, you can just pass in Console.WriteLine() instead of the NUnit.Logger method.

Although WebAii are looking at building a script recorder tool, I'm looking at modifying the WatinRecorder to allow us to record WebAii tests.. so far it looks like it'll be a fairly easy task.
Requesting Gravatar... Dave G Aug 02, 2007 12:08 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Thought I should add a few things..

We've quickly noticed, there are quite a few things in WebAii which can be painful to achieve, compared to previous testing frameworks we've used. Things like checking visibility of controls (i.e. for ASP.NET validation with client side script .. among many other cases). I've already asked for that one, but been told it's not likely for v1.0. Even the VERY regular tasks of checking for text on a page is the unreadable affair, of:

Assert.IsNotNull(Find.ByContent("p:sometext", InnerText));

compared to, say in selenium:

Assert.IsTrue(isTextPresent("someText"));

BTW just to add, there is nothing stopping you from driving Selenium tests from MbUnit, or CruiseControl .. We've been doing that for a while now as native C# tests, similar to WebAii/WATIN. The FireFox IDE/recorder will also generate NUnit C# tests as well as Selenese/HTML (and easily modified for MbUnit), though this is not obvious without a bit of digging. To drive from xUnit, you need to get the Selenium Remote Control package for .NET, and fire up it's java service before running the tests.

I've got a NANT script for doing all this (bit tricky as the java server needs to be spawned .. something not easily done in NANT at the moment), and examples of MbUnit/ASP.NET helpers if anybody is interested.

We've just decided to try out WebAii as it is a bit lighter, and easier to setup, and has much better support for us reusing tests ... we're missing the recorder though!
Requesting Gravatar... Vadim Aug 02, 2007 2:52 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
There's a new kid on the block, SWAT. http://sourceforge.net/projects/ulti-swat/
SWAT stands for Simple Web Automation Toolkit.
Requesting Gravatar... Jason Haley Aug 03, 2007 8:03 AM
# Interesting Finds: August 3, 2007
Requesting Gravatar... Jay Kimble Aug 03, 2007 8:17 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Hey Phil,

Check out InCsif.net (trhat's the product name and the URL). It's very similar and it has recording capabilities. I actually found it to be superior to WebAii. The only downside for InCsif.net is the price. If you are looking for free then this won't fit the bill.

Jay
Requesting Gravatar... Tod Birdsall Aug 03, 2007 11:49 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
I have been pretty happy with SWExplorerAutomation (SWEA). I have been using it for over a year and comes with a recorder. Although it is not free, it is inexpensive at $79. The bugs that I have run into have been quickly fixed by the creator (Alex Furman).
Requesting Gravatar... Kevin Michalsky Aug 03, 2007 12:30 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Phil,

I have to agree with you, this is very exciting. I downloaded the RC0 version that is listed on ArtOfTest's Website today and I cannot stop digging into it. Their RC0 now has an HTML control set in it. You can do Generic finds and have a strongly typed html control....VERY NICE!

Thanks for the find!
Kevin
Requesting Gravatar... Jeroen van Menen Aug 03, 2007 3:40 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Hi Phill,

Could you (or your coworker) explain some more about the problems you experienced with WatiN. Love to improve WatiN even if you decide to move over to another web testing framework (although I think WatiN being open source gives you all the posibilities you might ever need).

Thanks in advance,
Jeroen van Menen
main developer of WatiN
Requesting Gravatar... Haacked Aug 03, 2007 7:00 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
@Jeroen Thanks for commenting. Here's the list of issues:

Details of WatiN issues
- WatiN doesn't focus control (control.focus() doesn't work)
- WatiN doesn't send events for typetext
- Watin doesn't set keyCode event parameter when keyDown/keyPress/keyUp are sent

The problem I had was I couldn't figure out how to verify a document.write statement. For example, I wanted to make sure that a document.write statement was executed.
Requesting Gravatar... Haacked Aug 03, 2007 7:01 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Also, the Firefox, IE, and Safari on Windows testing is useful for us.
Requesting Gravatar... Ftorres Aug 03, 2007 10:13 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
InCisif.net has already been mentioned, thanks Jay Kimble.

InCisif has a record mode and also a different Api from WatiN or Selenium.

We ship with the product a lot of samples in C#, VB.NET and IronPython.
We have a lot of videos on our web site - www.InCisif.net.
So you can judge for yourself.

InCisif.net does not support FireFox or Safari yet, that is a plus from WebAii.

InCisif Software is giving away free licenses for personal this summer.

Don't be afraid to try!
Requesting Gravatar... Steve Trefethen Aug 04, 2007 12:12 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
FWIW, Selenium does integrate with NUnit and in fact I just got done adding a bunch of Selenium/NUnit tests to our CI server. Check out Selenium RC and it's .NET support. I've written a blog post about it here.

I haven't tried WebAii so thanks for the pointer, I'll definitely take a look.
Requesting Gravatar... Jeroen van Menen Aug 04, 2007 5:06 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Phil,

Thanks for the info. The keycode issue is fixed in the current developement source and will be part of the next release. I will have a look at the other issues you mentioned.
Regarding the document.write issue. There must be something that has changed on your page after this statement has been executed. The simplest check you could do is IE.ContainsText("").
Supporting other browsers is on my TODO list but very near to the end of it.

Regards, Jeroen
Requesting Gravatar... Haacked Aug 04, 2007 9:34 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
It seemed to me that IE.ContainsText was finding the text every time because it found it in the original document.write statement. For example, with something like this:

<script type="text/javascript">
if(false) document.write('blah blah');
</script>

It appeared I was getting true for IE.ContainsText. When I get a chance, I'll try out that scenario again to double check.
Requesting Gravatar... Jeroen van Menen Aug 04, 2007 1:30 PM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Hmmm, your right about that.
Requesting Gravatar... Chris May Jan 10, 2008 11:57 AM
# WebAii Website Automated Testing Framework
Requesting Gravatar... Kieran Mar 31, 2008 4:29 AM
# re: Art Of Test WebAii Looks Promising For Testing The Web
Does anyone have any more opinions on webaii versus other testing frameworks? I've been using webaii for about 3 months though, and like it, but have never tried anything else. Is it worth learning something new or sticky with webaii?

What do you have to say?

(will show your gravatar)
Please add 2 and 4 and type the answer here: