Sunday, February 17, 2013

Validating WebDriver Tests Without the WebDriver API

One thing we have noticed working on WebTestingExplorer is just how excruciatingly slow WebDriver can be. Perhaps you have never noticed this because you only validate a few elements at the end of a WebDriver test. But in WebTestingExplorer use cases, we often need to examine hundreds and hundreds of WebElements and their attribute values after each action. And here, WebDriver's architecture where every call to the browser is a local RPC is a massive fail in terms of performance (and, I think, stability, although that is harder to pin down).

This is one of the (many) reasons I wish the testing practice was not standardizing on WebDriver: I feel like browser automation is far from a solved problem and would like to see what other smart people can continue to put forward. Nonetheless, we feel like we have to use it for WebTestingExplorer, both to provide synergy and integration between our tool and other testing efforts, and because there just aren't a lot of good alternatives under active development.

That said, we had been theorizing for awhile that we could improve life significantly if we started trying to implement WebTestingExplorer state checking and oracles on top of an alternative HTML-parsing library. A couple of weeks ago, I selected jsoup as the library and finally took a shot at this. The results were really incredible -- an order of magnitude or more speedup in many of our exploration and bug-checking scenarios, and fewer (read: zero) inexplicable, random exceptions flying out of WebDriver. Going forward, I hope to try and rework some of the WebTestingExplorer APIs to make this more natural.

But even if you're not using WebTestingExplorer, I think the overall approach is worth considering for any web unit/system/integration testing you might be doing. The idea is to command the browser using WebDriver as you normally do. Then, having completed that, grab the current DOM HTML of the page with WebDriver like this:

WebDriver driver;
// ...
JavascriptExecutor js = (JavascriptExecutordriver;
String domString = (Stringjs.executeScript("return document.documentElement.outerHTML;");

Done with WebDriver, feed that to the jsoup parser and enjoy its simple, fast API while writing your assertions. Yeah, you lose all the fancy element-is-displayed logic that WebDriver is constantly trying to perform, but hopefully (as in, you would think) it's not too hard to write assertions that don't care much about that. Enjoy!

Monday, February 4, 2013

Declaring Instances as Fields in Interfaces is a Testability Problem

Typically, when you see fields defined in Java interfaces, they are things you would normally think of as "constants" -- e.g. int's, Strings, maybe an enum or two. I had never really thought much about the fact that you can have instances of more complex classes as interface fields until I actually saw it in practice. Something like this:

public interface MyInterface {
  // Singleton instance.
  MyInterfaceImpl INSTANCE = new MyInterfaceImpl();
  // Interface methods...
  void doSomething();

Then you can use that field like a static variable, most likely as an old-school (pre-dependency-injection) singleton instance.

There are two problems with this as far as testability is concerned. First, you have the usual difficulty you get with static instances in unit tests, namely, that you cannot insert a mock, stub, or fake in all of the places where it is referenced. But if that is not enough to convince you to not use this pattern (hint: it should be), there is another problem that is arguably even more insidious.

Let's say that by some stroke of luck, you are trying to test a class that does not make static reference to the singleton instance and accepts the dependency properly. Now, when you go to create a mock or fake implementation of MyInterface for use in the test, Java still has to instantiate MyInterfaceImpl at classload-time. Depending on what all that involves, this may be expensive, unstable, or flat-out fail in the test environment, thereby thwarting your efforts to properly unit-test.

Obviously, the easiest way to solve this problem is to avoid it in the first place. But if you happen to run into it in a legacy system, you can start migrating away from it gradually by simply moving the instance to another class, or by using something like the value-holder pattern or initialization-on-demand holder idiom that moves the creation of the instance out of the classload sequence for the interface.