TDTD, or Unit testing Selenium tests
I liked the idea to call this "Test Driven Test Development"!
So this is time for my input to discussions "I'm writing Selenium tests. Do I need to unit test them?" You can replace "Selenium" with any of frameworks which uses it, if you wish. As the correct answer is "it depends", here's my opinion when it's worth doing this.
At the moment I've started automating huge system, which is not so complex when looking at what it does, but it's a challenge in another areas:
- main goal is to save and display lots of data about countries. Names, demographic data, national holidays, various geographic facts etc. etc.
- part of it is a legacy system. Some parts are being rewritten, some will stay as they are for a long time. Or maybe not. They can be rewritten any time. By "legacy" I mean - you remember those times, when main methods for formatting html page layout were "you put a table inside a table of a table of...."? I'm not talking about backend here (well, it's legacy also, but this is completely another story).
- on some environments our tests will have no options to modify data. We'll not know what data exists there also, but we need to run at least minimal regression tests to verify build was successful. So we guess we are talking about matching page component labels, verifying some mandatory fields have "some, but any text" etc.
- client loves GUI tests. Even more, he loves Gherkin syntax (using JBehave this time).
So given the above, we are dealing with:
- lots of XPath locators
- code which can be changed any time soon in any way
- very long scrollable pages and dynamic loading of elements
- no way to prepare stable data for tests
- unstable and unmaintained old tests, some of them are "passing" all the time (you know, test which does something on a page, fails, consumes exceptions silently, and test report looks good - test passed!)
- almost no page objects, lots of duplicates in code and methods of 100+ lines of code
For a few days I've been trying to understand old code, refactor it at least a bit, at the same time thinking what can be done with those terribly complex XPath locators. When GUI code will change (and it will!) I have no wish to spend lots of time trying to understand which "table inside table inside something" was converted to what. Especially if code changes will be only subtle - to catch why some tests are failing will be almost impossible.
At the moment it takes lots of time to test if locators are correct - though it's possible to tests lots of locators directly in browser, I need to verify that all Selenium methods, our custom asserts etc. return exactly what I expect to get from that complex page structure with no id's. Each such verify means:
- start test and wait for browser to load (~5 - 10 seconds)
- wait for base page to load (~5-10 seconds, as the system uses lots of redirects)
- login
- wait for page under test to load (again - ~5 - 10 seconds)
- check locator behavior by creating some demo tests and printing lots of debugging info
If something is wrong - repeat the same for several times. And this is for each component. Well, I'm too lazy to wait for so long just for debugging purposes.
So here the idea "I need unit tests" came in. In my case at the moment I'm testing only locators and custom asserts. As I start loving the process, think I'll test even more in future. This is how process looks like at the moment:
- Find a page with all the values filled in and save it as html page locally. Well, it's almost impossible to do this with real data, but in our dev environments we can add data where it's missing.
- Load this local page using Selenium's HtmlUnitDriver. It's a headless browser and loads really quickly.
- Write unit test for this page, where only a page object for this page with HtmlUnitDriver injected as WebDriver is used.
- Run it
Quick and easy, no logins, waiting for real browser loading, redirects etc.
Other benefits apart from saved time while debugging? Well, I see future process in this way:
- when the page changes, I'll know exactly which locators are failing: I'll just need to pass new format html page with same data values. If I have no way to get same data values - at least I can visually compare, if this is just a different value problem, or some more serious change in page structure.
- If some complex page structure changes will be needed, I'll already have unit tests. I just need to update locators / custom logic so that those tests were passing again.
- I have saved current page structure against which tests were written, so if e2e tests start to fail, I know exactly if I need to start investigating locators changes, or look for some more mysterious issues.
- I have and option to find what exactly has changed in page structure by using simple diff of pages. Sometimes it takes so long to find minimal changes, when you no longer remember how that page looked like at the time of writing complex XPath/CSS selectors.
Looks good at the moment, we'll see how this will work in the future. Have never used HtmlUnitDriver before. It says it can imitate how different browsers work with Javascript. I think I'll have option to try this also.
So this is time for my input to discussions "I'm writing Selenium tests. Do I need to unit test them?" You can replace "Selenium" with any of frameworks which uses it, if you wish. As the correct answer is "it depends", here's my opinion when it's worth doing this.
At the moment I've started automating huge system, which is not so complex when looking at what it does, but it's a challenge in another areas:
- main goal is to save and display lots of data about countries. Names, demographic data, national holidays, various geographic facts etc. etc.
- part of it is a legacy system. Some parts are being rewritten, some will stay as they are for a long time. Or maybe not. They can be rewritten any time. By "legacy" I mean - you remember those times, when main methods for formatting html page layout were "you put a table inside a table of a table of...."? I'm not talking about backend here (well, it's legacy also, but this is completely another story).
- on some environments our tests will have no options to modify data. We'll not know what data exists there also, but we need to run at least minimal regression tests to verify build was successful. So we guess we are talking about matching page component labels, verifying some mandatory fields have "some, but any text" etc.
- client loves GUI tests. Even more, he loves Gherkin syntax (using JBehave this time).
So given the above, we are dealing with:
- lots of XPath locators
- code which can be changed any time soon in any way
- very long scrollable pages and dynamic loading of elements
- no way to prepare stable data for tests
- unstable and unmaintained old tests, some of them are "passing" all the time (you know, test which does something on a page, fails, consumes exceptions silently, and test report looks good - test passed!)
- almost no page objects, lots of duplicates in code and methods of 100+ lines of code
For a few days I've been trying to understand old code, refactor it at least a bit, at the same time thinking what can be done with those terribly complex XPath locators. When GUI code will change (and it will!) I have no wish to spend lots of time trying to understand which "table inside table inside something" was converted to what. Especially if code changes will be only subtle - to catch why some tests are failing will be almost impossible.
At the moment it takes lots of time to test if locators are correct - though it's possible to tests lots of locators directly in browser, I need to verify that all Selenium methods, our custom asserts etc. return exactly what I expect to get from that complex page structure with no id's. Each such verify means:
- start test and wait for browser to load (~5 - 10 seconds)
- wait for base page to load (~5-10 seconds, as the system uses lots of redirects)
- login
- wait for page under test to load (again - ~5 - 10 seconds)
- check locator behavior by creating some demo tests and printing lots of debugging info
If something is wrong - repeat the same for several times. And this is for each component. Well, I'm too lazy to wait for so long just for debugging purposes.
So here the idea "I need unit tests" came in. In my case at the moment I'm testing only locators and custom asserts. As I start loving the process, think I'll test even more in future. This is how process looks like at the moment:
- Find a page with all the values filled in and save it as html page locally. Well, it's almost impossible to do this with real data, but in our dev environments we can add data where it's missing.
- Load this local page using Selenium's HtmlUnitDriver. It's a headless browser and loads really quickly.
- Write unit test for this page, where only a page object for this page with HtmlUnitDriver injected as WebDriver is used.
- Run it
Quick and easy, no logins, waiting for real browser loading, redirects etc.
Other benefits apart from saved time while debugging? Well, I see future process in this way:
- when the page changes, I'll know exactly which locators are failing: I'll just need to pass new format html page with same data values. If I have no way to get same data values - at least I can visually compare, if this is just a different value problem, or some more serious change in page structure.
- If some complex page structure changes will be needed, I'll already have unit tests. I just need to update locators / custom logic so that those tests were passing again.
- I have saved current page structure against which tests were written, so if e2e tests start to fail, I know exactly if I need to start investigating locators changes, or look for some more mysterious issues.
- I have and option to find what exactly has changed in page structure by using simple diff of pages. Sometimes it takes so long to find minimal changes, when you no longer remember how that page looked like at the time of writing complex XPath/CSS selectors.
Looks good at the moment, we'll see how this will work in the future. Have never used HtmlUnitDriver before. It says it can imitate how different browsers work with Javascript. I think I'll have option to try this also.
Comments
Post a Comment