Functional and visual E2E regression automation combined: Galen and Serenity
In out latest project I was free to choose what to use for E2E tests. No requirements from clients, awesome! I’m not impressed by Cucumber-based syntax for E2E tests. On the other
hand, I do understand client need for nice looking and readable E2E
automation reports. Let’s be honest, that’s why most of them choose
Cucumber - reports are business oriented when you write tests this way.
The project was very oriented towards "everything must look pixel-perfect" and lots of requirements which looked like "if amount of items received if 1, use Layout1; for 2 items use Layout2; else use Layout3". So I've thought I'll try to combine two frameworks: Serenity for pretty reporting and Galen for visual regression automation. Actually I loved both of them, and Serenity has much more than pretty reporting ;) Here’s a quick overview of “the good, the bad and the ugly” after project is finished.
Frameworks used:
Programming language: Java.
Galen features used: visual image comparison only, no CSS based layout validations.
No Galen specific syntax files used - all calls to Galen specific functions are made from same functional tests, reusing same page objects.
Single source code, single place to describe pages and page components.
Linking with Jira issues allows to access related issues from reports.
Multiple levels of code separation force to write tests which are easier to maintain when application code or requirements are changed:
Good looking (from client perspective), business-readable and very detailed Serenity reports. E2E reports where you can find all stack trace information when test fails - good!
Can incorporate REST tests (using REST Assured syntax) and they are displayed within same reports.
Visual validation was sucessfully used to:
Galen framework is intended to be used with its specific syntax. It is only mentioned “you can use it from Java tests directly". This means - you dig into framework code in order to use it. Dig a lot, and it takes some time.
On the other hand - I’m happy with result. I feel I've learned a lot as a developer also. Final code in feature file (test), which includes visual validation, at the moment looks something like this:
Baselines have to be manually updated. At first it looks time consuming, but at the end it is worth it (well, and some additional scripts help).
Well, nothing really ugly.
The most annoying thing is image baselines: single test may use several baseline images (e.g., one for browser size X1 x Y1, another for size X2 x Y2). Different images work really well, I’m happy with this. The problem is different configurations: you need as many baselines as many configurations you run tests on. Images differ: same browser size on Windows Chrome results in different results from same browser size on headless Chrome on CI server or even on Windows Chrome using laptop display.
Now I do understand why those with at least a bit of AI charge lots of money for visual comparison (I’m still dreaming of using Applitools Eyes in some project, and even more dreaming of working together with their team :))
Will use again, saved lots of time. Guess Serenity is the first framework where my emotions were more of “wow, framework owner thought of this situation also” than “ok, this works with standard html tags / simple elements. How the hell I do it with our very-customized-elements?“ most of the time.
Almost cannot imagine some kind of testing without visual tests anymore. It helped a lot. Especially when you catch regression bugs like this using automated tests:
The project was very oriented towards "everything must look pixel-perfect" and lots of requirements which looked like "if amount of items received if 1, use Layout1; for 2 items use Layout2; else use Layout3". So I've thought I'll try to combine two frameworks: Serenity for pretty reporting and Galen for visual regression automation. Actually I loved both of them, and Serenity has much more than pretty reporting ;) Here’s a quick overview of “the good, the bad and the ugly” after project is finished.
Frameworks used:
Programming language: Java.
Galen features used: visual image comparison only, no CSS based layout validations.
No Galen specific syntax files used - all calls to Galen specific functions are made from same functional tests, reusing same page objects.
The good
Single source code, single place to describe pages and page components.
Linking with Jira issues allows to access related issues from reports.
Multiple levels of code separation force to write tests which are easier to maintain when application code or requirements are changed:
- Features
- Steps
- Pages
- Page components
Good looking (from client perspective), business-readable and very detailed Serenity reports. E2E reports where you can find all stack trace information when test fails - good!
Visual validation was sucessfully used to:
- validate whole page (including long scrollable pages)
- validate only current viewport (saves time when not all page is needed)
- validate only selected components from page (good when layout of each component must be checked, and when same component is used for different cases only single test must fail)
- exclude some parts of objects from visual validation (e.g., dates which are changing every day)
The bad
Galen framework is intended to be used with its specific syntax. It is only mentioned “you can use it from Java tests directly". This means - you dig into framework code in order to use it. Dig a lot, and it takes some time.
On the other hand - I’m happy with result. I feel I've learned a lot as a developer also. Final code in feature file (test), which includes visual validation, at the moment looks something like this:
Baselines have to be manually updated. At first it looks time consuming, but at the end it is worth it (well, and some additional scripts help).
The ugly
Well, nothing really ugly.
The most annoying thing is image baselines: single test may use several baseline images (e.g., one for browser size X1 x Y1, another for size X2 x Y2). Different images work really well, I’m happy with this. The problem is different configurations: you need as many baselines as many configurations you run tests on. Images differ: same browser size on Windows Chrome results in different results from same browser size on headless Chrome on CI server or even on Windows Chrome using laptop display.
Now I do understand why those with at least a bit of AI charge lots of money for visual comparison (I’m still dreaming of using Applitools Eyes in some project, and even more dreaming of working together with their team :))
The verdict
Will use again, saved lots of time. Guess Serenity is the first framework where my emotions were more of “wow, framework owner thought of this situation also” than “ok, this works with standard html tags / simple elements. How the hell I do it with our very-customized-elements?“ most of the time.
Almost cannot imagine some kind of testing without visual tests anymore. It helped a lot. Especially when you catch regression bugs like this using automated tests:
Comments
Post a Comment