Playwright/Cypress Browser Support:
- Chrome
- Chromium
- Edge
- Firefox
Key differences in Browser support:
Cypress also supports Electron, but does NOT support Webkit (Safari) like Playwright does. If Safari is important to us then this is a borderline showstopper.
Pros and Cons for Playwright/Cypress
Below is a list of Pros and Cons for each tool prioritized by importance IMO followed by a general summary and my 2cents.
Cypress Pros:
- Cypress dashboard service which comes with some neat features besides just parallelizing tests
- Focus on a better experience for devs for writing and debugging E2E tests
- Focus on local testing
- Full package for all E2E tests including test runner (plug and play approach)
- Familiar JQuery syntax (cy instead of "$")
- Great documentation ranging not only from writing tests but to deploying Cypress in CI and Scaling since it's an "All in one" solution
- Less of a learning curve
- Optional Cypress component test runner (allows running component tests in browser and mocking native browser functionality among other things)
- More dev friendly for those not familiar with automated testing
Cypress Cons:
- No Webkit support (No Safari)
- Needs to be re-run with the other browser options (can't mix contexts at all) if you want to test even something as simple as multiple domains
- Must use their test runner
- Doesn't support working with multiple tabs (gotta force code to open in the same tab for testing)
- In order to test third-party logins, pop ups etc require a lot of stubbing
- Pain or very minimal support for iframes
- Domain restrictions, you cannot visit two domains of different origin in the same test without annoying workarounds
- Must subscribe to their SAAS to run tests in parallel or use a workaround
- Must pay a premium to get access to some dashboard features to do parallelization well, it requires vendor-locked software
- No "hover" support
- Tailored for local testing, may have to wrestle with this more when deploying to CI
- Does not work with self hosted Selenium Grid, must use a Saas Solution like Browserstack
- Slower run time than other E2E frameworks though probably not going to make a big difference in grand scheme of things
- Limited to Javascript and transpiled languages as it runs in the browser (though doesn't matter here since we are a JS shop)
- Must use their file structure
Playwright Pros:
- Playwright is promise-based and can run several different browsers and different user contexts in the same test
- Supports Webkit
- Flexible with choosing which test runner to use
- iFrame support
- Uses Puppeteer under the hood which means higher compatibility rate for cross browser testing and interfacing with other tooling
- The ability to create object instances allows us to run multiple tabs, browsers and user contexts at the same time (good for testing 3P integrations too)
- Support for parallel tests for free out of the box
- Support for many codecept.io and other BDD tools like CucumberJS (limited or none for Cypress)
- Great synergy with VSCode, allowing you to track the browser's behavior, use breakpoints, inspect variables in memory, etc
- Ability to generate tests (https://playwright.dev/docs/codegen#generate-tests)
- Syntax is Javascript first (await/async) instead of this promised based chaining pattern so feels more natural if you're familiar with Javascript and doesn't require having to learn about the Cypress event loop happening behind the scenes
- Language independent (though doesn't matter here since we are a JS shop)
- Supported by Microsoft (good QA automation tooling IMO)
- Compatible with a self hosted selenium grid (doesn't matter if we use a SaaS solution like Browserstack though)
Playwright Cons:
- Newer tool which can mean not as good support (only two years old)
General Findings
Based on my research it seems that while both E2E frameworks are fairly new and have a lot of similar features and may seem like similar picks, the key differences start to appear when you look at the limitations of Cypress.
I had an incredibly hard time finding downsides to Playwright besides just being fairly new. Though it’s new, it is very feature rich and continues to build on tools/paradigms before it whereas Cypress tries to rewrite the paradigm and just isn’t quite there yet for large scale applications IMO.
Limitations
My gut feeling just tells me that we are going to spend more time wrestling with the limitations of Cypress than it’s going to be worth. Things like testing multiple domains, iframes, and third party integrations are going to be key to a lot of true E2E browser tests and to me defeats the purpose if you have to mock your browser tests though I can see the attraction to being able to override native browser behavior.
The features that Cypress does offer isn’t enough to offset all the limitations we’d be running into. We also have strong enough engineers that the hand holding that Cypress includes isn’t as impactful. Cypress focuses on the experience of writing a test which is why the GUI is great but Playwright’s GUI is also very slick.
I’m not a huge fan of being forced to use a specific Test Runner for Cypress whereas Playwright lets you choose whatever Test Runners you want and it’s easier to integrate modern BDD Syntax Assertion libraries if that’s something we want to do as well.
Closing Thoughts
If I had to bet my money on a horse I’d bet it on Playwright. It’s only 2 years old but the devs who worked on Puppeteer went on to create this at Microsoft. When it comes to QA Automation tools, Microsoft has a good track record and gives me more confidence that Playwright is enterprise ready and I’m sure they are dog-fooding it.
Cypress I can see becoming enterprise ready in a few years but even then its goal is to be a one stop shop for all E2E testing so it’s inherently opinionated and aimed for devs who are new to tests. That’s not something we want if we have the engineering skills to build out what’s best for us and make sure it scales with our needs/software without sacrificing any development speed.
Comparison Links I Found Useful:
