Repository navigation
test: port the upstream page and library tests - #2024
Merged
Yury Semikhatsky (yury-s) merged 93 commits intoOct 28, 2026
Merged
Conversation
Pavel Feldman (pavelfeldman)
approved these changes
Oct 27, 2026
This was referenced Oct 27, 2026
…rary tests A test server with the Node.js request and response shapes, WebSocket support and the upstream certificate; the testInfo, contextFactory, launchPersistent, createUserDataDir and pageWithHar fixtures; the utils of tests/config; the upstream assets, golden screenshots and test registry; a checker that every registry entry is ported or listed as not ported; @fixme for strict expected failures; Surefire reports named after the upstream test titles. Three spec files are ported as examples, under the rules of src/test/AGENTS.md.
…stry The registry shows the describe path and markdown escapes around the title.
…error, eval-on-selector*
…d-*-tag, page-autowaiting-*
…p, page-dispatchevent, page-dialog, page-expose-function, elementhandle-bounding-box, elementhandle-click
…st, network-post-data, page-set-extra-http-headers, page-basic, page-set-content, page-transition, page-event-popup, queryselector, retarget
…et chunked encoding
…electors-register, page-strict, page-select-option, page-screenshot, page-localstorage, page-cache-storage, page-wait-for-function
…filechooser, firefox-network-response, page-fill, page-focus, frame-evaluate, frame-frame-element
…se the client Threads blocked in Playwright calls polled and dispatched messages concurrently, so a response could be dispatched before the __create__ of the object it names, failing with "Object doesn't exist". A fair lock around poll+dispatch and around sending keeps the order; the object and callback maps are concurrent.
…, locator-list, locator-misc-*, locator-query, locator-wait-for-function, page-mouse, wheel, page-goto, page-navigation, page-history, page-event-load
…css, selectors-text, selectors-role
…-script*, jshandle-*, page-keyboard, page-add-locator-handler, locator-any-frame, locator-click, locator-convenience, locator-element-handle, locator-evaluate
…page-wait-for-navigation, page-wait-for-url, page-wait-for-response, workers
…,pages,page-event}, browser, browser-server, browsertype-basic, browsertype-launch, launcher, beforeunload, capabilities, modernizr, client-certificates
…ne-id,user-agent,viewport,service-worker-policy}, video, webmcp*, shared-worker, web-socket, route-web-socket
The wait info sent for waitForConsoleMessage() and the other event waits had no event name and no title, so the trace viewer showed them as __waitInfo__. The before message of a wait now carries the event and the title of the Node.js client, "Wait for event "console"" or "Wait for navigation", set for that message only so that the calls made from the callback of the wait stay visible. The wait of BrowserContext.waitForPage() was reported under the name of BrowserContext.close(). TR-041 passes and loses its fixme.
upstream_base.txt names the microsoft/playwright commit whose tests/registry.md is upstream_registry.md and whose spec files the ports follow, with the commands that list the upstream changes still to port.
The upstream test checks that page.reload() ignores a pushState that happens while the reload is pending. A port without concurrent calls can only arrange that with timers, so it checks the fix only while the timing holds. The fix is in the driver, and the Node.js test covers it for all the ports. Record the upstream rewrite of the other tests in microsoft/playwright#43656.
The ported tests use @ParameterizedClass, which JUnit added in 5.13.
The headed tests need an X server, as upstream runs its Linux tests under xvfb-run. With the ported tests a run takes about four times longer, and the macOS and Windows jobs did not finish in 30 minutes.
Playwright Test sets testIdAttribute before every test. Without it, SE-075 failed when SE-072 or SE-073 ran before it in the same worker.
On Windows the temporary directory and the checkout are on different drives, where Path.relativize throws and path.relative returns the absolute path.
…y occur Fixme gets an os attribute and can be repeated, like DisabledOnBrowser. CA-030, PR-007 and FI-013 pass in the WebKit of some platforms, DD-008 fails in the WebKit of macOS and Windows, and EH-077 matches the goldens in CI.
The stack trace collector skipped the frames of the tests in com.microsoft.playwright.tests, so TR-014 and TR-016 found no source files. TestTracing now sets PLAYWRIGHT_JAVA_SRC, as the Node.js client always sends the stack frames.
…rocess The driver is installed once per JVM, and it kept the env of the first Playwright.create() call. The env of later instances did not get to their driver process.
Like the Node.js client: when the page or the context closes while a route handler runs, the update of the interception patterns no longer throws from the call that dispatched the route event, such as context.close().
…l its pid taskkill got pid -1 and killed nothing, so the teardown of the child processes waited forever and TestBrowsertypeConnect never finished.
…nutes The method timeout does not cover the fixtures, and a hang there stopped the whole job with no trace of the cause.
After the Fixme timeout, the paused call kept reading the connection of the worker while the teardown used it, and the test class could hang.
TZ-006 waits for the log message of the worker, because Chrome on Windows can log the 404 error of /favicon.ico first. NE-024 is an expected failure in the WebKit of driver 1.64.1 on Windows.
Like the Node.js client, waitForNavigation, waitForURL and waitForLoadState throw the target closed error when the page is already closed, instead of the close reason of the connection, which differs between platforms.
The test classes run in parallel in one JVM, so the fixed port 9339 of upstream collides with the old TestChromium.
Java does not escape the quotes in an argument on Windows, so the remote server got its JSON options without the quotes.
On JDK 8 the fork-join pool of JUnit adds a thread when a pool thread blocks, and the new thread starts another test. On the 3-core macOS runners about 30 tests ran at once and the jobs hit the time limit.
Chromium 156 of driver 1.64 sometimes never answers a Page.captureScreenshot that is sent right after a document of the redirect loop commits, and the screenshot times out after 30 seconds.
The teardown uses the connection of the test, which is not thread-safe; a timed-out connect() kept reading it while the worker closed the browser.
Yury Semikhatsky (yury-s)
force-pushed
the
upstream-tests
branch
from
October 27, 2026 19:58
6c15f7b to
d90d726
Compare
Concurrent extractions shared the zip file system of the jar, and the first one to finish closed it under the others: ClosedFileSystemException when several Playwright instances were created at once.
A class ran its methods one at a time, so a few long classes such as TestBrowsertypeConnect kept a run going long after the others had finished. The classes whose tests share state or a display opt out.
With the methods in parallel, 32 Firefox tests at once on the Windows runner timed out by the hundreds. The limit of TestScope follows the configured parallelism, since JDK 8 can add threads to the JUnit pool.
A port that was free when it was picked can be taken before the server binds it, now that many tests run at once: Address already in use. The tests of a class also create their server only once.
Yury Semikhatsky (yury-s)
merged commit Oct 28, 2026
441d9f0
into
microsoft:main
20 of 21 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/pageandtests/librarytoplaywright/src/test/java/com/microsoft/playwright/tests, one class per spec file. Each test has the title and the identifier of the upstream test.upstream_registry.mdis the upstream test registry at the commit inupstream_base.txt.scripts/check_test_registry.pychecks that each test is ported or is inupstream_not_ported.txtwith a reason.@Fixme.Tests not yet ported
3342 of the 4936 upstream tests are ported. The other 1594 are in
upstream_not_ported.txt.Keep listed as not ported
To port next
comparePNGslocator.withinchrome://crash; Firefox and WebKit stay listedreq.pause()on the test server, which now reads the whole body before the route runsReasons to correct
java: concurrent.locator.withintests are out of date.