Skip to content

[Bug]: Browser plugin: a test that fails on its first check is reported as risky, and an earlier page's server exception reaches later pages #2009

Open
pestphp/pest-plugin-browser
#324

Description

@Muhammadjewel

pestphp/pest-plugin-browser has issues turned off, so I file this here.
Each case below reproduces in a fresh laravel/laravel app with only Pest and the browser plugin added, on plugin 5.0.1 and on 5.2.0 (the 5.x head).
Line numbers are for 5.2.0, with 5.0.1 in brackets where it differs.

Versions: pestphp/pest-plugin-browser 5.0.1 and 5.2.0, pestphp/pest 5.2.1, phpunit/phpunit 13.3.4, PHP 8.5.8, Playwright 1.63.0 (Chromium), laravel/framework 13.39.0, macOS.
Routes used by the tests:

Route::get('/ok', fn () => 'Hello');
Route::get('/fail', fn () => throw new RuntimeException('Thrown by /fail.'));

1. A test that fails on its first browser check is also risky

it('fails on its first check', function () {
    visit('/ok')->assertSee('Goodbye');
});

Run: vendor/bin/pest tests/Browser/FirstCheckFailsTest.php --log-junit junit.xml --log-events-text events.txt
Expected: 1 failed test with 1 assertion and no risky event, as for a failed PHPUnit, Pest or Laravel assertion.
Actual: Tests: 1 failed (0 assertions), assertions="0" in junit.xml, and in events.txt:

Test Considered Risky (P\Tests\Browser\FirstCheckFailsTest::__pest_evaluable_it_fails_on_its_first_check)
This test did not perform any assertions

laravel/pao 1.1.5 reports "assertions":0,"failed":1,"risky":1.
A failed first action does the same: visit('/ok')->click('No such button') fails with Timeout 5000ms exceeded., 0 assertions, risky.
A test that passes one check before it fails is not affected.

Cause: these calls throw ExpectationFailedException directly, with no assertion counted: assertSee() and assertDontSee() (src/Api/Concerns/MakesElementAssertions.php:61, :79), which count one only when they pass, a failed action in Client::execute() (src/Playwright/Client.php:127 [106]), GuessLocator::for() (src/Support/GuessLocator.php:75 [65]) and ArrayablePendingAwaitablePage::ensureOnWebpage() (src/Api/ArrayablePendingAwaitablePage.php:151).
PHPUnit's own assertions count before they throw, and PHPUnit marks a test with 0 assertions as risky unless it errored, was skipped or is incomplete (src/Framework/TestRunner/TestRunner.php:224-234), so a failed test is not exempt.

Suggested fix: count one assertion before each of these throws, as assertPortIs() already does (src/Api/Concerns/MakesUrlAssertions.php:109-110).
With expect(true)->toBeTrue(); before the throw in assertSee(), the test above gives 1 failed (1 assertions) and no risky event.

2. The server exception of an earlier page reaches later pages of the same test

it('does not throw the exception of an earlier page again', function () {
    $this->withoutExceptionHandling();

    expect(fn () => visit('/fail')->assertSee('Server Error'))
        ->toThrow(RuntimeException::class, 'Thrown by /fail.');

    visit('/ok')->assertSee('Hello');
});

it('names only the exception of the page that failed', function () {
    visit('/no-such-page')->assertSee('Not Found');

    visit('/ok')->assertSee('Goodbye');
});

Expected: the first test passes, and the second fails on Goodbye with no server exception in its message, because /ok throws nothing.
Actual: the first test fails with RuntimeException: Thrown by /fail., thrown again by visit('/ok')->assertSee('Hello'); the same call passes in a test of its own.
The failure message of the second test contains:

The following exception was thrown by the HTTP server:
Symfony\Component\HttpKernel\Exception\NotFoundHttpException: The route no-such-page could not be found.

Cause: LaravelHttpServer keeps the last request exception in $lastThrowable (src/Drivers/LaravelHttpServer.php:352, :368 [281, 293]) and clears it only in flush(), which runs in the plugin's afterEach (src/Plugin.php:55 [53]).
Until then, AwaitableWebpage::__call() throws it again after each page call under withoutExceptionHandling() (src/Api/AwaitableWebpage.php:89, :100 [61, 72]), and each failed check adds it to its message (src/Exceptions/BrowserExpectationFailedException.php:60 [51]).

Suggested fix: clear $lastThrowable when throwLastThrowableIfNeeded() throws it, and when Page::goto() starts a new page load.
With both changes, both tests behave as expected, and visit('/fail')->assertSee('Goodbye') still names RuntimeException: Thrown by /fail. in its message.

3. Request: keep the plugin's frames out of failure traces

The failure of the test in case 1 is located in the plugin: the trace in junit.xml starts at vendor/pestphp/pest-plugin-browser/src/Exceptions/BrowserExpectationFailedException.php:69 [60], and laravel/pao reports that file and line.
Pest puts its own src on PHPUnit's exclude list (src/Bootstrappers/BootExcludeList.php), but the plugin does not.
With PHPUnit\Util\ExcludeList::addDirectory() for the plugin's src, the trace is only at tests/Browser/FirstCheckFailsTest.php:4.
PHPUnit then also ignores the warnings that the plugin suppresses with @, such as the rmdir() warnings that pestphp/pest-plugin-browser#250 and pestphp/pest-plugin-browser#278 fixed in 5.2.0.

Activity

  1. vasikmansuri commented on Nov 2, 2026

    @vasikmansuri

    I opened pestphp/pest-plugin-browser#298 for case 1 (counting a failed check as an assertion). Cases 2 and 3 are not in it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions