Repository navigation
Conversation
Goal is to make it faster to triage individual test failures while keeping the full existing regrtest fidelity. Fairly minimal integration to hopefully save some human eyeball time. Setup so only the test summary process should output the summary, subprocesses have the env var which will cause output unset.
|
Can you temporarily trigger a test failure so we can see it in action? |
|
Yea, planning to pass + fail + iterate on failure to get it pretty then get back to passing and mark as ready for review :) |
cmaloney
changed the title
|
I think having a screenshot in the description could help out a ton (I know that is the reason for the intentional failures, but may be easier to glance and get more exposure) |
Locate annotations at the last traceback frame in a test file instead of mapping each test to its file. Pass annotations to workers in RunTests, like use_junit, so tests which run regrtest don't inherit GITHUB_STEP_SUMMARY. Shrink the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fall back to the first traceback frame (the test) instead of passing the test module file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I think this is ready for review now. I updated the PR comment with full details. My hope is in the longer term the Reporter could be used if we, for instance, want buildbot jobs to annotate the github commit (or if moving places other than github). With this PR there is the text, |
| class Reporter: | ||
| """Report test failures, for example to a CI service. A no-op here.""" | ||
|
|
||
| def run_started(self, runtests: RunTests) -> None: |
There was a problem hiding this comment.
| def run_started(self, runtests: RunTests) -> None: | |
| def test_started(self, runtests: RunTests) -> None: |
Maybe, for parity with the other methods?
There was a problem hiding this comment.
The "run" concept is different than a "test" or "test suite" hence the different name. Parallels RunTests vs. a TestResult
| if not failed: | ||
| return | ||
| ncases = sum(len(result.failed_test_cases) for result in failed) | ||
| blocks = [f"## {decolor(state)}: " |
There was a problem hiding this comment.
| blocks = [f"## {decolor(state)}: " | |
| blocks = [f"## 🛑 {decolor(state)}: " |
Maybe it makes it a little easier to parse vs just markdown text?
Maybe also lowercase the state? I think it is all caps currently
There was a problem hiding this comment.
If we output success as well as failure definitely, this only outputs failures to github summaries. I tried outputting both success and failures but we run 30+ test jobs and that just made a lot of summaries that needed to be browsed past and weren't particularly useful.
| filename, line = frames[0].groups() | ||
| exception = traceback[frames[-1].end():] | ||
| # Prefer the last frame in a test file, the assertion | ||
| for frame in frames: |
There was a problem hiding this comment.
Maybe I'm wrong but will this get executed only when the above comparison is True?
There was a problem hiding this comment.
Correct.In general this tries to find a filename from the traceback but if we fail that's okay (it shouldn't block the test run from finishing). The if frames translates to "if the length of the list of frames is > 1".
The loop then finds the last frame with a name that starts with "test_" and uses that for the annotation. Definitely not a perfect metric but seems to work reasonably for getting back to the test filename.
3 participants
Update libregrtest to output GitHub Job Summary and Error Annotations for each failing job, test when run in Github Actions. Screenshots of the changes below.
This creates a number of UI adjustments which should make it a little easier to get around. These only happen if run in Github. Goal is to make triage a little bit simpler/faster but keep the full fidelity of the existing libregrtest output.
note: iOS and Android are excluded for now, as are other kinds of jobs like
docsandmypy. Can expand support to those but would rather do as separate well scoped PRs.Job Summaries
If you click on "checks" tab on a PR then the "tests" you get taken to the summary page (ex. https://github.com/python/cpython/actions/runs/38330186853). At the top is the workflow graph, if you scroll past that you see a job summary per failed job. If there are many failures they are "collapsed". Just a few will start expanded. The tracebacks are colored with Python traceback syntax highlighting.
Error Annotations
If you click on "Files Changed" in a PR there is a button to "Open Alerts Panel". This will show all the error annotations on all the files. Individual test failures are mapped back to the source line when possible, and that line in the Diff view will show an indicator that there is an error which can be expanded. Clicking on "show details" will take you to the specific job logs.
On individual test jobs all the annotations from that job are included at the top of the job page.
Unfortunately this does result in redundant printing of the assertion failures as we print both the
libregrtestform and a form of the github annotation line is presented. I chose for this to be just after the existing failure message which I think reads reasonably.edit: Rewrote with screenshots to show features.