RFC: update Pull Requests with summary of CI failures

[mmarchini]

ncu-ci is quite useful to get summary of failures in our CI, but it still requires us to use an external program to fetch the information from Jenkins. We could use our github-bot to post the information generated by ncu-ci back to the PR, either as a new comment or by editing the bot's comment on that PR. This would also allow us to keep a history of CI failures (since our Jenkins jobs are not persistent).

Is that something other collaborators want as well? I'm looking for feedback before implementing it.

If folks want example of what this would look like, I've updated all CI runs with their respective summaries in #32116. It would look like this:

Or, as a standalone comment:

[sam-github]

My thoughts, FWTW:

On the one hand:

• anything making it easier for collaborators to understand the CI failures seems like a positive.

On the other hand:

• Often, if something basic is wrong, a PR will fail on every platform, so this might cause a masive amount of large comments... this could be addressed by using the github <details>...</details> guards around the ncu-ci output.
• Its yet another bit of build infrastructure we have to maintain.
• I feel sad that we consider our CI tools to have UIs that are so difficult to navigate that we'd even consider this, but that's out of our control.

As long as it doesn't overwhelm the comment history, I wouldn't block it, though when something fails, I personally haven't really had tons of trouble clicking through to find what the error was, even though it can be tedious.

[mmarchini]

Thanks for the feedback!

Often, if something basic is wrong, a PR will fail on every platform, so this might cause a masive amount of large comments... this could be addressed by using the github

Details

...

guards around the ncu-ci output.

Agreed. In the examples above I used details to address this issue.

I feel sad that we consider our CI tools to have UIs that are so difficult to navigate that we'd even consider this, but that's out of our control

That's mostly a Jenkins problem IMO, but I don't think there's any other CI compatible with all platforms we support.

As long as it doesn't overwhelm the comment history, I wouldn't block it

If the bot edits its own comment instead of creating new ones, this shouldn't be an issue.

though when something fails, I personally haven't really had tons of trouble clicking through to find what the error was, even though it can be tedious.

For me it depends on how many platforms failed. If it's more than two it usually takes way too much time for a task that should be simple. If node-test-commit and node-test-pull-request aggregated result from subprocesses, the situation would be a lot better, but still wouldn't solve the "preserve failures" issue.

[sam-github]

That's mostly a Jenkins problem IMO

I don't find Travis much better, just differently annoying.

If the bot edits its own comment instead of creating new ones, this shouldn't be an issue.

That might leave the CI result comment floating in the middle of the conversation, instead of where it should be, at the bottom, near the CI results.

For me it depends on how many platforms failed. If it's more than two it usually takes way too much time for a task that should be simple.

Many people fix one problem, and re-CI, I think, particularly after checking two failed platforms and seeing the same error. Even with ncu-ci output, we still have to click through to every build, and then to the console output (usually) to see the problems.

It might be better to spend time better advertising ncu-ci, particularly in onboarding, and getting it to the point where everyone prefers running it to clicking through to Jenkins. Its easy to sell automating a process that everyone already does.

And again, no objections from me. Maybe if everyone sees ncu-ci output, it'll even push folks to help improve the tool.

[mmarchini]

Many people fix one problem, and re-CI, I think, particularly after checking two failed platforms and seeing the same error.

I see this as a problem in our process today. Jenkins is too cumbersome to check for failures on every platform, leading people to ignore some platforms before rerunning CI, which can hide flaky tests.

Even with ncu-ci output, we still have to click through to every build, and then to the console output (usually) to see the problems.

Unless it is a build failure, there's no need to click on the console output. Expanding <details> also takes way less time than loading the whole Jenkins output.