[2026-03-02] Incident Thread

[github-actions[bot]]

❗ An incident has been declared:

Delayed visibility of newly added issues on project boards

Subscribe to this Discussion for updates on this incident. Please upvote or emoji react instead of commenting +1 on the Discussion to avoid overwhelming the thread. Any account guidance specific to this incident will be shared in thread and on the Incident Status Page.

[github-actions[bot]]

Update

Newly added issues can take up to 30 minutes to appear on project boards. We're investigating the cause of this delay.

[github-actions[bot]]

Update

Newly added issues are taking 30–60 minutes to appear on project boards, compared to the normal near-real-time behavior. We're investigating the root cause and possible mitigations.

[github-actions[bot]]

Update

The impact extends beyond adding issues to project boards. Adding pull requests and updating fields such as "Status" may also be affected. We're continuing to investigate the root cause.

[github-actions[bot]]

Update

Project board updates — including adding issues, pull requests, and changing fields such as "Status" — are currently delayed by 1–2 hours. Normal behavior is near-real-time. We're actively investigating the root cause.

[github-actions[bot]]

Update

The delay for project board updates has increased to up to 3 hours. We've identified a potential cause and are working on remediation.

[github-actions[bot]]

Update

We're deploying a fix targeting the increased delay in GitHub Projects updates. The rollout should complete within 60 minutes. If successful, the current delay of up to 4 hours should begin to decrease.

[github-actions[bot]]

Update

The fix is still building and is expected to deploy within 60 minutes. The current delay for GitHub Projects updates has increased to up to 5 hours.

[github-actions[bot]]

Update

The fix has been deployed and processing speeds have returned to normal. There is a backlog of delayed updates that will continue to be worked through — we're estimating how long that will take and will provide an update in the next 60 minutes.

[github-actions[bot]]

Update

The backlog of delayed updates is expected to fully clear within approximately 1 hour, after which project board updates will return to near-real-time.

[github-actions[bot]]

Update

This incident has been resolved. Project board updates are now processing in near-real-time.

[github-actions[bot]]

Incident Resolved

This incident has been resolved.

[github-actions[bot]]

Incident Summary

Between March 2, 21:42 UTC and March 3, 05:54 UTC project board updates, including adding new issues, PRs, and draft items to boards, were delayed from 30 minutes to over 2 hours, as a large backlog of messages accumulated in the Projects data denormalization pipeline.

The incident was caused by an anomalously large event that required longer processing time than expected. Processing this message exceeded the Kafka consumer heartbeat timeout, triggering repeated consumer group rebalances. As a result, the consumer group was unable to make forward progress, creating head-of-line blocking that delayed processing of subsequent project board updates.

We mitigated the issue by deploying a targeted fix that safely bypassed the offending message and allowed normal message consumption to resume. Consumer group stability recovered at 04:10 UTC, after which the backlog began draining. All queued messages were fully processed by 05:53 UTC, returning project board updates to normal processing latency.

We have identified several follow-up improvements to reduce the likelihood and impact of similar incidents in the future, including improved monitoring and alerting, as well as introducing limits for unusually large project events.