2025-Q4
Oct 1, 2025 - Dec 31, 2025
Total Downtime
Total Incidents
Worst Component
Service Features
Time-based uptime calculation for the 132,480 minutes in this quarter
Downtime Definition: Minutes with >5% error rate (approximated from incident data)
| Component | Uptime % | Downtime | Incidents | Status | Service Credit |
|---|---|---|---|---|---|
| Git Operations | 99.8635% | 3h 1m | 4 | Violation | 10% |
| API Requests | 99.8952% | 2h 19m | 3 | Violation | 10% |
| Issues | 99.8445% | 3h 26m | 4 | Violation | 10% |
| Pull Requests | 99.9044% | 2h 7m | 3 | Pass | None |
| Webhooks | 99.9111% | 1h 58m | 2 | Pass | None |
| Pages | 99.9111% | 1h 58m | 2 | Pass | None |
Actions
Execution-based calculation (workflow success rate)
| Component | Uptime % | Downtime | Incidents |
|---|---|---|---|
| Actions | 99.3971% | 13h 19m | 11 |
Packages
Hybrid calculation with two separate metrics
1. Package Transfers: (Total transfers - Failed transfers) / Total transfers × 100
2. Package Storage: (Total minutes - Minutes with >5% error rate) / Total minutes × 100
| Component | Uptime % | Downtime | Incidents |
|---|---|---|---|
| Packages | 99.9221% | 1h 43m | 2 |
Incidents in 2025-Q4
55 incidents occurred during this quarter
Incident with Issues and Pull Requests
5 updates
On December 23, 2025, between 09:15 UTC and 10:32 UTC the Issues and Pull Requests search indexing service was degraded and caused search results to contain stale data up to 3 minutes old for roughly 1.3 million issues and pull requests. This was due to search indexing queues backing up from resource contention caused by a running transition.<br /><br />We mitigated the incident by cancelling the running transition.<br /><br />We are working to implement closer monitoring of search infrastructure resource utilization during transitions to reduce our time to detection and mitigation of issues like this one in the future.
Issues and Pull Requests are operating normally.
We are seeing recovery in search indexing for Issues and Pull Requests. The queue has returned to normal processing times, and we continue to monitor service health. We'll post another update by 11:00 UTC.
We're experiencing delays in search indexing for Issues and Pull Requests. Search results may show data up to three minutes old due to elevated processing times in our indexing pipeline. We're working to restore normal performance. We'll post another update by 10:30 UTC.
We are investigating reports of degraded performance for Issues and Pull Requests
Disruption with some GitHub services
6 updates
On December 22, 2025, between 22:01 UTC and 22:32 UTC, unauthenticated requests to github.com were degraded, resulting in slow or timed out page loads and API requests. Unauthenticated requests from Actions jobs, such as release downloads, were also impacted. Authenticated traffic was not impacted. This was due to a severe spike in traffic, primarily to search endpoints.<br /><br />Our immediate response focused on identifying and mitigating the source of the traffic increase, which along with automated traffic management restored full service for our users.<br /><br />We improved limiters for load to relevant endpoints and are continuing work to more proactively identify these large changes in traffic volume, improve resilience in critical request flows, and improve our time to mitigation.
All services at healthy levels. We're finalizing the change to prevent future degradations from the same source.
We're investigating elevated traffic affecting GitHub services, primarily impacting logged-out users with some increased latency on Issues. We're preparing additional mitigations to prevent further spikes.
We are experiencing elevated traffic affecting some GitHub services, primarily impacting logged-out users. We're actively investigating the full scope and working to restore normal service. We'll post another update by 23:45 UTC.
Issues is experiencing degraded performance. We are continuing to investigate.
We are investigating reports of impacted performance for some GitHub services.
Disruption with some GitHub services
7 updates
On December 18, 2025, between 16:25 UTC and 19:09 UTC the service underlying Copilot policies was degraded and users, organizations, and enterprises were not able to update any policies related to Copilot. No other GitHub services, including other Copilot services were impacted. This was due to a database migration causing a schema drift.<br /><br />We mitigated the incident by synchronizing the schema. We have hardened the service to make sure schema drift does not cause any further incidents, and will investigate improvements in our deployment pipeline to shorten time to mitigation in the future.
Copilot is operating normally.
We have observed full recovery with updating copilot policy settings, and are validating that that there is no further impact.
Copilot is experiencing degraded performance. We are continuing to investigate.
We have identified the source of this regression and are preparing a fix for deployment. We will update again in one hour.
We are seeing an increase in errors on the User and Org policy settings page when updating policies. The errors are affecting the user copilot policies settings page, org copilot policies settings page when updating a policy. <br />
We are investigating reports of impacted performance for some GitHub services.
4 updates
On December 18th, 2025, from 08:15 UTC to 17:11 UTC, some GitHub Actions runners experienced intermittent timeouts for Github API calls, which led to failures during runner setup and workflow execution. This was caused by network packet loss between runners in the West US region and one of GitHub’s edge sites. Approximately 1.5% of jobs on larger and standard hosted runners in the West US region were impacted, 0.28% of all Actions jobs during this period.<br /><br />By 17:11 UTC, all traffic was routed away from the affected edge site, mitigating the timeouts. We are working to improve early detection of cross-cloud connectivity issues and faster mitigation paths to reduce the impact of similar issues in the future.
We are observing recovery with request from GitHub-hosted Actions runners and will continue to monitor.
Since approximately 8:00 UTC, we have observed intermittent failures on GitHub-hosted actions runners. The failures have been observed both during runner setup, and workflow execution. We are continuing to investigate.<br /><br />Self-hosted runners are not impacted.
We are investigating reports of degraded performance for Actions
Intermittent networking issues across hosted runners
3 updates
During an investigation into a unrelated issue, this issue was elevated to public status by mistake, with a title intended for the other incident. We immediately resolved this in order to ensure that our internal investigation was aligned with the correct public status.
Since approximately 8:00 UTC, we have observed intermittent failures on GitHub-hosted actions runners. The failures have been observed both during runner setup, and workflow execution. We are continuing to investigate.<br /><br />Self-hosted runners are not impacted.
We are investigating reports of impacted performance for some GitHub services.
Incident With Copilot
1 update
From 11:50-12:25 UTC, Copilot Coding Agent was unable to process new agent requests. This affected all users creating new jobs during this timeframe, while existing jobs remained unaffected. The cause of this issue was a change to the actions configuration where Copilot Coding Agent runs, which caused the setup of the Actions runner to fail, and the issue was resolved by rolling back this change. As a short term solution, we hope to increase our alerting criteria so that we can be alerted more quickly when an incident occurs, and in the long term we hope to harden our runner configuration to be more resilient against errors.
3 updates
On December 15, 2025, between 15:15 UTC and 18:22 UTC, Copilot Code Review experienced a service degradation that caused 46.97% of pull request review requests to fail, requiring users to re-request a review. Impacted users saw the error message: “Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.” The remaining requests completed successfully.<br /><br />The degradation was caused by elevated response times in an internal, model-backed dependency, which led to request timeouts and backpressure in the review processing pipeline, resulting in sustained queue growth and failed review completion.<br /><br />We mitigated the issue by temporarily bypassing fix suggestions to reduce latency, increasing worker capacity to drain the backlog, and rolling out a model configuration change that reduced end-to-end latency. Queue depth and request success rates returned to normal and remained stable through peak traffic.<br /><br />Following the incident, we increased baseline worker capacity, added instrumentation for worker utilization and queue health, and are improving automatic load-shedding, fallback behavior, and alerting to reduce time to detection and mitigation for similar issues.
We have seen recovery for Copilot Code Review requests and are investigating long-term availability and scaling strategies
We are investigating reports of impacted performance for some GitHub services.
Incident with Copilot Grok Code Fast 1
4 updates
On Dec 15th, 2025, between 14:00 UTC and 15:45 UTC the Copilot service was degraded for Grok Code Fast 1 model. On average, 4% of the requests to this model failed due to an issue with our upstream provider. No other models were impacted.<br /><br />The issue was resolved after the upstream provider fixed the problem that caused the disruption. GitHub will continue to enhance our monitoring and alerting systems to reduce the time it takes to detect and mitigate similar issues in the future.
We are continuing to work with our provider on resolving the incident with Grok Code Fast 1. Users can expect some requests to intermittently fail until all issues are resolved.
We are experiencing degraded availability for the Grok Code Fast 1 model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.
We are investigating reports of degraded performance for Copilot
Disruptions in Login and Signup Flows
6 updates
Between 13:25 UTC and 18:35 UTC on Dec 11th, GitHub experienced an increase in scraper activity on public parts of our website. This scraper activity caused a low priority web request pool to increase and eventually exceed total capacity resulting in users experiencing 500 errors. In particular, this affected Login, Logout, and Signup routes, along with less than 1% requests from within Actions jobs. At the peak of the incident, 7.6% of login requests were impacted, which was the most significant impact of this scraping attack.<br /><br />Our mitigation strategy identified the scraping activity and blocked it. We also increased the pool of web requests that were impacted to have more capacity, and lastly we upgraded key user login routes to higher priority queues. <br /><br />In future, we’re working to more proactively identify this particular scraper activity and have faster mitigation times.
We see signs of full recovery and will post a more in-depth update soon.
We are continuing to monitor and continuing to see signs of recovery. We will update when we are confident that we are in full recovery.
We've applied a mitigation to fix intermittent failures in anonymous requests and downloads from GitHub, including Login, Signup, Logout, and some requests from within Actions jobs. We are seeing improvements in telemetry, but we will continue to monitor for full recovery.
We currently have ~7% of users experiencing errors when attempting to sign up, log in, or log out. We are deploying a change to mitigate these failures.
We are investigating reports of impacted performance for some GitHub services.
7 updates
Between 13:25 UTC and 18:35 UTC on December 11th, GitHub experienced elevated traffic to portions of GitHub.com that exceeded previously provisioned capacity for specific request types. As a result, users encountered intermittent 500 errors. Impact was most pronounced on Login, Logout, and Signup pages, peaking at 7.6% of login requests. Additionally, fewer than 1% of requests originating from GitHub Actions jobs were affected. <br /><br />This incident was driven by the same underlying factors as the previously reported <a href="https://www.githubstatus.com/incidents/40730vhmg6y8">disruption to Login and Signup flows</a><br /><br />Our immediate response focused on identifying and mitigating the source of the traffic increase. We increased available capacity for web request handling to relieve pressure on constrained pools. To reduce recurrence risk, we also re-routed critical authentication endpoints to a different traffic pool, ensuring sufficient isolation and headroom for login related traffic.<br /><br />In future, we’re working to more proactively identify these large changes in traffic volume and improve our time to mitigation.<br /><br />
Git Operations is operating normally.
We believe that we have narrowed down our affected users to primarily those that are signing up or signing in as well as logged out usage. We are currently continuing to investigate the root cause and are working multiple mitigation angles.
We are experiencing intermittent web request failures across multiple services, including login and authentication. Our teams are actively investigating the cause and working on mitigation.
Codespaces, Copilot, Git Operations, Packages, Pages, Pull Requests and Webhooks are experiencing degraded performance. We are continuing to investigate.
API Requests and Actions are experiencing degraded performance. We are continuing to investigate.
We are investigating reports of degraded performance for Issues
Some macOS Actions jobs routing to Ubuntu instead
3 updates
Between December 9th, 2025 21:07 UTC and December 10th, 2025 14:52 UTC, 177 macos-14-large jobs were run on an Ubuntu larger runner VM instead of MacOS runner VMs. The impacted jobs were routed to a larger runner with incorrect metadata. We mitigated this by deleting the runner.<br /><br />The routing configuration is not something controlled externally. A manual override was done previously for internal testing, but left incorrect metadata for a large runner instance. An infrastructure migration caused this misconfigured runner to come online which started the incorrect assignments. We are removing the ability to manually override this configuration entirely, and are adding alerting to identify possible OS mismatches for hosted runner jobs.<br /><br />As a reminder, hosted runner VMs are secure and ephemeral, with every VM reimaged after every single job. All jobs impacted here were originally targeted at a GitHub-owned VM image and were run on a GitHub-owned VM image.
We've applied a mitigation to ensure all macOS jobs route to macOS fulfillers and are monitoring for full recovery.
We are investigating reports of degraded performance for Actions
Some Actions customers experiencing run start delays
8 updates
On December 10, 2025 between 08:50 UTC and 11:00 UTC, some GitHub Actions workflow runs experienced longer-than-normal wait times for jobs starting or completing. All jobs successfully completed despite the delays. At peak impact, approximately 8% of workflow runs were affected.<br /><br />During this incident, some nodes received a spike in workflow events that led to queuing of event processing. Because runs are pinned to nodes, runs being processed by these nodes saw delays in starting or showing as completed. The team was alerted to this at 8:58 UTC. Impacted nodes were disabled from processing new jobs to allow queues to drain.<br /><br />We have increased overall processing capacity and are implementing safeguards to better balance load across all nodes when spikes occur. This is important to ensure our available capacity can always be fully utilized.
Actions is operating normally.
We have validated the mitigation and are no longer seeing impact.
We are seeing improvements in telemetry and are monitoring for full recovery.
We've applied a mitigation to fix the issues with queuing and running Actions jobs. We will continue monitoring to confirm whether this resolves the issue.
The team continues to investigate issues with some Actions jobs being queued for a long time. We will continue providing updates on the progress towards mitigation.
We're investigating Actions workflow runs taking longer than expected to start.
We are investigating reports of degraded performance for Actions
Disruption with some GitHub services
5 updates
On December 8, 2025, between 21:15 and 22:24 UTC, Copilot code completions experienced a significant service degradation. During this period, up to 65% of code completion requests failed.<br /><br />The root cause was an internal feature flag that caused the primary model supporting Copilot code completions to appear unavailable to the backend service. The issue was resolved once the flag was disabled.<br /><br />To prevent recurrence, we expanded test coverage for Copilot code completion models and are strengthening our detection mechanisms to better identify and respond to traffic anomalies.
We are beginning to see signs of resolution after applying a mitigation. We expect full resolution within approximately 30 minutes.
We're continuing to investigate and mitigate issues with the GPT 4o model for Copilot completions. Users can currently work around this issue by updating their VS Code settings with "github.copilot.advanced.debug.overrideEngine": "gpt-41-copilot".
We are currently investigating failures with the GPT 4o model for Copilot completions.
We are investigating reports of impacted performance for some GitHub services.
3 updates
On November 26th, 2025, between approximately 02:24 UTC and December 8th, 2025 at 20:26 UTC, enterprise administrators experienced a disruption when viewing agent session activities in the Enterprise AI Controls page. During this period, users were unable to list agent session activity in the AI Controls view. This did not impact viewing agent session activity in audit logs or directly navigating to individual agent session logs, or otherwise managing AI Agents.<br /><br />The issue was caused by a misconfiguration in a change deployed on November 25th that unintentionally prevented data from being published to an internal Kafka topic responsible for feeding the AI Controls page with agent session activity information.<br /><br />The problem was identified and mitigated on December 8th by correcting the configuration issue. GitHub is improving monitoring for data pipeline dependencies and enhancing pre-deployment validation to catch configuration issues before they reach production.
We are investigating an incident affecting missing Agent Session data on the AI Settings page on Agent Control Plane.
We are investigating reports of impacted performance for some GitHub services.
4 updates
On December 5th, 2025, between 12:00 pm UTC and 9:00 pm UTC, our Team Synchronization service experienced a significant degradation, preventing over 209,000 organization teams from syncing their identity provider (IdP) groups. The incident was triggered by a buildup of synchronization requests, resulting in elevated Redis key usage and high CPU consumption on the underlying Redis cluster.<br /><br />To mitigate further impact, we proactively paused all team synchronization requests between 3:00 pm UTC and 8:15 pm UTC, allowing us to stabilize the Redis cluster. Our engineering team also resolved the issue by flushing the affected Redis keys and queues, which promptly stopped runaway growth and restored service health. Additionally, we scaled up our infrastructure resources to improve our ability to process the high volume of synchronization requests. All pending team synchronizations were successfully processed following service restoration.<br /><br />We are working to strengthen the Team Synchronization service by implementing a killswitch, adding throttling to prevent excessive enqueueing of synchronization requests, and improving the scheduler to avoid duplicate job requests. Additionally, we’re investing in better observability to alert when job drops occur. These efforts are focused on preventing similar incidents and improving overall reliability going forward.
We believe we reached a scaling limit and are increasing the amount of resources available to reduce the delays for the team synchronization process.
We're continuing to investigate the delays in the team synchronization and will report back once we have more information.
We are investigating reports of impacted performance for some GitHub services.
Webhooks delivery degradation
1 update
On December 3, 2025, between 22:21 UTC and 23:44 UTC, the Webhooks service experienced a degradation that delayed writes of webhook delivery records to our database. During this period, many webhook deliveries were not visible in the webhook delivery UI or API for more than an hour after they were sent. As a result, customers were temporarily unable to request redeliveries for those delayed records. The underlying cause was throttling of database writes due to high replication lag. We mitigated the incident by temporarily disabling delivery history for a small number of very high‑volume webhook owners to reduce write pressure and stabilize the service. We are contacting the affected customers directly with more details. We are improving our webhook delivery storage architecture so it can scale with current and future webhook traffic, reducing the likelihood and impact of similar issues.
Incident with Copilot
4 updates
On November 28th, 2025, between approximately 05:51 and 08:04 UTC, Copilot experienced an outage affecting the Claude Sonnet 4.5 model. Users attempting to use this model received an HTTP 400 error, resulting in 4.6% of total chat requests during this timeframe failing. Other models were not impacted.<br /><br />The issue was caused by a misconfiguration deployed to an internal service which made Claude Sonnet 4.5 unavailable. The problem was identified and mitigated by reverting the change. GitHub is working to improve cross-service deploy safeguards and monitoring to prevent similar incidents in the future.
We have rolled out a fix and are monitoring for recovery.
We are investigating degraded performance with the Claude Sonnet 4.5 model in Copilot.
We are investigating reports of degraded performance for Copilot
Disruption with some GitHub services
6 updates
On November 24, 2025, between 12:15 and 15:04 UTC, Codespaces users encountered connection issues when attempting to create a codespace after choosing the recently released VS Code Codespaces extension, version 1.18.1. Users were able to downgrade to the 1.18.0 version of the extension during this period to work around this issue. At peak, the error rate was 19% of connection requests. This was caused by mismatching version dependencies for the released VS Code Codespaces extension.<br /><br />The connection issues were mitigated by releasing the VS Code Codespaces extension version 1.18.2 that addressed the issue. Users utilizing version 1.18.1 of the VS Code Codespaces extension are advised to upgrade to version >=1.18.2.<br /><br />We are improving our validation and release process for this extension to ensure functional issues like this are caught before release to customers and to reduce detection and mitigation times for extension issues like this in the future.
Version 1.18.2 of the GitHub Codespaces VSCode extension has been released. This version should resolve the connection issues, and we are continuing to monitor success rate for Codespaces creation.
We are testing a new version of the GitHub Codespaces VSCode extension that should resolve the connection issues, and expect that to be available in the next 30 minutes.
Codespaces is experiencing degraded performance. We are continuing to investigate.
We are seeing Codespaces connection issues related to the latest version of the VSCode Codespaces extension (1.18.1). Users can select the 1.18.0 version of the extension to avoid issues (View -> Command Palette, run "Extensions: Install specific version of Extension..."), while we work to remove the affected version.
We are currently investigating this issue.
Disruption with some GitHub services
5 updates
Between November 20, 2025 17:16 UTC to November, 2025 19:08 UTC some users experienced delayed or failed Git Operations for raw file downloads. On average, the error rate was less than 0.2%. This was due to a sustained increase in unauthenticated repository traffic.<br /><br />We mitigated the incident by applying regional rate limiting and are taking steps to improve our monitoring and time to mitigation for similar issues in the future.
Mitigation has been applied and operations have returned to normal.
We continue to see a small number of errors when accessing raw file content. We are deploying a mitigation.
We're investigating elevated error rates for a small amount of customers when accessing raw file content.
We are currently investigating this issue.
Incident with Actions
4 updates
On November 19, between 17:36 UTC and 18:04 UTC, GitHub Actions service experienced degraded performance that caused excessive latency in queueing and updating workflow runs and job statuses. Operations related to artifacts, cache, job steps and logs also had significantly increased latency. At peak, 67% of workflow jobs queued during that timeframe were impacted, and the median latency for impacted operations increased by up to 35x.<br /><br />This was caused by a significant change in load pattern on Actions Cache-related operations, leading to a saturated shared resource on the backend. The impact was mitigated by mitigating the new load pattern.<br /><br />To reduce the likelihood of a recurrence, we are improving rate-limiting measures in this area to ensure a more consistent experience for all customers. We are also evaluating changes to reduce the scope of impact.
We have applied mitigation and are seeing recovery
We are investigating delays in actions runs and possible errors in artifact and cache creation.
We are investigating reports of degraded performance for Actions
Disruption with some GitHub services
4 updates
Between November 19th, 16:13UTC and November 21st, 12:22UTC, the GitHub Enterprise Importer (GEI) service was in a degraded state, during which time, customers of the service experienced a delay when reclaiming mannequins post-migration.<br /><br />We have taken steps to prevent similar incidents from occurring in the future.
Processing of these jobs has resumed.
GitHub Enterprise Importer migration systems are currently impacted by a pause to Migration Mannequin Reclaiming. At 19:43 UTC on 2025-11-19, we paused the queue that processes Mannequin Reclaiming work done at the end of a migration. This was done after observing load that threatened the health of the overall system. The cause has been identified, and efforts to fix are underway. In the current state: - all requests to Reclaim Mannequins will be held in a queue - those requests will be processed when repair work is complete and the queue unpaused, at which time the incident will be closed This does not impact processing of migration runs using GitHub Enterprise Importer, only mannequin reclamation.
We are currently investigating this issue.
Git operation failures
11 updates
From Nov 18, 2025 20:30 UTC to Nov 18, 2025 21:34 UTC we experienced failures on all Git operations, including both SSH and HTTP Git client interactions, as well as raw file access. These failures also impacted products that rely on Git operations.<br /><br />The root cause was an expired TLS certificate used for internal service-to-service communication. We mitigated the incident by replacing the expired certificate and restarting impacted services. Once those services were restarted we saw a full recovery.<br /><br />We have updated our alerting to cover the expired certificate and are performing an audit of other certificates in this area to ensure they also have the proper alerting and automation before expiration. In parallel, we are accelerating efforts to eliminate our remaining manually managed certificates, ensuring all service-to-service communication is fully automated and aligned with modern security practices.
Git Operations is operating normally.
We are seeing full recovery after rolling out the fix and all services are operational.
Codespaces is operating normally.
We have shipped a fix and are seeing recovery in some areas and will continue to provide updates.
We have identified the likely cause of the incident and are working on a fix. We will provide another update as we get closer to deploying the fix.
Codespaces is experiencing degraded availability. We are continuing to investigate.
We are currently investigating failures on all Git operations, including both SSH and HTTP.
We are seeing failures for some git http operations and are investigating
Git Operations is experiencing degraded availability. We are continuing to investigate.
We are currently investigating this issue.
Disruption with some GitHub services
3 updates
Between November 17, 2025 21:24 UTC and November 18, 2025 00:04 UTC the gists service was degraded and users were unable to create gists via the web UI. 100% of gist creation requests failed with a 404 response. This was due to a change in the web middleware that inadvertently triggered a routing error. We resolved the incident by rolling back the change. We are working on more effective monitoring to reduce the time it takes to detect similar issues and evaluating our testing approach for middleware functionality.
We are investigating reports of 404s creating gists.
We are currently investigating this issue.
Disruption with some GitHub services
6 updates
From Nov 17, 2025 00:00 UTC to Nov 17, 2025 15:00 UTC Dependabot was hitting a rate limit in GitHub Container Registry (GHCR) and was unable to complete about 57% of jobs.<br /><br />To mitigate the issue we lowered the rate at which Dependabot started jobs and increased the GHCR rate limit.<br /><br />We’re adding new monitors and alerts and looking into more ways to decrease load on GHCR to help prevent this in the future.
We continue to see recovery and dependabot jobs are currently processing as expected.
We are applying a configuration change and will monitor for recovery.
We are continuing to investigate dependabot failures and a configuration change to mitigate.
We are investigating dependabot job failures affecting approximately 50% of version updates and 25% of security updates.
We are currently investigating this issue.
3 updates
From Nov 13, 2025 14:50 UTC to Nov 13, 2025 15:01 UTC we experienced failures on all Git Push and SSH operations. An internal service became unhealthy due to a scaling configuration change. We reverted the change and are evaluating our health monitoring and processes to prevent similar incidents.
Git Operations is experiencing degraded performance. We are continuing to investigate.
We are currently investigating this issue.
Disruption with some GitHub services
4 updates
On November 12, 2025, between 22:10 UTC and 23:04 UTC, Codespaces used internally at GitHub were impacted. There was no impact to external customers. The scope of impact was not clear in the initial steps of incident response, so it was considered public until confirmed otherwise. One improvement from this will be improved clarity of internal versus public impact for similar failures to better inform our status decisions going forward.
We are continuing to investigate connectivity issues with codespaces
We are investing reports of codespaces no longer appearing in the UI or API. Users may experience connectivity issues to the impacted codespaces.
We are currently investigating this issue.
Delay in notification deliveries
6 updates
On November 12th, 2025, from 13:10 - 17:40 UTC, notifications service was degraded, showing an increase in web notifications latency and increasing delays in notification deliveries. A change to the notifications settings access path introduced additional load to the settings system, degrading its response times. This impacted both requests to web notifications (with p99 response times as high as 1.5s, while lower percentiles remained stable) and notification deliveries, which reached a peak delay of 24 minutes on average. System capacity was increased around 15:10 UTC and the problematic change was fully reverted soon after that, restoring the latency of web notifications and increasing notification delivery throughput, decreasing the delay in notification deliveries. The notification queue was fully emptied around 17:40 UTC.<br /><br />We are working to adjust capacity in the affected systems and to improve the time needed to address these capacity issues.
We are continuing to monitor our mitigations to delays in notification deliveries. Some users may still experience delays of over 10 minutes.
We are continuing to work on mitigating delays in notification deliveries. Some users may still experience delays of over 10 minutes.
We are continuing to work on mitigating delays in notification deliveries. Some users may experience delays of over 10 minutes.
We are investigating delays of up to 10 minutes in notification deliveries. Our team has identified the likely cause and is actively working to mitigate the issue.
We are currently investigating this issue.
Larger hosted runners experiencing delays
5 updates
On November 11, 2025, between 16:28 UTC and 20:54 UTC, GitHub Actions larger hosted runners experienced degraded performance, with 0.4% of overall workflow runs and 8.8% of larger hosted runner jobs failing to start within 5 minutes. The majority of impact was mitigated by 18:44, with a small tail of organizations taking longer to recover.<br /><br />The impact was caused by the same database infrastructure issue that caused similar larger hosted runner performance degradation on October 23rd, 2025. In this case, it was triggered by a brief infrastructure event in this incident rather than a database change.<br /><br />Through this incident, we identified and implemented a better solution for both prevention and faster mitigation. In addition to this, a durable solution for the underlying database issue is rolling out soon.
Mitigation is complete and all new jobs targeting Larger Hosted Runners should not experience delays.
The team is continuing to apply the mitigation for Large Hosted Runners. We will provide updates as we progress.
The team continues to investigate delays with Large Hosted Runners. We will continue providing updates on the progress towards mitigation.
We are currently investigating this issue.
Incident with Copilot
4 updates
Between November 5, 2025 23:27 UTC and November 6, 2025 00:06 UTC, ghost text requests experienced errors from upstream model providers. This was a continuation of the service disruption for which we statused Copilot earlier that day, although more limited in scope.<br /><br />During the service disruption, users were again automatically re-routed to healthy model hosts, minimizing impact to users and we are updating our monitors and failover mechanism to mitigate similar issues in the future.
We have recovered from our earlier performance issues. Copilot code completions should be functioning normally at this time.
Copilot Code Completions are partially unavailable. Our engineering team is engaged and investigating.
We are investigating reports of degraded performance for Copilot
Copilot Code Completions partially unavailable
4 updates
On November 5, 2025, between 21:46 and 23:36 UTC, ghost text requests experienced errors from upstream model providers that resulted in 0.9% of users seeing elevated error rates.<br /><br />During the service disruption, users were automatically re-routed to healthy model hosts but may have experienced increased latency in response times as a result of re-routing.<br /><br />We are updating our monitors and tuning our failover mechanism to more quickly mitigate issues like this in the future.
We have identified and resolved the underlying issues with Code Completions. Customers should see full recovery.
We are investigating increased error rates affecting Copilot Code Completions. Some users may experience delays or partial unavailability. Our engineering team is monitoring the situation and working to identify the cause.
We are investigating reports of degraded performance for Copilot
Incident With GitHub Enterprise Importer
1 update
On November 4, 2025, GitHub Enterprise Importer experienced a period of degraded migration performance and elevated error rates between 18:04 UTC and 23:36 UTC. During this interval customers queueing and running migrations experienced prolonged queue times and slower processing. The degradation was ultimately connected to higher than normal system load, once load was reduced error rates returned to normal. The investigation is ongoing to pinpoint the precise root cause and prevent future recurrence. Long-term work is planned to strengthen system resilience under high load and promote better visibility into migration status for customers.
Incident with Packages
8 updates
On November 3, 2025, between 14:10 UTC and 19:20 UTC, GitHub Packages experienced degraded performance, resulting in failures for 0.5% of Nuget package download requests. The incident resulted from an unexpected change in usage patterns affecting rate limiting infrastructure in the Packages service.<br /><br />We mitigated the issue by scaling up services and refining our rate limiting implementation to ensure more consistent and reliable service for all users. To prevent similar problems, we are enhancing our resilience to shifts in usage patterns, improving capacity planning, and implementing better monitoring to accelerate detection and mitigation in the future.
We have applied the mitigation and are starting to see signs of recovery. We will continue to monitor the health of the system.
We are continuing to work on mitigation.
Progress on mitigation continues but no recovery seen yet to error rates. We will continue to provide updates as we have them.
We are continuing to see high error rates for package downloads. Our team is working on ways to mitigate this urgently<br /><br />Next update in 20 minutes
Our investigations are continuing and we are working to mitigate impact. Thank you for your patience as we work on this.
We are seeing increased failure rates of up to 15% for GitHub Packages downloads with users experiencing 5xx errors.<br /><br />We are investigating and working towards mitigation. We will continue to provide updates as they are available.
We are investigating reports of degraded performance for Packages
Incident with using workflow_dispatch for Actions
6 updates
On November 1, 2025, between 2:30 UTC and 6:14 UTC, Actions workflows could not be triggered manually from the UI. This impacted all customers queueing workflows from the UI for most of the impact window. The issue was caused by a faulty code change in the UI, which was promptly reverted once the impact was identified. Detection was delayed due to an alerting gap for UI breaks in this area when all underlying APIs are still healthy. We are implementing enhanced alerting and additional automated tests to prevent similar regressions and reduce detection time in the future.
Actions is operating normally.
We have mitigated the issue for manually dispatching workflows via the UI
We have identified the cause of the issue and are working towards a mitigation
We are investigating issues manually dispatching workflows via the GitHub UI for Actions. The Workflow Dispatch API is unaffected.
We are investigating reports of degraded performance for Actions
Disruption with some GitHub services
4 updates
On October 30th we shipped a change that broke 3 links in the "Solutions" dropdown of the marketing navigation seen on https://github.com/home. We noticed internally the broken links and declared an incident so our users would know no other functionality was impacted. We were able to revert a change and are evaluating our testing and rollout processes to prevent future incidents like these.
Links on GitHub's landing https://github.com/home are not working. Primary user workflows (PRs, Issues, Repos) are not impacted.
Dotcom main navigation broken links.
We are currently investigating this issue.
Disruption with Copilot Bing search tool
2 updates
A cloud resource used by the Copilot bing-search tool was deleted as part of a resource cleanup operation. Once this was discovered, the resource was recreated. Going forward, more effective monitoring will be put in place to catch this issue earlier.
We are currently investigating this issue.
Experiencing connection issues across Actions, Codespaces, and possibly other services
17 updates
On October 29th, 2025 between 14:07 UTC and 23:15 UTC, multiple GitHub services were degraded due to a broad outage in one of our service providers:<br /><br />- Users of Codespaces experienced failures connecting to new and existing Codespaces through VSCode Desktop or Web. On average the Codespace connection error rate was 90% and peaked at 100% across all regions throughout the incident period.<br />- GitHub Actions larger hosted runners experienced degraded performance, with 0.5% of overall workflow runs and 9.8% of larger hosted runner jobs failing or not starting within 5 minutes. These recovered by 20:40 UTC.<br />- The GitHub Enterprise Importer service was degraded, with some users experiencing migration failures during git push operations and most users experiencing delayed migration processing.<br />- Initiation of new trials for GitHub Enterprise Cloud with Data Residency were also delayed during this time.<br />- Copilot Metrics via the API could not access the downloadable link during this time. There were approximately 100 requests during the incident that would have failed the download. Recovery began around 20:25 UTC.<br /><br />We were able to apply a number of mitigations to reduce impact over the course of the incident, but we did not achieve 100% recovery until our service provider’s incident was resolved.<br /><br />We are working to reduce critical path dependencies on the service provider and gracefully degrade experiences where possible so that we are more resilient to future dependency outages.
Codespaces is operating normally.
Codespaces continues to recover
Actions is operating normally.
Actions has fully recovered.<br /><br />Codespaces continues to recover. Regions across Europe and Asia are healthy, so US users may want to choose one of those regions following these instructions: https://docs.github.com/en/codespaces/setting-your-user-preferences/setting-your-default-region-for-github-codespaces.<br /><br />We expect full recovery across the board before long.
Codespaces is experiencing degraded performance. We are continuing to investigate.
We are beginning to see small signs of recovery, but connections are still inconsistent across services and regions. We expect to see gradual recovery from here.
We continue to see improvement in Actions larger runners jobs. Larger runners customers may still experience longer than normal queue times, but we expect this to rapidly improve across most runners. <br /><br />ARM64 runners, GPU runners, and some runners with private networking may still be impacted.<br /><br />Usage of Codespaces via VS Code (but not via SSH) is still degraded.<br /><br />GitHub and Azure teams continue to collaborate towards full resolution.
Codespaces is experiencing degraded availability. We are continuing to investigate.
Codespaces is experiencing degraded performance. We are continuing to investigate.
Impact to most larger runner jobs should now be mitigated. ARM64 runners are still impacted. GitHub and Azure teams continue to collaborate towards full resolution.
Codespaces is experiencing degraded availability. We are continuing to investigate.
Additional impact from this incident:<br /><br />We’re currently investigating an issue causing the Copilot Metrics API report URLs to fail for 28-day and 1-day enterprise and user reports. We are collaborating with Azure teams to restore access as soon as possible.
We are seeing ongoing connection failures across Codespaces and Actions, including Enterprise Migrations. <br /><br />Linux ARM64 standard hosted runners are failing to start, but Ubuntu latest and Windows latest are not affected at this time. <br /><br />SSH connections to Codespaces may be successful, but connections via VS Code are consistently failing. <br /><br />GitHub and Azure teams are coordinating to mitigate impact and resolve the root issues.
Actions impact is primarily limited to larger runner jobs at this time. This also impacts enterprise migrations.
Codespaces is experiencing degraded performance. We are continuing to investigate.
We are investigating reports of degraded performance for Actions
Inconsistent results when using the Haiku 4.5 model
4 updates
From October 28th at 16:03 UTC until 17:11 UTC, the Copilot service experienced degradation due to an infrastructure issue which impacted the Claude Haiku 4.5 model, leading to a spike in errors affecting 1% of users. No other models were impacted. The incident was caused due to an outage with an upstream provider. We are working to improve redundancy during future occurrences.
The issues with our upstream model provider have been resolved, and Claude Haiku 4.5 is once again available in Copilot Chat and across IDE integrations.<br /><br />We will continue monitoring to ensure stability, but mitigation is complete.
Usage of the Haiku 4.5 model with Copilot experiences is currently degraded. We are investigating and working to remediate. Other models should be unaffected.
We are currently investigating this issue.
5 updates
Between October 23, 2025 19:27:29 UTC and October 27, 2025 17:42:42 UTC, users experienced timeouts when viewing repository landing pages. We observed the timeouts for approximately 5,000 users across less than 1,000 repositories including forked repositories. The impact was limited to logged in users accessing repositories in organizations with more than 200,000 members. Forks of repositories from affected large organizations were also impacted. Git operations were functional throughout this period.<br /><br />This was caused by feature flagged changes impacting organization membership. The changes caused unintended timeouts for organization membership count evaluations which led to repository landing pages not loading.<br /><br />The flag was turned off and a fix addressing the timeouts was deployed, including additional optimizations to better support organizations of this size. We are reviewing related areas and will continue to monitor for similar performance regressions.
We have deployed the fix and resolved the issue.
The fix for this issue has been validated and is being deployed. This fix will also resolve related timeouts on the Access settings page of the impacted repositories and forks.
Viewing code in repositories in or forked from very large organizations (200k+ members) are not loading in the desktop web UI, showing a unicorn instead. A fix has been identified and is being tested. Access via git and access to specific pages within the repository, such as pull requests, are not impacted, nor is accessing the repository via the mobile web UI.
We are currently investigating this issue.
1 update
On UTC Oct 24 2:55 - 3:15 AM, githubstatus.com was unreachable due to service interruption with our status page provider. During this time, GitHub systems were not experiencing any outages or disruptions. We are working our vendor to understand how to improve availability of githubstatus.com.
3 updates
From Oct 22, 2025 15:00 UTC to Oct 24, 2025 14:30 UTC git operations via SSH saw periods of increased latency and failed requests, with failure rates ranging from 1.5% to a single spike of 15%. Git operations over http were not affected. This was due to resource exhaustion on our backend ssh servers. <br /><br />We mitigated the incident by increasing the available resources for ssh connections. We are improving the observability and dynamic scalability of our backend to prevent issues like this in the future.
We have found the source of the slowness and mitigated it. We are watching recovery before we status green but no user impact is currently observed.
We are currently investigating this issue.
Incident with Actions - Larger hosted runners
8 updates
On October 23, 2025, between 15:54 UTC and 19:20 UTC, GitHub Actions larger hosted runners experienced degraded performance, with 1.4% of overall workflow runs and 29% of larger hosted runner jobs failing to start or timing out within 5 minutes.<br /><br />The full set of contributing factors is still under investigation, but the customer impact was due to database performance degradation, triggered by routine database changes causing a load profile that triggered a bug in the underlying database platform used for larger runners.<br /><br />Impact was mitigated through a combination of scaling up the database and reducing load. We are working with partners to resolve the underlying bug and have paused similar database changes until it is resolved.
Actions is operating normally.
Actions larger runner job start delays and failure rates are recovering. Many jobs should be starting as normal. We're continuing to monitor and confirm full recovery.
We continue to investigate problems with Actions larger runners. We're continuing to see signs of improvement, but customers are still experiencing jobs queueing or failing due to timeout.
We continue to investigate problems with Actions larger runners. We're seeing limited signs of recovery, but customers are still experiencing jobs queueing or failing due to timeout.
We continue to investigate problems with Actions larger runners. Some customers are experiencing jobs queueing or failing due to timeout.
We're investigating problems with larger hosted runners in Actions. Our team is working to identify the cause. We'll post another update by 17:03 UTC.
We are investigating reports of degraded performance for Actions
Incident with API Requests
5 updates
On October 22, 2025, between 14:06 UTC and 15:17 UTC, less than 0.5% of web users experienced intermittent slow page loads on GitHub.com. During this time, API requests showed increased latency, with up to 2% timing out. <br /><br />The issue was caused by elevated loads on one of our databases caused by a poorly performing query, which impacted performance for a subset of requests.<br /><br />We identified the source of the load and optimized the query to restore normal performance. We’ve added monitors for early detection for query performance, and we continue to monitor the system closely to ensure ongoing stability. <br />
API Requests is operating normally.
We have identified a possible source of the issue and there is currently no user impact but we are continuing to investigate and will not resolve this incident until we have more confidence in our mitigations and investigation results.
Some users may see slow, timing out requests or not found when browsing repos. We have identified slowness in our platform and are investigating.
We are investigating reports of degraded performance for API Requests
Disruption with some GitHub services
6 updates
On October 21, 2025, between 13:30 and 17:30 UTC, GitHub Enterprise Cloud Organization SAML Single Sign-On experienced degraded performance. Customers may have been unable to successfully authenticate into their GitHub Organizations during this period. Organization SAML recorded a maximum of 0.4% of SSO requests failing during this timeframe.<br /><br />This incident stemmed from a failure in a read replica database partition responsible for storing license usage information for GitHub Enterprise Cloud Organizations. This partition failure resulted in users from affected organizations, whose license usage information was stored on this partition, being unable to access SSO during the aforementioned window. A successful SSO requires an available license for the user who is accessing a GitHub Enterprise Cloud Organization backed by SSO.<br />The failing partition was subsequently taken out of service, thereby mitigating the issue. <br /><br />Remedial actions are currently underway to ensure that a read replica failure does not compromise the overall service availability.<br />
Mitigation continues, the impact is limited to Enterprise Cloud customers who have configured SAML at the organization level.
We continuing to work on mitigation of this issue.
We’ve identified the issue affecting some users with SAML/OIDC authentication and are actively working on mitigation. Some users may not be able to authenticate during this time.
We're seeing issues for a small amount of customers with SAML/OIDC authentication for GitHub.com users. We are investigating.
We are currently investigating this issue.
Incident with Actions
6 updates
On October 21, 2025, between 07:55 UTC and 12:20 UTC, GitHub Actions experienced degraded performance. During this time, 2.11% workflow runs failed to start within 5 minutes, with an average delay of 8.2 minutes. The root cause was increased latency on a node in one of our Redis clusters, triggered by resource contention after a patching event became stuck. <br /><br />Recovery began once the patching process was unstuck and normal connectivity to the Redis cluster was restored at 11:45 UTC, but it took until 12:20 UTC to clear the backlog of queued work. We are implementing safeguards to prevent this failure mode and enhancing our monitoring to detect and address problems like this more quickly in the future.
We were able to apply a mitigation and we are now seeing recovery.
We are seeing about 10% of Actions runs taking longer than 5 minutes to start, we're still investigating and will provide an update by 12:00 UTC.
We are still seeing delays in starting some Actions runs and are currently investigating. We will provide updates as we have more information.
We are seeing delays in starting some Actions runs and are currently investigating.
We are investigating reports of degraded performance for Actions
Disruption with Grok Code Fast 1 in Copilot
5 updates
From October 20th at 14:10 UTC until 16:40 UTC, the Copilot service experienced degradation due to an infrastructure issue which impacted the Grok Code Fast 1 model, leading to a spike in errors affecting 30% of users. No other models were impacted. The incident was caused due to an outage with an upstream provider.
The issues with our upstream model provider continue to improve, and Grok Code Fast 1 is once again stable in Copilot Chat, VS Code and other Copilot products.
We are continuing to work with our provider on resolving the incident with Grok Code Fast 1 which is impacting 6% of users. We’ve been informed they are implementing fixes but users can expect some requests to intermittently fail until all issues are resolved.<br />
We are experiencing degraded availability for the Grok Code Fast 1 model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.
We are investigating reports of degraded performance for Copilot
Codespaces creation failling
6 updates
On October 20, 2025, between 08:05 UTC and 10:50 UTC the Codespaces service was degraded, with users experiencing failures creating new codespaces and resuming existing ones. On average, the error rate for codespace creation was 39.5% and peaked at 71% of requests to the service during the incident window. Resume operations averaged 23.4% error rate with a peak of 46%. This was due to a cascading failure triggered by an outage in a 3rd-party dependency required to build devcontainer images.<br /><br />The impact was mitigated when the 3rd-party dependency recovered.<br /><br />We are investigating opportunities to make this dependency not a critical path for our container build process and working to improve our monitoring and alerting systems to reduce our time to detection of issues like this one in the future.
We are now seeing sustained recovery. As we continue to make our final checks, we hope to resolve this incident in the next 10 minutes.
We are seeing early signs of recovery for Codespaces. The team will continue to monitor and keep this incident active as a line of communication until we are confident of full recovery.
We are continuing to monitor Codespace's error rates and will report further as we have more information.
We are seeing increased error rates with Codespaces generally. This is due to a third party provider experiencing problems. This impacts both creation of new Codespaces and resumption of existing ones.<br /><br />We continue to monitor and will report with more details as we have them.
We are investigating reports of degraded availability for Codespaces
Disruption with push notifications
3 updates
On October 17th, 2025, between 12:51 UTC and 14:01 UTC, mobile push notifications failed to be delivered for a total duration of 70 minutes. This affected github.com and GitHub Enterprise Cloud in all regions. The disruption was related to an erroneous configuration change to cloud resources used for mobile push notification delivery.<br /><br />We are reviewing our procedures and management of these cloud resources to prevent such an incident in the future.
We're investigating an issue with mobile push notifications. All notification types are affected, but notifications remain accessible in the app's inbox. For 2FA authentication, please open the GitHub mobile app directly to complete login.
We are currently investigating this issue.
Disruption with some GitHub services
2 updates
On October 14th, 2025, between 18:26 UTC and 18:57 UTC a subset of unauthenticated requests to the commit endpoint for certain repositories received 503 errors. During the event, the average error rate was 3%, peaking at 3.5% of total requests.<br /><br />This event was triggered by a recent configuration change and some traffic pattern shifts on the service. We were alerted of the issue immediately and made changes to the configuration in order to mitigate the problem. We are working on automatic mitigation solutions and better traffic handling in order to prevent issues like this in the future.
We are currently investigating this issue.
Disruption with GPT-5-mini in Copilot
6 updates
On Oct 14th, 2025, between 13:34 UTC and 16:00 UTC the Copilot service was degraded for GPT-5 mini model. On average, 18% of the requests to GPT-5 mini failed due to an issue with our upstream provider.<br /><br />We notified the upstream provider of the problem as soon as it was detected and mitigated the issue by failing over to other providers. The upstream provider has since resolved the issue.<br /><br />We are working to improve our failover logic to mitigate similar upstream failures more quickly in the future.
GPT-5-mini is once again available in Copilot Chat and across IDE integrations.<br /><br />We will continue monitoring to ensure stability, but mitigation is complete.
We are continuing to see degraded availability for the GPT-5-mini model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We continue to work with the model provider to resolve the issue.<br />Other models continue to be available and working as expected.
We continue to see degraded availability for the GPT-5-mini model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We continue to work with the model provider to resolve the issue.<br />Other models continue to be available and working as expected.
We are experiencing degraded availability for the GPT-5-mini model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br />Other models are available and working as expected.
We are investigating reports of degraded performance for Copilot
Incident with Webhooks
25 updates
On October 9th, 2025, between 14:35 UTC and 15:21 UTC, a network device in maintenance mode that was undergoing repairs was brought back into production before repairs were completed. Network traffic traversing this device experienced significant packet loss.<br /><br />Authenticated users of the github.com UI experienced increased latency during the first 5 minutes of the incident. API users experienced up to 7.3% error rates, after which it stabilized to about 0.05% until mitigated. Actions service experienced 24% of runs being delayed for an average of 13 minutes. Large File Storage (LFS) requests experienced minimally increased error rate, with 0.038% of requests erroring.<br /><br />To prevent similar issues, we are enhancing the validation process for device repairs of this category.
All services have fully recovered.
Actions has fully recovered but Notifications is still experiencing delays. We will continue to update as the system is fully restored to normal operation.
Actions is operating normally.
Pages is operating normally.
Git Operations is operating normally.
Actions and Notifications are still experiencing delays as we process the backlog. We will continue to update as the system is fully restored to normal operation.
Pull Requests is operating normally.
Actions is experiencing degraded performance. We are continuing to investigate.
We are seeing full recovery in many of our systems, but delays are still expected for actions. We will continue to update as the system is fully restored to normal operation.
Webhooks is operating normally.
Webhooks is experiencing degraded performance. We are continuing to investigate.
Issues is operating normally.
Pull Requests is experiencing degraded performance. We are continuing to investigate.
API Requests is operating normally.
We identified a faulty network component and have removed it from the infrastructure. Recovery has started and we expect full recovery shortly.
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Git Operations is experiencing degraded performance. We are continuing to investigate.
Actions is experiencing degraded availability. We are continuing to investigate.
We are investigating widespread reports of delays and increased latency in various services. We will continue to keep users updated on progress toward mitigation.
Issues is experiencing degraded availability. We are continuing to investigate.
API Requests is experiencing degraded performance. We are continuing to investigate.
Pages is experiencing degraded performance. We are continuing to investigate.
Actions is experiencing degraded performance. We are continuing to investigate.
We are investigating reports of degraded availability for Webhooks
Multiple GitHub API endpoints are experiencing errors
3 updates
Between 13:39 UTC and 13:42 UTC on Oct 9, 2025, around 2.3% of REST API calls and 0.4% Web traffic were impacted due to the partial rollout of a new feature that had more impact on one of our primary databases than anticipated. When the feature was partially rolled out it performed an excessive number of writes per request which caused excessive latency for writes from other API and Web endpoints and resulted in 5xx errors to customers. <br /><br />The issue was identified by our automatic alerting and reverted by turning down the percentage of traffic to the new feature, which led to recovery of the data cluster and services. <br /><br />We are working to improve the way we roll out new features like this and move the specific writes from this incident to a storage solution more suited to this type of activity. We have also optimized this particular feature to avoid its rollout from having future impact on other areas of the site. We are also investigating how we can even more quickly identify issues like this.
A feature was partially rolled out that had high impact on one of our primary databases but we were able to roll it back. All services are recovered but we will monitor for recovery before statusing green.
We are currently investigating this issue.
Disruption with some GitHub services
7 updates
On October 7, 2025, between 7:48 PM UTC and October 8, 12:05 AM UTC (approximately 4 hours and 17 minutes), the audit log service was degraded, creating a backlog and delaying availability of new audit log events. The issue originated in a third-party dependency.<br /><br />We mitigated the incident by working with the vendor to identify and resolve the issue. Write operations recovered first, followed by the processing of the accumulated backlog of audit log events.<br /><br />We are working to improve our monitoring and alerting for audit log ingestion delays and strengthen our incident response procedures to reduce our time to detection and mitigation of issues like this one in the future.
We are seeing recovery of audit log ingestion and continue to monitor recovery.
We are seeing recovery of audit log ingestion and continue to monitor recovery.
We continue to apply mitigations and monitor for recovery.
We have identified an issue causing delayed audit log event ingestion and are working on a mitigation.
Ingestion of new audit log events is delayed
We are currently investigating this issue.
Incident with Copilot
5 updates
<p>On October 3rd, between approximately 10:00 PM and 11:30 Eastern, the Copilot service experienced degradation due to an issue with our upstream provider. Users encountered elevated error rates when using the following Claude models: Claude Sonnet 3.7, Claude Opus 4, Claude Opus 4.1, Claude Sonnet 4, and Claude Sonnet 4.5. No other models were impacted.</p><p>The issue was mitigated by temporarily disabling affected endpoints while our provider resolved the upstream issue. GitHub is working with our provider to further improve the resiliency of the service to prevent similar incidents in the future.</p>
This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
The upstream provider is implementing a fix. Services are recovering. We are monitoring the situation.
We’re seeing degraded experience across Anthropic models. We’re working with our partners to restore service.
We are investigating reports of degraded performance for Copilot
Degraded Gemini 2.5 Pro experience in Copilot
8 updates
Between October 1st, 2025 at 1 AM UTC and October 2nd, 2025 at 10:33 PM UTC, the Copilot service experienced a degradation of the Gemini 2.5 Pro model due to an issue with our upstream provider. Before 15:53 UTC on October 1st, users experienced higher error rates with large context requests while using Gemini 2.5 Pro. After 15:53 UTC and until 10:33 PM UTC on October 2nd, requests were restricted to smaller context windows when using Gemini 2.5. Pro. No other models were impacted.<br /><br />The issue was resolved by a mitigation put in place by our provider. GitHub is collaborating with our provider to enhance communication and improve the ability to reproduce issues with the aim to reduce resolution time.
We have confirmed that the fix for the lower token input limit for Gemini 2.5 Pro is in place and are currently testing our previous higher limit to verify that customers will experience no further impact.
The underlying issue for the lower token limits for Gemini 2.5 Pro has been identified and a fix is in progress. We will update again once we have tested and confirmed that the fix is correct and globally deployed.
We are continuing to work with our provider to resolve the issue where some Copilot requests using Gemini 2.5 Pro return an error indicating a bad request due to exceeding the input limit size.
We are continuing to investigate and test solutions internally while working with our model provider on a deeper investigation into the cause. We will update again when we have identified a mitigation.
We are testing other internal mitigations so that we can return to the higher maximum input length. We are still working with our upstream model provider to understand the contributing factors for this sudden decrease in input limits.
We are experiencing a service regression for the Gemini 2.5 Pro model in Copilot Chat, VS Code and other Copilot products. The maximum input length of Gemini 2.5 prompts been decreased. Long prompts or large context windows may result in errors. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.
We are investigating reports of degraded performance for Copilot
Degraded Performance for GitHub Actions MacOS Runners
15 updates
On October 1, 2025 between 07:00 UTC and 17:20 UTC, Mac hosted runner capacity for Actions was degraded, leading to timed out jobs and long queue times. On average, the error rate was 46% and peaked at 96% of requests to the service. XL and Intel runners recovered by 10:10 UTC, with the other types taking longer to recover.<br /><br />The degraded capacity was triggered by a scheduled event at 07:00 UTC that led to a permission failure on Mac runner hosts, blocking reimage operations. The permission issue was resolved by 9:41 UTC, but the recovery of available runners took longer than expected due to a combination of backoff logic slowing backend operations and some hosts needing state resets.<br /><br />We deployed changes immediately following the incident to address the scheduled event and ensure that similar failures will not block critical operations in the future. We are also working to reduce the end-to-end time for self-healing of offline hosts for quicker full recovery of future capacity or host events.
We are seeing some recovery for image queueing and continuing to monitor.
We are continuing work to restore capacity for our MacOS ARM runners.
Our team continues to work hard on restoring capacity for the Mac runners.
Work continues on restoring capacity on the Mac runners.
MacOS ARM runners continue to be at reduced capacity, causing queuing of jobs. Investigation is ongoing.
Work continues to bring the full runner capacity back online. Resources are focused on improving the recovery of certain runner types.
We are continuing to see recovery of some runner capacity and investigating slow recovery of certain runner types.
We are seeing recovery of some runner capacity, while also investigating slow recovery of certain runner types.
MacOS runners are coming back online and starting to process queued work.
We are continuing to deploy the necessary changes to restore MacOS runner capacity.
We have identified the cause and are deploying a change to restore MacOS runner capacity.
Customers using GitHub Actions Mac OS runners are experiencing job start delays and failures. We are aware of this issue and actively investigating.
Actions is experiencing degraded performance. We are continuing to investigate.
We are currently investigating this issue.