Sarah Gooding · October 30, 2020 · 17 Comments
WordPress’ Core systems team had an eventful Friday when an error in the auto-update system caused sites to update to WordPress 5.5.3-alpha-49449, including live production sites with no auto-update constants defined.
Those who received an email about the update logged into their sites to see the message: “BETA TESTERS: This site is set up to install updates of future beta versions automatically.”
Shaun Rieman logged the first ticket about sites being updated to 5.5.3-alpha-49449, which was also incidentally his first WordPress trac ticket. More users and developers confirmed the issue.
“It’s worth noting that there’s no functional difference between 5.5.2 and 5.5.3-alpha, so there’s no need to worry in that regard,” core committer John Blackbourn said.
Sites that were accidentally updated also installed all the default themes released over the last decade, as well as Akismet. Developers will need to manually delete the bundled themes that they don’t need.
In under an hour, all affected sites were automatically returned to 5.5.2, but the incident has eroded trust and damaged confidence in the auto-update system. Several commenting on the ticket asked how they can explicitly disable development updates.
“The worrying thing is that a single developer can do this, seemingly without any checking or confirmation by other developers,” UK-based developer Paul Stenning said. “This is a serious security concern as a rogue developer could push out malicious code in an update that nobody else checks.”
WordPress agency owner Rob Migchels, who had approximately 50 websites affected, tracked 18 minutes between the the trac ticket (#51679) and receiving the fix.
“The 5.5.3-alpha issue is a side effect of another issue that occurred on 5.5.2,” WordPress engineer and 5.6 Triage release lead Tonya Mork said. Jake Spurlock published an official statement regarding the incident as part of the 5.5.3 release notes:
“Earlier today — between approximately 15:30 and 16:00 UTC — the auto-update system for WordPress updated some sites from version 5.5.2 to version 5.5.3-alpha. This auto-update was due to an error in the Updates API caused by the 5.5.3 release preparations.”
Spurlock elaborated on the technical details in a separate post on the WordPress development blog:
While work was being done to prepare for WordPress 5.5.3, the release team attempted to make 5.5.2 unavailable for download on WordPress.org to limit the spread of the issue noted in the section above, as the error only affected fresh installations. This action resulted in some installations being updated to a pre-release “5.5.3-alpha” version.
In a situation like this, where users who haven’t elected to run their live sites on beta releases are getting a forced update, site owners might wonder whether this update is actually arriving from WordPress, or if the system has been hijacked.
Security researcher Slavco Mihajloski, who commented last week on the lack of transparency regarding how automatic updates are tested and performed, said this incident highlights the need for more openness surrounding the process.
“Why is transparency important? Because procedure will become public and when public, the community will be able to contribute in order to improve it,” Mihajloski said. “At the moment it is more than obvious that this process and the whole security at WP.dot org lacks QA and QC. Each task is left to an individual or closed group. Imagine the following: what if an automatic security update could be pushed only if: – X out of Y (where X < Y) authorities agree that update is fine – have a pilot update on let’s say 100 different servers (I hope .org could afford this) where regression tests will be fired against each one. The current problems would not occur.”
Automatic background updates for minor releases have saved developers thousands of hours in updating sites. A UI for allowing users to opt into automatic updates for major releases is on the roadmap for WordPress 5.6, expected in early December.
This particular accidental update has betrayed for many developers what was already a somewhat fragile trust in the auto-update system. It doesn’t shore up more confidence for selling the idea of core updates when 5.6 is released, but it doesn’t mean that auto-updates are not a good idea. WordPress.org will need to put better processes in place in order to win back users’ trust.
The incident affected more than 100 sites for WordPress agency owner Robert Staddon. He reports that they all displayed the “Update now” button with the confusing and incorrect text seen below. Staddon said the incident has not yet caused him to change his approach to allowing clients to receive auto-updates.
“I was very grateful for the extraordinarily fast response time to get the problem fixed,” Staddon said. “However, it did shake my confidence in the WordPress auto-update process. Considering the number of websites using WordPress, a mistake of this magnitude could end up having a rather catastrophic effect around the web. I would hope that the core team would be able to evaluate how this happened and consider putting some checks in place to make sure it doesn’t happen again.”
