ALVSCE Web Server 1pm Update (FINAL)

[Agriculture, Life and Veterinary Sciences, and Cooperative Extension Weekly Bulletin]

Good News Division,

UITS has concluded that our entire web server was not compromised, but limited to a vulnerability within a single WordPress website. They have given us the green light to bring back the dozens of unaffected sites online.

WordPress is a common attack surface due to its popularity and low barrier of use. While we had the capability to host these sites, we will move forward with migrating our handful of WordPress users to another provider while maintaining and increasing our security around the sites we actively manage. Thank you again for your patience during this outage. I've included the summary of events below for those who are interested.

Aug 30, 16:59: CCT-WebDev noticed the issue with a WordPress website on our server
Aug 30 17:00: CCT killed the process linked to the WordPress Website
Aug 30, 17:11: CCT disabled the WordPress website
Aug 30, 17:34: UITS reached out to Matt Rahr regarding activity on the WordPress Website
Aug 30, 18:08: UITS and Matt conclude that the incident was adequately neutralized and no further immediate action was needed.
Aug 30 19:31: UITS notifies me via email and Teams that they have reversed course and will take our webserver off the network until further notice. Websites are brought offline.
Aug 30, 21:33: CCT learns that our entire server has been taken off the network. Begins strategizing workarounds with backup instances of the server.
Aug 30, 23:08: CCT restores backup instance (not serving pages, no web access). Backup was generated from a snapshot from Aug 30 00:21:06 (before the attack took place)
Aug 30, 23:08 - 2:50: CCT investigates evidence of similar attack symptoms being present on the backup restoration
Aug 31, 02:50: CCT removes all WordPress sites on the server restored from backup and replaces them with static HTML maintenance pages with no codebases
Aug 31, 06:18 UITS blesses the solution with our backup server while we investigate the compromised server in parallel. Websites are brought online.
Aug 31, 07:14 CCT meets with Executive Directors at UITS to inform us they no longer trust the backup server solution and want to begin forensic analysis of the entire server. Outline next steps.
Aug 31, 08:30 CCT meets with the UITS Forensic Security team to discuss the data required for the analysis and the proper/secure transfer of data (snapshots, logs, etc.) to UITS.
Aug 31, 10:03 CCT has provided an initial round of logs and file snapshots. Investigation begins.
Aug 31, 10:30 CCT's backup instance is taken offline at UITS-ISO request and returned to the maintenance message. Websites are brought offline.
Sept 1, 11:00 - 12:00 CCT and UITS meet to discuss preliminary findings and conclude with "moderate confidence" that the incident is limited to the WordPress environment on the ALVSCE web server. UITS gives CCT The green light to restore websites.
Sept 1, 12:23: CCT restores all websites, minus 4 WordPress websites. Websites are brought online.

I want to thank you, our users, for your patience and understanding. I received several emails of support throughout this process. We have a great team that worked throughout the night to limit our downtime while UITS investigated our server. Please reach out if you have any questions.

Cheers,
Matt


[The University of Arizona block 'A' logo.]
Matt Rahr
Director, Cyber & Information Technologies
Division of Agriculture, Life & Veterinary Sciences and Cooperative Extension
THE UNIVERSITY OF ARIZONA

Forbes Building, 230
1140 E South Campus Dr | Tucson, AZ 85721
Office: 520-621-1212
rahr at ag.arizona.edu<mailto:rahr at ag.arizona.edu>

Communications & Cyber Technologies Unit
Landmark Stories | CCT Data Science Team
cct.arizona.edu<https://cct.cals.arizona.edu/>
landmarkstories.arizona.edu<https://landmarkstories.arizona.edu/>
datascience.cals.arizona.edu<https://datascience.cals.arizona.edu/>
facebook<https://www.facebook.com/LandmarkStories/> | twitter<https://twitter.com/StoriesLandmark> | instagram<https://www.instagram.com/landmarkstories/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.cals.arizona.edu/pipermail/alvsce_bulletin/attachments/20230901/3fd511fa/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 14175 bytes
Desc: image001.png
URL: <https://list.cals.arizona.edu/pipermail/alvsce_bulletin/attachments/20230901/3fd511fa/attachment-0001.png>