<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Good News Division,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">UITS has concluded that our <i>entire</i> web server was not compromised, but limited to a vulnerability within a single WordPress website. They have given us the green light to bring back the dozens of unaffected sites online.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">WordPress is a common attack surface due to its popularity and low barrier of use. While we had the capability to host these sites, we will move forward with migrating our handful of WordPress users to another provider while maintaining
and increasing our security around the sites we actively manage. Thank you again for your patience during this outage. I've included the summary of events below for those who are interested.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Aug 30, 16:59: CCT-WebDev noticed the issue with a WordPress website on our server<br>
Aug 30 17:00: CCT killed the process linked to the WordPress Website<br>
Aug 30, 17:11: CCT disabled the WordPress website<o:p></o:p></p>
<p class="MsoNormal">Aug 30, 17:34: UITS reached out to Matt Rahr regarding activity on the WordPress Website<o:p></o:p></p>
<p class="MsoNormal">Aug 30, 18:08: UITS and Matt conclude that the incident was adequately neutralized and no further immediate action was needed.<o:p></o:p></p>
<p class="MsoNormal">Aug 30 19:31: UITS notifies me via email and Teams that they have reversed course and will take our webserver off the network until further notice.
<b>Websites are brought offline.</b><o:p></o:p></p>
<p class="MsoNormal">Aug 30, 21:33: CCT learns that our entire server has been taken off the network. Begins strategizing workarounds with backup instances of the server.<o:p></o:p></p>
<p class="MsoNormal">Aug 30, 23:08: CCT restores backup instance (not serving pages, no web access). Backup was generated from a snapshot from Aug 30 00:21:06 (before the attack took place)<br>
Aug 30, 23:08 - 2:50: CCT investigates evidence of similar attack symptoms being present on the backup restoration<o:p></o:p></p>
<p class="MsoNormal">Aug 31, 02:50: CCT removes all WordPress sites on the server restored from backup and replaces them with static HTML maintenance pages with no codebases<br>
Aug 31, 06:18 UITS blesses the solution with our backup server while we investigate the compromised server in parallel.
<b>Websites are brought online.<o:p></o:p></b></p>
<p class="MsoNormal">Aug 31, 07:14 CCT meets with Executive Directors at UITS to inform us they no longer trust the backup server solution and want to begin forensic analysis of the entire server. Outline next steps.<o:p></o:p></p>
<p class="MsoNormal">Aug 31, 08:30 CCT meets with the UITS Forensic Security team to discuss the data required for the analysis and the proper/secure transfer of data (snapshots, logs, etc.) to UITS.<o:p></o:p></p>
<p class="MsoNormal">Aug 31, 10:03 CCT has provided an initial round of logs and file snapshots. Investigation begins.<o:p></o:p></p>
<p class="MsoNormal">Aug 31, 10:30 CCT’s backup instance is taken offline at UITS-ISO request and returned to the maintenance message.
<b>Websites are brought offline.<o:p></o:p></b></p>
<p class="MsoNormal">Sept 1, 11:00 – 12:00<b> </b>CCT and UITS meet to discuss preliminary findings and conclude with “moderate confidence” that the incident is limited to the WordPress environment on the ALVSCE web server. UITS gives CCT The green light to
restore websites.<o:p></o:p></p>
<p class="MsoNormal">Sept 1, 12:23: CCT restores all websites, minus 4 WordPress websites.
<b>Websites are brought online.<o:p></o:p></b></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal">I want to thank you, our users, for your patience and understanding. I received several emails of support throughout this process. We have a great team that worked throughout the night to limit our downtime while UITS investigated our server.
Please reach out if you have any questions.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<br>
Matt<o:p></o:p></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td valign="top" style="padding:0in 0in 0in 0in">
<p class="MsoNormal"><img width="120" height="96" style="width:1.25in;height:1.0in" id="Picture_x0020_1" src="cid:image001.png@01D9DCCC.F3591140" alt="The University of Arizona block 'A' logo."><span style="mso-ligatures:none"><o:p></o:p></span></p>
</td>
<td width="500" valign="top" style="width:375.0pt;padding:0in 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:14.0pt;color:#0C234B;mso-ligatures:none">Matt Rahr</span></b><span style="font-size:10.0pt;color:#495057;mso-ligatures:none"><br>
Director, Cyber & Information Technologies<br>
Division of Agriculture, Life & Veterinary Sciences and Cooperative Extension<br>
THE UNIVERSITY OF ARIZONA<br>
<br>
Forbes Building, 230<br>
1140 E South Campus Dr | Tucson, AZ 85721<br>
Office: 520-621-1212 <br>
<a href="mailto:rahr@ag.arizona.edu"><span style="color:#495057">rahr@ag.arizona.edu</span></a><br>
<br>
Communications & Cyber Technologies Unit</span><span style="mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;color:#495057;mso-ligatures:none">Landmark Stories | CCT Data Science Team</span><span style="mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><u><span style="font-size:10.0pt;color:#8B0015;mso-ligatures:none"><a href="https://cct.cals.arizona.edu/"><span style="color:#8B0015">cct.arizona.edu</span></a>
<br>
</span></u></b><u><span style="font-size:10.0pt;color:#8B0015;mso-ligatures:none"><a href="https://landmarkstories.arizona.edu/"><span style="color:#8B0015">landmarkstories.arizona.edu</span></a><br>
<a href="https://datascience.cals.arizona.edu/"><span style="color:#8B0015">datascience.cals.arizona.edu</span></a></span></u><span style="mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#495057;mso-ligatures:none"><a href="https://www.facebook.com/LandmarkStories/"><span style="font-size:9.0pt;color:#8B0015">facebook</span></a></span><span style="font-size:9.0pt;color:#767676;mso-ligatures:none">
| </span><span style="font-size:10.0pt;color:#495057;mso-ligatures:none"><a href="https://twitter.com/StoriesLandmark"><span style="font-size:9.0pt;color:#8B0015">twitter</span></a></span><span style="font-size:9.0pt;color:#767676;mso-ligatures:none"> |
</span><span style="font-size:10.0pt;color:#495057;mso-ligatures:none"><a href="https://www.instagram.com/landmarkstories/"><span style="font-size:9.0pt;color:#8B0015">instagram</span></a></span><span style="font-size:9.0pt;color:#767676;mso-ligatures:none">
</span><span style="mso-ligatures:none"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-ligatures:none"> </span><o:p></o:p></p>
</div>
</body>
</html>