Network Outage

Summary:

The core WOU network router pair failed to pass traffic beginning at 9:30am on January 14, 2015.  Partial network throughput was restored at 12:40pm and a full recovery occured at 9:00pm January 14, 2015.

Timeline:

  • Campus network outage began at approximately 9:30am on January 14, 2015
  • UCS responded immediately and went into diagnostic mode
  • Cisco TAC support was engaged at 10:30am
  • High CPU utilization was identified as an issue on the core campus router pair at 11:00am
  • Call placed to local Cisco representative for additional support at 11:30
  • Call placed to NERO (the WOU ISP) engineer at 12:30
  • NERO diagnostics led to finding a server that was identified as pushing an excessive amount of ARP request to the router.  The server was removed from the network at 12:40pm
  • Several networks were pulled out from behind the firewall, allowing network traffic to flow again
  • CPU utilization went from 99% to 86% after server was removed from the network
  • About 12:50 the CPU utilization had climbed back to 99% even though the server had not been reconnected to the network
  • Additional Cisco support provided about 1:00pm — at this point we had three Cisco engineers on the phone and connected to our router pair via a Webex call.
  • By late afternoon, I requested additional on-site support from Mt. States Networking.
  • A Mt. States engineer was on site by 6:00pm
  • At ~8:15pm, the router netflow process was identified as a culprit in the high CPU utilization.  After the netflows were removed, the CPU utilization fell from 99% to 23%
  • All networks were moved behind the firewall and traffic continued to flow properly.
  • The suspect host that was removed in the morning was returned to service and the CPU utilization on the router immediately climbed to 99%
  • The suspect host was removed

Forensics:

  • February 15, 2015
    • Our unix systems administrator has been reviewing the suspect servers logs and discovered the server had been compromised.  This server is running openstack OS.
    • We know that whoever compromised the server did not gain direct access to it via ssh or telnet
    • Forensics work continues…

E-mail ramblings

This rambling includes e-mail security/insecurity, archiving and content filtering.

 

The University of Washington has posted the following information on their web site:  (http://www.washington.edu/itconnect/connect/email/)

  • Privacy:  The UW email systems are provided to support UW activities and are subject to state laws and UW policy.
  • Do not send sensitive information by email. This includes personnel data, patient records, student information, and financial information.
  • Email messages can be kept and forwarded. Never assume email is private, even when using encryption technologies. The message you send to one person can easily spread to many more.

 

What e-mail security infrastructure is in place at WOU?

  • SSL is the predecessor to TLS
  • What is TLS?
    • Transport layer security is an encryption protocol that is implemented by e-mail systems and other services to prevent eavesdropping and tampering.
  • Always choose TLS rather than SSL, if TLS is an option
  • What does Google say about TLS?  Why should I use TLS?
    • This encryption makes it more difficult for hackers to intercept and read messages.
  • Transport Layer Security is available for incoming and outgoing WOU mail accounts.
    • TLS is on by default when you use the web-based mail client.
    • If you are using an external mail program such as Macmail, Outlook or a mobile device, confirm that TLS or SSL is turned on for both inbound and outbound mail.
  • When mail is sent from a WOU mail account to another WOU mail account the transmission will utilize TLS/SSL, as long as sender and recipient have TLS/SSL turned on.
  • When sending mail to a non-WOU account, all bets are off.
  • You can ensure encryption on the mail receiver end by using active encryption
    • Information regarding active encryption can be found here.

 

Is WOU e-mail filtered?

  • Spam, anti-virus and content filtering is configured.
    • Examples of content filtering include:
      • Nigerian get rich schemes, etc.
      • When there is content with malicious intent that gets through the content management system, we manually enter a rule into the system.  This would include the “Your Mailbox Is Full” email.

 

Do you have trouble finding your e-mails on occasion?

  • They may be in your spam folder
  • All unfiltered e-mails are sent to the mail archiver.
    • The archiver currently contains about three years of e-mail.
    • Your mail that ended up in trash, was deleted or sent to the spam bucket, can be found on the mail archiver.
    • The archiver is located at https://archive.wou.edu
    • Use your Pawprint credentials to login to the archiver.
    • I recommend that you click on the Advanced Search link.

 

Moodle Updates

Moodle development continues in an effort to provide 99.999% up-time and good page load-times.  Current up-time is 99.967%.  Page load times are good, except during large “Course Development” events.

Recent changes:

  • Added additional “Course Development” servers
  • When a server reaches a high enough load to render the web-page unusable, a restart script will automatically be executed on the high load server

Current design:

  • Whether you enter Moodle through a “Course Participation” link or a “Course Development” link, you are accessing centralized data that is shared by all web servers.
  • The current entry point for “Course Participation” is https://moodle.wou.edu
    • One of five Moodle web-servers will respond to https://moodle.wou.edu
      • The algorithm used for selecting which of the five servers will be selected for the current requester, is “least load”.
  • The current entry points for “Course Development” is http://dep-web.wou.edu http://faculty-moodle.wou.edu and http://video-moodle.wou.edu
    • Course Development” is separated from “Course Participation” in order to provide a more consistent user experience for the students.  “Course Development“, tends to have a high load impact on the web-server being used for development.

MoodleCluster

Next iteration:

Moodle access will include the following changes, as represented by the graphic below.

  • Course Participation” remains unchanged
  • The three access points for “Course Development“, will be consolidated into a single link, https://coursedev.wou.edu
    • This change will balance the load across all three “Course Development” servers
    • All three servers, dep-web.wou.edu, faculty-moodle.wou.edu and video-moodle.wou.edu will respond to the URL https://coursedev.wou.edu
      • The algorithm used for selecting which of the three servers will be utilized by the current requester, is “least load”.
    • The three old URLs will continue to function, but will re-direct to https://coursedev.wou.edu

MoodleCluster2

 

One month load average as of 11/25/2014

CryptoWall has arrived

CryptoWall infected a staff members computer recently.  All the files on his C: and H: drive have been encrypted and he no longer has access to them.  Fortunately the files on his H: drive are backed up in several locations, so those are recoverable.  The files on the C: drive have been lost.

The information below the horizontal line came from the Sophos web-site, the WOU anti-virus software company.

Below the double horizontal lines, you will find recommendations from Sophos on how to prevent being infected with this type of malware.


 

CryptoWall and CryptoDefense

New variants of file-encrypting ransomware called CryptoWall and CryptoDefense have been popping up since at least April 2014.

SophosLabs threat researcher Anand Ajjan says CryptoWall has the same code as CryptoDefense, and only differs in the name.

If you see a message like the one below, you’re in trouble – many, if not most, of the data files on your hard drive or any connected drives will be scrambled, and it’s simply not practicable to crack the encryption used by the crooks.

(You don’t have to pay, of course. Despite losing data, police in the New Hampshire town of Durham showed a bit of public resistance to the crooks, announcing that they were “definitely not paying any ransom.”)

The message gives instructions on how to use the Tor anonymizing proxy to access a website where you can pay to unlock your files:

CryptoDefense-pay-screen-500

If you do go to the payment website, you come to a screen that shows a clock counting down the time you have left to pay the ransom.

Leave it too long and the price to decrypt your files doubles:

cw-cost-500

In broken but intelligible English, the website tells you:

We are present a special software - CryptoWall Decrypter - which is allow to decrypt and return control to all your encrypted files.

This website (blocked by Sophos) includes links to payment options, and offers you the chance to “Decrypt 1 file for FREE”:

cw-ransom-500

Unlike the crooks SophosLabs found who are trying to copy CryptoLocker but without actually encrypting your files, CryptoWall’s encryption can’t be reversed without the key.

That means if your files get locked, you either have to pay up, or “do a Durham,” and kiss your files goodbye.

According to SophosLabs, a common way of spreading CryptoWall infections is through exploit kits called RIG (also known as “Goon”) and Angler.

Exploit kits are web pages containing pre-packaged exploits that can be used to deliver malware of your choice to unsuspecting victims.

Often, one group of cybercrooks will simply “rent” exploit kit services from other cybercrooks on a pay-per-install basis.

So, whereas some ransomware attacks use social engineering in spam to trick you into downloading the malware, CryptoWall can get onto your computer just by visiting a website that is rigged up with an exploit kit.

Sophos Anti-Virus (in endpoint and gateway products) detects and blocks the various components of this threat with the following names:

  • HPmal/Ransom-I: the Cryptowall/Cryptodefense malware itself.
  • Troj/ExpJS-KX: web pages containing the RIG exploit kit.
  • Mal/Generic-S and Mal/ExpJava-AF: other exploit kit pages associated with this threat.

 


 

IT admin:

  • prevent SPAM email from reaching end users.
  • educate users not to open any attachments that they are not expecting.
  • ensure local anti-virus is up to date on all computers and is active (ensure the user has not disabled the protection).
  • ensure your central shares (that endpoints update from) are receiving updates from Sophos Update Manager – check your console.

Regular user:

  • avoid opening any attachment emailed to you that you were not expecting.
  • watch out for emails with attachments suggesting you must reply quickly or ‘act fast’ and hence feel compelled to open the attachment quickly – without considering the source.
  • check your Sophos shield in the system tray and make sure it does not have a red cross or warning triangle.
    Good Bad
      

    Move your mouse point over the shield and ensure ‘On-access scanning: disabled’ is not shown.

    Good
    Bad

    Double-click the Sophos shield to open the program.  On the left hand side, under the ‘Status’ panel make sure the ‘Last updated’ value is recent…

  • At WOU “Web control” is Disabled.  When this is turned on, it will block you from going to specific categories of web sites, some of which my be required for your research.

    …the date shown when hovering the mouse point over the shield does not indicate a recent update in protection, but only that it checked with the update source and is in sync.
  • contact your IT department if in any doubt.

Wireless Update

Two methods of guest wireless access are now available on the WOU campus.

  1. Sponsored Guest
    1. A guest account can be created prior to the guest arriving on campus.  Accounts can be created by the APA’s, Service Request Desk, Hamersly Library reference desk or the Werner University Center Information Desk.
    2. The guest will select SSID: wou-guest
    3. The SSID password will be provided by the sponsor.
    4. The guest will login using the credentials provided by the sponsor.
  2. Self-serve Guest
    1. A guest account is created by the guest when they arrive on campus.
    2. The guest will select SSID: wou-guest-open
    3. When the guest selects SSID wou-guest-open, they will be presented with the web pages seen below.
    4. The guest login credentials will be sent by both text message and e-mail, once they complete the form below.
    5. The guest will login using the credentials sent to them as a result of their form submission.
  3. Campus users will continue to use SSID: wou-secure.

Screen Shot 2014-10-20 at 10.21.51 AM

Screen Shot 2014-10-20 at 9.54.39 AM Screen Shot 2014-10-20 at 9.55.00 AM

 

NetApp EF-550 SSD (solid state drive) update

The WOU EDW (data warehouse), VDI and Moodle have all been moved from spinning drive storage to SSD storage, utilizing the NetApp EF550.  VDI is running on a RAID 0 pool, while Moodle, Oracle and Cognos are running on RAID 6. 

 

The server side utilizes the Cisco UCS blade platform.  The RedHat operating system runs on VMware.  The communication between server and storage is via a HA pair of Cisco Nexus 5000’s, utilizing 8Gb fiber-channel SFP’s.

 

Significant data throughput improvements has been noted on the following three applications:

  • Data Warehouse  (utilizes both Cognos Insight and Oracle 12c database)
    • Reports that previously took 25 – 30 seconds to complete now finish in 5 – 10 seconds
    • The user experience, including moving through menu items, creating queries and reports within Cognos, has increased significantly
  • VDI (virtual Windows 7 desktop lab environment running on VMware)
    • 200 concurrent lab VMs, all running on an NetApp EF550.  Previously these 200 VMs were spread accross two FAS-3250 heads, a FAS-2240-2 with flash pool and a FAS-2240-2
    • Windows logins that previously took 45 – 80 seconds, now take 20 – 25 seconds
  • Moodle version 2.7.1
    • The user load is currently peaking at about 186 users on the current term Moodle server.  See graph here
    • The http response time on the current term, running on EF550 SSD drives averages 25 milli-seconds, with 60 milli-second peaks.  See graph here
    • The http response time on last summer term, running on spinning drives is averages 291 milli-seconds, with 1,635 milli-second peaks.  See graph here
    • Differences in latency between spinning drives and SSD drives were significant during several load test.  See graph here  The large spike on the right side of the graph was 580 users utilizing spinning drives, while the small bump further to the right of the spike was 5,125 users utilizing SSD drives.

E-mail encryption

WOU uses Barracuda to provide e-mail encryption.

Barracuda both actively and passively encrypts e-mails.  Active encryption can be accomplished by entering the keyword #secure# in the subject line of your e-mail.  Passive encryption occurs when Barracuda finds social security or credit card numbers in the body of the e-mail.

Encryption can only be performed on e-mail that is sent to an e-mail address other than @wou.edu E-mail sent to @wou.edu will not be encrypted.

There will be a link embedded in the e-mail that the recipient will click on.  The first time a recipient receives an encrypted e-mail from WOU, they will be asked to create an account.  For all future encrypted e-mails, the recipient will use the login credentials they created the first time they received an encrypted message.

After the user logs into the encrypted site, via the link that is embedded in the e-mail, they will be able to view the contents of the e-mail at the encrypted site.

 

Below is a sample of the recipient’s view of an encrypted e-mail.


 

You have a new encrypted message from <username>@wou.edu

WOU_encryptedMailService

You have received an email message from <username>l@wou.edu that has been encrypted for privacy and security by the Barracuda Email Encryption Service.

To view the email message, click here to log into the Barracuda Message Center. You’ll be prompted to either create a password or enter the one you may already have. You can also paste the following URL into your browser to access the Barracuda Message Center:

https://encrypt.barracudanetworks.com/login?nid=xxxxx

The secure message will expire in 30 days. Need Help?

Disclaimer: This email is confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.

Copyright 2013 Barracuda Networks, Inc. All rights reserved

Guest wireless

A self-serve guest wireless portal will be placed into production by Thanksgiving 2014.  When using the new guest portal, a guest wireless user will be able to self-register for wireless through a web-page.

Eduroam access to wireless will be available by January 2015.  This service allows guest wireless users to authenticate to their native participating institution.

Currently, guest wireless can be obtained by contacting the following resources:

  • Service Request Desk  88925
  • Werner Information Desk  88261
  • Hamersly Library Reference Desk
  • Academic Program Assistant from you department / division

Telecommunications update

All of our incoming phone calls and local outgoing phone calls are now routed through Minet across two diverse fiber paths.

Minet is the Internet Service Provider serving Monmouth and Independence with cutting edge fiber connections. We collaborated with Minet to migrate some of our telecommunications services over the summer. All of our incoming phone calls and local outgoing phone calls are now routed through Minet.

Our local calling area has always extended to Dallas, Independence, and Salem. Any phone calls to numbers within our local calling area are made “without the 1” and without your long distance code. Any phone calls to numbers outside of our local calling area require the 1 and will prompt for your long distance code at the end of dialing.

You may have found some phone numbers in Woodburn, McMinville, Beaverton, etc., that – somehow – connected without the 1. Chalk those up to our old provider being preoccupied with reinventing itself amid a changing technological landscape, or simply their oversight. The routing with Minet is more accurate and will only connect calls to Dallas, Independence, and Salem. Any call made to a number outside of our local calling area will require the 1, then a long distance code, and be connected through our existing long distance provider, AT&T.

Because we share the area code overlay of 503 and 971 with the greater Portland area, you may not know beforehand if the number you’re dialing is local or long distance. We’ve programmed our system to remove the 1 if it’s dialed by accident for local telephone numbers. Conversely, if a 503 or 971 area code long distance number is dialed without the 1, Minet will play a “please dial a 1 when dialing this number” message.

Google licensing

WOU adopted Google Apps shortly after the Oregon University System signed an agreement with Google that was acceptable to the OUS attorney.  The agreement includes FERPA and HIPPA compliance language.  The core-suite included in this agreement includes:

  • Calendar
    • Organize your schedule and share events with friends
  • Classroom
    • Lets teachers create and organize assignments, provide feedback and easily communicate with their classes
  • Contacts
    • Manage your contacts
  • Drive
    • With Google Drive, you can create, share and keep all your stuff in one place. Share files with others, and edit them together in real time.
  • Gmail
    • Get a fresh start with email that has less spam
  • Groups for Business
    • Create mailing lists and discussion groups
  • Mobile
    • Google Sync for Mobile
  • Sites
    • Create, share and publish websites
  • Talk / Hangouts
    • Talk, IM, and share files with your friends for free

The following non-core Google services are also available:

  • Google Analytics
  • Google Wallet
  • Google+
  • Location History
  • Picasa Web Albums
  • YouTube

The non-core/additional apps are not governed by the OUS contract with Google, but rather by the consumer/personal Terms of Service and Privacy Policy. This means that in using the non-core/additional apps, you are agreeing to the Google’s Terms of Service.

Beyond the core suite of Google Apps, there are many additional apps that you can pair with your Google account. For help with these apps, the best source of support will be existing vendor support articles, although UCS will make best efforts to assist.