Ottawa Valley SAGE

Something I had noticed a couple of posts ago was bothering me today and I decided to search out the problem and see if there was a fix for it.

The problem being the tags for a post were no longer working. I had used them for the first post, but when I decided to use them a couple of posts ago, no luck. I ignored it as something to look at later and when I posted my HDD woes, the tags were still broken. I tried again today and still broken…

However, there was a firebug warning for the page. It was complaining about a JavaScript error. Some problem that I hadn’t seen before. Anyway, rather than wade in, I decided to see if anyone else had the problem and there was a post and a solution. It appears that the event plugin includes a deprecated JavaScript module and if you comment it out, things appear to go back to normal. While this isn’t the official patch – probably because there isn’t one, it seems to work. The URL for the article is: http://wordpress.org/support/topic/326448

Title: Ottawa Area Security Klatch
Location: Microsoft Canada, 100-152 Queen St., Ottawa, ON, Canada
Link out: Click here
Description: Talk #1: DNS Security: The Seven Deadliest Sins

Speaker: Derrick Webber

A vulnerable DNS allows attackers to compromise everything else in the organization: your web sites, servers, SSL, VPNs, even desktops. This short presentation covers the very worst mistakes in the design and operation of the Domain Name Service and how to fix them.
Talk #2: Log-based Intrusion Detection (LIDS) using OSSEC+Splunk

Speaker: Dale Neufeld

OSSEC is multi-platform, open source Host-Based Intrusion Detection System (HIDS). It has a powerful correlation and analysis engine which integrates log analysis, file integrity checking, Windows registry monitoring, centralized policy enforcement, rootkit detection, real-time alerting and active response.

Splunk is a platform for IT search that enables you to build completely customizable dashboards, to have alerting and forensic capabilities for security, availability and application troubleshooting, and more.

We’ll focus on the following items:

* Benefits of centralized logging
* Overview of OSSEC architecture
* OSSEC tuning
* OSSEC custom rules for your environment
* Overview of Splunk architecture
* Splunk and OSSEC integration

Start Time: 17:30
Date: 2010-03-16

Well, not so much fun…

Tuesday last week, I guess there was a power bump or some such event and my main drive on my mac decided to take a nose dive. I should have expected this, as the power is not exactly the most stable in this neck of the woods. I do have backups, but I was lazy the past 2 months and didn’t do a backup. Time machine isn’t quite suitable (no real control) and I haven’t bothered anything beyond a full dump at intervals and that takes time.

The resulting sequence was interesting (yes, with all the connotations of interesting time):

• Screensaver is not responding, so power cycle the machine
• Get nice white OS X screen, no apple logo
• Repeat power cycle, same
• Remove power and wait a few, motherboard may be retaining some oddities
• Try again, logo and spinner this time… 5 minutes later, still spinning
• Remove USB hard drives. This has happened before after a power issue and it loses it’s brains on what drive to boot from.
• Try again… logo, spinner, 1 minute later, power off
• Odd… power on machine again, this time booting with the standard unix boot visible
• It shows a normal startup, an fsck and a couple of errors and then powers down
• Try again in single user mode and manually run fsck…
• Well, look at that, boot drive has 2 unrecoverable errors, recommend booting from OS DVD and running disk utility to clear errors
• Try booting from DVD and running utility
• Even better message, Not recoverable. Please back up all of your readable data from the suspect disk and repartition and install a new file system. Supposedly this will result in a usable disk.
• I don’t have any internal drives available that are the same size or larger to copy to, so I get a 1TB USB that I had some backup data on and start copying the 300GB of data in my account to the external drive.
• I don’t want to redo the drive until I am happy with the backup
• In the interest of getting things running, I grab a 400GB drive and use that for the OS and home directories.
• After that, I migrate the applications off of the old main drive and copy a reduced set of my data off the drive.
• All looks good so far
• Most things are working as expected and I haven’t lost my business info or my email. Life is reasonable.
• Now for slightly better response. Go out and get replacement drives for the 4 internals, just in case it’s a drive issue, as the 4 drives are in serial number.
• Hmm, 1.5TB drives are cheaper than the 500s I bought 2 years ago. I’ll go with that, 2TB are still a little expensive.
• Boot off the Install DVD again and this time take 2 drives and mirror them. No point in being screwed over a second time.
• After installation and reboot, I have a 1.5TB mirrored boot disk and 2 1.5TB data disks.
• Account migration, some application reinstallation and some recovery I’m back to a properly functioning system.

So at the end of the exercise, I’m in better shape, I need to keep a better backup regime and I think hard drives vs. tapes is probably my answer, as 1TB hard drives are under $100 and the tapes and tape drives I have are not near the same capacity as my my hard drives and tapes are not “live” file systems, allowing for easier recovery and viewing. This could have been much worse.

I think it’s bacula time and time to put in a somewhat power efficient NAS. I have enough SATA drives and a hardware RAID controller to do this, I can backup sets to USB drive from the NAS, and I can store the drives in a protected environment. I have no desire to get a cheap NAS appliance, I’ve tried that and they are not suitable for my needs. Time to create either an OpenFiler or a FreeNAS box. This will probably be a topic for after the current care and feeding of a virtual Linux box practical exercise we are doing.

Anybody want an ethernet enabled hard drive enclosure? It also does USB 2.o

Well, registration for the site doesn’t automatically give you the ability to create articles, just to comment. As I didn’t have to register, I didn’t notice. I have upgraded access for the current members and if it doesn’t work, leave a comment for me.

I have a friend who has decided to get a new computer. The old one just doesn’t seem to be able to handle the load on it.

This is rather amusing, as the “load” is the use of a web browser. I asked and that is pretty much the sum of the computer use. I guess there may be a game of solitaire in there somewhere, but that is supposedly the entire sum of the computer’s existence.

This article is called “opportunity” and I decided to take the opportunity to take them down a different path. Linux Mint. This is a Ubuntu based distro that goes out of it’s way to be a desktop OS. It’s clean, simply laid out and I have to admin, relatively pretty. The green scheme doesn’t really do it for me, but that is my problem. I know I could have tried PC-BSD or Ubuntu or some other item, maybe even puppy , but I wanted this to be painless and easy to keep up to date. They will be taking the machine this weekend and I’m hoping they won’t even realize it’s another OS until a little later. If you have the opportunity to get someone else over to the Linux side, this may be the distribution you want to use. The URL is linuxmint.com.

Matt Simmons sent a tweet this morning regarding today’s xkcd comic. His observation was that it would make a large number of sysadmin blogs and I’d have to agree. The rest of his commentary from his blog is also quite apt. I remember having this discussion at the LISA conference multiple times, specifically the hero complex and how dangerous it is to stability and reliability for all the reasons listed.

In our profession (yes, it is one), invisibility is the name of the game. This unfortunately has the side effect of no one really understanding what you do or why they employ you. If you are good, there are no problems, so why do they need you? On the other hand, you get the person who isn’t as experienced yet and who runs around fixing the symptoms rather than the problem and, due to the visible results, gets praised. This leads into the hero complex and it’s a difficult thing to turn around, as everyone likes to know they are doing a good job.

It is quite the set of standards we have:

• We only get a call when something is wrong
• Few people know what we do
• If we are doing a good job, we mustn’t be working
• If we are doing a bad job (not sufficiently experienced), we are praised for fixing the problems, as they had risen to a noticeable level
• The better we get, the less recognition we get
• As time goes on, we become better generalists, rather than specialists

The real problem is more along the lines of promoting the idea that if all is quiet, we are working effectively and if all is chaos we are not. From an external point of view, the apparent effort is inverse to the actual situation which is counter intuitive to most people. This problem is exacerbated with the convention that people expect computers to have problems, so outages are the norm, not the exception. Realistically, we should be handling exceptions to the norm, rather than our visibility being the norm.

So far the only thing that comes to mind is a set of shameless self-promotion items which show that the work we do is affecting the bottom line by avoiding problems rather than trying to correct them as they happen. It’s kind of like the Y2K issue – perception is that nothing happened, so we wasted all that effort on nothing. The real statement should be that we had minimal issues because we fixed the problems before they hit. I know I spent a lot of time in advance patching machines and installing new versions of software in a controlled manner in the months leading up to Y2K. Personally I’d prefer to have fixed it all in advance rather then be scrambling after the fact. I guess I’m just lazy.

Comments are welcome.

As I have been promising, I have finally pushed the new site live.

Unfortunately, I haven’t had the time to migrate data and users from the old site, so I will be bringing over the users from the old site. If you are impatient, register again and I’ll avoid duplicating you. Due to a difference in the password encoding mechanisms used, you will have to use the lost password feature to get a new one.

The content will migrate over the next week or two, as I get the time. The ultimate goal is to get all the prior content, all the way back to the original content from the first mailing list.

Features will be added as I get a chance and as far as I can tell, this is a much more manageable system for handling upgrades. I have gone with a grittier look for the front page, and some of you might even recognize the background image.

This will be a more open system then the previous one and the content will be readable by anyone who wishes to. Comments and posts will still require registration and the first ones will require approval. That should handle most issues. I will be adding the Creative Commons logos as well.