Der Flounder

Using ddrescue on a failing hard drive

rtrouton — Tue, 31 Jan 2012 22:13:57 +0000

While I was out at MacIT last week, our Casper server sent me a notification that the “Check for failing hard drive” smart group had a member. Since that’s a club that nobody wants to be a part of, I forwarded the notification to the team at home to let them know. They were able to copy the user’s home folder data off to the server and left the box itself and its failing drive for me to take a look at when I got home.

Once I took a look, the SMART status report of Failing and the weird noises from the drive made me certain that it was only a few steps short of the Choir Eternal. However, I still wanted to see if I could get the maximum amount of data off of it before its final demise. Time for ddrescue.

Installing ddrescue

I had a bootable utility disk and a separate drive named Backup with enough storage to hold the complete contents of the failing drive. Now I just needed to install ddrescue on the utility drive. To do this, I installed MacPorts on the utility drive and then installed ddrescue.

1. Installed MacPorts from http://www.macports.org

2. Following installation, ran the following command to get MacPorts up to date:

sudo port -v selfupdate

3. Next, ran the following command to install ddrescue:

sudo port install ddrescue

Running ddrescue

1. Get the information on the drive you want to recover from by running the following command:

diskutil list

For the purposes of this example, we’ll assume that the failing drive is /dev/disk0s2

2. Use the following command to have ddrescue recover data using verbose mode to a disk image on the Backup drive:

sudo ddrescue -v /dev/disk0s2 /Volumes/Backup/failing_drive_backup.dmg failing_drive_backup.log

The failing_drive_backup.log part of the command will create a log file that ddrescue writes to. In the event that you need to stop ddrescue and start it again, ddrescue will read the log and use the information to pick up where it left off.

Monitoring ddrescue

Using the “-v” flag in ddrescue means that ddrescue will display its output in verbose mode. It should look similar like this:

About to copy 9223 PBytes from /dev/disk0s2 to /Volumes/Backup/failing_drive_backup.dmg Starting positions: infile = 0 B, outfile = 0 B Copy block size: 128 sectors Sector size: 512 bytes Max retries: 0 Direct: no Sparse: no Split: yes Truncate: no

Press Ctrl-C to interrupt Initial status (read from logfile) rescued: 0 B, errsize: 0 B, errors: 0 Current status rescued: 69792 MB, errsize: 0 B, current rate: 18153 kB/s ipos: 69792 MB, errors: 0, average rate: 18429 kB/s opos: 69792 MB, time from last successful read: 0 s Copying non-tried blocks...

You can also use fs_usage to monitor ddrescue‘s read/write progress by using the following command:

sudo fs_usage | grep ddrescue

Recovery

Once your data has been recovered to the disk image, hopefully you can double-click on it and have it mount. However, in the event it does not, you can try restoring from the disk image to another disk partition using the following steps.

1. Get the information on the drive you want to recover from by running the following command:

diskutil list

For the purposes of this example, we’ll assume that the drive we’re restoring to is /dev/disk1s3

2. Use the following command to have ddrescue recover data using verbose mode from the failing_drive_backup disk image on the Backup drive to /dev/disk1s3:

sudo ddrescue -v /Volumes/Backup/failing_drive_backup.dmg /dev/disk1s3 recovery_drive.log

Once on the disk partition, you should be able to run whatever repairs are needed to get the drive to mount and access your files.

OpenBSM auditing on Mac OS X

rtrouton — Mon, 30 Jan 2012 21:44:32 +0000

Way back in 10.3.x, Apple submitted Mac OS X and Mac OS X Server to the National Information Assurance Partnership for Common Criteria certification. Common Criteria certification means that the the covered hardware and software has been tested and evaluated to make sure that it meets an established set of requirements for security and data protection. 10.3.6 and 10.3.6 Server were tested and were found to meet Evaluation Assurance Level 3 (EAL3) for Common Criteria certification.

As part of that certification effort, a new piece of software appeared from Apple: the Common Criteria Tools audit software. This software was OpenBSM, which is an open source implementation of Sun’s Basic Security Module (BSM) security audit API and file format. From 10.3.x – 10.5.x, this software needed to be installed and configured separately. As of 10.6.x and 10.7.x, it’s installed along with the OS. In fact, if you’re running 10.6.x and 10.7.x or their Server equivalents, it’s running now on your box unless you went in and turned it off. If you’re interested in learning more, see below the jump.

The good news was that this software (potentially) records all events on your Mac or Mac server, which would be invaluable to a forensic examination of a compromised box.

The bad news was that the audit logs themselves weren’t sent to syslog, but instead were saved in BSM’s own binary file format. Also, the files themselves had a unique file name scheme, where the complete timestamp of the beginning time and the complete timestamp of the end time were used as the file name. Consequently, the filenames look like this:

The log currently in use is usually easy to identify, as it’s the one named something like 20120108210134.not_terminated.

What’s being logged and to where?

What’s being logged depends on how /etc/security/audit_control is configured. On my 10.7.2 laptop, /etc/security/audit_control looks like this:

# # $P4: //depot/projects/trustedbsd/openbsm/etc/audit_control#8 $ # dir:/var/audit flags:lo,aa minfree:5 naflags:lo,aa policy:cnt,argv filesz:2M expire-after:10M superuser-set-sflags-mask:has_authenticated,has_console_access superuser-clear-sflags-mask:has_authenticated,has_console_access member-set-sflags-mask: member-clear-sflags-mask:has_authenticated

The first value (dir:) specifies where the audit logs are being stored. By default, they’re stored in /var/audit.

The second value listed (flags:) is for audit flags. These tell the system how to record those events which are audited for all users on the machine (i.e. the events that can be tied back to a specific user.) The flags themselves are usually two letter combinations, which in turn represent system calls. Here’s a partial listing of them:

no = no_class - Null value for turning off event preselection fr = file_read - Read of data, open for reading fw = file_write - Write of data, open for writing fa = file_attr_acc - Access of file attributes: stat, pathconf fm = file_attr_mod - Change of file attributes: chown, chmod, flock fc = file_creation - Creation of file fd = file_deletion - Deletion of file cl = file_close - File being closed after reading or writing pc = process - Process operations: fork, exec, exit nt = network - Network events: bind, connect, accept ip = ipc - System V IPC operations na = non_attrib - Nonattributable events ad = administrative - Administrative actions lo = login_logout - Login and logout events ap = application - Application-defined event io = ioctl - ioctl system call ex = exec - program execution to = other - Miscellaneous (your system shrugging its shoulders and saying "I saw it, but don't know what it is.") all = all - All flags set

To be honest, trying to fine tune the flags is hard so I recommend using the great catch-all of flags: all. That way, you know you’re getting everything.

Note: If you’re using flags: all, be aware that this will cause your audit software to dump a large amount of data into the current audit log.

The third value listed (minfree:) defines the minimum free-space level on the disk for the audit software. This is a percentage value, so that the auditing software can be told to do something if free space on the boot drive drops below that amount. Once this minimum percentage is hit, the auditing function runs the /etc/security/audit_warn script. You should edit this script to perform whatever functions you deem necessary.

The fourth value listed (naflags:) is for nonattributable flags. These tell the system how to record those events which can’t be tied back to a specific user. Once again, I recommend using the great catch-all of naflags: all if you want to make sure you’re getting everything. Same warning as above, using all will cause the audit software to record a LOT of data to your audit logs.

The fifth value listed (policy:) is a bit of mystery to me. The FreeBSD man pages explain what they do, but a lot seems to not be implemented at this time.

The sixth value (filesz:) defines the maximum size that an individual audit log can grow to. Once that limit is hit, the audit process closes out the file and starts a new log file.

Values for the disk space used are numbers with the following suffixes:

B – Disk space used in Bytes.

K – Disk space used in Kilobytes.

M – Disk space used in Megabytes.

G – Disk space used in Gigabytes.

The seventh value (expire-after:) specifies when audit log files will expire and be removed. This may be after a time period has passed since the audit file was last written to, when the aggregate of all the audit files have reached a specified size or a combination of both. If no expire-after parameter is given then audit log files will not expire and be removed by the audit control system.

Values for the audit log file age are numbers with the following suffixes:

s – Log file age in seconds.

h – Log file age in hours.

d – Log file age in days.

y – Log file age in years.

Values for the disk space used are numbers with the following suffixes:

B – Disk space used in Bytes.

K – Disk space used in Kilobytes.

M – Disk space used in Megabytes.

G – Disk space used in Gigabytes.

The suffixes on the values are case sensitive. You can also use AND and OR values to determine when audit log files expire. In the case of AND, both values must be met before the audit file can be removed. In the case of OR, either condition may expire the log file.

For example:

expire-after: 60d AND 1G

Setting this value will expire files that are older than 60 days but only if 1 gigabyte of disk space total is being used by the audit logs. However, setting this value will expire files if the directory is over 1 GB in size or those files in the directory that are older than 60 days:

expire-after: 60d OR 1G

All the values below expire-after are a complete mystery to me and I don’t know what they do. If someone does know, please let me know in the comments.

Reading the audit logs

Reading the audit logs can be a bit of a challenge, as they’re stored in BSM’s binary file format. Fortunately, there is now an application named Audit Explorer available from the Mac App Store that helps simplify this process a lot. NetSquared, the makers of Audit Explorer, have also posted videos to show you sample audit trails for particular events to help get you started with analyzing your audit files.

As /var/audit‘s permissions are set so that only root has access, you may need to copy out the files that you want to analyze to a location that your user account can access.

Modular OS Deployment session at MacIT 2012

rtrouton — Sat, 28 Jan 2012 21:07:27 +0000

Thank you to all the folks who turned out early on a Saturday morning to hear Mike Boylan and myself give our modular OS deployment session. I was very surprised and gratified that it turned out to be a standing room-only session.

For those who attended and want a reference copy, I’ve posted our slides here in PDF format.

Opening Inaccessible Attachments in Outlook 2011

rtrouton — Fri, 20 Jan 2012 17:13:26 +0000

One of my users ran into an unusual display issue in Outlook 2011, where emails with attachments will sometimes not be displayed in the reading window.

These messages will show up with the paper clip icon that indicates that there’s an attachment (indicated by the red square in the picture below.)

When opened, the message will not show the attachment line

(Note: the recipients have been redacted from the screenshot below.)

I have not found a fix for this issue, but I’ve found a workaround that allows access to the attachment. See below the jump for the procedure

1. Select the message where the attachment is not showing up.

2. Go to the Message menu and select Edit Message.

3. A new window will open up, with the attachment showing in the attachment line.

Attaching Files to Meeting Invitations in Outlook 2011

rtrouton — Thu, 12 Jan 2012 17:51:23 +0000

I was asked by one of our users today how to attach files to a meeting invitation in Outlook 2011. After some research, here’s how you do it:

1. Set up a new meeting invitation in Outlook 2011.

2. Once the meeting invite opens up, drag your file you want to attach into the blank gray area to the right of Duration.

Your attachment will then appear in a newly-appearing attachment line below the start and end time for the meeting.

Interestingly enough, you can’t add files to Outlook 2011 appointments. If you need to add a file attachment to an appointment, click the Invite button in your Appointment window.

The appointment will then turn into a meeting invite and allow you to attach files. Invite your own email address and hit the Send button to add the event to your calendar.

Encrypting 10.7 non-boot volumes without erasing them

rtrouton — Fri, 06 Jan 2012 15:03:31 +0000

In addition to using FileVault 2 to encrypt your boot partition, it’s possible to encrypt your non-boot storage on 10.7 using the same CoreStorage-based encryption. Apple provided a way to do this via Disk Utility, where you would need to erase the drive and have the new volume be set up as an encrypted volume.

It is also possible to encrypt the drive without erasing it first from the command line. This allows your existing data to stay on your drive while the drive is being encrypted. See below the jump for the procedure.

To encrypt a drive, run the following command:

diskutil cs convert /Volumes/your_drive_name_here -passphrase

Note: You can use diskutil corestorage in place of diskutil cs

You’ll then be prompted for the passphrase you want to set and get output similar to what’s shown below:

computer name:~ username$ diskutil cs convert /Volumes/your_drive_name_here -passphrase New passphrase for converted volume: Confirm new passphrase: Started CoreStorage operation on disk2s1 your_drive_name_here Resizing disk to fit Core Storage headers [ - 0%..10%.............................................. ] Creating Core Storage Logical Volume Group Attempting to unmount disk2s1 Switching disk2s1 to Core Storage Waiting for Logical Volume to appear Mounting Logical Volume Core Storage LVG UUID: D1EAB2C3-EC21-41DA-AD60-75E1302E247B Core Storage PV UUID: 991C89E9-A628-408C-AAFF-39A561FCB95C Core Storage LV UUID: 36483526-6C2C-43FA-A4B7-6F503473F1C2 Core Storage disk: disk3 Finished CoreStorage operation on disk2s1 your_drive_name_here Encryption in progress; use `diskutil coreStorage list` for status

You’ll want to leave the Mac up and running while the drive is encrypting, as the encryption process will fail if interrupted. To check on its progress, run the following command:

diskutil corestorage list

Check the list displayed for the name of the drive you’re encrypting. When you find it, look for Size (Converted):, as that will tell you how far along the encryption is.

Once the encryption is finished, it should report Conversion Status: Complete

In order to kick off the encryption process, the drive needs to unmount and remount. If the drive does not unmount, you may see output like this appear:

Started CoreStorage operation on disk2s2 your_drive_name_here Resizing disk to fit Core Storage headers Creating Core Storage Logical Volume Group Attempting to unmount disk2s2 Switching disk2s2 to Core Storage Couldn't unmount disk2s2; converted volume won't appear until it's unmounted Core Storage LVG UUID: E8855450-E9B1-4DC7-84E3-27FE682A7FAB Core Storage PV UUID: 74610185-5485-44B0-B76A-B5C5E1616D6E Core Storage LV UUID: 3704705E-9645-4628-ABB7-A0E6999697C6 Finished CoreStorage operation on disk2s2 iLoad4 Encryption in progress; use `diskutil coreStorage list` for status

If the drive does not unmount, unmount it and unplug it (to make sure the Mac isn’t seeing it anymore), then replug it in. At that point, you should be prompted for the password you had set and the encryption process should begin.

If you check on its progress and encryption still has not begun, try unmounting, unplugging and replugging again. At that point, it should kick off.

To decrypt a drive from the command line, you would run the following command:

diskutil cs revert /Volumes/your_drive_name_here -passphrase

You’ll be prompted for authentication before it will decrypt, so enter the password you had set when encrypting.

Once entered, the drive will begin to decrypt.

To check on its progress, run the following command:

diskutil corestorage list

Check the list displayed for the name of the drive you’re decrypting. When you find it, look for Size (Converted):, as that will tell you how far along the decryption is. Also, when decrypting, the conversion direction should be reported as Conversion Direction: backward

You can also decrypt your encrypted drive using Disk Utility. To decrypt a drive from Disk Utility, use the following procedure:

1. Open Disk Utility.

2. Select your locked hard drive.

3. Under the File menu, select Turn Off Encryption…

4. When prompted for a password, enter the password you had set when encrypting.

You may need to unplug the drive and replug it in before it begins decryption. When it’s replugged in, the drive should mount normally without requiring your encryption password. At this time, Disk Utility will not show you the decryption progress, so to check on its progress, you’ll need to open Terminal and run the following command:

diskutil corestorage list

Hidden users with hidden home folders not migrated when upgrading to 10.7

rtrouton — Wed, 04 Jan 2012 15:14:12 +0000

In a number of Mac environments, it’s advantageous for Mac admins to hide the IT administrator account so that it can’t be deleted or altered by other users on those Macs. In other cases, like Jamf’s Casper, the system management tool needs an account in order to do its work. In both cases, hiding the affected account and its associated home folder is a good strategy to keep unwanted attention from noticing the account.

One way you can hide the account is to create it using a UID that’s lower than 500. Apple uses UIDs of 501 and higher for its accounts. UIDs of 500 and lower are assumed to be system-only accounts and should not show up at either the login window or in the Accounts or Users & Groups listing in System Preferences.

The downside to this is that these hidden accounts may not be migrated when upgrading your Mac to a new OS, which may leave you without your usual administrator account following the upgrade. I first noticed this with 10.7.x, but I’ve heard that it also affects hidden accounts when migrating from 10.5.x to 10.6.x.

How can you tell if your hidden account will be migrated? Here’s what works and doesn’t as of Mac OS X 10.7.x:

Note: In the description below, Visible refers to a user account that shows up and is editable in the Accounts or Users & Groups listing in System Preferences. Hidden refers to an account with a UID that’s lower than 500.

Successfully migrates:

Visible user account, where the home folder is stored in /Users

Hidden user account, where the home folder is stored in /Users

Visible user account, where the home folder is stored somewhere other than /Users

Does not successfully migrate:

Hidden user account, where the home folder is stored somewhere other than /Users

If you have a hidden user account with a home folder stored outside of /Users, there’s a couple of solutions that you may be able to leverage as part of the upgrade process to get those hidden admin accounts back.

1. If you’re upgrading to 10.7.x, use CreateLionUser to build installer packages that recreate your hidden user accounts following the upgrade. These installer packages should be incorporated into your upgrade workflow and set to run after the main 10.7 upgrade process has finished.

2. If the hidden user is needed by your system management tool, check to see if the needed user is created by the agent installer. If it is, then re-running the agent installer should put back the needed hidden user account.

2011 in review

rtrouton — Sat, 31 Dec 2011 23:38:23 +0000

The WordPress.com stats helper monkeys prepared a 2011 annual report for Der Flounder.

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 50,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 19 sold-out performances for that many people to see it.

Click here to see the complete report.

Clearing the font cache to fix an Outlook 2011 hanging problem

rtrouton — Wed, 28 Dec 2011 21:20:43 +0000

I had an issue today where Outlook 2011 was giving the spinning beachball right after opening. When I looked at the process list in Activity Monitor, I saw that the Microsoft Database Daemon process was using over 50% of CPU and sometimes going as high as 80% while the beachball was spinning. I also noticed that the fontd process was occasionally popping up to the top of the list of active processes, then going back to normal processor usage. After fifteen minutes, the spinning beachball went away and Outlook started behaving normally.

Since the fontd process had caught my attention, I decided to go with a sudden hunch and cleared the font cache system-wide. After that, I logged out of the user’s account and had them log back in. This time, Outlook opened right away. No beachball and no heavy Microsoft Database Daemon CPU usage. Based on that, Outlook was having some issues with something buried in the font cache and forcing a rebuild fixed the issue. In case someone else has a similar issue, here’s the commands I ran:

sudo atsutil databases -remove

(removes all user and system font caches)

sudo atsutil server -shutdown

(stops the Apple Type Services service that manages the font caches)

sudo atsutil server -ping

(restarts the Apple Type Services service)

I’ve also posted a script for automating the font clearing and ATS stop/restart on my GitHub repo. It’s available here.

Checking which accounts on a Mac have administrator rights

rtrouton — Thu, 22 Dec 2011 21:10:36 +0000

Something a number of Mac admins need to know about the Macs in their environment is being able to detect which accounts have admin rights on a particular Mac. This can be particularly important not just in secured networks, but also in schools. Savvy users can be inventive about finding ways to grant themselves admin rights, so admins need to be just as savvy about identifying which accounts have admin rights and shouldn’t.

To help with the task of identifying which accounts have admin rights, I found that Ryan Manly had posted a script-based Extension Attribute for use with Casper to detect which accounts had admin rights on a particular Mac. For those who need it, I’ve posted the Extension Attribute to my own GitHub repo as well as modifying it for use as an Absolute Manage Custom Information Item or as a generic standalone script.