Tales of SMB

In late 2016 we started moving our many many network shares from CLARiiON (SMB2) to Isilon (SMB3). I quickly started getting interesting reports from some users on shares that had moved to Isilon about “files and folders disappearing”.  This didn’t appear to be affecting all users that had shares moved to Isilon, nor was it affecting older clients (10.9 negotiating SMB2). Typically this would occur with shares that had nested directories. I was able to replicate this behavior by writing 1000 empty directories nested 2 deep. E.x. access //testshare/test1 and back out to //testshare  and random directories are now “missing”.

1000.jpg

777.png

Un-mounting and remounting the share would typically resolve the issue for a short period of time.

Working with Apple and some very courteous end users it was determined that there is a directory caching bug in 10.11 and 10.12 wherein macOS ignores an enumeration currently in play. The issue becomes more prominent when connecting to shares with large numbers of file/directories. We tested disabling directory caching by modifying /etc/nsmb.conf as follows.

[default]
dir_cache_max_cnt=0

This did resolve the issue with Finder enumerating items in Isilon SMB3 shares but created a new problem. When connecting to the DNS name of the Isilon share it presents a single ip address, but in our Isilon configuration Isilon round robins IP addresses. macOS fails to verify the share is already mounted in this scenario and keeps remounting the same share every time a new IP address is presented to the client.

dir caching disabled.png

In testing with some of our users affected by the enumeration bug they felt that this was even more disruptive than the “disappearing items” bug. We reported and discussed this issue with Apple and they validated this was also a bug on their end. Back to the drawing board.

We tested and validated that forcing SMB1 resolves this issue, since SMB1 does not have this concept of directory caching. Again I tested forced SMB1 connections with a sample of affected users. Success! While cifs performance isn’t great they appreciated being able to functionally use and trust their GUI save dialog boxes again.

I had this idea that we didn’t want to force SMB1 for all clients when the enumeration issue only seemed to affect a subset of the population (and doesn’t affect clients connecting to shares still on CLARiiON storage. I set out to create a self service policy to allow affected users to force SMB1 if they were having the Finder enumeration issue as follows.

  1. Create nsmb.conf to force SMB1
  2. Create installer package for nsmb.conf
  3. Create Extension Attribute to check nsmb.conf for forced status
  4. Create Jamf Pro policy for Self Service.

The config for forcing SMB1 on 10.11 and later with nsmb.conf is as follows

[default]
protocol_vers_map=1

I created the nsmb.conf file in BBEdit but any standard text editor should work fine.

Using Composer (or your preferred method for creating a package) create a package to place the nsmb.conf file in /etc with root:wheel 744 permissions. Upload the .pkg to your distribution point.

composer.jpg

Here is a quick and dirty Extension Attribute to check for the SMB1 forced value in nsmb.conf

#!/bin/sh

#Reports if 10.11+ client has SMB1 forced

if grep “protocol_vers_map=1″ /etc/nsmb.conf; then
SMB1_forced=”Yes”
else
SMB1_forced=”No”
fi
echo “<result>$SMB1_forced</result>”

To make the Self Service policy key a smart group off the value being returned “No”

smart group.jpg

 

Create a policy in Jamf Pro using the aforementioned smart group as the scope target, leaving the trigger as Self Service, the Execution Frequency ongoing, and install the package we just created. I added some boilerplate text for the description to encourage folks to not use this unless they were experiencing the enumeration issue.

Only use this tool if you are experiencing issues with files and directories “disappearing” while connected to SMB shares. You will need to un-mount and re-mount any shares for this to take effect.

fix finder.jpg

Now you have a Self Service policy that will fix Finder enumeration issues between macOS and Isilon hosted SMB shares.

Here is a quick look at that in our self service portal.

 

self service 2.png

self service.png

You can validate the change is enforced with `smbutil statshares -a`

e.x. Connecting to Isilon hosted share before SMB1 enforcement.

statshares.png

e.x. Connecting to Isilon hosted share after SMB1 enforcement.

stat shares smb1.png

 

 

I have verified with Apple that both of these of these bugs are on the macOS side and need to be fixed by Apple, however they are regarded as low priority and it is not currently expected that either bug will be fixed in macOS Sierra. If you or your organization are currently experiencing one or both of these issues I encourage you to file radar or enterprise cases with Apple to get upward pressure on the prioritization of these bugs.

 

 

Advertisements
Tales of SMB

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s