What is SMB3 with RDMA / SMB Direct

Following the full release of Violin’s WFA product, I can now share some of my understanding gained from using it over the last six months or so. Violin’s WFA – “Windows Flash Array”-  is a storage system that embeds a specially optimised version of Windows 2012 R2 on our gateway modules and provides very fast performance to SQL, Hyper-V and other application environments over SMB3. For more details see HERE, or for Microsofts view you can review the TechNet blog here.

This is the first of a number of blog entries to provide some further technical detail on the product and the benefits it provides.

 

Screen Shot 2014-04-22 at 11.14.22 AM

While it is essentially a simple premise and a product that is very easy to use and manage, it has taken me a while to get my head around the new vocabulary associated with using the latest version of SMB3. SMB Direct, iWARP, RoCE, SMB-Multichannel  are all technologies and acronyms that combine to deliver the biggest benefits to SMB3 over previous versions of the protocol. What are those benefits I hear you ask? In short they fall into two areas: performance and availability.

cheeter

Performance: with the improvements in the protocol, we can now drive very high bandwidth and very low latency from a modest Windows server only using a small amount of CPU resource. I have done my own testing and can confirm that the latencies are significantly lower then FC to the same array for nearly every IO profile…. For full details of the performance numbers please contact your Violin representative.

Why is it faster? The simple answer is RDMA. Microsoft have added the capability to perform data transfers via “Remote Direct Memory Access” to SMB3 and named it SMB Direct. In my simple understanding, RDMA allows one system to perform a data copy by directly reading or writing from the memory of another computer, avoiding many levels of protocol overhead. There are two competing implementations that are both supported by SMB Direct; iWARP and RoCE and interfaces either support one or the other.

The other nice thing about SMB-Direct is that it requires no setup, if the environment supports SMB-Direct it just works. The easiest way to tell if it’s working is by looking at the SMB-Direct counters in perfmon, or using the powershell “commandlet” Get-SmbMultichannelConnection.

Screen Shot 2014-04-22 at 11.10.48 AM

 

Availability: Microsoft have added a new capability in to the SMB protocol called SMB-Multichannel. This automatically detects and utilises all available paths to an SMB share that are running at the fastest speed. It is like MPIO or Ethernet link bonding, but requires no manual setup or additional software and provides very fast failover and failback. Failover between cluster nodes on the Violin WFA are pretty much undetectable to a client server; even when physically pulling a network link on a client under very heavy load IO continues with little interruption. The below example is created by saturating 4 links on a client before pulling one link, then reinserting it 30 seconds later.

linkremoval and re-add

Historically I have been reluctant to recommend file based shared storage solutions for applications or databases that require the lowest latency and the highest reliability. However, with SMB3 and SMBDirect we get higher bandwidth and lower latency storage connectivity with automatic resiliency. When combined with the Violin 6000 array as WFA you get a very high performance solution that is very easy to manage.

Look out for the next blog to provide comparisons with FC & SMB3 traffic and also SMB3 with and without RDMA…