Storage Wars–HCI edition

There is alot of fuzz these days around hyperconverged, software defined storage etc.. especially since VMware announced VSAN 6.2 earlier this week, that trigge alot of good old brawling on social media. Since VMware was clearly stating that they are the marked leader in the HCI marked, if that is true or not I don’t know. So therefore I decided to write this post, just to clear up the confusion on what HCI actually is and what the different vendors are delivering in terms of features and how their  architecture differentiates. Just hopefully someone is as confused as I was in the beginning..

Now after a while now I’ve been working for this quite some time now, so in this post I have decided to focus on 4 different vendors in terms of features and what their architecture looks like.

• VMware
• Nutanix
• Microsoft

PS: Things changes, features get updated, if something is wrong or missing let me know!

The term hyper-converged actually comes from the term converged infrastructure, where vendors started to provide a pre-configured bundle of software and hardware into a single chassis. This was to try minimize compability issues that we would have within the traditional way we did infrastructure, and of course make it easy to setup a new fabric.  So within hyperconverged we integrate these components even further so that they cannot be broken down into seperate components. So by using software-defined storage it allows us to deliver high-performance, highly available storage capacity to our infrastructure without the need of particular/special hardware. So instead of having the traditional three-tier architecture, which was the common case in the converged systems. We have servers where we combine the compute and storage, then we have software on the top which aggreagates the storage between mulitple nodes to create a cluster.

So in conclusion of this part, you cannot get hyperconverged without using some sort of software-defined storage solution.

Now back to the vendors. We have of course Microsoft and VMware which are still doing a tug o war with their relases, but their software-defined storage option has one thing in common. It is in the kernel. Now VMware was the first of the two to release a fully hyperconverged solution and as of today they released version 6.2 which added alot of new features. On the other hand Microsoft is playing this safe, and with Windows Server 2016 they are releasing a new version of Storage Spaces which now has an hyperconverged deployment option. Now belive it or not, Microsoft has had alot of success with the Storage Spaces feature, since it has been a pretty cheap setup and included that with some large needed improvements to the SMB protocol. So therefore let us focus on how VSAN 6.2 and Windows Server 2016 Storage Spaces Direct which both have “in-kernel” ways of delivery HCI.

VMware VSAN 6.2

Deployment types: Hybrid (Using SSD and spinning disks) or All-flash
Protocol support: Uses its own proprietary protocol within the cluster
License required: License either Hybrid or All-Flash
Supported workloads: Virtual machine storage
Hypervisor support: ESXi
Minimum nodes in a cluster: 2 (With a third node as witness)https://blogs.vmware.com/virtualblocks/2015/09/11/vmware-virtual-san-robo-edition/)
Hardware supported: VSAN Ready Nodes, EVO:RAIL and Build Your Own based on the HCL –> http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan
Disk requirements: Atleast one SSD and one HDD
Deduplication support: Yes, starting from 6.2 near-line (only within an all flash array only
Compression support: Yes, starting from 6.2 near-line (only within an all flash array only)
Resilliency factor: Resiliency,  Fault Tolerance Method (FTM) Raid-1 Mirroring. Raid 5/6 are in the 6.2 release
Disk scrubbing: Yes, as of 6.2 release.
Storage QoS: Yes, as of 6.2 release. (Based upon a 32KB block size ratio) can be attached to virtual machines or datastores.
Read Cache: 0,4% of Host memory is used for read cache, where the VMs are located.
Data Locality: Sort of, it does not do client-side local read cache.
Network infrastructure needed: 1Gb or 10Gb ethernet network. (10GB only for all-flash) multicast enabled
Maximum number of nodes: 64 nodes pr cluster

Things that are important to remember is that VMware VSAN stores data within an object. So for instance if we are to create a virtual machine on a Virtual SAN datastore, VSAN would create an object for each virtual disk, snapshot and so on. It also creates a container object that stores all the metadata files of the virtual machine. So the availability factor can be configured pr object.  These objects are stored on one or multiple magnetic disks and hosts, and VSAN can access these objects remotely both read and write wise. VSAN does not have the concept of a pure data locality model like others do, a machine can be running on one host but the objects be stored on another, this gives a consistent performance if we for instance were to migrate a virtual machine from one host to another. VSAN has the ability to read for multiple mirror copies at the same time to distribute the IO equally.

Also VSAN has the concept of stripe width, since in many cases we may need to stripe and object across multiple disks. the largest component size in VSAN is 255 GB, so if we have an VMDK which is 1 TB, VSAN needs to stripe that VDMK file out to 4 components. The maximum strip width is 12. The SSD within VSAN act as an read cache and for a write buffer.

Windows Server 2016 Storage Spaces Direct*

*Still only in Tech Preview 4

Deployment types: Hybrid (Using SSD and spnning disks) or All-flash
Protocol support: SMB 3
License required: Windows Server 2016 Datacenter
Supported workloads: Virtual machine storage, SQL database, General purpose fileserver support
Hypervisor support: Hyper-V
Hardware supported: Storage Spaces HCL (Not published yet for Windows Server 2016)
Deduplication support: Yes but still only limited support workloads (VDI etc)
Compression support: No
Minimum nodes in a cluster: 2 *And using some form a witness to maintain quorom)
Resilliency factor: Two way mirror, three way mirror and Dual Parity
Disk scrubbing: Yes, part of chkdisk
Storage QoS: Yes, can be attached to virtual machines or shares
Read Cache: CSV Read cache (Which is part of the RAM on the host) also depending on deployment type. In Hybrid mode, SSD is READ & WRITE cache, therefore SSD is not used for persistent storage.
Data Locality: No
Network infrastructure needed: RDMA enabled network adapters, including iWARP and RoCE
Maximum number of nodes: 12 nodes pr cluster as of TP4
You can read more about under the hood about Storage Spaces Direct here –> http://blogs.technet.com/b/clausjor/archive/2015/11/19/storage-spaces-direct-under-the-hood-with-the-software-storage-bus.aspx
Hardware info: http://blogs.technet.com/b/clausjor/archive/2015/11/23/hardware-options-for-evaluating-storage-spaces-direct-in-technical-preview-4.aspx

Important thing to remember here is that we have an CSV volume which is created on top of a SMB file share. Using Storage Spaces Direct, Microsoft leverages mulitple features of the SMB 3 protocol using SMB Direct and SMB Multichannel. Another thing to think about is that since there is no form for data locality here, Microsoft is dependant on using RDMA based technology to with low-overhead read and write data from another host in the network. With has much less overhead then TCP based networks. Unlike VMware, Microsoft uses extents to spread data across nodes these are by default on 1 GB each.

Now in terms of difference between these two, well first of its the way the manage reads and writes of their objects. VMware has a distributed read cache, while on the other hand Microsoft requires RDMA but allows to read/write with very low overhead and latency from different hosts. Microsoft does not have any virtual machine policies that define how resillient the virtual machine is, but this is placed on the share (which is virtual disk) which defines what type of redundacy level it is. Now there are still things that are still not documentet on the Storage Spaces Direct solution.

So let us take a closer look at Nutanix.

Nutanix

Deployment types: Hybrid (Using SSD and spinning disks) or All-flash
Protocol support: SMB 3, NFS, iSCSI
Editions: http://www.nutanix.com/products/software-editions/
Supported workloads: Virtual machine storage, general purpose file service* (Tech Preview n
Hypervisor support: ESX, Hyper-V, Acropolis (Cent OS KVM custom build)
Hardware supported: Nutanix uses Supermicro general purpose hardware for their own models, but they have an OEM deal with Dell and Lenovo
Deduplication support: Yes, both Inline and post clusted based
Compression support: Yes both Inline and post process
Resilliency factor: RF2, RF3 and Erasure-Coding
Storage QoS: No, equal share
Read Cache: Unified Cache (Consists of RAM and SSD from the CVM)
Data Locality: Yes, read and writes are aimed at running on the local host which the compute resources is running on.
Network infrastructure needed: 1Gb or 10Gb ethernet network. (10GB only for all-flash)
Maximum number of nodes: ?? (Not sure if there are any max numbers here.

The objects on Nutanix are broken down to vDisks, which are composed of multiple extents.

Source: http://nutanixbible.com/

Unlike Microsoft, Nutanix operates with an extent size of 1MB, and the IO path is in most cases locally on the physical host.

When a virtual machine running on a virtualization platform does and write operations it will write to a part of the SSD on the physical machine called the OpLog (Depending on the resilliency factor, OpLog will then replicate the data to other node Oplog to achive the replication factor that is defined in the cluster. Reads are served from the Unified Cache which consists of RAM and SSD from the Controller VM which it runs on. If the data is not available on the cache it can get it from the extent store, or from another node in the cluster.

Source: http://nutanixbible.com/

Now all three vendors all have different ways to achive this. In case of Vmware and Microsoft which both have their solution in-kernel, Microsoft focused on using RDMA based technology to allow for low latency, high bandwidth backbone (which might work in the advantage, when doing alot of reads from other hosts in the network (when the traffic is becoming in-balanced)

Now VMware and Nutanix on the otherhand only require regular ethernet 1/10 GB network. Now Nutanix uses data locality and with new hardware becoming faster and faster that might work in their advantage since the internal data buses on a host can then generate more & more troughput, the issues that might occur which VMware published in their VSAN whitepaper on why they didn’t create VSAN with data locality in mind is when doing alot of vMotion which would then require alot of data to be moved between the different hosts to maintain data locality again.

So what is the right call? don’t know, but boy these are fun times to be in IT!

NB: Thanks to Christian Mohn for some clarity on VSAN! (vNinja.net)

#microsoft, #nutanix, #storage-spaces-direct, #vmware, #vsan

Comparison Microsoft Storage Spaces Direct and Nutanix

There has been a lot of fuzz around storage spaces direct coming with Windows Server 2016, and I have been getting alot of questions around it lately. “Will it solve my storage issues?” “can we replace our existing SAN?” “When to choose SPD over SAN” and so on.

Now as of right now, not all the technical details are known around the feature itself and not all features are 100% in place but this blogpost will do a comparison between Nutanix and Storage Spaces Direct and how they differenciate. Now Storage Spaces direct is a more advances Storage Spaces setup, but uses the same capabilities but now we can agreegate local disks inside servers to setup a SMB 3.0 based fileservice.

This is an overview of how a Storage Spaces Direct setup might look like, since it has a requirement for 4 nodes and having a backbone RDMA, Im come back to why this is a requirement. Now as I have mentioned previously is that Storage Spaces direct has an issue and that is with data locality, Microsoft treats storage and compute as two seperate entities and that is reflected in the Storage Spaces Direct setup. Since it can be setup as two seperate components SMB Scale out file server or using hyperconvereged.

When setting up as Hyperconverged the following happens

Let us say that we have a VM01 running on NODE1 which is running on top of a storage spaces direct vdisk01 running as a two-way mirror. What will happen is that Storage Spaces will create 1 GB extent of the vDisk and spread the chucks across seperate nodes, so even thou the VM01 is running on a specific host, the storage is placed random on the different hosts within the cluster, which will indicate that this will generate alot of east-west traffic within the cluster, and that is why Microsoft has set a requirement that we have RDMA network backbone on our Storage Spaces Direct cluster since it will require low-latency – high troughput traffic in order to be efficient in this type of cluster setup, since Microsoft just looks at the different nodes as a bunch of disks.

On the other hand, Nutanix solves this in another matter, which I also think that Microsoft should think about which is data locality, in case of a VM running on a particular host, most of the content is served locally from the host that the VM is running on, using the different tieres (Content Cache), (Extent Store), (Oplog)

Which removes the requirement of any particular high speed backbone.

#nutanix, #storage-spaces-direct

Setting up Storage Spaces Direct on Windows Server 2016 TP3

This is a step-by-step guide on how to setup a minimal Storage Spaces direct cluster on virtual machines running on Vmware workstation. It is also meant to enlighten people abit about the functionality which Microsoft is coming with and what it is lacking at the moment.

Important thing to remember about Storage Spaces Direct it is Microsoft’s first step into a converged infrastructure. Since it allows us to setup servers using locally attached Storage and created a cluster on top. Kinda lika VSAN and Nutanix, but not quite there yet. On top of the cluster functionality it uses Storage Spaces to create a pool of different vDisks on top to store virtual machines. Storage spaces Direct is not at the same level as VSAN and Nutanix, but It can also be used for general file server usage.

This setup is running Vmware workstation 11 with 2 virtual machines for scale-out file server and 1 domain controller.
The two scale out file servers have attached 4 virtual harddrives and 2 NICs.

Important that the harddrives are SATA based

After setting up the virtual machines install file server and failover cluster manager

Install-WindowsFeature –Name File-Services, Failover-Clustering –IncludeManagementTools

Then just create a failover cluster using Failover cluster manager or using PowerShell

New-Cluster –Name hvcl –Node <hv02,hv01> –NoStorage

After the Cluster setup is complete we need to define that this is going to be a Storage Spaces Direct SAN

Enable-ClusterStorageSpacesDirect

Then do a validation test to make sure that the Storage Spaces direct cluster shold work as inteded

Now you might get a warning that the disks on both nodes have the same identifier In case you need to shut down on of the VMs and change SATA disk identifier

Then define cluster network usage

The Storage Spaces Replication network will be on Cluster Only usage. Now that we have the bunch of disks available we need to create a disk pool. This can either be done using Failover cluster or using Powershell

 

But either way you should disable Writebackcache on a Storage Spaces Direct cluster which can be done after the creating using set-storagepool –friendlyname “nameofpool” –writecachesizedefault 0

Now we can create a new virtual disk, then configure settings like storage resilliency and such

Then we are done with the vDisk

Now when going to the virtual disk partion setup, make sure that you set it to ReFS

Now we can see that default values of the storage spaces direct vdisk

Now I can create a CSV volume of that vDisk

After we have created the CSV we need to add the Scale-out file server role as a clustered role

Next we need to add a file share to explose the SMB file share to external applications such as Hyper-V

And we are done!

We can now access the storage spaces direct cluster using the client access point we defined. Now during file transfer we can see which endpoint is being used for the reciving end. in this case it is this host 192.168.0.30 which is getting a file transfer from 192.168.0.1 and will then replicate the data to 10.0.0.2 across the cluster network.

The SMB client uses DNS to do an initial request to the SMB file server. Then they agree upon the dialect to use in the process. (This is from Message Analyzer)

Now what it is missing ?

Data locality! I have not seen any indication that Hyper-V clusters running on top of Storage Spaces direct in a hyperconverged deployment have the ability to “place” or run virtual machines on the node that they are running on top on. This will create a fragementation of storage / compute which is not really good. Maybe this is going to be implemented in the final version of Windows Server 2016, but not the SMB protocol does not have any built-in mechinims that handles this. For instance Nutanix has the built-in since the Controller will see if the VM is running locally or not and will start replicating bits and bytes until the processing is running locally.

#nutanix, #storage-spaces-direct, #windows-server-2016, #yper-v