Hyper-V and Storage features deep-dive comparison with Nutanix

So another blogpost in this storage series with Hyper-V, in the previous posts I discussed a bit about what features Hyper-V has and the issues with them. Well time to take that to the next level. Just to show how Nutanix solves the performance issues and how Microsoft does it with their Windows Server features.

First of we have the native capabilities with Windows Server and Storage Spaces. We can benefit from SMB 3 and for instance mutlichannel with RSS and Jumbo frames which allows for much less overhead in a TCP network, of course it requires some knowledge on congestion algoritms to use as well to be able to use the full troughput

We can also use tiering in the back-end with the default write-back cache feature (which by default is on 1 GB) and during night the tiering feature run an optimization task that moves the hot data to the SSD tier and the cold data to the HDD tier.

On the other hand we can have a RDMA deplouyment which in essence removes the TCP/IP stack completly and does zero-copy network capabilities, and we can use this in conjunction with CSV cache which only provides benefits for read-only unbuffered I/Os in RAM on the host, this feature can be enabled on a CSV disk level and is integrated into failover cluster manager and is leveraged on all the hosts in a cluster. but… this feature is disabled for a tiered stoarge space CSV therefore they can not be both activated on the same deployment.


In the Nutanix I/O Path things are a bit different, since the CVM (Controller VM) serves content locally from the node to the hyper-V host using SMB using disk passtrough locally.


The I/O fabric in a Nutanix node consists of many different logical stores. First of we have the Content Cache which is an deduplicated read cache which consists of both memory and SSD. Which is serverd from the memory of the CVM. Here we have the ability to leverage from inline deduplication.

Then we have the OpLog which is built to handle random I/O, when dealing with bursts of random I/O it coalesce them and then sequentially drains it to the other Store (Extent Store) The oplog is on the SSD tier. In case of sequencial Write I/O  the Oplog is bypassed and is then writen directly to the Extent Store.  The Oplog is also replicated to one or more nodes in a cluster to handle high-availabilty.

The Extent Store serves as persistent data storage in a Nutanix node, which consists of SSD and HDD. Data coming into the extent store is either directly as sequential write I/O or drained from the Oplog. The Extent store can also leverage from deduplication, this is a cross cluster deduplication feature, meaning that all nodes participate. 

So as we can see Nutanix leverages tiering, deduplication, in-memory caching while maintaining availability for data across nodes in a cluster, and combining this with data locality to deliver the lowest form of latency.

How Nutanix works with Hyper-V and SMB 3.0

In my previous blog post I discussed a bit about software defined options using Hyper-V https://msandbu.wordpress.com/2015/07/28/software-defined-storage-options-for-hyper-v/ and that Windows Server is getting alot of good built-in capabilities but lacks the proper scale out solution with performance, which is also something that is coming with Windows Server 2016.

Now one of the vendors which I talked about which has a proper scale-out SDS solution for Hyper-V with support for SMB 3 is Nutanix, which is the subject for this blogpost where I will describe how it works for SMB based storage, now before I head on over to that I want to talk a little bit about how SMB 3 and some of the native capabilities and why they do not work for a proper HCI setup.

With SMB 3.0 Microsoft Introduced two great new features, which was SMB Direct and Multichannel, which are features that are aimed for higher troughput over lower latency.

SMB Multichannel (leverages multiple TCP connections across multiple CPU cores using RSS)

SMB Direct (allowing for RDMA based network transfer, which does bypasses the TCP stack and moving data from memory to memory which gives low overhead, low latency connections.

Now both these features allow us to leverage better NIC utilization, but is aimed for a traditional configuration where storage is still a seperate resource from computing. My guess is that when we are going to deploy a Storage Spaces Direct cluster on Windows Server 2016 in a HCI deployment these features will be disabled.

So how does Nutanix work with SMB 3 ?


First of, important to understand the underlaying structure of the Nutanix OS. First of all local storage in the Nutanix nodes from a cluster are added to a unified pool of storage which are part of the Nutanix distributed filesystem. On top of this we create containers which have their settings like compression, dedup and replication factor which defines the amount of copies of data within a container. The reason for these copies are for fault-tolerance in case of a node failure or disk failure. So in essence you can think about this is a DAG (Database availability Groups) but for virtual machines.

So for SMB we can have shares which are represented as containers which again are created on top of a Nutanix cluster.  Which are then presented to the Hyper-V hosts for VM placement.

Also important to remember that even thou we have a distributed file system across different nodes, the data is always run locally for a node (reason for this is so that the network does not becoming a point of congestion) Nutanix has a special role called the Curator (Which runs on the CVM)which is responsible for moving the hot data as local to the VM as possible. So if we for instance do a migration from host 1 to host 2, the CVM on host 1 might still contain the VM data and then reads and writes will from host 2 to CVM on host 1 the CVM will start to cache the data locally.

Now since this architecture leverages data locallity there is no need for feature like SMB Direct and SMB multichannel so therefore these features are not required in a Nutanix deployment for Hyper-V, however is does support SMB transparent failover which allows for continuously available file shares.

Now I haven’t started to explain yet how this architecture handles I/O yet, this is where the magic happens. Stay tuned.

Trouble with Hyper-V, Virtual Machine manager and XenClient

So in my Hyper-V enviroment all of hosts are administered by Virtual Machine Manager. The other day I needed to deploy Citrix XenClient to a hyper-v host (Since its the only hypervisor that is supported for the syncronizer part)

Now by default when installing XenClient it sets up the TomCat service running on port 443. After the XenClient installation was complete and I didn’t think much about it for the next week or so.


After that I needed to deploy a new virtual machine from a template to the same host, and then I started getting some strange error messages on the job status in VMM

“A Hardware Management error has occured trying to contact server”


Now I could either change the ports used for BITS in VMM by following the instructions here –> http://support.microsoft.com/kb/2405062 or I could change the ports of the TomCat engine by following the setup her –> http://support.citrix.com/article/CTX134691

So in my case I changed VMM To use different ports for BITS (Since I have other products that might run on 443 on a Hyper-V server.


After I changed the port, VM deployment worked as it should again!

Veeam Management pack for Hyper-V and Vmware walktrough

Yesterday, Veeam released their new management pack which for the first time includes support for both Vmware and Hyper-V. Now I have gotten a lot of questions regarding (Why have Hyper-V monitoring if Microsoft has it ?) well Veeam’s pack has alot more features included, such as capacity planning, heat maps and so on.

The management pack can be downloaded as an free trial from veeam’s website here –> http://www.veeam.com/system-center-management-pack-vmware-hyperv.html

Now as for the architecture of the functionality here it’s quite simple


First of there are two components.

* Veeam Virtualization Extesions (Service and UI) it manages connections to VMware systems and the Veeam Collector(s), controling licensing, load balancing, and high availability

* Veeam Collector component gathers data from VMware and injects its information into the Ops Agent.

It is possible to install all of these components on the management server itself. You can also install the collector service on other servers which have the Opsmgr agent installed. The virtualization extension service must be installed on the management server.

In my case I wanted to install this on the mangement server itself, since I have a small enviroment. Before I started the installation I needed to make sure that the management server was operating in proxy mode.


Next I started the installation on the management server. Now as with all of Veeams setup it can automatically configure all prerequisites and is pretty straight forward. (Note it will automatically import all required management packs into SCOM1

If you have a large enviroment it is recommended to split ut collectors into different hosts and create a resource pool (There is an online calculator which can help you find out how many collectors you need) http://www.veeam.com/support/mp_deployment.html

You can also define if collector roles should be automatically deployed


After the installation is complete (using the default ports) you will find the extensions shortcut on the desktop


By default this opens a website on the localhost (using port 4430) from here we need to enter the connection information to Vmware (Hyper-V hosts are discovered automatically when they have the agent installed) Same with Veeam Backup servers as well.


After you have entered the connection info you will also get a header saying the recommended number of collector hosts.


After this is finished setup you can open the OpsMgr console. From here there is one final task that is needed. Which is to Configure the Health Service, this can be dome from tasks under _All_active_Alerts under VMware monitoring pane.


After this is done you need to expect atleast 15 min before data is populated into your OpsMgr servers, depending on the load. You can also view the events logs on the Opsmgr servers to see that data is correctly imported.


and after a while, voila!

I can for instance view info about storage usage



Vm information


Now I could show grafs and statistics all day but one of the cool stuff in this release, is the cloud capacity planning reports.


They allow it to see for instance how many virtual machines I would need in Azure (and what type) to move them there.


Microsoft Virtual Machine Converter 2.0

So this is such a great update I have to blog about it, I have been in many projects involving migrating from VMware to Hyper-V and there of course many options to choose from there. Alas Microsoft had its own Virtual Machine Converter but didn’t have support for the latest version.

Microsoft today released a new version of Virtual MAchine Converter which contains the following updates:

With the release today, you will be able to access many updated features including:

  • Added support for vCenter & ESX(i) 5.5
  • VMware virtual hardware version 4 – 10 support
  • Linux Guest OS migration support including CentOS, Debian, Oracle, Red Hat Enterprise, SuSE enterprise and Ubuntu.

We have also added two great new features:

  • On-Premises VM to Azure VM conversion: You can now migrate your VMware virtual machines straight to Azure. Ease your migration process and take advantage of Microsoft’s cloud infrastructure with a simple wizard driven experience.
  • PowerShell interface for scripting and automation support: Automate your migration via workflow tools including System Center Orchestrator and more. Hook MVMC 2.0 into greater processes including candidate identification and migration activities.


So alot of great new features which should make it even easier to convert Virtual Machines. Also another important factor here is this.

At this time, we are also announcing the expected availability of MVMC 3.0 in fall of 2014. In that release we will be providing physical to virtual (P2V) machine conversion for supported versions of Windows.

Since Microsoft removed this option from SCVMM in R2 release its great that it is coming back. You can download the tool from here –> http://www.microsoft.com/en-us/download/details.aspx?id=42497

Study resources 74-409 Server Virtualization with Windows Server Hyper-V and System Center

NOTE: This is work in progress
Now its a long time made since I made any of these, but been busy Smilefjes
Here is a new exam from Microsoft which just released earlier this november, this is the first Microsoft exam which contains Azure technology from an “it-pro” perspetive and it also contains stuff from the latest 2012 R2 release.
The exam also goes trough stuff like Generation 2 VMs, Hyper-V recovery manager and so on.
You can read more about the exam here –> http://www.microsoft.com/learning/en-us/exam.aspx?ID=74-409 This exam replaces the earlier MCTIP Server Virtualization for Windows Server 2008.

The exam will contain the following. So im addind study resources under each section.

Configure Hyper-V

  • Create and configure virtual machine settings.
    • This objective may include but is not limited to: Configure dynamic memory; configure smart paging; configure Resource Metering; configure guest integration services; create and configure Generation 1 and 2 virtual machines; configure and use extended session mode, and configure RemoteFX

Dynamic Memory –> http://technet.microsoft.com/en-us/library/hh831766.aspx
Enable Resource Metering –> http://technet.microsoft.com/en-us/library/hh848481.aspx
Configure Guest Integration –> http://www.techrepublic.com/blog/data-center/configure-integration-services-options-for-hyper-v-vms/
Create Gen 2 VMs –>http://blogs.technet.com/b/jhoward/archive/2013/10/24/hyper-v-generation-2-virtual-machines-part-1.aspx
Extended session –> http://technet.microsoft.com/en-us/library/dn282274.aspx
Configure RemoteFX –> http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

  • Create and configure virtual machine storage.
    • This objective may include but is not limited to: Create VHDs and VHDx; configure differencing drives; modify VHDs; configure pass-through disks; manage checkpoints; implement a virtual Fibre Channel adapter; configure storage Quality of Service

Create VHD and VHDX –> http://technet.microsoft.com/en-us/library/hh848503.aspx
Create Differeing disks –> http://lyncdup.com/2012/06/creating-hyper-v-3-differencing-disks-in-server-2012-with-gui-and-powershell/
Pass-trough disks –> http://www.petri.co.il/convert-hyper-v-pass-through-disk-to-a-vhdx.htm
Implement virtual fibre channel –> http://www.virtualizationadmin.com/articles-tutorials/microsoft-hyper-v-articles/storage-management/first-look-hyperv-vs-virtual-fibre-channel-feature-part2.html
Configure Storage QoS –> http://technet.microsoft.com/en-us/library/dn282276.aspx
Modify VHD –> http://technet.microsoft.com/en-us/library/dn282284.aspx

  • Create and configure virtual networks.
    • This objective may include but is not limited to: Configure Hyper-V virtual switches; optimize network performance; configure MAC addresses; configure network isolation; configure synthetic and legacy virtual network adapters; configure NIC teaming in virtual machines

Configure Hyper-V virtual Switches –> http://www.serverwatch.com/server-tutorials/harnessing-the-power-of-hyper-v-network-virtual-switches.html
Optimize network performance –> http://www.aidanfinn.com/?p=15414
Configure network isolation –> http://technet.microsoft.com/en-us/library/jj679878.aspx#bkmk_pvlan
Configure NIC teaming in virtual machines –> http://www.msserverpro.com/configuring-windows-server-2012-nic-teaming-to-a-hyper-v-virtual-machine/

Configure and Manage Virtual Machine High Availability

  • Configure failover clustering with Hyper-V.
    • This objective may include but is not limited to: Configure shared storage; configure Quorum; configure cluster networking; restore single node or cluster configuration; implement Cluster Aware Updating; upgrade a cluster; configure and optimize clustered shared volumes; and configure clusters without network names

Configure shared storage –> http://blogs.technet.com/b/keithmayer/archive/2012/12/12/step-by-step-building-a-free-hyper-v-server-2012-cluster-part-1-of-2.aspx
Configure Quorum –> http://technet.microsoft.com/en-us/library/jj612870.aspx
Configure cluster networking –> http://www.msserverpro.com/implementing-windows-server-2012-hyper-v-failover-clustering/
Optimizate clustered shared volumes –> http://technet.microsoft.com/en-us/library/jj612868.aspx
Restore cluster configuration –>
Configure clusters without network names –> http://technet.microsoft.com/en-us/library/dn265970.aspx
Cluster aware updating –> http://technet.microsoft.com/en-us/library/hh831694.aspx

  • Manage failover clustering roles.
    • This objective may include but is not limited to: Configure role-specific settings including continuously available shares; configure VM monitoring; configure failover and preference settings; and configure guest clustering

Configure VM monitoring –> http://blogs.msdn.com/b/clustering/archive/2012/04/18/10295158.aspx
Configure guest cluestering –>  http://technet.microsoft.com/en-us/library/dn265980.aspx 

  • Manage virtual machine movement.
    • This objective may include but is not limited to: Perform Live Migration; perform quick migration; perform storage migration; import, export, and copy VMs; configure Virtual Machine network health protection; configure drain on shutdown; manage Physical-to-Virtual (P2V) and Virtual-to-Virtual (V2V) migrations; and implement virtual machine migration between clouds

Live Migration –> http://technet.microsoft.com/en-us/library/hh831435.aspx http://technet.microsoft.com/en-us/library/jj860434.aspx
Virtual Machine network health protection –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_VMHealth
Virtual Machine Drain on shutdown –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_VMDrain
Physical-to-Virtual –> http://blogs.technet.com/b/scvmm/archive/2013/10/03/how-to-perform-a-p2v-in-a-scvmm-2012-r2-environment.aspx
V2V migration –> http://technet.microsoft.com/en-us/library/gg610672.aspx

Implement a Server Virtualization Infrastructure

  • Implement virtualization hosts.
    • This objective may include but is not limited to: implement delegation of virtualization environment (hosts, services, and virtual machines) including self-service capabilities; implement multi-host libraries including equivalent objects; implement host resource optimization; integrate third-party virtualization platforms; and deploying Hyper-V hosts to bare metal

Bare Metal –> http://technet.microsoft.com/en-us/library/gg610634.aspx
Host Resource optimization –> http://technet.microsoft.com/en-us/library/gg675109.aspx
Selv-service capabilites –> http://technet.microsoft.com/en-us/library/gg610573.aspx
Integrate third-party virtualization –> http://technet.microsoft.com/en-us/library/gg610687.aspx

  • Implement virtual machines.
    • This objective may include but is not limited to: Implement highly available VMs; implement guest resource optimization including shared VHDx; configure placement rules; create a Virtual Machine Manager template

Shared VHDx –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_SharedVHDX
Placement rules –> http://technet.microsoft.com/en-us/library/jj860428.aspx
Create template –> http://technet.microsoft.com/en-us/library/hh427282.aspx

  • Implement virtualization networking.
    • This objective may include but is not limited to: Configure Virtual Machine Manager logical networks including virtual switch extensions and logical switches; configure IP address and MAC address settings across multiple Hyper-V hosts including network virtualization; configure virtual network optimization; plan and implement Windows Server Gateway; implement VLANs and pVLANs; plan and implement virtual machine networks; and implement converged networks
  • Implement virtualization storage.
    • This objective may include but is not limited to: Configure Hyper-V host clustered storage; configure Hyper-V virtual machine storage including virtual Fibre Channel, Internet SCSI (iSCSI), and shared VHDx; plan for storage optimization; and plan and implement storage by using SMB 3.0 file shares
  • Manage and maintain a server virtualization infrastructure.
    • This objective may include but is not limited to: Manage dynamic optimization and resource optimization; integrate Operations Manager with System Center Virtual Machine Manager and System Center Service Manager; update virtual machine images in libraries; implement backup and recovery of a virtualization infrastructure by using System Center Data Protection Manager (DPM)

Monitor and Maintain a Server Virtualization Infrastructure

  • Plan and implement a monitoring strategy.
    • This objective may include but is not limited to: planning considerations including monitoring servers using Audit Collection Services (ACS) and System Center Global Service Monitor, performance monitoring, application monitoring, centralized monitoring, and centralized reporting; implement and optimize System Center 2012 Operations Manager management packs; and plan for monitoring Active Directory
  • Plan and implement a business continuity and disaster recovery solution.
    • This objective may include but is not limited to: plan a backup and recovery strategy; planning considerations including Active Directory domain and forest recovery, Hyper-V replica including using Windows Azure Hyper-V Recovery Manager, domain controller restore and cloning, and Active Directory object and container restore using authoritative restore and Recycle Bin; and plan for and implement backup and recovery by using System Center Data Protection Manager (DPM)

Securing Hyper-V 2012R2 hosts and VMs

Microsoft has implemented a lot of new cool security features in Hyper-V on the 2012R2 release, and most importently statefull firewall and network inspection features.

From the 2012 release, Microsoft introduced features like
* ARP Guard https://msandbu.wordpress.com/2013/04/03/arp-guard-in-hyper-v-2012/
* DHCP Guard
* Router Guard
(These three functions are also included in regular network devices from most vendors)


The use of Bandwidth control as well is useful for limiting for instance DDOS attacks.
* Bitlocker with Network Unlock (To protect a VM from theft)
* NVGRE (Network virtualization, which is not a security feature but it can be used to define each customer to its own network segment without the use of VLANs (This offers security since it is not able for instance to use VLAN-hopping)
* PVLAN (In many cases the use of VLANS still has its purpose for instance you can define three types of PVLANs (Isolated, Promiscuous and Community)
* VM stateless firewalls (Not on the indvidual VM but on the Hyper-V traffic going to the VMs) But these had pretty limited functionality (Which was restricted to IP-ACL, couldn’t define port or TCP EST)
* Bitlocker for CSV (Encrypt everything in a cluster)

So what else has Microsoft implemented of Security mechanisms in the OS-stack with the new R2 release ?

Not much info here yet.. but they are mostly related to hyper-v networking rules, new generation VMs with UEFI boot options (UEFI enable secure boot which makes it harder for rootkits to get installed)

What else can you do to secure your hosts and VM*s running on Hyper-V?

Microsoft has released a built-in baseline configuration that you can start from Server Manager this has some rules that It can use to scan if your hosts are according to best-practice, this offers you tips on what you should do.


Microsoft also offers other tools that can be used deploy security according to best practice  (This uses Group Policy for deployment of security settings)  for instance Security Compliance Manager http://www.microsoft.com/en-us/download/details.aspx?displayLang=en&id=16776


Installing all Hyper-v hosts as Server Core will also limit the attack surface on the hosts since it does not install all the unnecessery components like Internet explorer, .Net framework etc.
Which makes the host less open for attacks. (And also don’t use RDP there have been many security holes here which hackers have taken advantage of so If you need to enable RDP use NLA as well)

Monitoring / Antivirus and Patching

Integration with System Center also can prove to be quite useful for many reasons.
Which can offer you features like
* Anti-malware / Anti-virus (Configuration Manager)
* Patch management (Virtual Machine Manager / Configuration Manager)
* Baselining and remediation (Configuration Manager / Virtual Machine Manager)
* Monitoring (Operations Manager)

But this will require a number of agents being installed on all VM’s for instance Configuration Manager with Endpoint Protection and Operations Manager (and VMM agent on Hyper-v hosts)
(NOTE: You can enable baseline configuration in Operations Manager as well, instead of using Server Manager and with the integration of System Center Advisor you will get more intel)


Now Microsoft recommends that the parent partition to be as clean as possible, therefore they recommend not installing AV on the Hyper-V hosts (Since you will also suffer some performance loss), but if it is a part of the company policy.
Remember that if you install endpoint protection for Hyper-V hosts, put exclusions for these folders.“%PROGRAMDATA%\Microsoft\Windows\Hyper-V”
You can read more about it here –> http://social.technet.microsoft.com/wiki/contents/articles/2179.hyper-v-anti-virus-exclusions-for-hyper-v-hosts.aspx

When regarding firewalls, each host running Windows has Windows Firewall enabled by default, should we then use Hyper-V port ACLs also ?
Hyper-V port ACLs follow the virtual machines so if you move them to another host, the ACL sticks. But they have different features.
The built-in firewall from Windows can allow Applications to communicate and is not restricted to a port or protcol, the firewall can also use IPsec.
While a Hyper-V port ACL can check if it is a statefull connection while the built-in firewall cannot. Hyper-V port ACL can also measure the traffic bandwidth that goes trough.
For many reasons you should use for built-in firewall for most cases (Create Group policies for the most common use server roles) and in more extreme cases where you need to lock down more and controll the traffic flow more you deploy and hyper-v port ACL.

You should also move your management traffic to a dedicated NIC outside of other traffic so it is not so easy to “sniff” on your traffic.

RBAC (Role Based Access Control) an easy rule of thumb is to split user rights where you can.
For instance an hyper-v administrator should not have admin-rights on VMs and vice versa.
If  you are using SCVMM you should create custom User Roles (For instance you can define a user role that (Group 1) has access to which can be used to administrate their hosts (Which is under a host group) and access to certain run as roles)


Sysinternals also should be used when evaluating your security for instance to see if there are any open ports that shouldn’t be open by using TCPView

Make sure that your internal network is configured as it should.
By disabling CDP on access ports (If you are using Cisco)
Enabling all ports as Access Ports (Portfast) so you can’t be hijacked by STP attacks.


Other resources:
http://www.microsoft.com/en-us/download/details.aspx?id=16650 This is an old security guide from Microsoft but alot of it still applies today.

Might also mention that there are some third party solutions that you can use to secure Hyper-V.

5-Nine –> http://www.5nine.com/
Watchguard –> http://www.watchguard.com