How Nutanix works with Hyper-V and SMB 3.0

In my previous blog post I discussed a bit about software defined options using Hyper-V https://msandbu.wordpress.com/2015/07/28/software-defined-storage-options-for-hyper-v/ and that Windows Server is getting alot of good built-in capabilities but lacks the proper scale out solution with performance, which is also something that is coming with Windows Server 2016.

Now one of the vendors which I talked about which has a proper scale-out SDS solution for Hyper-V with support for SMB 3 is Nutanix, which is the subject for this blogpost where I will describe how it works for SMB based storage, now before I head on over to that I want to talk a little bit about how SMB 3 and some of the native capabilities and why they do not work for a proper HCI setup.

With SMB 3.0 Microsoft Introduced two great new features, which was SMB Direct and Multichannel, which are features that are aimed for higher troughput over lower latency.

SMB Multichannel (leverages multiple TCP connections across multiple CPU cores using RSS)

SMB Direct (allowing for RDMA based network transfer, which does bypasses the TCP stack and moving data from memory to memory which gives low overhead, low latency connections.

Now both these features allow us to leverage better NIC utilization, but is aimed for a traditional configuration where storage is still a seperate resource from computing. My guess is that when we are going to deploy a Storage Spaces Direct cluster on Windows Server 2016 in a HCI deployment these features will be disabled.

So how does Nutanix work with SMB 3 ?

image

First of, important to understand the underlaying structure of the Nutanix OS. First of all local storage in the Nutanix nodes from a cluster are added to a unified pool of storage which are part of the Nutanix distributed filesystem. On top of this we create containers which have their settings like compression, dedup and replication factor which defines the amount of copies of data within a container. The reason for these copies are for fault-tolerance in case of a node failure or disk failure. So in essence you can think about this is a DAG (Database availability Groups) but for virtual machines.

So for SMB we can have shares which are represented as containers which again are created on top of a Nutanix cluster.  Which are then presented to the Hyper-V hosts for VM placement.

Also important to remember that even thou we have a distributed file system across different nodes, the data is always run locally for a node (reason for this is so that the network does not becoming a point of congestion) Nutanix has a special role called the Curator (Which runs on the CVM)which is responsible for moving the hot data as local to the VM as possible. So if we for instance do a migration from host 1 to host 2, the CVM on host 1 might still contain the VM data and then reads and writes will from host 2 to CVM on host 1 the CVM will start to cache the data locally.

Now since this architecture leverages data locallity there is no need for feature like SMB Direct and SMB multichannel so therefore these features are not required in a Nutanix deployment for Hyper-V, however is does support SMB transparent failover which allows for continuously available file shares.

Now I haven’t started to explain yet how this architecture handles I/O yet, this is where the magic happens. Stay tuned.

Trouble with Hyper-V, Virtual Machine manager and XenClient

So in my Hyper-V enviroment all of hosts are administered by Virtual Machine Manager. The other day I needed to deploy Citrix XenClient to a hyper-v host (Since its the only hypervisor that is supported for the syncronizer part)

Now by default when installing XenClient it sets up the TomCat service running on port 443. After the XenClient installation was complete and I didn’t think much about it for the next week or so.

image

After that I needed to deploy a new virtual machine from a template to the same host, and then I started getting some strange error messages on the job status in VMM

“A Hardware Management error has occured trying to contact server”

image

Now I could either change the ports used for BITS in VMM by following the instructions here –> http://support.microsoft.com/kb/2405062 or I could change the ports of the TomCat engine by following the setup her –> http://support.citrix.com/article/CTX134691

So in my case I changed VMM To use different ports for BITS (Since I have other products that might run on 443 on a Hyper-V server.

image

After I changed the port, VM deployment worked as it should again!

Veeam Management pack for Hyper-V and Vmware walktrough

Yesterday, Veeam released their new management pack which for the first time includes support for both Vmware and Hyper-V. Now I have gotten a lot of questions regarding (Why have Hyper-V monitoring if Microsoft has it ?) well Veeam’s pack has alot more features included, such as capacity planning, heat maps and so on.

The management pack can be downloaded as an free trial from veeam’s website here –> http://www.veeam.com/system-center-management-pack-vmware-hyperv.html

Now as for the architecture of the functionality here it’s quite simple

image

First of there are two components.

* Veeam Virtualization Extesions (Service and UI) it manages connections to VMware systems and the Veeam Collector(s), controling licensing, load balancing, and high availability

* Veeam Collector component gathers data from VMware and injects its information into the Ops Agent.

It is possible to install all of these components on the management server itself. You can also install the collector service on other servers which have the Opsmgr agent installed. The virtualization extension service must be installed on the management server.

In my case I wanted to install this on the mangement server itself, since I have a small enviroment. Before I started the installation I needed to make sure that the management server was operating in proxy mode.

8

Next I started the installation on the management server. Now as with all of Veeams setup it can automatically configure all prerequisites and is pretty straight forward. (Note it will automatically import all required management packs into SCOM1

If you have a large enviroment it is recommended to split ut collectors into different hosts and create a resource pool (There is an online calculator which can help you find out how many collectors you need) http://www.veeam.com/support/mp_deployment.html

You can also define if collector roles should be automatically deployed

2

After the installation is complete (using the default ports) you will find the extensions shortcut on the desktop

4

By default this opens a website on the localhost (using port 4430) from here we need to enter the connection information to Vmware (Hyper-V hosts are discovered automatically when they have the agent installed) Same with Veeam Backup servers as well.

5

After you have entered the connection info you will also get a header saying the recommended number of collector hosts.

7

After this is finished setup you can open the OpsMgr console. From here there is one final task that is needed. Which is to Configure the Health Service, this can be dome from tasks under _All_active_Alerts under VMware monitoring pane.

image

After this is done you need to expect atleast 15 min before data is populated into your OpsMgr servers, depending on the load. You can also view the events logs on the Opsmgr servers to see that data is correctly imported.

image

and after a while, voila!

I can for instance view info about storage usage

image

 

Vm information

image

Now I could show grafs and statistics all day but one of the cool stuff in this release, is the cloud capacity planning reports.

image

They allow it to see for instance how many virtual machines I would need in Azure (and what type) to move them there.

image

Microsoft Virtual Machine Converter 2.0

So this is such a great update I have to blog about it, I have been in many projects involving migrating from VMware to Hyper-V and there of course many options to choose from there. Alas Microsoft had its own Virtual Machine Converter but didn’t have support for the latest version.

Microsoft today released a new version of Virtual MAchine Converter which contains the following updates:

With the release today, you will be able to access many updated features including:

  • Added support for vCenter & ESX(i) 5.5
  • VMware virtual hardware version 4 – 10 support
  • Linux Guest OS migration support including CentOS, Debian, Oracle, Red Hat Enterprise, SuSE enterprise and Ubuntu.

We have also added two great new features:

  • On-Premises VM to Azure VM conversion: You can now migrate your VMware virtual machines straight to Azure. Ease your migration process and take advantage of Microsoft’s cloud infrastructure with a simple wizard driven experience.
  • PowerShell interface for scripting and automation support: Automate your migration via workflow tools including System Center Orchestrator and more. Hook MVMC 2.0 into greater processes including candidate identification and migration activities.

 

So alot of great new features which should make it even easier to convert Virtual Machines. Also another important factor here is this.

At this time, we are also announcing the expected availability of MVMC 3.0 in fall of 2014. In that release we will be providing physical to virtual (P2V) machine conversion for supported versions of Windows.

Since Microsoft removed this option from SCVMM in R2 release its great that it is coming back. You can download the tool from here –> http://www.microsoft.com/en-us/download/details.aspx?id=42497

Study resources 74-409 Server Virtualization with Windows Server Hyper-V and System Center

NOTE: This is work in progress
Now its a long time made since I made any of these, but been busy Smilefjes
Here is a new exam from Microsoft which just released earlier this november, this is the first Microsoft exam which contains Azure technology from an “it-pro” perspetive and it also contains stuff from the latest 2012 R2 release.
The exam also goes trough stuff like Generation 2 VMs, Hyper-V recovery manager and so on.
You can read more about the exam here –> http://www.microsoft.com/learning/en-us/exam.aspx?ID=74-409 This exam replaces the earlier MCTIP Server Virtualization for Windows Server 2008.

The exam will contain the following. So im addind study resources under each section.

Configure Hyper-V

  • Create and configure virtual machine settings.
    • This objective may include but is not limited to: Configure dynamic memory; configure smart paging; configure Resource Metering; configure guest integration services; create and configure Generation 1 and 2 virtual machines; configure and use extended session mode, and configure RemoteFX

Dynamic Memory –> http://technet.microsoft.com/en-us/library/hh831766.aspx
Enable Resource Metering –> http://technet.microsoft.com/en-us/library/hh848481.aspx
Configure Guest Integration –> http://www.techrepublic.com/blog/data-center/configure-integration-services-options-for-hyper-v-vms/
Create Gen 2 VMs –>http://blogs.technet.com/b/jhoward/archive/2013/10/24/hyper-v-generation-2-virtual-machines-part-1.aspx
Extended session –> http://technet.microsoft.com/en-us/library/dn282274.aspx
Configure RemoteFX –> http://social.technet.microsoft.com/wiki/contents/articles/16652.remotefx-vgpu-setup-and-configuration-guide-for-windows-server-2012.aspx

  • Create and configure virtual machine storage.
    • This objective may include but is not limited to: Create VHDs and VHDx; configure differencing drives; modify VHDs; configure pass-through disks; manage checkpoints; implement a virtual Fibre Channel adapter; configure storage Quality of Service

Create VHD and VHDX –> http://technet.microsoft.com/en-us/library/hh848503.aspx
Create Differeing disks –> http://lyncdup.com/2012/06/creating-hyper-v-3-differencing-disks-in-server-2012-with-gui-and-powershell/
Pass-trough disks –> http://www.petri.co.il/convert-hyper-v-pass-through-disk-to-a-vhdx.htm
Implement virtual fibre channel –> http://www.virtualizationadmin.com/articles-tutorials/microsoft-hyper-v-articles/storage-management/first-look-hyperv-vs-virtual-fibre-channel-feature-part2.html
Configure Storage QoS –> http://technet.microsoft.com/en-us/library/dn282276.aspx
Modify VHD –> http://technet.microsoft.com/en-us/library/dn282284.aspx

  • Create and configure virtual networks.
    • This objective may include but is not limited to: Configure Hyper-V virtual switches; optimize network performance; configure MAC addresses; configure network isolation; configure synthetic and legacy virtual network adapters; configure NIC teaming in virtual machines

Configure Hyper-V virtual Switches –> http://www.serverwatch.com/server-tutorials/harnessing-the-power-of-hyper-v-network-virtual-switches.html
Optimize network performance –> http://www.aidanfinn.com/?p=15414
Configure network isolation –> http://technet.microsoft.com/en-us/library/jj679878.aspx#bkmk_pvlan
Configure NIC teaming in virtual machines –> http://www.msserverpro.com/configuring-windows-server-2012-nic-teaming-to-a-hyper-v-virtual-machine/

Configure and Manage Virtual Machine High Availability

  • Configure failover clustering with Hyper-V.
    • This objective may include but is not limited to: Configure shared storage; configure Quorum; configure cluster networking; restore single node or cluster configuration; implement Cluster Aware Updating; upgrade a cluster; configure and optimize clustered shared volumes; and configure clusters without network names

Configure shared storage –> http://blogs.technet.com/b/keithmayer/archive/2012/12/12/step-by-step-building-a-free-hyper-v-server-2012-cluster-part-1-of-2.aspx
Configure Quorum –> http://technet.microsoft.com/en-us/library/jj612870.aspx
Configure cluster networking –> http://www.msserverpro.com/implementing-windows-server-2012-hyper-v-failover-clustering/
Optimizate clustered shared volumes –> http://technet.microsoft.com/en-us/library/jj612868.aspx
Restore cluster configuration –>
Configure clusters without network names –> http://technet.microsoft.com/en-us/library/dn265970.aspx
Cluster aware updating –> http://technet.microsoft.com/en-us/library/hh831694.aspx

  • Manage failover clustering roles.
    • This objective may include but is not limited to: Configure role-specific settings including continuously available shares; configure VM monitoring; configure failover and preference settings; and configure guest clustering

Configure VM monitoring –> http://blogs.msdn.com/b/clustering/archive/2012/04/18/10295158.aspx
Configure guest cluestering –>  http://technet.microsoft.com/en-us/library/dn265980.aspx 

  • Manage virtual machine movement.
    • This objective may include but is not limited to: Perform Live Migration; perform quick migration; perform storage migration; import, export, and copy VMs; configure Virtual Machine network health protection; configure drain on shutdown; manage Physical-to-Virtual (P2V) and Virtual-to-Virtual (V2V) migrations; and implement virtual machine migration between clouds

Live Migration –> http://technet.microsoft.com/en-us/library/hh831435.aspx http://technet.microsoft.com/en-us/library/jj860434.aspx
Virtual Machine network health protection –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_VMHealth
Virtual Machine Drain on shutdown –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_VMDrain
Physical-to-Virtual –> http://blogs.technet.com/b/scvmm/archive/2013/10/03/how-to-perform-a-p2v-in-a-scvmm-2012-r2-environment.aspx
V2V migration –> http://technet.microsoft.com/en-us/library/gg610672.aspx

Implement a Server Virtualization Infrastructure

  • Implement virtualization hosts.
    • This objective may include but is not limited to: implement delegation of virtualization environment (hosts, services, and virtual machines) including self-service capabilities; implement multi-host libraries including equivalent objects; implement host resource optimization; integrate third-party virtualization platforms; and deploying Hyper-V hosts to bare metal

Bare Metal –> http://technet.microsoft.com/en-us/library/gg610634.aspx
Host Resource optimization –> http://technet.microsoft.com/en-us/library/gg675109.aspx
Selv-service capabilites –> http://technet.microsoft.com/en-us/library/gg610573.aspx
Integrate third-party virtualization –> http://technet.microsoft.com/en-us/library/gg610687.aspx

  • Implement virtual machines.
    • This objective may include but is not limited to: Implement highly available VMs; implement guest resource optimization including shared VHDx; configure placement rules; create a Virtual Machine Manager template

Shared VHDx –> http://technet.microsoft.com/en-us/library/dn265972.aspx#BKMK_SharedVHDX
Placement rules –> http://technet.microsoft.com/en-us/library/jj860428.aspx
Create template –> http://technet.microsoft.com/en-us/library/hh427282.aspx

  • Implement virtualization networking.
    • This objective may include but is not limited to: Configure Virtual Machine Manager logical networks including virtual switch extensions and logical switches; configure IP address and MAC address settings across multiple Hyper-V hosts including network virtualization; configure virtual network optimization; plan and implement Windows Server Gateway; implement VLANs and pVLANs; plan and implement virtual machine networks; and implement converged networks
  • Implement virtualization storage.
    • This objective may include but is not limited to: Configure Hyper-V host clustered storage; configure Hyper-V virtual machine storage including virtual Fibre Channel, Internet SCSI (iSCSI), and shared VHDx; plan for storage optimization; and plan and implement storage by using SMB 3.0 file shares
  • Manage and maintain a server virtualization infrastructure.
    • This objective may include but is not limited to: Manage dynamic optimization and resource optimization; integrate Operations Manager with System Center Virtual Machine Manager and System Center Service Manager; update virtual machine images in libraries; implement backup and recovery of a virtualization infrastructure by using System Center Data Protection Manager (DPM)

Monitor and Maintain a Server Virtualization Infrastructure

  • Plan and implement a monitoring strategy.
    • This objective may include but is not limited to: planning considerations including monitoring servers using Audit Collection Services (ACS) and System Center Global Service Monitor, performance monitoring, application monitoring, centralized monitoring, and centralized reporting; implement and optimize System Center 2012 Operations Manager management packs; and plan for monitoring Active Directory
  • Plan and implement a business continuity and disaster recovery solution.
    • This objective may include but is not limited to: plan a backup and recovery strategy; planning considerations including Active Directory domain and forest recovery, Hyper-V replica including using Windows Azure Hyper-V Recovery Manager, domain controller restore and cloning, and Active Directory object and container restore using authoritative restore and Recycle Bin; and plan for and implement backup and recovery by using System Center Data Protection Manager (DPM)

Securing Hyper-V 2012R2 hosts and VMs

Microsoft has implemented a lot of new cool security features in Hyper-V on the 2012R2 release, and most importently statefull firewall and network inspection features.

From the 2012 release, Microsoft introduced features like
* ARP Guard https://msandbu.wordpress.com/2013/04/03/arp-guard-in-hyper-v-2012/
* DHCP Guard
* Router Guard
(These three functions are also included in regular network devices from most vendors)

image

The use of Bandwidth control as well is useful for limiting for instance DDOS attacks.
* Bitlocker with Network Unlock (To protect a VM from theft)
* NVGRE (Network virtualization, which is not a security feature but it can be used to define each customer to its own network segment without the use of VLANs (This offers security since it is not able for instance to use VLAN-hopping)
* PVLAN (In many cases the use of VLANS still has its purpose for instance you can define three types of PVLANs (Isolated, Promiscuous and Community)
* VM stateless firewalls (Not on the indvidual VM but on the Hyper-V traffic going to the VMs) But these had pretty limited functionality (Which was restricted to IP-ACL, couldn’t define port or TCP EST)
* Bitlocker for CSV (Encrypt everything in a cluster)

So what else has Microsoft implemented of Security mechanisms in the OS-stack with the new R2 release ?

Not much info here yet.. but they are mostly related to hyper-v networking rules, new generation VMs with UEFI boot options (UEFI enable secure boot which makes it harder for rootkits to get installed)
image

What else can you do to secure your hosts and VM*s running on Hyper-V?

Microsoft has released a built-in baseline configuration that you can start from Server Manager this has some rules that It can use to scan if your hosts are according to best-practice, this offers you tips on what you should do.

image

Microsoft also offers other tools that can be used deploy security according to best practice  (This uses Group Policy for deployment of security settings)  for instance Security Compliance Manager http://www.microsoft.com/en-us/download/details.aspx?displayLang=en&id=16776

image

Installing all Hyper-v hosts as Server Core will also limit the attack surface on the hosts since it does not install all the unnecessery components like Internet explorer, .Net framework etc.
Which makes the host less open for attacks. (And also don’t use RDP there have been many security holes here which hackers have taken advantage of so If you need to enable RDP use NLA as well)

Monitoring / Antivirus and Patching

Integration with System Center also can prove to be quite useful for many reasons.
Which can offer you features like
* Anti-malware / Anti-virus (Configuration Manager)
* Patch management (Virtual Machine Manager / Configuration Manager)
* Baselining and remediation (Configuration Manager / Virtual Machine Manager)
image
* Monitoring (Operations Manager)

But this will require a number of agents being installed on all VM’s for instance Configuration Manager with Endpoint Protection and Operations Manager (and VMM agent on Hyper-v hosts)
(NOTE: You can enable baseline configuration in Operations Manager as well, instead of using Server Manager and with the integration of System Center Advisor you will get more intel)

image

Now Microsoft recommends that the parent partition to be as clean as possible, therefore they recommend not installing AV on the Hyper-V hosts (Since you will also suffer some performance loss), but if it is a part of the company policy.
Remember that if you install endpoint protection for Hyper-V hosts, put exclusions for these folders.“%PROGRAMDATA%\Microsoft\Windows\Hyper-V”
C:\ClusterStorage
You can read more about it here –> http://social.technet.microsoft.com/wiki/contents/articles/2179.hyper-v-anti-virus-exclusions-for-hyper-v-hosts.aspx

When regarding firewalls, each host running Windows has Windows Firewall enabled by default, should we then use Hyper-V port ACLs also ?
Hyper-V port ACLs follow the virtual machines so if you move them to another host, the ACL sticks. But they have different features.
The built-in firewall from Windows can allow Applications to communicate and is not restricted to a port or protcol, the firewall can also use IPsec.
While a Hyper-V port ACL can check if it is a statefull connection while the built-in firewall cannot. Hyper-V port ACL can also measure the traffic bandwidth that goes trough.
For many reasons you should use for built-in firewall for most cases (Create Group policies for the most common use server roles) and in more extreme cases where you need to lock down more and controll the traffic flow more you deploy and hyper-v port ACL.

You should also move your management traffic to a dedicated NIC outside of other traffic so it is not so easy to “sniff” on your traffic.

RBAC (Role Based Access Control) an easy rule of thumb is to split user rights where you can.
For instance an hyper-v administrator should not have admin-rights on VMs and vice versa.
If  you are using SCVMM you should create custom User Roles (For instance you can define a user role that (Group 1) has access to which can be used to administrate their hosts (Which is under a host group) and access to certain run as roles)

image

Sysinternals also should be used when evaluating your security for instance to see if there are any open ports that shouldn’t be open by using TCPView
http://technet.microsoft.com/en-US/sysinternals
image

Make sure that your internal network is configured as it should.
By disabling CDP on access ports (If you are using Cisco)
Enabling all ports as Access Ports (Portfast) so you can’t be hijacked by STP attacks.

image

Other resources:
http://www.microsoft.com/en-us/download/details.aspx?id=16650 This is an old security guide from Microsoft but alot of it still applies today.

Might also mention that there are some third party solutions that you can use to secure Hyper-V.

5-Nine –> http://www.5nine.com/
Watchguard –> http://www.watchguard.com

ARP guard in Hyper-V 2012

So I decided to try the ARP guard functionality in Hyper-V 2012 and see how it works, and in the same case check if it is possible to change the Mac address.

I took a look at what documentation Microsoft had around the subject
http://blogs.technet.com/b/wincat/archive/2012/11/18/arp-spoofing-prevention-in-windows-server-2012-hyper-v.aspx
http://technet.microsoft.com/en-us/library/hh831823.aspx

And what they say here is that

 I am sure you already browsed the new Hyper-V Manager UI and found a couple of new settings like DHCP Guard, Router Guard but nothing specific for ARP Spoofing.
Well, the feature you are looking for is called Port Access Control Lists and is implemented in the new Hyper-V switch and must be configured via PowerShell.

Arp Spoofing is a technique that allows for man-in-the-middle attack.

I can for instance place my computer in the middle of another user and intercept all the traffic going between the end-user and the gateway and place a sniffer on my computer and scan all the traffic going in and out.
Without the user even knowing it. This can happen because of how the Arp protocol is built. It is built on trust, and how computers can find other computers on the same subnet and was never thought of as a secure protocol.

So in order to test this out I had to setup a minor lab built with a couple of VM’s running on a hyper-v 2012 virtual switch.
1: with Windows Server 2008 R2
1: one domain controller
1: Linux Backtrack (which I will use arp spoof and mac changer on)

So when I start my newly installed WS2008 server It has a clean arp table (which consists of the broadcast address)

And as you can see this computer has the IP address 10.0.0.56
So what happens when I ping this server from the backtrack computer ? First the arp request (who owns this ip ? )

You can see the arp request first, then the ICMP protocol start. Then the Arp table is updated.

As an dynamic update. Then I ping the domain controller, which has ip 10.0.0.1,

and it has added itself to the list, look at difference between the mac addresses of 1 and 77.
Next I start the arp-spoof attack from my backtrack computer.

And I can see in wireshark that I am spamming with ARP traffic

And notice here I am saying that IP 10.0.0.1 is at another MAC address.
If you check the arp table now on the other computer you can see that the arp table is updated (poisoned)

And after I activate IP forwarding on the backtrack server I can «act» as a man in the middle.
As you can see now when I try to ping 10.0.0.1 I get a response

but from my Backtrack server instead of my domain controller. And according to my server it responds fine from 10.0.0.1

So how does the arpguard in Windows Server fit in here? In addition, where can I configure it?
The answer is Port Access Control Lists via PowerShell.

This is configured on the Hyper-V host I find it a best to do it via the PowerShell ISE.
so what can I do ? First, I have to create a port ACL that defines that the virtual machine can ONLY communicate out with the IP address of 10.0.0.77 and not any other.

So when I apply this port ACL and try to ping 10.0.0.1 It will not receive a response, and since it does not get a response I tries an ARP request again and my backtrack computer is unable to respons because of the Port ACL

And the arp table is restored to its default.