Setting up NFS Direct Veeam against Nutanix cluster

So the last couple of days I have tried to wrap my head around Direct NFS support which is coming in Veeam v9. The cool thing about this feature is that Veeam has a custom built NFS agent, which will go directly to the NFS share (only needs READ access) and export the snapshot data when doing a backup.

Now important that Veeam is configured against a vCenter server ( I tried many times against an ESX directly and then NFS Direct didn’t really work.

When setting up a Direct NFS backup solution, we need to first setup a Veeam Backup Proxy as we would in other scenarioes. We need to include the Veeam Backup Proxy in the virtual vSwitch that Nutanix provisions within ESX (Note: Do not change the vSwitch, just add the VM to the vSwitch network)


Then define an IP address to the Veeam Backup Proxy within the vSwitch so it can communicate with the Controller VM.


Note that since the vSwitch is an internal only switch, we should setup a Backup proxy per node to maximize the performance. Even thou in this scenario it will work to do NFS direct on this node against other node as well, but then we will be pushing the traffic across the Controller VM network. So when setting up backup jobs try to make it so it uses the local proxy on the host which the virtual machines recides on, this will give the best troughput.

We also need to whitelist the IP address of the proxy so that it can allow access ot the NFS share (Which in the case of Nutanix will be the Storage Container which virtual machines resides on) This can be done on a container level or at a cluster level.


Next we need to “force” Veeam to use the storage network on the proxies to do backup traffic. Which can be done in the central management pane within Veeam.


Lastly we need to rescan the storage attached to the infrastructure which will allow Veeam to see the new NFS datastores and see that they can access it using NFS direct. This can be done here.


We can see from the statistics of this job that it is using NFS in the first screenshot


We can also see in the backup job log file for the VM


and that we are using regular hotadd in the second one.


Virtual Machine backup in Azure using Veeam Endpoint

A while back I blogged about Veeam Endpoint while it is aimed at Physical computers / servers it has another purpose that I just discovered.

In Azure, Microsoft has currently a preview feature called Azure VM backup, which in essence is a image based backup of virtual machines in Azure. Now since this currently has alot of limitations I figured what other options do we have?

While some people do Windows Server Backup directly to another Azure VM disk, I figured why not give Veeam a try with Data disk and use it in conjunction with SMB files. The reason why is that we can use Veeam Endpoint do to backup to an data disk (which is attached to an individual VM) then create a task to move the backup to an SMB files store (in case the virtual machines crashes or is unavailable we have the backup on an SMB file share and that makes it accessable for all other virtual machines within that storage account. NOTE: Doing Veeam backup directly to SMB file shares is not working

So we create a virtual machine in Azure and then use the portal to attach an empty data disk for the virtual machine


This new disk is going to be the repostiory for Veeam Endpoint within the VM

SMB files is a SMB like feature which is currently in preview and is available for each storage account. In order to use it we must first create a SMB file share using PowerShell

$FSContext=New-AzureStorageContext storageaccount storageaccountkey

$FS = New-AzureStorageShare sampleshare -Context $FSContext

New-AzureStorageDirectory -Share $FS -Path sampledir

After we have created the fileshare we need to add the network path to the virtual machine inside Azure. First we shold use CMDkey to add the username and password to the SMB file share to that it can reconnect after reboot

cmdkey /add: /user:useraccount /pass:<storage Access Key>

And then use net use z: \\\sampleshare


After the network drive is mapped up, we can install Veeam Endpoint.


Now Veeam Endpoint is a free backup solution, it can integrate with existing Veeam infrastructure such as repositories for more centralized backup solution. It also has some limitations regarding application-aware processing but works well with tradisional VMs in Azure.

After setup is complete we can setup our backup schedule




Then I run the backup job. Make sure that the backup job is run correnctly, not that as best-pratice is not to store any appliaction or such on C:\ drive, I also got VSS error messages while backing up data on c:\ therefore you should have another data disk where you store applications and files if neccessery.

Now after the backup is complete we have our backup files on a data disk that is attached to a virtual machine. We have two options here in case we need to restore data on another virtual machine.

1: We can run the restore wizard from the backup files on another virtual machine against the copied backup files on the SMB file share


2: Deattach and reattach the virtual disk to another virtual machine.
this is cumbersome to do if we have multiple virtual harddrives


Now the attaching a virtual disk is done on the fly, when we run the restore wizard from Veeam, the wizard will automatically detect the backup volume and give us the list of restore points available on the drive


Note that while running the file recovery wizard does not give us an option to restore back directly to the same volume, so we can only copy data out from a backup file.


Well there you have it, using Veeam endpoint protection for virtual machine in Azure against a data drive. After given it a couple of test runs I can tell its working as intended and gives alot better functionality over the built-in windows server backup. If you want to you can also set it up with Veeam FastSCP for Azure and allowing it to download files from Azure VMs down to an on-premises setup.

New award – Veeam Vanguard

Received some good news today, (Which I have known for quite some time) but it is only now that I am allowed to talk about it Smilefjes

I have been quite active regarding Veeam on my blog and much work related since I am a Veeam Instructor and a general evangelist for their products, so therefore I was quite thrilled when Veeam announced a new community award called Veeam Vanguard and that I was one of the awardees!

and now I join the ranks of other skilled IT-pros in the community such as, Thomas Maurer, Rasmus Haslund and a fellow Norwegian Christian Mohn

Thanks to Veeam!

More info on the Vanguard page here —

Building up a Veeam Cloud Connect infrastructure in Azure

Now before I start, I have already been blogging about settings up Veeam Cloud Connect in Auzre

And its important to remember the Veeam Cloud Connect is only available for Veeam Service Providers (or VCP Veeam Cloud Providers)

This is more of a technical overview of the solution.


On-premise Veeam customers which have version 8 (should also have patch 1 installed) Can add a service provider from their console, this can be a IaaS solution running in Azure.

End customers are given a usage quota on the cloud repositories. This shows how much data they can store on their cloud repostitory.

So how to setup this in Azure ?

  • Use either the template from Veeam which is in the Azure Marketplace (NOTE: This requires a paid subscribtion in order to be activated)
  • Download the BITS and install it ourselves.

Now when setting this up in Azure there are a few things to take notice of.

Firstly always check of where the closest datacenter to the customers are, you can use this third-party website as a reference –>

The first two virtual machines are used as a cloud gateway proxy. They will handle the incoming data but not store the data. Important things to take note of here is the bandwidth requirements depending on how many customers, since they operate as a proxy I would try to keep them as cheap as possible. So if we look at the A-instance virtual machines


A2 gives us 200 Mbps bandwidth and should be adequate for Gateway proxy performance. On a side note here, A instances do not have SSD drives, so if we want to setup customers using WAN acceleration we should use the D-series (Which has SSD enabled drivers on the D:\ partition) Which gives it a good boost on doing the digest work of comparing blocks. (Ref blogpost IOPS performance in Azure –>


There are also some other limits that need to be taken in account. First of when planning for repositories. Data disks in Azure only support up to 1 TB pr disk, meaning that if you need to store data over 1 TB you need to setup Storage spaces running across many drives (Note that storage spaces and geo-replication are not supported)

Also there is a cap for 500 IOPS or data disk, this can be increased a bit by using storage spaces as well. For a regular A4 instance (there as maximum of 16 data disks) look at this reference sheet there is higher amounts of IOPS for D and G-series. Also allows for higher amounts of stored data.

Then you might think (well thats not much data? a maximum amount of 32 TB) important to note that this is not a replacement for on-premise backup. And that moving 32 TB of data from Azure during an outage back on-premice might restrict itself because of the internet bandwidth available at the customer. Just for info, moving 1000 GB over 100 MBps link requires 23 hours… (If your customers require more data and better bandwidth and lower latency, well Azure is not the right solution Smilefjes

Lastly its important to setup load balancing for our cloud gateways. Now the cloud gateways already have built-in load balancing, and will redirect internally based upon traffic. What we need is to load balance the initial request to the Cloud Gateway, since after the first connection, Veeam will keep a list of the availabe cloud gateways.

Now there are two ways to do this using Azure. Either we can use regular DNS based round robin, this means that we have multiple A-records for the same FQDN. When Veeam connects it is able to download all the A-records and try them one after one. Problem with DNS round is that it has no option to check health, and therefore it might take more time.

We can also use Traffic Manager (Which is Azure Load balancing) which has the ability to do health probes to check if they are alive or not. The negativ of this is that when a DNS request is make to our Traffic Manager DNS alias it will only respond with one IP-address & FQDN.

Setting up traffic manager in Azure is a pretty simple case, you just setup it up, give it a URL (Which then needs to be attached using CNAME to a FQDN of your choice on your domain.


And note that this requires that we have multiple cloud services (Which again have their own public IP address)


Now the monitoring part here is a bit tricky, since it by default uses HTTP GET commands to verify the existence of a server. Either using HTTP or HTTPS, which require installation of IIS and then setup ACL’s on the endpoints to only respond from Microsoft Traffic manager.

The instances running as a cloud gateway need to be put in a availabilty group in order to get SLA from Microsoft. When in a availability group, Microsoft knows they can take one of the roles down in the group when they have maintance, and allowing for the other one to keep running.

The repositories can be customer specific (depending on the size) but should not be placed in a availability group (since there are no options for shared storage in the backend to keep it redundant) if a virtual machine is not placed in a availability group the azure administrator will get a notice 2 weeks before hand, and in most cases it will just cause the virtual machine to reboot once and it will be up and running again.

Veeam Endpoint backup free

Today at the VeeamON conference, they announced a new tool called Veeam Endpoint Backup free. This tool which will ship H1 next year Allows us to take backup of physical servers, computers, laptops and such.

It can integrate with existing Veeam repositories or to a NAS share. The best part it of course that it is going to be free! Smilefjes

Stay tuned as the preview comes later in November, but this allows us to lastly do backup av Physical servers in a Veeam enviroment without the need to buy more licenses.

Veeam Management pack for Hyper-V and Vmware walktrough

Yesterday, Veeam released their new management pack which for the first time includes support for both Vmware and Hyper-V. Now I have gotten a lot of questions regarding (Why have Hyper-V monitoring if Microsoft has it ?) well Veeam’s pack has alot more features included, such as capacity planning, heat maps and so on.

The management pack can be downloaded as an free trial from veeam’s website here –>

Now as for the architecture of the functionality here it’s quite simple


First of there are two components.

* Veeam Virtualization Extesions (Service and UI) it manages connections to VMware systems and the Veeam Collector(s), controling licensing, load balancing, and high availability

* Veeam Collector component gathers data from VMware and injects its information into the Ops Agent.

It is possible to install all of these components on the management server itself. You can also install the collector service on other servers which have the Opsmgr agent installed. The virtualization extension service must be installed on the management server.

In my case I wanted to install this on the mangement server itself, since I have a small enviroment. Before I started the installation I needed to make sure that the management server was operating in proxy mode.


Next I started the installation on the management server. Now as with all of Veeams setup it can automatically configure all prerequisites and is pretty straight forward. (Note it will automatically import all required management packs into SCOM1

If you have a large enviroment it is recommended to split ut collectors into different hosts and create a resource pool (There is an online calculator which can help you find out how many collectors you need)

You can also define if collector roles should be automatically deployed


After the installation is complete (using the default ports) you will find the extensions shortcut on the desktop


By default this opens a website on the localhost (using port 4430) from here we need to enter the connection information to Vmware (Hyper-V hosts are discovered automatically when they have the agent installed) Same with Veeam Backup servers as well.


After you have entered the connection info you will also get a header saying the recommended number of collector hosts.


After this is finished setup you can open the OpsMgr console. From here there is one final task that is needed. Which is to Configure the Health Service, this can be dome from tasks under _All_active_Alerts under VMware monitoring pane.


After this is done you need to expect atleast 15 min before data is populated into your OpsMgr servers, depending on the load. You can also view the events logs on the Opsmgr servers to see that data is correctly imported.


and after a while, voila!

I can for instance view info about storage usage



Vm information


Now I could show grafs and statistics all day but one of the cool stuff in this release, is the cloud capacity planning reports.


They allow it to see for instance how many virtual machines I would need in Azure (and what type) to move them there.


Veeam B&R 7 a list of issues and solutions

Now  I’ve been working with Veeam for a while now, and I’ve seen thatt mostly the case that when a backup job fails (or a surebackup job fails) or something fails, its most often not Veeam’s fault.

Veeam is a powerful product but it is dependant on alot of external features to function properly in order to do its job right. For instance in order to backup from a Vmware host, you need a vmware license in place in order to allow Veeam to access the Vmware VADP API’s.
If not Veeam can’t backup your virtual machines running on Vmware.

Also in order to do incremental backups properly Veeam is also dependant on CBT working properly on the hypervisor. So the real purpose of this blog post is mostly for my own part, but having a list of problems/errors that I come across in Veeam and what the fix is for it.

Now in most cases, when running jobs the job indicator will give a good pinpoint what the problem is. If not look into the Veeam logs which are located under C:\Programdata\Veeam\Logs (Programdata is a hidden folder) there is also possible to generate support logs directly from the Veeam console –>

Issue nr 1# Cannot use CBT when running backup jobs
Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: ‘<InvalidArgumentFault xmlns=»urn:internalvim25″ xsi:type=»InvalidArgument»><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>’, endpoint: »

If CBT is for some reason not available and it not being used, Veeam has its own filter which it uses in these cases. Veeam will then process the entire VM and then on its own compare the block of the VM and the backup and see which blocks have changed, and the copy only the changed blocks to the repository. This makes processing time alooooot longer. Now in order to fix this you need to reset CBT on the guest VM. This can be done by following the instructions here –> and one for Hyper-V CBT

Issue nr 2# Sure backup jobs fail with error code 10061 when running applications tests. This is most likey when a firewall is configured on the guest VM which only allows specific VMs. I have also seen this when a guestVM is a restarting state. If you do not have a guestVM firewall active, doing a restart of the guestVM and then do a new backup should allow the surebackup job to run successfully.

Issue nr 3# WAN accelerator failes to install. This might happen if a previous Veeam install has failed on a server. When you try to install the WAN accelerator the setup just stops without no parent reason. Something makes the installpath of the WAN cache folder to the wrong drive. You need to go into the registry of the VM and change the default paths as seen here –>

Issue nr 4# Backup of GuestVMs running on a hyper-v server with Windows Server 2012 R2 update 1, this is a known issue from Microsoft which requires an update from Microsoft –>

Issue nr 5# Application-aware image processing skipped on Microsoft Hyper-V server, this is of course related to many possible features. In most cases it is integration services, a list of the different causes and solutions are listed here –>

Issue nr 6# Logs not getting truncated on Exchange/SQL guest VMs, this requires application aware image processing and define that the backup job should truncate logs –>

Issue nr 7# Backup of vCenter servers –>

Issue nr 8# Backup using Hyper-V and Dell Equallogic VSS –>

Issue nr 9# Incredible slow backup over the network and no load on the servers, make sure that all network switches are full-duplex.

Issue nr 10# Win32 error: the network path was not found. When doing application aware image processing veeam needs to access the VM using the admin share with the credentials that are defined in the backup job. (For Vmware if the VM does not have network access Vmware VIX is used) It is possible to change the priority of these protocols –>