Veeam 9 Scale-out backup repositories

So what did backup with v8 used to look like? A backup-job was attached to a file repository, the problem was that if the repository was getting low on space, which might happen from time to time. We could either clean up, try to expand the space or do even more damage. Even thou we might have multiple repositories available for use, we would need to move the backup data from one repo to another and then update the database, using for instance this KB https://www.veeam.com/kb1729

image

Luckily this has changed from V9 with the introduction of the awesome! Scale-out backup repositories.
This allows us to group together a mix of different repositories or extents into a group, which allows us to group together the size of all our repositories into a single “pool”

To setup a Scale-Out repo there must exist on the repository server, then we can go and create, and choose Add.

image

By default the Use per-VM backup files setting is enabled, this means that it will place a per-vm backup file instead of a full backup job on a single extent.

image

We can also enable full backup, if a required extent (which hosts the existing incremental backup files) is offline. We also have options to define policies, which is either based upon data locality or performance.

With data locality option switched on, Veeam will place all dependant backups files on the same extent, which is typically the case with incremental backups + its full backup file. If we choose performance policy, we can define for instance which extents should have the full backups and which should have the incremental.

image

In my case I have two repositories, one fast and one slow. In that case I want all my full backups to be place on my fast repository and the incrementals on my slow repository. The backup jobs will point to the same “scale-out group” but the repository will handle the data differently.

image

image

So what happens if we need to do maintance or need to move the data from a repository that is being retired?

Then we have the ability to firstly set maintance mode on, when a repository is set into maintaince mode we have the option called «Evacuate» backups. Now when doing evacuation, we have no option to define which extents should get the data being moved. If we have for instance multiple repositories setup and have data locality policy enabled, Veeam will try to honor that policy, same goes if we have define multiple incremental repositories and multiple full backup repositories. If we were to evacuate a repository with only full backups, Veeam will try to move that data to another repository which has full backup policy enabled.

image

Note however that there are some limitations to Scale-Out repository, depending on license and backup type:

  • The scale-out backup repository functionality is available only in Enterprise and Enterprise Plus editions of Veeam Backup & Replication.
  • Configuration backup job
  • Replication jobs
  • Endpoint backup jobs
  • You cannot add a backup repository as an extent to the scale-out backup repository if any job of unsupported type is targeted at this backup repository or if the backup repository contains data produced by jobs of unsupported types (for example, replica metadata). To add such backup repository as an extent, you must first target unsupported jobs to another backup repository and remove the job data from the backup repository.
  • You cannot use a scale-out backup repository as a cloud repository. You cannot add a cloud repository as an extent to the scale-out backup repository.
  • If a backup repository is added as an extent to the scale-out backup repository, you cannot use it as a regular backup repository.
  • You cannot add a scale-out backup repository as an extent to another scale-out backup repository.
  • You cannot add a backup repository as an extent if this backup repository is already added as an extent to another scale-out backup repository.
  • You cannot add a backup repository on which some activity is being performed (for example, a backup job or restore task) as an extent to the scale-out backup repository.
  • If you use Enterprise Edition of Veeam Backup & Replication, you can create 1 scale-out backup repository with 3 extents for this scale-out backup repository. Enterprise Plus Edition has no limitations on the number of scale-out backup repositories or extents.

#scale-out-repositories, #veeam

Advanced backup options for Hyper-V 2012 R2 on Veeam

Some questions that come up again and again are advanced backup features in Hyper-V using Veeam. How does Veeam take backup from a Hyper-V host?

in a simple day to day virtual machine life, the read & writes consist of I/O traffic from a virtual machines to a VHD/VHDX. Residing on a network share. SAN/SMB and such.

image

When we setup Veeam to take backup of a virtual machine, what will happen is the following. First thing is that Veeam will trigger a snapshot using the Hyper-V Integration Services Shadow Copy Provider on that particular Hyper-V host that the virtual machine resides on. What will happen is that a AVHDX. This can either be done using an hardware VSS provier or software VSS provider.

image

A hardware provider manages shadow copies at the hardware level by working in conjunction with a hardware storage adapter or controller. A software provider manages shadow copies by intercepting I/O requests at the software level between the file system and the volume manager. The number of VMs in a group is limite depending on VSS provider. For a software VSS provider — 4 VMs, for a hardware VSS provider — 8 VMs.

NOTE: Using Offhost-proxy, requires an storage solution which supports an hardware transferrable shadow copies against a SAN. If we for instance use SMB based storage for Hyper-V we do not require this –> http://helpcenter.veeam.com/backup/hyperv/smb_off-host_backup.html

Using onhost backup, means that the transport role will be using on a Hyper-V host which has access to the running virtual machines.

Make sure that the integration services and running and up to date before doing online backup, you can check this from Hyper-V PowerShell –> Get-VM | FT Name, IntegrationServicesVersion
More troubleshooting on interation services here –> https://www.veeam.com/kb1855

So what will happen in a online backup is (If all the requirements are meet)

1: Veeam will interact with the Hyper-V host VSS service and request backup of the specific VM

2: The VSS writer on the hyper-v host will then forward the reques tto the Hyper-V Integration components inside the VM guest OS

3: The integration components will then communicate with the VSS framework inside the guest OS and request backup of all VSS-aware application inside the VM

4: The VSS writers of application aware VSS will then get application data suiteable for backup

5: After the applications are quiesced the VSS inside the Virtual machine takes an internal snapshot using the software based VSS

6: The integration service component notifices the Hypervisor that the VM is ready for backup, and Hyper-V will then take a snapshot of the volume which the Virtual machine is located on. Then a AVHDX file will be generated, all WRITES will be redirected there.

7: The volume snapshot is presented to Veeam either using Off-host or on-host backup. (If the Off-host proxy is not available it will fallback to on-host proxy on a designeted host)

8: Data will then be processed on the proxy server and then be moved to the repository

image

NOTE: Off-host setup requires an dedicated Hyper-V host (It requires Hyper-V to have access to the VSS providers) and in case of using Off-host it cannot be part of the Hyper-V cluster, and make sure it has READ only access to the LUN and that your storage vendors supports readable shadow volume copies.

On-host backup will use the Veeam transport service on the Hyper-V machine. If the volume is placed on a CSV volume, the CSV Software Shadow Copy Provider will be used for the snapshot creation process.

NOTE: During the backup process, Veeam will try to use its own CBT driver on Hyper-V host to make sure that it only takes backup of only the changed blocks. (Since Hyper-V does not natively provide CBT, this will change in Windows Server 2016)

NOTE: If CBT is not working on Veeam run the command Reset-HvVmChangeTracking PowerShell cmdlet http://helpcenter.veeam.com/backup/80/powershell/reset-hvvmchangetracking.html, or if the virtual machines are being shut down during backup process, try to disable ODX)

If Change block tracking is not enabled or not working as it should, the backup proxy will copy the virtual machine and use Veeam’s proprietary filtering mechanism. so Instead of tracking changed blocks of data, Veeam Backup & Replication filters out unchanged data blocks. During backup, Veeam Backup & Replication consolidates virtual disk content, scans through the VM image and calculates a checksum for every data block. Checksums are stored as metadata to backup files next to VM data.

So what about the more advanced features for Hyper-V

Hyper-V Settings

  • Enable Hyper-V guest quiescene

In case of application aware, The VM OS is suspsended and the content of the system memory and CPU is written to a dump file, in order to be able to perserve the data integrity of files with for instance transactional applications (This is known as offline backup)

Note that using this feature Veeam will not be able to perform application tasks like

    • Applying application-specific settings to prepare applications for VSS-aware restore at the next VM startup
    • Truncating transaction logs after successful backup or replication.
  • Take Crach consistent backup instead of suspending VM

If you do not want to suspend the virtual machine during backup, you can use crach consistent backup instead of suspending the virtual machine. This is equal to a hard reset of a virtual machine, this does not involve any downtime to a virtual machine but it does not preserve the data integrity of open files and may result in data loss.

  • Use changed block tracking data

Use the Veeam filter driver to look at changed blocks before data is copied to the offhost-veeam proxy or on-host proxy to the repository

  • Allow Processing of multiple VMs with a single volume snapshot

If you have multiple virtual machines within the same job, this feature will help reduce the load on the Hyper-V hosts.As this will trigger a volume snapshot for mulitple machines instead of a single virtual machine.

NOTE: The virtual machines much be located on the same host and must reside on a file share which uses the same VSS provider.

This is the first post of series – Veeam post and Hyper-V processing.

#backup, #hyper-v, #veeam

Setting up Veeam Managed backup portal in Azure

Veeam now has available a new managed backup portal in the Azure marketplace, which will make it easier to do on-boarding / monitoring and multi-tenancy.

Integrated with Veeam Cloud Connect for Service Providers and available in the Microsoft Azure Marketplace, Veeam Managed Backup Portal for Service Providers makes it easy to acquire new customers and build new revenue streams through the following capabilities:

  • Simplified customer on-boarding: With a service provider administration portal, creating new customer accounts, provisioning services, and even managing customer billing and invoicing is easier than ever 
  • Streamlined remote monitoring and remote management: Daily monitoring and management of customers’ jobs is made simple and convenient, and can be done securely through a single port over SSL/TLS (no VPN required)
  • Multi-tenant customer portal: Clients remain engaged with a customer portal where they can set up users and locations, easily monitor backup health, review cloud repository consumption and manage monthly billing statements.

Now this as of now in tech preview available from Azure marketplace.

image

Which can deployed either using resource manager or using classic mode. After the deployment is done, you should do one last configuraiton which is to add a custom endpoint to be allowed to manage the setup externally over https. Which can be done under the security group endpoint settings.

image

NOTE: Before managing anything from the portal you need to add a license to the Veeam console, you can get a trial license here –> http://go.veeam.com/managed-backup-portal-trial-ty.html (Then connect to the virtual machine using RDP)

NOTE: The cloud connect seutp is already enabled, ports are also setup.

After adding the firewall rules for (destination port:443) source any we can configure the portal using the public IP address and port 443 (From there we login with our machine username and password, which was provisioned using the Azure portal)

image

After logging in into the portal I am greeted with the configuration wizard.

image

So we can start by creating a new customer

image

So we go trough the settings like a reguler setup and we choose a subscription plan

image

Next time I now logout and login again, I have a new portal dashboard, which gives me the quick overview.

image

We can also see that there is a new user created with description Veeam portal

image

now after we add a cloud gateway on the Azure machine, we can connect to it using an existing Veeam infrastructure

image

And configure and backup copy job and start doing copies to Azure. The end customer has its own portal (website) that they can access to see their status. They need to login using companyname\username and password on the same portal.

image

This is just a small post on what is to come!

#veeam

Setting up NFS Direct Veeam against Nutanix cluster

So the last couple of days I have tried to wrap my head around Direct NFS support which is coming in Veeam v9. The cool thing about this feature is that Veeam has a custom built NFS agent, which will go directly to the NFS share (only needs READ access) and export the snapshot data when doing a backup.

Now important that Veeam is configured against a vCenter server ( I tried many times against an ESX directly and then NFS Direct didn’t really work.

When setting up a Direct NFS backup solution, we need to first setup a Veeam Backup Proxy as we would in other scenarioes. We need to include the Veeam Backup Proxy in the virtual vSwitch that Nutanix provisions within ESX (Note: Do not change the vSwitch, just add the VM to the vSwitch network)

image

Then define an IP address to the Veeam Backup Proxy within the vSwitch so it can communicate with the Controller VM.

image

Note that since the vSwitch is an internal only switch, we should setup a Backup proxy per node to maximize the performance. Even thou in this scenario it will work to do NFS direct on this node against other node as well, but then we will be pushing the traffic across the Controller VM network. So when setting up backup jobs try to make it so it uses the local proxy on the host which the virtual machines recides on, this will give the best troughput.

We also need to whitelist the IP address of the proxy so that it can allow access ot the NFS share (Which in the case of Nutanix will be the Storage Container which virtual machines resides on) This can be done on a container level or at a cluster level.

image

Next we need to “force” Veeam to use the storage network on the proxies to do backup traffic. Which can be done in the central management pane within Veeam.

image

Lastly we need to rescan the storage attached to the infrastructure which will allow Veeam to see the new NFS datastores and see that they can access it using NFS direct. This can be done here.

image

We can see from the statistics of this job that it is using NFS in the first screenshot

image

We can also see in the backup job log file for the VM

image

and that we are using regular hotadd in the second one.

image

#nutanix, #veeam

Virtual Machine backup in Azure using Veeam Endpoint

A while back I blogged about Veeam Endpoint https://msandbu.wordpress.com/2014/12/01/veeam-endpoint-backup-a-new-free-backup-solution-for-computers-and-physical-servers/ while it is aimed at Physical computers / servers it has another purpose that I just discovered.

In Azure, Microsoft has currently a preview feature called Azure VM backup, which in essence is a image based backup of virtual machines in Azure. Now since this currently has alot of limitations I figured what other options do we have?

While some people do Windows Server Backup directly to another Azure VM disk, I figured why not give Veeam a try with Data disk and use it in conjunction with SMB files. The reason why is that we can use Veeam Endpoint do to backup to an data disk (which is attached to an individual VM) then create a task to move the backup to an SMB files store (in case the virtual machines crashes or is unavailable we have the backup on an SMB file share and that makes it accessable for all other virtual machines within that storage account. NOTE: Doing Veeam backup directly to SMB file shares is not working

So we create a virtual machine in Azure and then use the portal to attach an empty data disk for the virtual machine

image

This new disk is going to be the repostiory for Veeam Endpoint within the VM

SMB files is a SMB like feature which is currently in preview and is available for each storage account. In order to use it we must first create a SMB file share using PowerShell

$FSContext=New-AzureStorageContext storageaccount storageaccountkey

$FS = New-AzureStorageShare sampleshare -Context $FSContext

New-AzureStorageDirectory -Share $FS -Path sampledir

After we have created the fileshare we need to add the network path to the virtual machine inside Azure. First we shold use CMDkey to add the username and password to the SMB file share to that it can reconnect after reboot

cmdkey /add: storageaccountpost.file.core.windows.net /user:useraccount /pass:<storage Access Key>

And then use net use z: \\storageaccount.file.core.windows.net\sampleshare

image

After the network drive is mapped up, we can install Veeam Endpoint.

image

Now Veeam Endpoint is a free backup solution, it can integrate with existing Veeam infrastructure such as repositories for more centralized backup solution. It also has some limitations regarding application-aware processing but works well with tradisional VMs in Azure.

After setup is complete we can setup our backup schedule

image

image

image

Then I run the backup job. Make sure that the backup job is run correnctly, not that as best-pratice is not to store any appliaction or such on C:\ drive, I also got VSS error messages while backing up data on c:\ therefore you should have another data disk where you store applications and files if neccessery.

Now after the backup is complete we have our backup files on a data disk that is attached to a virtual machine. We have two options here in case we need to restore data on another virtual machine.

1: We can run the restore wizard from the backup files on another virtual machine against the copied backup files on the SMB file share

image

2: Deattach and reattach the virtual disk to another virtual machine.
this is cumbersome to do if we have multiple virtual harddrives

image

Now the attaching a virtual disk is done on the fly, when we run the restore wizard from Veeam, the wizard will automatically detect the backup volume and give us the list of restore points available on the drive

image

Note that while running the file recovery wizard does not give us an option to restore back directly to the same volume, so we can only copy data out from a backup file.

image

Well there you have it, using Veeam endpoint protection for virtual machine in Azure against a data drive. After given it a couple of test runs I can tell its working as intended and gives alot better functionality over the built-in windows server backup. If you want to you can also set it up with Veeam FastSCP for Azure and allowing it to download files from Azure VMs down to an on-premises setup.

#azure, #veeam

New award – Veeam Vanguard

Received some good news today, (Which I have known for quite some time) but it is only now that I am allowed to talk about it Smilefjes

I have been quite active regarding Veeam on my blog and much work related since I am a Veeam Instructor and a general evangelist for their products, so therefore I was quite thrilled when Veeam announced a new community award called Veeam Vanguard and that I was one of the awardees!

and now I join the ranks of other skilled IT-pros in the community such as, Thomas Maurer, Rasmus Haslund and a fellow Norwegian Christian Mohn

Thanks to Veeam!

More info on the Vanguard page here — http://www.veeam.com/vanguard.html

#veeam, #veeam-vanguard

Building up a Veeam Cloud Connect infrastructure in Azure

Now before I start, I have already been blogging about settings up Veeam Cloud Connect in Auzre https://msandbu.wordpress.com/2014/11/12/veeam-cloud-connect-for-microsoft-azure-walkthrough/

And its important to remember the Veeam Cloud Connect is only available for Veeam Service Providers (or VCP Veeam Cloud Providers)

This is more of a technical overview of the solution.

image

On-premise Veeam customers which have version 8 (should also have patch 1 installed) Can add a service provider from their console, this can be a IaaS solution running in Azure.

End customers are given a usage quota on the cloud repositories. This shows how much data they can store on their cloud repostitory.

So how to setup this in Azure ?

  • Use either the template from Veeam which is in the Azure Marketplace (NOTE: This requires a paid subscribtion in order to be activated)
  • Download the BITS and install it ourselves.

Now when setting this up in Azure there are a few things to take notice of.

Firstly always check of where the closest datacenter to the customers are, you can use this third-party website as a reference –> http://www.azurespeed.com/

The first two virtual machines are used as a cloud gateway proxy. They will handle the incoming data but not store the data. Important things to take note of here is the bandwidth requirements depending on how many customers, since they operate as a proxy I would try to keep them as cheap as possible. So if we look at the A-instance virtual machines

image

A2 gives us 200 Mbps bandwidth and should be adequate for Gateway proxy performance. On a side note here, A instances do not have SSD drives, so if we want to setup customers using WAN acceleration we should use the D-series (Which has SSD enabled drivers on the D:\ partition) Which gives it a good boost on doing the digest work of comparing blocks. (Ref blogpost IOPS performance in Azure –> https://msandbu.wordpress.com/2013/07/16/azure-and-iops-performance/)

image

There are also some other limits that need to be taken in account. First of when planning for repositories. Data disks in Azure only support up to 1 TB pr disk, meaning that if you need to store data over 1 TB you need to setup Storage spaces running across many drives (Note that storage spaces and geo-replication are not supported)

Also there is a cap for 500 IOPS or data disk, this can be increased a bit by using storage spaces as well. For a regular A4 instance (there as maximum of 16 data disks) look at this reference sheet https://msdn.microsoft.com/en-us/library/azure/dn197896.aspx there is higher amounts of IOPS for D and G-series. Also allows for higher amounts of stored data.

Then you might think (well thats not much data? a maximum amount of 32 TB) important to note that this is not a replacement for on-premise backup. And that moving 32 TB of data from Azure during an outage back on-premice might restrict itself because of the internet bandwidth available at the customer. Just for info, moving 1000 GB over 100 MBps link requires 23 hours… (If your customers require more data and better bandwidth and lower latency, well Azure is not the right solution Smilefjes

Lastly its important to setup load balancing for our cloud gateways. Now the cloud gateways already have built-in load balancing, and will redirect internally based upon traffic. What we need is to load balance the initial request to the Cloud Gateway, since after the first connection, Veeam will keep a list of the availabe cloud gateways.

Now there are two ways to do this using Azure. Either we can use regular DNS based round robin, this means that we have multiple A-records for the same FQDN. When Veeam connects it is able to download all the A-records and try them one after one. Problem with DNS round is that it has no option to check health, and therefore it might take more time.

We can also use Traffic Manager (Which is Azure Load balancing) which has the ability to do health probes to check if they are alive or not. The negativ of this is that when a DNS request is make to our Traffic Manager DNS alias it will only respond with one IP-address & FQDN.

Setting up traffic manager in Azure is a pretty simple case, you just setup it up, give it a URL (Which then needs to be attached using CNAME to a FQDN of your choice on your domain.

image 

And note that this requires that we have multiple cloud services (Which again have their own public IP address)

image

Now the monitoring part here is a bit tricky, since it by default uses HTTP GET commands to verify the existence of a server. Either using HTTP or HTTPS, which require installation of IIS and then setup ACL’s on the endpoints to only respond from Microsoft Traffic manager.

The instances running as a cloud gateway need to be put in a availabilty group in order to get SLA from Microsoft. When in a availability group, Microsoft knows they can take one of the roles down in the group when they have maintance, and allowing for the other one to keep running.

The repositories can be customer specific (depending on the size) but should not be placed in a availability group (since there are no options for shared storage in the backend to keep it redundant) if a virtual machine is not placed in a availability group the azure administrator will get a notice 2 weeks before hand, and in most cases it will just cause the virtual machine to reboot once and it will be up and running again.

#azure, #cloudconnect, #veeam