New award – Veeam Vanguard

Received some good news today, (Which I have known for quite some time) but it is only now that I am allowed to talk about it Smilefjes

I have been quite active regarding Veeam on my blog and much work related since I am a Veeam Instructor and a general evangelist for their products, so therefore I was quite thrilled when Veeam announced a new community award called Veeam Vanguard and that I was one of the awardees!

and now I join the ranks of other skilled IT-pros in the community such as, Thomas Maurer, Rasmus Haslund and a fellow Norwegian Christian Mohn

Thanks to Veeam!

More info on the Vanguard page here — http://www.veeam.com/vanguard.html

Building up a Veeam Cloud Connect infrastructure in Azure

Now before I start, I have already been blogging about settings up Veeam Cloud Connect in Auzre https://msandbu.wordpress.com/2014/11/12/veeam-cloud-connect-for-microsoft-azure-walkthrough/

And its important to remember the Veeam Cloud Connect is only available for Veeam Service Providers (or VCP Veeam Cloud Providers)

This is more of a technical overview of the solution.

image

On-premise Veeam customers which have version 8 (should also have patch 1 installed) Can add a service provider from their console, this can be a IaaS solution running in Azure.

End customers are given a usage quota on the cloud repositories. This shows how much data they can store on their cloud repostitory.

So how to setup this in Azure ?

  • Use either the template from Veeam which is in the Azure Marketplace (NOTE: This requires a paid subscribtion in order to be activated)
  • Download the BITS and install it ourselves.

Now when setting this up in Azure there are a few things to take notice of.

Firstly always check of where the closest datacenter to the customers are, you can use this third-party website as a reference –> http://www.azurespeed.com/

The first two virtual machines are used as a cloud gateway proxy. They will handle the incoming data but not store the data. Important things to take note of here is the bandwidth requirements depending on how many customers, since they operate as a proxy I would try to keep them as cheap as possible. So if we look at the A-instance virtual machines

image

A2 gives us 200 Mbps bandwidth and should be adequate for Gateway proxy performance. On a side note here, A instances do not have SSD drives, so if we want to setup customers using WAN acceleration we should use the D-series (Which has SSD enabled drivers on the D:\ partition) Which gives it a good boost on doing the digest work of comparing blocks. (Ref blogpost IOPS performance in Azure –> https://msandbu.wordpress.com/2013/07/16/azure-and-iops-performance/)

image

There are also some other limits that need to be taken in account. First of when planning for repositories. Data disks in Azure only support up to 1 TB pr disk, meaning that if you need to store data over 1 TB you need to setup Storage spaces running across many drives (Note that storage spaces and geo-replication are not supported)

Also there is a cap for 500 IOPS or data disk, this can be increased a bit by using storage spaces as well. For a regular A4 instance (there as maximum of 16 data disks) look at this reference sheet https://msdn.microsoft.com/en-us/library/azure/dn197896.aspx there is higher amounts of IOPS for D and G-series. Also allows for higher amounts of stored data.

Then you might think (well thats not much data? a maximum amount of 32 TB) important to note that this is not a replacement for on-premise backup. And that moving 32 TB of data from Azure during an outage back on-premice might restrict itself because of the internet bandwidth available at the customer. Just for info, moving 1000 GB over 100 MBps link requires 23 hours… (If your customers require more data and better bandwidth and lower latency, well Azure is not the right solution Smilefjes

Lastly its important to setup load balancing for our cloud gateways. Now the cloud gateways already have built-in load balancing, and will redirect internally based upon traffic. What we need is to load balance the initial request to the Cloud Gateway, since after the first connection, Veeam will keep a list of the availabe cloud gateways.

Now there are two ways to do this using Azure. Either we can use regular DNS based round robin, this means that we have multiple A-records for the same FQDN. When Veeam connects it is able to download all the A-records and try them one after one. Problem with DNS round is that it has no option to check health, and therefore it might take more time.

We can also use Traffic Manager (Which is Azure Load balancing) which has the ability to do health probes to check if they are alive or not. The negativ of this is that when a DNS request is make to our Traffic Manager DNS alias it will only respond with one IP-address & FQDN.

Setting up traffic manager in Azure is a pretty simple case, you just setup it up, give it a URL (Which then needs to be attached using CNAME to a FQDN of your choice on your domain.

image 

And note that this requires that we have multiple cloud services (Which again have their own public IP address)

image

Now the monitoring part here is a bit tricky, since it by default uses HTTP GET commands to verify the existence of a server. Either using HTTP or HTTPS, which require installation of IIS and then setup ACL’s on the endpoints to only respond from Microsoft Traffic manager.

The instances running as a cloud gateway need to be put in a availabilty group in order to get SLA from Microsoft. When in a availability group, Microsoft knows they can take one of the roles down in the group when they have maintance, and allowing for the other one to keep running.

The repositories can be customer specific (depending on the size) but should not be placed in a availability group (since there are no options for shared storage in the backend to keep it redundant) if a virtual machine is not placed in a availability group the azure administrator will get a notice 2 weeks before hand, and in most cases it will just cause the virtual machine to reboot once and it will be up and running again.

Veeam Endpoint backup free

Today at the VeeamON conference, they announced a new tool called Veeam Endpoint Backup free. This tool which will ship H1 next year http://www.veeam.com/blog/announcing-veeam-endpoint-backup-free.html. Allows us to take backup of physical servers, computers, laptops and such.

It can integrate with existing Veeam repositories or to a NAS share. The best part it of course that it is going to be free! Smilefjes

Stay tuned as the preview comes later in November, but this allows us to lastly do backup av Physical servers in a Veeam enviroment without the need to buy more licenses.

Veeam Management pack for Hyper-V and Vmware walktrough

Yesterday, Veeam released their new management pack which for the first time includes support for both Vmware and Hyper-V. Now I have gotten a lot of questions regarding (Why have Hyper-V monitoring if Microsoft has it ?) well Veeam’s pack has alot more features included, such as capacity planning, heat maps and so on.

The management pack can be downloaded as an free trial from veeam’s website here –> http://www.veeam.com/system-center-management-pack-vmware-hyperv.html

Now as for the architecture of the functionality here it’s quite simple

image

First of there are two components.

* Veeam Virtualization Extesions (Service and UI) it manages connections to VMware systems and the Veeam Collector(s), controling licensing, load balancing, and high availability

* Veeam Collector component gathers data from VMware and injects its information into the Ops Agent.

It is possible to install all of these components on the management server itself. You can also install the collector service on other servers which have the Opsmgr agent installed. The virtualization extension service must be installed on the management server.

In my case I wanted to install this on the mangement server itself, since I have a small enviroment. Before I started the installation I needed to make sure that the management server was operating in proxy mode.

8

Next I started the installation on the management server. Now as with all of Veeams setup it can automatically configure all prerequisites and is pretty straight forward. (Note it will automatically import all required management packs into SCOM1

If you have a large enviroment it is recommended to split ut collectors into different hosts and create a resource pool (There is an online calculator which can help you find out how many collectors you need) http://www.veeam.com/support/mp_deployment.html

You can also define if collector roles should be automatically deployed

2

After the installation is complete (using the default ports) you will find the extensions shortcut on the desktop

4

By default this opens a website on the localhost (using port 4430) from here we need to enter the connection information to Vmware (Hyper-V hosts are discovered automatically when they have the agent installed) Same with Veeam Backup servers as well.

5

After you have entered the connection info you will also get a header saying the recommended number of collector hosts.

7

After this is finished setup you can open the OpsMgr console. From here there is one final task that is needed. Which is to Configure the Health Service, this can be dome from tasks under _All_active_Alerts under VMware monitoring pane.

image

After this is done you need to expect atleast 15 min before data is populated into your OpsMgr servers, depending on the load. You can also view the events logs on the Opsmgr servers to see that data is correctly imported.

image

and after a while, voila!

I can for instance view info about storage usage

image

 

Vm information

image

Now I could show grafs and statistics all day but one of the cool stuff in this release, is the cloud capacity planning reports.

image

They allow it to see for instance how many virtual machines I would need in Azure (and what type) to move them there.

image

Veeam B&R 7 a list of issues and solutions

Now  I’ve been working with Veeam for a while now, and I’ve seen thatt mostly the case that when a backup job fails (or a surebackup job fails) or something fails, its most often not Veeam’s fault.

Veeam is a powerful product but it is dependant on alot of external features to function properly in order to do its job right. For instance in order to backup from a Vmware host, you need a vmware license in place in order to allow Veeam to access the Vmware VADP API’s.
If not Veeam can’t backup your virtual machines running on Vmware.

Also in order to do incremental backups properly Veeam is also dependant on CBT working properly on the hypervisor. So the real purpose of this blog post is mostly for my own part, but having a list of problems/errors that I come across in Veeam and what the fix is for it.

Now in most cases, when running jobs the job indicator will give a good pinpoint what the problem is. If not look into the Veeam logs which are located under C:\Programdata\Veeam\Logs (Programdata is a hidden folder) there is also possible to generate support logs directly from the Veeam console –> http://www.veeam.com/kb1832

Issue nr 1# Cannot use CBT when running backup jobs
Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: ‘<InvalidArgumentFault xmlns=»urn:internalvim25″ xsi:type=»InvalidArgument»><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>’, endpoint: »

If CBT is for some reason not available and it not being used, Veeam has its own filter which it uses in these cases. Veeam will then process the entire VM and then on its own compare the block of the VM and the backup and see which blocks have changed, and the copy only the changed blocks to the repository. This makes processing time alooooot longer. Now in order to fix this you need to reset CBT on the guest VM. This can be done by following the instructions here –> http://www.veeam.com/kb1113 and one for Hyper-V CBT http://www.veeam.com/kb1881

Issue nr 2# Sure backup jobs fail with error code 10061 when running applications tests. This is most likey when a firewall is configured on the guest VM which only allows specific VMs. I have also seen this when a guestVM is a restarting state. If you do not have a guestVM firewall active, doing a restart of the guestVM and then do a new backup should allow the surebackup job to run successfully.

Issue nr 3# WAN accelerator failes to install. This might happen if a previous Veeam install has failed on a server. When you try to install the WAN accelerator the setup just stops without no parent reason. Something makes the installpath of the WAN cache folder to the wrong drive. You need to go into the registry of the VM and change the default paths as seen here –> http://www.veeam.com/kb1828

Issue nr 4# Backup of GuestVMs running on a hyper-v server with Windows Server 2012 R2 update 1, this is a known issue from Microsoft which requires an update from Microsoft –> http://www.veeam.com/kb1863

Issue nr 5# Application-aware image processing skipped on Microsoft Hyper-V server, this is of course related to many possible features. In most cases it is integration services, a list of the different causes and solutions are listed here –> http://www.veeam.com/kb1855

Issue nr 6# Logs not getting truncated on Exchange/SQL guest VMs, this requires application aware image processing and define that the backup job should truncate logs –> http://www.veeam.com/kb1878

Issue nr 7# Backup of vCenter servers –> http://www.veeam.com/kb1051

Issue nr 8# Backup using Hyper-V and Dell Equallogic VSS –> http://www.veeam.com/kb1844

Issue nr 9# Incredible slow backup over the network and no load on the servers, make sure that all network switches are full-duplex.

Issue nr 10# Win32 error: the network path was not found. When doing application aware image processing veeam needs to access the VM using the admin share with the credentials that are defined in the backup job. (For Vmware if the VM does not have network access Vmware VIX is used) It is possible to change the priority of these protocols –> http://www.veeam.com/kb1230

VMCE study guide

Now for those working with Veeam, a hot topic these days is the VMCE (Veeam Certified Engineer) certification. In order to take this exam you first need to attend a 3-day technical course which covers the syllabus, then you are allowed to take the exam.

The exam consists of 50 random questions which are multiple choice, and you need 70% to pass the exam.

Now as an Veeam instructor I get questions regarding where can I find more info about the different subjects and a bit more regarding best practice regarding each subject ?

Therefore I created this study guide which consists of links to each module in the syllabus. First of you need to take the course, get yourself familiarized with the GUI and where options are stored in the GUI. Know the different components, where they can be placed and how traffic flows between the different components and look at some sample scenarios for instance which are listed in the evaluators guide.

Sample guides:
Support for Hypervisors:
Hyper-V: http://veeampdf.s3.amazonaws.com/datasheet/product-info-veeam-support-for-windows-server-2012-r2.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398891896&Signature=9TThHxabKQdTtUd7SGpFtqmhPdU%3D
VMware: http://veeampdf.s3.amazonaws.com/datasheet/product-info-veeam-support-for-vsphere-5-5.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398891896&Signature=HSvgsxXXvEk%2B%2BovZDrD%2FMat%2BvMU%3D

Best-practice for backup and replication deployment:
http://veeampdf.s3.amazonaws.com/guide/veeam_backup_7_0_deployment_vmware.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398891903&Signature=F5KVQ5urIkTL%2BFXbRAcp6T5kHgs%3D

Best-practice for HP storage and Veeam:
http://veeampdf.s3.amazonaws.com/guide/wp_veeam_hp_configuration_2.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398891915&Signature=Qm6cHw%2FqTRwZbLaqQB3pZzBlaMk%3D

Evaluators guide for VMware:
http://veeampdf.s3.amazonaws.com/guide/veeam_backup_evaluators_guide_7_vmware.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398891922&Signature=Qj0COAOPG5qq8O8r3HvVRbfLVFU%3D

Syllabus:

Backup Methods
http://helpcenter.veeam.com/backup/70/hyperv/index.html?backup_methods.html

Scheduling
http://helpcenter.veeam.com/backup/70/hyperv/scheduling.html
 
Changed Block Tracking (CBT)
http://helpcenter.veeam.com/backup/70/hyperv/changed_block_tracking.html

Compression and Deduplication
http://helpcenter.veeam.com/backup/70/hyperv/compression_deduplication.html

Retention Policy
http://helpcenter.veeam.com/backup/70/hyperv/retention_policy.html

Auto Discovery of Backup and Virtual Infrastructure
http://helpcenter.veeam.com/one/70/vsphere/configuring_veeam_one_monitor.html

Business Categorization
http://helpcenter.veeam.com/one/70/vsphere/assigning_categorization_value.html

Pre-Defined Alerting
http://helpcenter.veeam.com/one/70/vsphere/appendix_alarm_rules_events.html

http://helpcenter.veeam.com/one/70/vsphere/alarms.html

Agentless data gathering
http://helpcenter.veeam.com/one/70/vsphere/introducing_veeam_one_business_view.html?zoom_highlightsub=agentless

Hyper-V specific features
http://veeampdf.s3.amazonaws.com/guide/veeamone_7_0_deployment_guide.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398887835&Signature=CMfYdlrWg9qEN4kbcOJdWH%2Fidps%3D

Veeam One Deployment
http://veeampdf.s3.amazonaws.com/guide/veeamone_7_0_deployment_guide.pdf?AWSAccessKeyId=AKIAJI4MX44AEVG3NBLA&Expires=1398887835&Signature=CMfYdlrWg9qEN4kbcOJdWH%2Fidps%3D

http://helpcenter.veeam.com/backup/70/vsphere/install_vbr.html

Deployment Scenarios
http://helpcenter.veeam.com/backup/70/vsphere/deployment_scenarios.html
http://helpcenter.veeam.com/backup/70/vsphere/components.html

Prerequisites
http://helpcenter.veeam.com/backup/70/vsphere/planning.html
 
Upgrading Veeam Backup & Replication
http://helpcenter.veeam.com/backup/70/vsphere/upgrade_vbr.html

Adding Servers
http://helpcenter.veeam.com/backup/70/vsphere/setup_addserver.html

Adding a VMware Backup Proxy
http://helpcenter.veeam.com/backup/70/vsphere/add_vmware_proxy.html

Adding a Hyper-V Offhost Backup Proxy
http://helpcenter.veeam.com/backup/70/hyperv/add_hyperv_proxy.html

Adding Backup Repositories
http://helpcenter.veeam.com/backup/70/hyperv/setup_addrepo.html

Performing Configuration Backup and Restore
http://helpcenter.veeam.com/backup/70/hyperv/export_vbr_config.html
http://helpcenter.veeam.com/backup/70/hyperv/restore_vbr.html

Creating Backup Jobs
http://helpcenter.veeam.com/backup/70/hyperv/backup_job.html
http://helpcenter.veeam.com/backup/70/hyperv/options_parallel_processing.html

Creating VM Copy Jobs
http://helpcenter.veeam.com/backup/70/vsphere/index.html?vm_copy.html

Instant VM Recovery
http://helpcenter.veeam.com/backup/70/hyperv/performing_instant_recovery.html

Insight into Replication
http://helpcenter.veeam.com/backup/70/hyperv/index.html?intro.html

Insight into Failover
http://helpcenter.veeam.com/backup/70/hyperv/performing_failover.html

Insight into Failback
http://helpcenter.veeam.com/backup/70/hyperv/performing_failback.html

SureBackup Recovery Verification
http://helpcenter.veeam.com/backup/70/vsphere/recovery_verification.html

SureReplica
http://helpcenter.veeam.com/backup/70/vsphere/recovery_verification_surereplica.html

Restoring Microsoft Exchange and SharePoint objects
http://helpcenter.veeam.com/backup/70/vsphere/vex.html
http://helpcenter.veeam.com/backup/70/vsphere/working_with_vesp.html

Working with Veeam Backup & Replication Utilities
http://helpcenter.veeam.com/backup/70/hyperv/extract_utility_console_restore.html

3-2-1 rule
http://www.veeam.com/blog/how-to-follow-the-3-2-1-backup-rule-with-veeam-backup-replication.html

Working with Tape Media
http://helpcenter.veeam.com/backup/70/vsphere/working_with_tape_media.html

Wan Accelerator
http://helpcenter.veeam.com/backup/70/vsphere/wan_add.html
http://helpcenter.veeam.com/backup/70/vsphere/wan_acceleration.html

Offsite Backup Copy Job
http://helpcenter.veeam.com/backup/70/hyperv/offhost_proxy_advanced.html
http://helpcenter.veeam.com/backup/70/hyperv/backup_copy_job.html

Delegate file and VM restores with Veeam Backup Enterprise Manager
http://helpcenter.veeam.com/backup/70/em/performing_1-click_file_restore.html
http://helpcenter.veeam.com/backup/70/em/1click_vm_restore.html

Veeam Backup Enterprise Manager RESTful API
http://helpcenter.veeam.com/backup/70/em/used_ports.html

HP StoreVirtual VSA
http://helpcenter.veeam.com/backup/70/vsphere/hp_san_support.html
http://helpcenter.veeam.com/backup/70/vsphere/hp_san.html

Product Editions Comparison
http://helpcenter.veeam.com/backup/70/vsphere/editions.html

My first sponsor – Veeam

Many might have noticed that little banner on the right side of my blog, well of right now I have my first official sponsor and of course it had to be veeam Smilefjes
I have had some offers before but Veeam is something I work with everyday and enjoy working with.

Veeam is an interesting product and I will when I have more time focus on more posts with Veeam in the middle.