Netscaler and real performance tuning

So yesterday I held a session at Citrix User Group in norway regarding Netscaler and performance tuning, not so much I can really say about performance tuning in 45 minutes but I think I managed alright.

The agenda on my list was

* TCP profiles, Multipath TCP, Path MTU
* SSL profiles and tuning
* Autonegotitation and duplex
* Netscaler VPX
* Jumbo frames and LACP
* Last but not least mobilestream

Now most of this is core Netscaler optimization features, expect Mobilestream which is more related to features standing behind Netscaler. So therefore I wanted to write a blogpost about it as well.

Firstly is the TCP profiles. By default there is an TCP profile which hasen’t changed since 1999. So the Netscaler profile is by default there for compability and not for the best performance, but of course there are alot of different factors invovled here. For instance what kind of network infrastructure you have, packet loss, bandwidth, jitter, firewalls and so on.

But, the main thing is that the default profile does not:

Have Window Scaling activated (Window scaling is usefull send more packets inse the scaling window meaning that we can easier send more data)

Have Selective Acknoledgement activated (Means that we don’t need to resend all the data after a packet loss. Meaning that if we sendt packets 1, 2, 3, 4 , 5 and the sender didn’t receive packet 3 we don’t need to resend 4, 5)

Have Nagle alogrithm activated (Gathers up more data and waits until it reaches the full MTU and then sends the data)

So for instance the ICA-protocol which is very chatty and uses small packets (Which uses alot of overhead) means that it is not suiteable for the regular TCP-profile, so this is where the tcp profile

nstcp_xa_xd_profile (Which has all the features I mentioned above enabled in the policy) but of course you also have the mobile users who are jumping back and forth between different WLAN points or mobile antennas which means there is a point with total packet loss. In the default TCP profile it uses TCP reno, which tries to cut the congestion window in half when it detected a packet loss, not going to do the mobile users any good Smilefjes

Therefore Citrix impletented a variant of the TCP congestion features called Westwood+ which tries to determine the current bandwidth with the device and then it cuts the congestion window to reflect the current bandwidth. Which means that the mobile users can faster get to higher speeds again.

Now also with 10.5 ( I belive) is the option to enable MTCP (Multipatch TCP) so meaning that if you have mobile devices which support two atennas (one for mobile data and one for WIFI which can be used at the same time) we can have two TCP connections from the same device used to access content on the netscaler, its just a policy setting and we are good to go.

The problem is that you need to have specific applications written to leverage MTCP (Not all are there yet)

So go into System –> Profiles –> TCP Profiles (you can either use an existing one or create a new one)

image

Check for Window Scaling

image

And here for MTCP (If you need it) SACK and for Nagle.
Now there is also an downfall for Nagle since it waits until it waits until a full MTU has been reached before it sends it across the wire and the mobile user has a lot of packet loss, in theory there might be alot of data that needs to be resent across the wire. So for SQL instances for instance, don’t use Nagle! Smilefjes 

and the cool part is that these policies can be applied on each vServer and of course services, so dependant on the services it is hosting you can create a differnet policy.

The other thing is SSL tuning, there is a few tips here as well. First thing is quantum size. Bu default the quantum size is 8 KB meaning that the Netscaler will get 8 KB of data that is going to be sent across the wire and the sent it to the SSL chips for encrypting. We can also chance this quantum size to 16 KB meaning that more data is allowed inside the encrypted package.

image

So for solutions exposing for instnace downloading of large files, a 16 KB quantum size is to prefer. Regular websites which has alot of small data I recommend sticking to the 8 KB.

And then there is of course the autonegititation and duplex, which is something that everybody expects to work fine these days, but…

I still see some having issues with this and specific network devices, so you should always try to manually set the speed and duplex on the netscaler and the switch/router/firewall it is connected to.

For the VPX alot of tuning tips are the same as the MPX but….

For instnace the VPX has support for multiple packet engines meaning that you have a specific engine inside the Netscaler which runs all the different policies, handles encryption and so on. So for a regular VPX it is by default setup with 2 vCPU (One CPU for mangement and another for the packet engine) So if you have an VPX 3000 (2 vCPU and 2 GB ram might not be enough) so if you are using XenServer og Vmware you have the option to add more CPU and RAM to gain additional packet engines. (NOTE: Hyper-v does not support this feature and is capped at 2 vCPU and 2 GB ram and 2vNIC DON’T add 3 vNic)

But of course if you are running Hyper-V and Netscaler VPX make sure you have the newest drivers and make sure that VMQ (Virtual Machine Queing)

VMQ means that a VM has a dedicated Queue on the physical network card if VMQ is not working the VM has to use the default queue along with all the other VMs, with alot of Broadcom drivers that VMQ does not work.

And there is also LACP (NIC teaming, Port Channel, 802.3ad) which allows for aggreating and failover/redundacy on physical NICs (Note that this requires configuration on the switche/s and the Netscaler and it only works on the MPX and the SDX.

There is also a new feature which came with 10.5 is the suppor for Jumbo frames, this allows us to send up to 9000 MTU in an ethernet frame (the default 1500 MTU) which allows for much less overhead since there is more data in a single frame that requires less ACKs)

image

This only works on MPX/SDX as well, since a VPX is reliant on what the hypervisor provides.
This can be configured on per interface. But note that this requires support for jumbo frames on the switch / server, but note that this does not work out over the WAN since it stops at the router or the ISP (This they mostly support the default MTU)

But note the Netscaler also has the Path MTU feature (Which allows) to Netscaler to see the path ahead and see what the lowest minimum MTU is. This feature uses ICMP to determine what the lowest MTU is on a next-hop device. Problem is that since it uses ICMP the next hop devices might be firewalls and such and therefore it might not work. This feature is used to avoid IP fragmentation on the network.

That’s it for now, stay tuned for more Netsacler Smilefjes

Multiple vNet site-to-site configuration in Microsoft Azure

In many cases you would need to establish a site-2-site VPN connection between different subscribtions in Microsoft Azure, now this is a pretty simple process in Azure and can be easily done using the management portal.

Example: We have 2 vNETs configured in Microft Azure within the same region (Note that this does not consume bandwith cost, only gateway hours)

vNets:

vNEt 1 (Test1) IP adsress subnet space 10.0.0.0/24 and with a Gateway address of 23.100.60.100

vNet 2 (Test 2) IP address subnet space 10.10.0.0/24 and with a Gateway address of 23.100.70.100

In order to setup a Site-to-site VPN connection I just need to define both of these as local networks as well to each other.

Local Networks:

Local vNet 1(Test1) IP address subnet space 10.0.0.0/24 and with a Gateway address of 23.100.60.100

Local vNet 2(Test2) IP address subnet space 10.10.0.0/24 and with a Gateway addres of 23.100.70.100

So in the management portal I can just define them as local networks to each other

vNet 2 –> Local vNet 1

vNet 1 –> Local vNet 2

and from there just add a same shared key and allow them to connect.

What if we want a third vNet to integrate with one of the other vNets using a Site-to-Site VPN? Is it possible ? Sure it is. With Microosft Azure it is possible to create up to 10 different VPN tunnels, problem is that the management portal only allows for one VPN tunnel at the time for one vNet. So we need to use PowerShell and a custom network xml file in order to finish the configuration here.

We need to create a new virtual entwork called vNet 3 (Test 3) IP address subnet space 10.20.0.0/24 and with a Gateway address of 100.100.20.100 (This also has to be created as a local network site as well in order to bind it up to another vNet.

In this examples we will bind vNet 3 to vNet 1, which already has an VPN tunnel activated for vNet 2.

image

First we need to download the vNet configuration XML, which can be done using the command

get-azurevnetconfig –exporttofile c:\folder\name.xml

Open it up and locate the virtualnetwork site for vNet1

<VirtualNetworkSites>
    <VirtualNetworkSite name=»test» Location=»North Europe»>
      <AddressSpace>
        <AddressPrefix>10.0.0.0/24</AddressPrefix>
      </AddressSpace>
      <Subnets>
        <Subnet name=»Subnet-1″>
          <AddressPrefix>10.0.0.0/27</AddressPrefix>
        </Subnet>
        <Subnet name=»GatewaySubnet»>
          <AddressPrefix>10.0.0.32/29</AddressPrefix>
        </Subnet>
      </Subnets>
      <DnsServersRef>
        <DnsServerRef name=»10.0.0.100″ />
      </DnsServersRef>
      <Gateway>
        <ConnectionsToLocalNetwork>
          <LocalNetworkSiteRef name=»test2″>
            <Connection type=»IPsec» />
          </LocalNetworkSiteRef>
          <LocalNetworkSiteRef name=»test3″>
            <Connection type=»IPsec» />
          </LocalNetworkSiteRef>

        </ConnectionsToLocalNetwork>
      </Gateway>

Here is where we need to define our local network we which this vNet to connect to. For vNet 3 which does not have any VPN connection set up we can do this via the managmenet portal. or add a

          <LocalNetworkSiteRef name=»test1″>
            <Connection type=»IPsec» />
          </LocalNetworkSiteRef>

In the vnet xml file. After we are done adding the connection path to vNet we need to import the XML file to our azure subscribtion.

This can be done using the set-azurevnetconfig –configurationpath c:\folder\file.xml

After this is done we need to change the sharedkey so that the vNets have the same key.

Set-AzureVnetGatewayKey –VnetName test1 –Localnetworksitename test3 –SharedKey 12345QWERT

Set-AzureVnetGatewayKey –VnetName test3 –Localnetworksitename test1 –SharedKey 12345QWERT

After this is done the connections should be established. Note that if they don’t you need to go into the management portal, into vNet 3 and choose connect.

Then you can go into vNet 1 and see the connection is setup against two vNets.

1

Citrix Netscaler supported for Lync 2013

QUICK NOTE:

Microsoft just updated its support matrix for Lync 2013 (Finally) Where Netscaler is listed as supported for Reverse Proxy and for Load balancing –> http://technet.microsoft.com/en-us/office/dn788945

You can also read the deployment guide for Netscaler and Lync here –> http://www.citrix.com/content/dam/citrix/en_us/documents/partner-documents/microsoft-lync-2013-citrix-netscaler-deployment-guide.pdf

Upcoming speaking events

Alot is happening these days, and I am not standing still. So therefore I wanted to list my speaking events in the upcoming future.

Citrix User Group in Norway is having a boat-trip in the end of October –> http://cugtech.no/?page_id=766 (If you are working with Citrix this is an excellent oppurtunity to learn more)

Here I have two sessions (In norwegian)

  • Netscaler and Performance tuning
  • Netscaler and security features

Also I was confirmed as an speaker at next years NIC (Nordic Infrastructure Conference) http://www.nicconf.com/ haven’t gotten any confirmation regardiing what sessions yet, but it’s either Azure or 3D graphics!

  • Azure AD
  • Azure RemoteApp
  • 3d graphics

Citrix Netscaler and SSL3 “poodle” exploit

Earlier today, Google published a article regaring how hackers can exploit a vulnerability in the SSL 3.0 protocol. Which you can read more about here –> http://googleonlinesecurity.blogspot.no/2014/10/this-poodle-bites-exploiting-ssl-30.html

You can also read more about the specific attack in detail here –> https://www.openssl.org/~bodo/ssl-poodle.pdf

Microsoft recommends that you disable SSL 3.0 using Group Policy on Windows Computer, since it is by default enabled, you can read more about it here –> https://technet.microsoft.com/en-us/library/security/3009008.aspx

UPDATE::: Citrix has added a article on this exploit as well –> http://support.citrix.com/article/CTX200238

AND NOTE THAT IN THE SCREENSHOT DENY SSL RENEGOTIATION IS SET TO NO, THIS SHOULD BE PUT TO YES TO PROTECT AGAINST BEAST ATTACK.

Citrix Netscaler we can be fore flexible. For Netscaler Gateway we can define which type of SSL profiles or protocols which are going to be enabled for the session. We can create a new front-end SSL profile which we can attach to the Netscaler Gateway. Front end policies are used when a client is connecting to a vServer

image

Here I define that TLSv1 is enabled, and that the client cannot use SSLv3. (This is a screenshot from a VPX) and therefore TLSv1.1 and 1.2 cannot be enabled for this profile, and by default Citrix Receiver only supports TLS1 not the newer versions.

After I created the protocol I can bind it to a Gateway vServer

image

Now If I have other load balanced vServer I can also disable SSL for these vServers, but it is important to check if the clients that are connecting actually support TLS.

NOTE: I have not verified that this works for most browsers but I verified that my client can connect to the gateway vServer using TLS and not SSL3.

Netscaler Gateway and content switching

today is the day! Citrix annonced earlier today a new enhacement release for Netscaler Gateway which allows us to use Netscaler Gateway together with Content Switching.

This means that we can have a Gateway vServer together with content switching policy. So when we create a Netscaler gateway together with content switching we need to define content switching policies. For instance if we have the vServer gateway 10.0.0.1 and we have two content switching policies for the URLS /zm/ and /xm/ will point to a load balanced vServer. Others urls which are not being catched by a content switching policy will be redirected to the Gateway vServer.

So the content switching rules are checked first, before it goes on with session policies for the gateway vServer.

Now another thing that is cool with this release is that it supports SSO to RD solutions.

So this is the new screen when we create a new vServer.

image

We have the RDP info setup directly here. And we can also define CS policy bindings. So I can add a new content switching policy and add it to the vServer

image

And as I mentioned these rules will be evaluated before session policies.

But note that this is an enhacement build, and should/can be used for testing you can read more about the e versions here –> http://blogs.citrix.com/2013/03/29/citrix-access-gateway-demystifying-the-e-releases/

You can download the new build from citrix downloads here —> https://www.citrix.com/downloads/netscaler-adc/virtual-appliances/netscaler-vpx-release-105e.html

Software defined Storage? Dell’s got you covered

Earlier I’ve discucced a bit on Software defined storage and how this is an growing market with new vendors appearing all the time. Some of the concept behind SDS is the ability to move features that have previously only been available to the hardware solutions into the software stack. http://msandbu.wordpress.com/2014/05/20/software-defined-storage-and-delivering-performance/

Now as I mentioned there are a lot of different vendors here, some focus on delivering high performance, some on delivering adequate I/O on commodity hardware, some on flexibility, and many in between.

So what do we choose ? Since there are so many different vendors here it might be a hard choice to choose one over the other. The big question is what do I need ? Do I need to run big OLTP databases running on an average 200,000 IOPS, do I need to have a hyper-v cluster setup using commodity hardware in order to have a low cost on my Storage? Do I have existing Vmware infrastructure that I want to improve my IOPS on ? Looking to buy new hardware to have next generation VDI platform ? Do I have a bunch of different backend NAS / DAS and SAN I want to pool into a large unit of storage?

So the question is what do I have, what do I want and where do I need to go

And as the title mentiones, when you are looking for a new solution/ platform for software-defined storage, well then Dell’s got you covered.

Dell is one of the few hardware vendors who is certified for most of the different SDS solutions such as.

VSAN: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

Storage Spaces: http://www.windowsservercatalog.com/results.aspx?&chtext=&cstext=&csttext=&chbtext=&bCatID=1642&cpID=16445&avc=79&ava=0&avq=0&OR=1&PGS=25&ready=0

EVO: RAIL http://www.vmware.com/products/evorail

Dell also has a strategic partnership with Nutanix (which is going to be Dell hardware shipping with Nutanix Software) called the XC-series

http://www.dell.com/learn/us/en/uscorp1/press-releases/2014-06-24-dell-software-defined-storage-portfolio

Also Dell has partnership with both Nexenta and Atlantis

http://www.dell.com/learn/us/en/04/campaigns/dell-nexenta-storage

http://en.community.dell.com/techcenter/extras/m/mediagallery/20439148/download

Dell has also included a partnership with SanDisk in 13th generation servers which allows for simple SSD tiering on servers –> http://www.sandisk.com/about-sandisk/press-room/press-releases/2014/sandisk-das-cache-software-now-available-for-next-generation-dell-poweredge-servers/

So Dell has many different SDS options on their solutions, and also their SC-series, Equallogic, Compellent for running traditional workloads.

Veeam Endpoint backup free

Today at the VeeamON conference, they announced a new tool called Veeam Endpoint Backup free. This tool which will ship H1 next year http://www.veeam.com/blog/announcing-veeam-endpoint-backup-free.html. Allows us to take backup of physical servers, computers, laptops and such.

It can integrate with existing Veeam repositories or to a NAS share. The best part it of course that it is going to be free! Smilefjes

Stay tuned as the preview comes later in November, but this allows us to lastly do backup av Physical servers in a Veeam enviroment without the need to buy more licenses.

Netscaler masterclass presentation Oktober 2014

Today I presented on the Netscaler masterclass on the subject,  System Center and Netscaler and here is my presentation –> https://www.slideshare.net/secret/uSy62iG3eeoaFY

My talk consisted about using the different integrations between System Center and Netscaler, primarly on

* Virtual Machine Manager and Netscaler (Using the load balancer extention to deploy load balancing rules for service templates)
* Operations Manager and Netscaler (How to setup monitoring for Netscaler and use it together with Distributed Applications)
* Orchestrator and Netscaler (How to setup automation tasks against Netsacler using the NITRO SDK)

And as promised in the presentation here is my scripts that I use for the different tasks.

 

Add-Server activity (Note that this requires that the SDK is added to C:\SDK folder and that the different DLL files are added to the global assembly cache.

Set-location «c:\sdk»
[System.Reflection.Assembly]::Load(«System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a»)
$publish = New-Object System.EnterpriseServices.Internal.Publish
$publish.GacInstall(«C:\sdk\lib\Newtonsoft.Json.dll»)

(ADD THE DLL files to the global assembly for Orcehstrator to use for reference)

 

Add-Server

$path1 = Resolve-Path «C:\sdk\lib\Newtonsoft.Json.dll»
[System.Reflection.Assembly]::LoadFile($path1)
$path = Resolve-Path «C:\sdk\lib\nitro.dll»
[System.Reflection.Assembly]::LoadFile($path)

$user = «»
$pass = «»
$nsip = «»

(NOTE THAT THE CODE ABOVE NEEDS TO BE ADDED TO EACH ACTIVITY)

$nitrosession = new-object com.citrix.netscaler.nitro.service.nitro_service($nsip,”http”)
$session = $nitrosession.login($user,$pass)

$server1 = New-Object com.citrix.netscaler.nitro.resource.config.basic.server
$server1.name = «»
$server1.ipaddress = «»
$ret_value=[com.citrix.netscaler.nitro.resource.config.basic.server]::add($nitrosession,$server1)

Add-Service

$service1 = New-Object com.citrix.netscaler.nitro.resource.config.basic.service
$service1.name = «»
$service1.servicetype = «»
$service1.monitor_name_svc = «»
$service1.port= «»
$service1.servername= «»
$ret_value=[com.citrix.netscaler.nitro.resource.config.basic.service]::add($nitrosession,$service1)

Create Load balanced Service

$nitrosession = new-object com.citrix.netscaler.nitro.service.nitro_service($nsip,”http”)
$session = $nitrosession.login($user,$pass)

$lbvserver1 = New-Object com.citrix.netscaler.nitro.resource.config.lb.lbvserver
$lbvserver1.name=»»
$lbvserver1.servicetype=»»
$lbvserver1.port=»»
$lbvserver1.ipv46=»»
$lbvserver1.lbmethod=»»
$lbvserver1.servicename=»»
$ret_value=[com.citrix.netscaler.nitro.resource.config.lb.lbvserver]::add($nitrosession,$lbvserver1)

$lb_to_service = New-object com.citrix.netscaler.nitro.resource.config.lb.lbvserver_service_binding
$lb_to_service.name = «»
$lb_to_service.servicename = «»
$ret_value=[com.citrix.netscaler.nitro.resource.config.lb.lbvserver_service_binding]::add($nitrosession,$lb_to_vserver)

MVP another year for Enterprise Client Management

I received an email today, saying that I am MVP for another year. I am honored since this represents many of the elite it-pros all around the world

Also on the same day, Microsoft released vNext previews of Windows Server and System Center also Windows 10. Alot of documentation has been released, but remember its a preview (alpha or beta stage)

But it can be downloaded from MSDN for those who have access there, I will add another blogpost when I have more information about the different releases.

Følg

Få nye innlegg levert til din innboks.

Bli med 45 andre følgere