So yesterday I held a session at Citrix User Group in norway regarding Netscaler and performance tuning, not so much I can really say about performance tuning in 45 minutes but I think I managed alright.
The agenda on my list was
* TCP profiles, Multipath TCP, Path MTU
* SSL profiles and tuning
* Autonegotitation and duplex
* Netscaler VPX
* Jumbo frames and LACP
* Last but not least mobilestream
Now most of this is core Netscaler optimization features, expect Mobilestream which is more related to features standing behind Netscaler. So therefore I wanted to write a blogpost about it as well.
Firstly is the TCP profiles. By default there is an TCP profile which hasen’t changed since 1999. So the Netscaler profile is by default there for compability and not for the best performance, but of course there are alot of different factors invovled here. For instance what kind of network infrastructure you have, packet loss, bandwidth, jitter, firewalls and so on.
But, the main thing is that the default profile does not:
Have Window Scaling activated (Window scaling is usefull send more packets inse the scaling window meaning that we can easier send more data)
Have Selective Acknoledgement activated (Means that we don’t need to resend all the data after a packet loss. Meaning that if we sendt packets 1, 2, 3, 4 , 5 and the sender didn’t receive packet 3 we don’t need to resend 4, 5)
Have Nagle alogrithm activated (Gathers up more data and waits until it reaches the full MTU and then sends the data)
So for instance the ICA-protocol which is very chatty and uses small packets (Which uses alot of overhead) means that it is not suiteable for the regular TCP-profile, so this is where the tcp profile
nstcp_xa_xd_profile (Which has all the features I mentioned above enabled in the policy) but of course you also have the mobile users who are jumping back and forth between different WLAN points or mobile antennas which means there is a point with total packet loss. In the default TCP profile it uses TCP reno, which tries to cut the congestion window in half when it detected a packet loss, not going to do the mobile users any good
Therefore Citrix impletented a variant of the TCP congestion features called Westwood+ which tries to determine the current bandwidth with the device and then it cuts the congestion window to reflect the current bandwidth. Which means that the mobile users can faster get to higher speeds again.
Now also with 10.5 ( I belive) is the option to enable MTCP (Multipatch TCP) so meaning that if you have mobile devices which support two atennas (one for mobile data and one for WIFI which can be used at the same time) we can have two TCP connections from the same device used to access content on the netscaler, its just a policy setting and we are good to go.
The problem is that you need to have specific applications written to leverage MTCP (Not all are there yet)
So go into System –> Profiles –> TCP Profiles (you can either use an existing one or create a new one)
Check for Window Scaling
And here for MTCP (If you need it) SACK and for Nagle.
Now there is also an downfall for Nagle since it waits until it waits until a full MTU has been reached before it sends it across the wire and the mobile user has a lot of packet loss, in theory there might be alot of data that needs to be resent across the wire. So for SQL instances for instance, don’t use Nagle!
and the cool part is that these policies can be applied on each vServer and of course services, so dependant on the services it is hosting you can create a differnet policy.
The other thing is SSL tuning, there is a few tips here as well. First thing is quantum size. Bu default the quantum size is 8 KB meaning that the Netscaler will get 8 KB of data that is going to be sent across the wire and the sent it to the SSL chips for encrypting. We can also chance this quantum size to 16 KB meaning that more data is allowed inside the encrypted package.
So for solutions exposing for instnace downloading of large files, a 16 KB quantum size is to prefer. Regular websites which has alot of small data I recommend sticking to the 8 KB.
And then there is of course the autonegititation and duplex, which is something that everybody expects to work fine these days, but…
I still see some having issues with this and specific network devices, so you should always try to manually set the speed and duplex on the netscaler and the switch/router/firewall it is connected to.
For the VPX alot of tuning tips are the same as the MPX but….
For instnace the VPX has support for multiple packet engines meaning that you have a specific engine inside the Netscaler which runs all the different policies, handles encryption and so on. So for a regular VPX it is by default setup with 2 vCPU (One CPU for mangement and another for the packet engine) So if you have an VPX 3000 (2 vCPU and 2 GB ram might not be enough) so if you are using XenServer og Vmware you have the option to add more CPU and RAM to gain additional packet engines. (NOTE: Hyper-v does not support this feature and is capped at 2 vCPU and 2 GB ram and 2vNIC DON’T add 3 vNic)
But of course if you are running Hyper-V and Netscaler VPX make sure you have the newest drivers and make sure that VMQ (Virtual Machine Queing)
VMQ means that a VM has a dedicated Queue on the physical network card if VMQ is not working the VM has to use the default queue along with all the other VMs, with alot of Broadcom drivers that VMQ does not work.
And there is also LACP (NIC teaming, Port Channel, 802.3ad) which allows for aggreating and failover/redundacy on physical NICs (Note that this requires configuration on the switche/s and the Netscaler and it only works on the MPX and the SDX.
There is also a new feature which came with 10.5 is the suppor for Jumbo frames, this allows us to send up to 9000 MTU in an ethernet frame (the default 1500 MTU) which allows for much less overhead since there is more data in a single frame that requires less ACKs)
This only works on MPX/SDX as well, since a VPX is reliant on what the hypervisor provides.
This can be configured on per interface. But note that this requires support for jumbo frames on the switch / server, but note that this does not work out over the WAN since it stops at the router or the ISP (This they mostly support the default MTU)
But note the Netscaler also has the Path MTU feature (Which allows) to Netscaler to see the path ahead and see what the lowest minimum MTU is. This feature uses ICMP to determine what the lowest MTU is on a next-hop device. Problem is that since it uses ICMP the next hop devices might be firewalls and such and therefore it might not work. This feature is used to avoid IP fragmentation on the network.
That’s it for now, stay tuned for more Netsacler
Microsoft just updated its support matrix for Lync 2013 (Finally) Where Netscaler is listed as supported for Reverse Proxy and for Load balancing –> http://technet.microsoft.com/en-us/office/dn788945
You can also read the deployment guide for Netscaler and Lync here –> http://www.citrix.com/content/dam/citrix/en_us/documents/partner-documents/microsoft-lync-2013-citrix-netscaler-deployment-guide.pdf
Earlier today, Google published a article regaring how hackers can exploit a vulnerability in the SSL 3.0 protocol. Which you can read more about here –> http://googleonlinesecurity.blogspot.no/2014/10/this-poodle-bites-exploiting-ssl-30.html
You can also read more about the specific attack in detail here –> https://www.openssl.org/~bodo/ssl-poodle.pdf
Microsoft recommends that you disable SSL 3.0 using Group Policy on Windows Computer, since it is by default enabled, you can read more about it here –> https://technet.microsoft.com/en-us/library/security/3009008.aspx
UPDATE::: Citrix has added a article on this exploit as well –> http://support.citrix.com/article/CTX200238
AND NOTE THAT IN THE SCREENSHOT DENY SSL RENEGOTIATION IS SET TO NO, THIS SHOULD BE PUT TO YES TO PROTECT AGAINST BEAST ATTACK.
Citrix Netscaler we can be fore flexible. For Netscaler Gateway we can define which type of SSL profiles or protocols which are going to be enabled for the session. We can create a new front-end SSL profile which we can attach to the Netscaler Gateway. Front end policies are used when a client is connecting to a vServer
Here I define that TLSv1 is enabled, and that the client cannot use SSLv3. (This is a screenshot from a VPX) and therefore TLSv1.1 and 1.2 cannot be enabled for this profile, and by default Citrix Receiver only supports TLS1 not the newer versions.
After I created the protocol I can bind it to a Gateway vServer
Now If I have other load balanced vServer I can also disable SSL for these vServers, but it is important to check if the clients that are connecting actually support TLS.
NOTE: I have not verified that this works for most browsers but I verified that my client can connect to the gateway vServer using TLS and not SSL3.
Today I presented on the Netscaler masterclass on the subject, System Center and Netscaler and here is my presentation –> https://www.slideshare.net/secret/uSy62iG3eeoaFY
My talk consisted about using the different integrations between System Center and Netscaler, primarly on
* Virtual Machine Manager and Netscaler (Using the load balancer extention to deploy load balancing rules for service templates)
* Operations Manager and Netscaler (How to setup monitoring for Netscaler and use it together with Distributed Applications)
* Orchestrator and Netscaler (How to setup automation tasks against Netsacler using the NITRO SDK)
And as promised in the presentation here is my scripts that I use for the different tasks.
Add-Server activity (Note that this requires that the SDK is added to C:\SDK folder and that the different DLL files are added to the global assembly cache.
[System.Reflection.Assembly]::Load(«System.EnterpriseServices, Version=188.8.131.52, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a»)
$publish = New-Object System.EnterpriseServices.Internal.Publish
(ADD THE DLL files to the global assembly for Orcehstrator to use for reference)
$path1 = Resolve-Path «C:\sdk\lib\Newtonsoft.Json.dll»
$path = Resolve-Path «C:\sdk\lib\nitro.dll»
$user = «»
$pass = «»
$nsip = «»
(NOTE THAT THE CODE ABOVE NEEDS TO BE ADDED TO EACH ACTIVITY)
$nitrosession = new-object com.citrix.netscaler.nitro.service.nitro_service($nsip,”http”)
$session = $nitrosession.login($user,$pass)
$server1 = New-Object com.citrix.netscaler.nitro.resource.config.basic.server
$server1.name = «»
$server1.ipaddress = «»
$service1 = New-Object com.citrix.netscaler.nitro.resource.config.basic.service
$service1.name = «»
$service1.servicetype = «»
$service1.monitor_name_svc = «»
Create Load balanced Service
$nitrosession = new-object com.citrix.netscaler.nitro.service.nitro_service($nsip,”http”)
$session = $nitrosession.login($user,$pass)
$lbvserver1 = New-Object com.citrix.netscaler.nitro.resource.config.lb.lbvserver
$lb_to_service = New-object com.citrix.netscaler.nitro.resource.config.lb.lbvserver_service_binding
$lb_to_service.name = «»
$lb_to_service.servicename = «»
With the recent announcement of the ShellShock vulnerability many vendors have done a great job with coming with patching / fixes to close the vulnerability. Citrix has released an knowledge article which shows what Citrix products are affected here –> http://support.citrix.com/article/CTX200217
But! Citrix has also released an update to AppFirewall signature to include fixes to services which are exposed via Netscaler. For instance if we have an load balanced service which is load balanced via Netscaler, and the services running in the back are affected or vulnerable we can use AppFirewall to protect them from the attack.
Then we can see that the new signature files include fixes for shellshock.
The actions are by default set to block. So when creating an appfirewall policy we can bind this to an particular vServer or URL.
Important to set signature action to block
But note that these rules only apply to services that are exposed via the Netscaler, and not the netscaler itself. Refer to the document which is posted above.
Had a case earlier today where a customer wanted to configure Netscaler to authenticate with UPN instead of SamAccountName. And using UPN instead of SamAccountName makes sense in many cases, since it easier for users to remember their email-address instead of their username. So in this scenario my samAccoutName is msandbu and my UPN is firstname.lastname@example.org
Now by default Netscaler is setup with samAccoutName under server logon name attribute. This defines what kind of account name you are allowed to logon with using Netscaler.
If you try to logon with UPN when SamAccountName is defined you will get this kind of error message on the StoreFront Server.
So Storefront strips the domain info sent from the Netscaler and tries to validate the credentials to Active Directory.
So how to fix this ?
You have to define the SSO name attribute in the LDAP credential, to samAccountName.
Then the Netscaler firstly validates the UPN, get the SamAccountName of the user and then forwards that to Storefront and logs in.
Important to remember that Storefront always tried to revalidate the info from Netscaler
On the next Netscaler Masteclass in October I will be presenting a session, regarding System Center and Netscaler. To talk about different forms of integration and monitoring.
For those who aren’t familiar with the Masterclass it is a webinar series that is hosted by Citrix, which are hosted once a month.
So sign up here if you want to know more –> http://www.citrix.com/events/netscaler-master-class.html
Now since the release of 10.5 I have been able to test alot of the new features in the latest release. Citrix has also released new versions of Insight and Endpoint clients for Windows & Mac to match the new release.
The upgrades have so far for my part have been non-problematic (in case of a custom GUI you may need to recreate it) from 9.3 and even 10.1 builds. For those that are in a migration plan please refer to the migration document from Citrix http://support.citrix.com/proddocs/topic/ns-faq-map-10-5/ns-faq-migration.html
I have also seen a performance increase in some scenarioes.
There has also been an update on the clustering features, which didn’t caught my eye at first. http://support.citrix.com/proddocs/topic/ns-system-10-map/ns-cluster-feat-supp-ref.html Which allows us to have a Netscaler Gateway vServer running on a local Netscaler node.
Now the new build is 99% pure HTML which is great! there are still some features which still requires JRE, but this is going to be fixed in a future release.
The following features or nodes still require JRE:
- Upgrade Wizard
- User Administration
- Command Policies
- Command Policy RegEx Editor
- Network > Network Visualizer
- Network > TCP/IP connections
- Traffic Management > Load Balancing > Visualizer
- Traffic Management > Content Switching > Visualizer
- Traffic Management > GSLB > Visualizer
- Application Firewall
- Application Firewall wizard
- Add/ Edit/ Import profiles
- Update Version
- Auto Update Settings
- Application Firewall
Citrix has also made easier integrations for their own products such as XenDesktop/XenMobile/Sharefile and so on, which makes it easier for consultants to deploy Netscaler solution to provide availability for other products.
Now all of the new features are listed here –> http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-5-map/netscaler-10-5-rn.html
One thing which I find is the most important featue in the latest build (besides the new GUI) is the front-end optimization feature which allows the Netscaler to reduce load and render times on web pages which are rendered on a client browser, after some intials tests with this feature I was able to save 60% of the load time. Since in most cases a web site is not optimized for speed, and therefore Netscaler might be an important piece there.
But to sum it up so far, I’m really impressed with the latest release and how Citrix has made Netscaler even more powerful with more then 100 more features, and makes it a more key component in most datacenters. Looking forward to the later releases to see what Citrix has up their sleeve!
A couple of days ago I was involved in a case where ICA sessions were suddenly disconnected and the users were unable to reconnect. The setup was a simple ICA-proxy access gateway using the latest build (126) and there were no error messages on the Storefront server.
After involving Citrix support they recommended that we disable AppFlow for the access gateway (since this deployment used Netscaler Insight to monitor ICA sessions) then suddenly things started to work again.
Now I knew that I’ve seen this issue before somewhere on twitter, a quick tweet discovered that someone else has seen the issue as well.
So apparently using Appflow with session reliability is a NO-GO!
If someone has managed to test this with 10.5 please give me some feedback if this has been fixed!
So there has been some fuzz regarding the latest release of Netscaler 10.5 (also codename Tagma) which should been the death of Java GUI within Netscaler. Not quite there yet..
So what has Citrix improved / added in this feature ? Well it is quite a lot. Citrix states that they have added over 100 new features in this release. Beta 1 has just been released to partners, and beta 2 is on its way which is coming mid may.
In the later betaes which are coming there are coming more templates to App and load balancing. But let us focus on the news that’s arrived now.
- HTML5 based GUI
- NITRO SDK for Python
- NITRO for File Operations
- NITRO for ZebOS system
- GSLB Static proximity sync
- SSL configuration Profiles
- CNAME record caching
- Multiple Port CS
- AAA Session Stickiness
- Kerberos Performance
- Jumbo Frames
- Link Redundancy
- TCP BIC and CUBIC
- SPDYv3 Gateway
- SDX Manageability
- Front End Optimization
- Insight Center Enhancements
First of is the GUI which is now mostly pure HTML 5 which makes it quite snappy! I would say that about 80% of the GUI is now HTML 5, some features such as running trace still uses Java (Im guessing this is something that is going to get fixed in a later release.
So what is new under licensing part ? We can see that there are some new features which appear here, such as Integrated Disk caching and RISE (Which is part of the Cisco platform)
There is also two new “features” within Traffic optimization
* Front End Optimization (Which converts data which is being sent back to the user, such as convert image files)
And we have content accelerator (Which is used for integration with Citrix ByteMobile)
Also setting up a new Netscaler Gateway is also alot easier since we don’t need the Java part anymore.
Also support for LLDP is here, which is a information exchange protocol kinda like CDP (from Cisco) So here is a comparison between the old GUI and the new GUI
There is also a list of new monitors which are built-in
Also support for LACP on interfaces which allows you to team NICs.
Citrix has also added some basic wizards which allow for easier setup against XenDesktop / Sharefile and such.
Also SSL profiles and DTLS profiles
We also have support for Jumbo Frames which allows for up to 9000 bytes of payload instead of 1500.
And one thing that is missing is Edgesight monitoring from Netscaler which looks like it has been removed for good. One thing which I didn’t find in the beta but is mentioned in the video is support for Oracle (which most likely coming in a later beta) o this is just my findings in the latest Beta. ill update when the next beta is coming! Looks like we have much to look forward to!