Citrix Netscaler and support for next generation web traffic protocols like SPDY & HTTP/2

Now with the ever growing pace of internet traffic, we are being faced with one challenge, an old protocol which is over 15 years old now and is now way any shape to continue in this race, and yes the one I am talking about is the HTTP protocol.

Now over the years, Google has done a great job trying to improve this way of communication with its own protocol called SPDY which uses prioritizing and multiplexing and with transmission headers are sent using GZIP or Deflate. You can read more about SPDY here –> https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3 

Now on the other hand you have the HTTP/2 protocol which the IEFT has worked one, which Google said will replace their own SPDY protocol (http://blog.chromium.org/2015/02/hello-http2-goodbye-spdy-http-is_9.html

You can read more about the RFC on HTTP/2 protocol here –> https://tools.ietf.org/html/rfc7540 but in essence its the same thing as SPDY, since the initial draft of HTTP/2 was based upon SPDY. Another thing that is important to note that communucation with HTTP2 is based upon a binary format since this is much easier to compress, while tradisional HTTP1.1 is upon human readable text. The people over at HTTP Watch did a comparision between tradisional HTTP, HTTP2 and SPDY https://blog.httpwatch.com/2015/01/16/a-simple-performance-comparison-of-https-spdy-and-http2/ and we can see that these new protocols works alot more efficient.

So what else is needed ? We need a web server that supports HTTP/2 or SPDY and we need web clients that support these protocols.

As we can see most web servers are already supported HTTP/2 https://en.wikipedia.org/wiki/HTTP/2#HTTP.2FHTTPS_servers Windows coming with in in Windows Server 2016 and the new version of IIS, and most web browsers support HTTP/2 as well, such as Chrome, Opera, Firefox, Internet Explorer and lastly Microsoft Edge

But for instance Firefox only supports HTTP/2 using TLS 1.2 https://wiki.mozilla.org/Networking/http2 meaning that even if the Netscaler can use HTTP2 over HTTP it will not work with most of the web browsers.

image

So how do I test that this stuff works ? the simplest thing is to download an addon to Chrome which is called HTTP/2 and SPDY indicator, which basically shows which sites are enabled for HTTP/2 and SPDY and so on.(This extension is available for FireFox as well)

So whenever we are on a site which has HTTP/2 enabled the icon will appear as such

image

We can also look at the interal table within Chrome by typing chrome://net-internals/#spdy in the address bar.

If this does not work on your chrome version you need to enable SPDY4/HTTP2 within Chrome which can be done using the chrome://flags/#enable-spdy4 flag.

In regards to setting this up on the Netscaler we have to create or alter a HTTP profile, and note this is only available from version 11 and upwards.

image

And choose enable under the checkbox for HTTP/2, if SPDY is also enabled the following preference is done when communicating with a vServer that has the HTTP profile bound

  • HTTP/2 (if enabled in the HTTP profile)
  • SPDY (if enabled in the HTTP profile)
  • HTTP/1.1

Now in most cases the backend servers are still using HTTP/1.1 In that case the Netscaler works as a proxy and decodes the traffic from the clients to HTTP 1.1 data and restrasmits the data to the backend servers.

It is however important to note that running HTTP/2 on VPX is not supported and hence the clients will fall back to SPDY which is supported on a VPX.

However there are some requirements that are worth noticing on VPX for SPDY as well:

Troubleshooting for SPDY

If SPDY sessions are not enabled even after performing the required steps, check the following conditions.

  • If the client is using a Chrome browser, SPDY might not work in some scenarios because Chrome sometimes does not initiate TLS handshake.
  • If there is a forward-proxy between the client and the NetScaler appliance, and the forward-proxy doesn’t support SPDY, SPDY sessions might not be enabled.
  • NetScaler does not support NPN over TLS 1.1/1.2. To use SPDY, the client should disable TLS1.1/1.2 in the browser.
  • Similarly, if the client wants to use SPDY, SSL2/3 must be disabled on the browser.

When to use Traffic Manager, Cloud Service Load balancing or Citrix Netscaler

Now that Citrix released their Netscaler appliance on Azure we have a huge option to do load balancing within the Azure platform. It is also important to think about the other options we have in Azure to do load balancing outside of Netscaler.

Traffic Manager is one of the first options which acts kinda like GSLB which is a DNS based load balancing feature. Which allow us to load balance between endpoints on a cloud service

1. Performance Load-Balancing

These services can be spread across different regions. This can either be load balanced based upon performance, round robin or failover.

Problem with DNS based load balancing is that is never gets a full overview of how the traffic is balanced since it basically just spreads the DNS responses. + at Traffic Manager has limited monitoring capabilities since it can only see on HTTP or HTTPS protocol.

We also now have support for nested profiles within Traffic Manager — http://azure.microsoft.com/blog/2014/10/29/new-azure-traffic-manager-nested-profiles/

2. Nested Load-Balancing, Performance   Weights

Now on the other side we have load balancing endpoints on Cloud Services.

image

When setting up Cloud Services Load balancing we have more option depending on load balancing distribution, ref https://msdn.microsoft.com/library/azure/dn495300

We can have persistency based upon sourceIP or destionationIP for instance, and that we have more monitoring endbpoint based options. This is a more L4 based load balancing approach, which is also a free option to in Azure.

While Netscaler is a complete L4 – L7 load balancing platform which can be used to load balanced based upon many different parameters. Now you can also combine the Netscaler appliance with a HA setup to get the best from both worlds. With also giving you a active/active Netscaler setup within a cloud service http://support.citrix.com/proddocs/topic/netscaler-vpx-10-5/vpx-azure-ha-config-con.html

So when do use the different services ?

If you have a simple web-service which does not require a advanced monitor capabilities and are setup on many different cloud services, use Traffic Manager

If you have a service which are setup within a cloud service which you need to setup a simple load balancing capability on while having low cost, use Load Balanced Endpoints

If you have a service which requires a more advanced service monitoring capabilities and special demand to distribute traffic use Netscaler within a Cloud Serivce.

Optimizing web content with Citrix Netscaler

This post, is based upon a session I had for a partner in Norway. How can we use Netscaler to optimize web content?

Let’s face it, the trends are chaging

* Users are becoming less patient (meaging that they demand that applications/services respond quicker. (more then 40% of users drop out if the website takes mroe then 5 – 10 seconds to load) think about how that can affect a WebShop or eCommerce site ?

* More and more mobile traffic (Mobile phones, ipads, laptops. Communicating using 3G/4G or WLAN for that matter) and to that we can add that there is more data being sent across the network as well. Site web applications become more and more complex, with more code and more components as well.

* More demands to availability (Users are demaing that services are available at almost every hour. If we think about it about 5 – 10 years ago, if something was down for about 10 min we didn’t think that much about it, but now ?

* More demands to have secure communication. It wasn’t that long ago that Facebook and Google switched to SSL as default when using their services. With more and more hacking attempts happening online it requires a certain amount of security.

So what can Netscaler do in this equation ?

* Optimizing content with Front-end optimization, Caching and Compression

With the latest 10.5 release, Citrix has made a good jump into web content optimization. With features like lazy loading of images, HTML comment removal, minify JS and inline CSS.  And adding it that after content is being optimized the content can be compressed using gZIP or DEFLATE and sent across the wire (NOTE: that most web servers like Apache and IIS support GZIP and Deflate but it is much more efficient to do this on a dedicated ADC)

And with using Caching to store often accessed data it makes the Netscaler a good web optimization platform.

* Optimizing based upon endpoints.

With the current trend and more users connecting using mobile devices which are using the internett with a wireless conenction. If needs a better way to communicate as well. A god example here is TCP congeston. On a wireless you have a higher amount of packet loss and this requires using for instance TCP Congestion Westwood which is much better suites on wireless connections. Also using features like MTCP (on supported devices) allows for higher troughput as well. And that we can place different TCP settings on different services makes it much more agile.

* High availability

Using features like load balancing and GSLB allows us to deliver a high availabilty and scale solution. And using features like AppQOE to allows us to prioritize traffic in a eCommerce setting might be a valuable asset. Think the scenario if we have a web shop, where most of our buying customers come from a regular PC while most mobile users that are connecting are mostly checking the latest offers. If we ever where to reach our peak in traffic it is useful to prioritize traffic based upon endpoint connecting.

* Secure content

With Netscaler it allows us to create specific SSL profile which we can attack to different services. For instance older applications which are used by everyone might not have the high requirement regarding security, but on the other hand PCI-DSS requires a high level of security. Add to the mix that we can handle many common DDoS attacks on TCP level and on HTTP. We can also use Application firewall which handles many application based attacks, when an own learning feature it can block users which are not following the common user pattern on a website. And that we can specify common URLs which users are not allowed to access.

So to summerize, the Netscaler can be a good component to optimizing and securing traffic, with a lot of exiting stuff happening in the next year! Smilefjes stay tuned.

Netscaler and Office365 SAML iDP setup

With Netscaler 10.5, Citrix announced the support for SAML Identity Provider on the Netscaler feature. That basically meant that we could in theory use the Netscaler as an identity provider for Office365 / Azure AD. Now I have been trying to reverse engineering the setup since Citrix hasen’t created any documentation regarding the setup.

But now! Citrix recently announced the setup of Netscaler iDP setup for Office365 setup http://support.citrix.com/article/CTX200818

Yay!

on another part Citrix also released a new build of Netscaler VPX (build 56.12) which fixes the CPU utilization bug on Vmware you can see more about the release note here –> http://support.citrix.com/article/CTX200818

And there is also a new PCI DSS report which shows compliance for version 3.

Setting up a secure XenApp enviroment– Netscaler

Now I had the pleasure of talking PCI-DSS compliant XenApp enviroment for a customer. Now after working with it for the last couple of days there are lot of usefull information that I thought I would share.

Now PCI-DSS compliance is needed for any merchant who accepts credit cards for instance an e-commerce size. Or using some sort of application. So this includes all sorts of

* Different procedures for data shredding and logging

* Access control

* Logging and authorization

Now the current PCI-DSS standard is in version 3 –> https://www.pcisecuritystandards.org/documents/PCI_DSS_v3.pdf

The different requirements and assesment procedures can be found in this document. Now Citrix has also created a document for how to setup a compliant XenApp enviroment https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/pci-dss-success-achieving-compliance-and-increasing-web-application-availability.pdf you can also find some more information here –> http://www.citrix.com/about/legal/security-compliance/security-standards.html

Now instead of making this post a pure PCI-DSS post I decided to do a more “howto secure yout XenApp enviroment” and what kind of options we have and where a weakness might be.

Now a typical enviroment might looks like this.

image

So let’s start by exploring the first part of the Citrix infrastructure which is the Netscaler, in a typical enviroment it might be located in the DMZ. Where the front-end firewall has statefull packet inspection to see what traffic goes back and forth. The best way to do a secure setup of Netscaler is one-armed mode and use routing to backend resources and then have another Firewall in between to do deep packet inspection.

First thing we need to do with Netscaler when setting up Netscaler Gateway for instance is to disable SSL 3.0 and default (We should have MPX do to TLS 1.1 and TLS 1.2 but with VPX we are limited to TLS 1.0

Also important to remember th use TRUSTED third party certificates from known vendors, without any known history. Try to avoid SHA-1 based certificates, Citrix now supports SHA256.

Important to setup secure access to management only (since it by default uses http)

image

This can be done by using SSL profiles which can be attached to the Netscaler Gateway

image

Also define NONSECURE SSL renegotiation. Also we need to define some TCP parameters. Firstly make sure that TCP SYN Cookie is enabled, this allows for protection against SYN flood attacks and that SYN Spoof Protection is enabled to protect against spoofed SYN packets.

image

Under HTTP profiles make sure that the Netscaler drops invalid HTTP requests

image

Make sure that ICA proxy migration is enabled, this makes sure that there is only 1 session at a time established for a user via the Netscaler

image

Double hop can also be an option if we have multiple DMZ sones or a private and internal zone.

Specify a max login attempts and a timeout value, to make sure that your services aren’t being hammered by a dictonary attack

image

Change the password for the nsuser!!!

image

Use an encrypted NTP source which allows for timestamping when logging. (Running at version 4 and above) and also verify that the timezones are running correctly.

image

Sett up a SNMP monitoring based solution or Command Center to get monitoring information from Netscaler, or use a Syslog as well to get more detailed information. Note that you should use SNMP v3 which gives both Authentication and encryption.

Use LDAPS based authetication against the local active directory server, since LDAP is pure-text based, and use TLS not SSL, and make sure that the Netscaler verifies the server certificate on the LDAP server

image

It also helps to setup two-factor authentication to provide better protection against user thefts. Make sure that if you are using a two factor authentication vendor that it uses CHAP authentication protocol instead of PAP. Since CHAP is much more secure authentication protocol then PAP

Use NetProfiles to control traffic flow from a particular SNIP to backend resources (This allows for easier management when setting up firewall rules for Access.

image

Enable ARP spoof validation, so we don’t have any forging ARP requests where the Netscaler is placed (DMZ Zone)

image

Use a DNSSEC based DNS server, this allows for signed and validated responses. This way you cannot its difficult to hijack a DNS or do MITM on DNS queries.  Note that this requires that you add a nameserver with both TCP and UDP enabled. (Netscaler can function as both a DNSSEC enabled authoritative DNS server and proxy mode for DNSSEC)

If you wish to use Netscaler as an VPN access towards the first DMZ zone, first things you need to do is

1: Update the SWOT library

image

Create a preauthetnication policy to check for updated antivirus software

image

Same goes for Patch updates

image

In most cases try to use the latest firmware, Citrix does release a new Netscaler firmware atleast one every three months which contains bug fixes and security patches as well.

Do not activate enhanced authentication feedback, this enabled hackers to learn more about lockout policies and or if the user is non existant or locked out, disabled and so on.

image

Set up STA communication using HTTPS (Which requires a valid certificate and that Netscaler trusts the root CA) You also need to setup Storefront using a valid certificate from a trusted Root CA. This should not be a internal PKI root CA since third party vendors have a much higher form a physical security.

If you for some reason cannot use SSL/TLS based communication with backend resources you can use MACSec which is a layer 2 feature which allows for encrypted traffic between nodes on ethernet.

Troubleshooting DNS and LDAP connections Netscaler

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

image

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

image

If you make a trace file you can also see that it works as it should.

image

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

image

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

image

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

Troubleshooting DNS and LDAP connections Netscaler

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

image

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

image

If you make a trace file you can also see that it works as it should.

image

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

image

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

image

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.