The case of the HTTP traffic not working NetScaler

So today I was asked to help troubleshoot an issue where an customer had setup a new pair of NetScaler MPXs which was setup using LACP and different VLANs, after the initial setup and started with a basic load balancing setup for Storefront 3.1, stuff were not working.

The service was working as it should, meaning that the SNIP traffic was working to the backend server. Opening a HTTP connection to the service didn’t do anything.. Even the networking tools in IE and Chrome didn’t see anything. I tried to do a ping but that was working fine

Then my first idea was WTF? Why isn’t HTTP traffic working but ICMP was working…

So like any good IT-guy we started at the bottom of the chain (Physical and data link layer)

Was the networkin working as it should?
VLAN configured properly? Check
LACP configured properly? Check
Routing properly configured? (Mac based forwarding didn’t work either)
After looking over the configuration, we noticed that outgoing traffic was going to one MAC address and then responding from another MAC address, which might pinpoint the issue, but that was a false positive since it was just HSRP protocol doing it thing…

Now since PING was working we just wanted to verify that all the other parts of the network was working as it should, and of course we didn’t see any firewall issues as well. Now comes the interesting part, we setup WireShark on a RDS server to see if the HTTP traffic was actually going back and forth to the NetScaler.

And it was, we saw the HTTP traffic going back and forth as it should, so looks like the network was working as it should.

Problem was now that the HTTP traffic packets were coming back to the client, but was NOT appearing in the browser and this seemed to happen all clients that tried to connect, and again

So what is happening now?? Then I get the information that they have an HTTP proxy solution in place. Which of course could be the culprit but this was not the case….

Now last night I actually saw a blog from Citrix here –>


AHA bingo! one quick look at the event log on the Storefront server we saw this.


So then we logged into the SSL parameter settings of the Service Group and saw this, after we disabled TLSv12, it worked!


So the solution was actually in the first lines of the blogpost… MPX and Storefront 3.1. Funny thing is that we actually had another similar case with SDX and VPXs where a consultant has upgraded their VPX appliances frmo 10.5.55 to 10.5.59, what happend is that the VPXs started to communicate using TLS 1.2 backend with older servers, which then caused the same problems. After upgrading the VPXs to the latest 11.64 version they were able to DISABLE TLS 1.2 backend for the SSL parameters and things started to work again.


Monitoring Syslog from OMS with non-oms agents

So this weekend I was tasked with trying to setup OMS syslog monitoring against Linux targets which was not supported as part of the OMS agents. Now the supported list of OMS Linux agents are the following:

Amazon Linux 2012.09 –> 2015.09 (x86/x64)
CentOS Linux 5,6, and 7 (x86/x64)
Oracle Linux 5,6, and 7 (x86/x64)
Red Hat Enterprise Linux Server 5,6 and 7 (x86/x64)
Debian GNU/Linux 6, 7, and 8 (x86/x64)
Ubuntu 12.04 LTS, 14.04 LTS, 15.04, 15.10 (x86/x64)
SUSE Linux Enteprise Server 11 and 12 (x86/x64)

Now since many have network devices which run non of these operating systems I needed to setup something which would allow me to forward the Syslog events from other devices and then forward it to OMS. So what I came up with was setting up a Syslog collector on a supported OMS agent operating system. So I setup a Ubuntu 14.04 virtual machine which was going to be used as a syslog collector


The simplest way was to use the built-in service rsyslog on ubuntu to configure it for remote collection, by default it is only used for local syslogging it does not accept remote syslogs.

Now as mentioned this requires a simple machine running Ubuntu 14.04 or 15.04. From the terminal we need to configure rsyslog.conf which is located under /etc folder

From there you need to change file, which can be done using VIM or VI. In the Conf file you need to remove # in front of the ModLoad and UDPServerRun which will allow the syslog daemon to gather from remote sessions.

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

Next you need to add this line before the GLOBAL DIRECTIVES part of the confing file.

$template RemoteLogs,»/var/log/%HOSTNAME%/%PROGRAMNAME%.log» *
*.*  ?RemoteLogs

This is used for the syslog daemon to create syslog files under /var/log where all the log files will be named after the remote host that forwards information.

After this is configured you need to restart the rsyslog feature,

sudo /etc/init.d/rsyslog restart


Now we should see that the syslog folder will be populated under the folder of the host name.

After this is done you need to install the OMS agent using the following commands

$> wget

$> chmod +x ./

$> md5sum ./


After the OMS agent is configured. Then we need to configure the syslog collector from within OMS


Then we can go into Log Search, we can go into the Syslog viewer and drill into the different alerts.


So in this case I just configured regular Syslog setup from a Cisco ASA and a Citrix NetScaler to forward to the Ubuntu server.

#netscaler, #oms, #syslog

NetScaler and traffic flow explained

So after reading trough alot of forum posts and blogposts on the subject and also getting alot of feedback on my previous ebook I decided to write a blogpost on the subject. Let us consider the following we have a NetScaler two-arm mode where we have an service located in DMZ and backend-server on another subnet.

So some important things to think about:

  • VIP is NOT an packet generating IP by default, it has the load balancing logic (Methods, Persistency and so on)
  • Internally on the NetScaler when traffic hits a VIP it will look up the service group(services) servers to see if the NetScaler is located on L2 directly to the backend servers, if not it will look at the routing table to see if it can talk to that network.
  • When adding a SNIP to the NetScaler, it will create a DIRECT ROUTE to that particular layer 2 network. ( In case of SNIP (It will add a DIRECT ROUTE where the GW is set to The SNIP is used for the route lookup capability, which the NetScaler is commonly used for when returing traffic.
  • Most of the monitors which are attached to a service are using the SNIP as Source IP


So when a client accesses a VIP all traffic will be directed to the VIP, where the destimation MAC  will be directed to Interface 1. Remember that on a NetScaler a IP address is not directly bound to a Interface, unless specifically configured.

The NetScaler has an interal table which looks at the servers  that are attached and will then using the closest IP from SNIP to talk with the backend server.

Now the problem with the example above is that it will not work with the default settings. Because since a VIP cannot generate outgoing packets on its own, the traffic flow will stop.

It is important that we have an L2 IP address on the same L2 network that we have the VIP. This is because SNIP is a Packet generating IP. So if we look at the example above again


This time we have a SNIP where the VIP is located. Meaning that we have a SNIP for each L2 network that we have. So in this the traffic flow will work like so.

Client –> VIP –> NetScaler –> SNIP (Closest L2 IP) –> Server, and when the NetScaler now responds back to the client

Server –> SNIP –> NetScaler (Session Table) –> VIP (Check if SNIP is present) –> Client

So even if we setup a SNIP on the same network where the VIP is located, this is never used as a source when going back to the client, it is just needed to establish a packet generating feature.

Mac based forwarding

Another option that we have, which many use which is called MAC Based forwarding.

Q. What is MBF?

A: MBF alters the way the NetScaler appliance routes the server replies back to clients. MBF caches the MAC address of the uplink router that forwarded the client request to the appliance. When a reply is received, it is passed through to the same router that sent the client request without going through any route lookup. If MBF is disabled, then the return path is determined by a route lookup, or is sent to the default route if no specific route exists.

Which is essentially turning of the routing lookup feature. IF we use this feature, then we do NOT require a SNIP where the VIP is located because then the packet generating feature is located on the interface itself. But still features like static routes will still work because when the NetScaler initiates a connection, it uses the route and ARP tables for the lookup function

Different backend subnets and monitors

What about the SNIP’s if we want to communicate with server 2 which is located on another network? Since we already have a SNIP in the default backend subnet. We can just add a new static route

Network to access:          netmask – 255         the gateway that I need to talk to with to reach that network                                   

This presumes that the Router 1 ( has knowledge of the second router on the network) another option that we have if we do not want to have that SNIP traversing between the different network is to add another SNIP directly to that subnet, with using for instance VLAN or direct attaching the interface.

What if I have multiple SNIPs connected to the same subnet?

The NetScaler would use the SNIPs in a round-robin fashion to communicate with the resources. If you want to dedicate a SNIP to a particular service you should create a net profile which allows us to restrict traffc to a particular subnet from a SNIP.

We can also specify a netprofile to a VIP, which allows us to seperate SNIP traffic from customer 1 to customer 2 for instance. For instance if we have two net profiles containing SNIP 1 and SNIP 2, and we bind those to a VIP 1 and VIP 2, we can distigunish traffic coming from which source to the same backend server

Client —> VIP 1 (netprofile) –> SNIP 1 –> Server1

Client –> VIP 2 (netprofile) –> SNIP 2 –> Server1

  • If there is no net profile on the virtual server or the service/service group, NetScaler uses the default method.
  • If there is a net profile only on the service/service group, NetScaler uses that net profile.
  • If there is a net profile only on the virtual server, NetScaler uses the net profile.
  • If there is a net profile both on the virtual server and service/service group, NetScaler uses the net profile bound to the service/service group.

We can also bind a particular SNIP to a monitor which in essence will only use a particular SNIP for monitoring purposes, which allows us to even more granulary filter out which SNIPs are used for monitoring and which SNIPs are used for direct backend communication. 

  • If there is a net profile bound to the monitor, NetScaler uses the net profile of the monitor. It ignores the net profiles bound to the virtual server or service/service group.
  • If there is no net profile bound to the monitor,
    • If there is a net profile on the service/service group, NetScaler uses the net profile of the service/service group.
    • If there is no net profile even on the service/service group, NetScaler uses the default method of selecting a source IP.

We can also specify a Net Profile to

called policy based routing, which allows us  to specify which type of traffic that should be routed and so on.

So for instance we want to route web traffic using the gateway (Since we have an application firewall) but regular monitoring can be going using the direct connection in that case we can specify a PBR (Policy Based Routing) to send HTTP traffic to a particular network to be processed via a Gateway ( Firewall) and therefore eliminating unecessery traffic going via the firewall

NOTE: PBR are evaluated before regular routes, so if there is traffic that is not evaluated via the PBR routes, the regular routing table will  be processed.

#afcvbou, #netscaler

NetScaler and PowerShell cmdlets

Now this is something I have been planning a post on for some time, ever since I started working with the C# library to do NITRO API calls against NetScaler. I was planning and started on a a PowerShell module for NetScaler, but still someone beat me to the race, so no reason to reinvent the wheel anymore Smilefjes

Someone at Citrix (Santiago Cardenas)  has already created an REST API based PowerShell module, which is placed on GitHub here –>

Now the scripts which contains many of the basic features, but ill give you an recipe which will allow you to create your own extensions to the scripts. Now using REST API, we have built-it documentation which is available on the NetScaler here –> Under the download page of the NetScaler


From the download you have an index.html file which will show you the different tasks

There are two main categories, Configuraiton and statistics, from there I can drill down into a specific feature. So for instance let us look at gateway vServer (Which is located under SSL VPN) which is also the same as Gateway vServer

So if we want to setup a Gateway vServer what do we need to specify ? If we from there choose vpnserver which is the base object
We get all the attributes that can be configured from the vpnvserver object.

name, servicetype, ipv46, range, port

Now its a long list, but if you scroll down the documentation page you can see a specific example if you for instance wish to add a vServer (The objects in red are the ones that ARE required)


Now using a REST API we need to use a POST command which will push the settings we specify using PowerShell. The github PowerShell cmdlets have already taken care of this, so the commands are built up llike this.

function GatewayvServer ($GatewayFQDN, $VIP) {
EnableFeatures SSLVPN

$body = @{
$body = ConvertTo-JSON $body
Invoke-RestMethod -uri «» -body $body -WebSession $NSSession `
-Headers @{«Content-Type»=»application/»} -Method POST

A funciton is the name we use when starting it from PowerShell and the variables are the ones that we can specify behind the cmdlet. Now all the specific attributes are part of a variable called $body, which then added to the HTTP Body. The is the direct name of the NetScaler.

Now what if we want to create a function that gets information about a particular vServer? We can see from the documentation that there is a “get” example


So an example Powershell function would look like this,

function Get-GatewayvServer {
# Login to NetScaler and save session to global variable
$gateway = Read-host -Prompt «Type VIP name»
$body = ConvertTo-JSON @{
Invoke-RestMethod -uri «” -WebSession $NSSession -Method GET
$Script:NSSession = $local:NSSession

As we can see from the URI (All we need is to specify the hostname of the NetScaler and that particular VPN vServer using the GET HTTP Method. So if you are unsure of the URI you can just open up a browser and connect to that particular URI


So the PowerShell cmdlets from Santiago Cardenas can be used as a starting point, adding your own PowerShell functions is pretty easy when you just look at what attributes and URI that are being used. So start scripting!

#netscaler, #powershell

NetScaler and basic functions, status of vServers and ICMP ARP operations

Sometimes when setting up a new NetScaler and migrating virtual servers from an old one to another, it is quite often that one might forget to disable or shutdown older vServers. Now NetScaler has features to disable different network settings, so this post I want to explain what each option does.

In a layer 2 network, the ARP protocol (In IPV4 network) is reponsible for mapping IP to MAC addresses. So if we have an vServer and we ping it from a host on the same subnet, it will run an ARP request to get the MAC address of that IP


So if we have an vServer running on that IP with port 80 and that is enabled. If that host is not in the ARP table, what will happen when we open up a network connection (Internet browser to that IP on that port)

  • ARP will run a broadcast
  • Get the MAC of that IP
  • Initiate a TCP connection to port 80 using HTTP

Next time we open up a connection, that MAC address will most likely be in the MAC table of the host and will no longer require an ARP request.

So let us say that we want to setup a new NetScaler to replace the old one, and we want to setup a new NetScaler using the same IP address. So I’m guessing that we can just disable the vServer right?


Wrong, what will happen is that the IP will still be in use and respond to ARP but the service running on port 80 will not be accessable.

Here we can see that NetScaler one and two responds at the same time, even thou the service is disabled.


So what if we disable ARP on the VIP on the older NetScaler ?


Yay! now only one NetScaler will respond (IF the ARP cache is cleaned up, on Windows is takes 2 minutes before the dynamic ARP table is cleared out) if you want to disable an old vServer (Disable the vServer first TCP service, then disable ARP and ICMP as well) which will not allow it to communicate at all)

Or what I recommend is that you define the response parameters of the VIP.


When we define these to ONE_VSERVER they will only respond to ARP and ICMP if one vserver which is attached to the VIP is in state up. If we then would disable a vServer for maintance or something, then ARP and ICMP would automatically be disabled on the VIP, which makes alot more sense when doing maintance, because if services are reponding to ICMP but not on the service itself, people tend to star troubleshooting pretty fast.


Free eBook on Optimizing Citrix NetScaler and services

So alas, it is here!

This is something I have been working on for some time now, and my intention is that this is just the beginning  of something bigger.. (Hopefully)

For a couple of years now I have been writing for Packt Publishing and authored some books on NetScaler which has been a fun and a good learning experience. The problem with that is… These projects take alot of time! and the problem these days is that the releases are becoming more and more frequent and same goes for other underlying infrastructure which makes it cumbersome to have up-to date content available.

This is the first step in an attempt to create a full (free) NetScaler eBook, for the moment in time I decided to focus on Optimzing NetScaler traffic features. Hopefully other people will tag along as well, since there are so many bright minds in this community!

So what’s included in this initial release.
CPU Sizing
Memory Sizing
NIC Teaming and LACP
VLAN tagging
Jumbo Frames
NetScaler deployment in Azure
NetScaler Packet flow
TCP Profiles
VPX SSL limitations
SSL Profiles
Front-end optimization
Tuning for ICA traffic

Also I would like to thank my reviewers which actually did the job of reading through it and giving me good feedback! (and of course correcting my grammar as well) a special thanks to Carl Stalhood ( a Citrix CTP who contributed with alot of content to this eBook as well.

Also to my other reviewers as well!

Carl Behrent

Dave Brett  (

How do I get it?
By signing up using your email in the contact form below, and ill send you an PDF copy after the book is finished editing sometime during the weekend, wanted to get this blogpost out before the weekend to see the interest.

The reason why I want to have an email address is that it makes it easier for me so send update after a new major version is available. Also I want some statistics to see how many are actually using it to see if I should continue on with this project or not. The email addresses I get will not be used to anything else, so no newsletters or selling info to the mafia…

Feedback and how to contribute?
Any feedback/corrections/suggestions please send them to my email address also if you want to contribute to this eBook please mail me! since I’m not an expert my all means, so any good ideas should be included so it can be shared with others.

#citrix, #front-end-optimization, #http2, #netscaler

Netscaler and AAA with CSW One VIP

As part of the latest release from Citrix Netscaler V11, there was an interesting feature added to the firmware. Which in essence allows ut to add a NO-IP Virtual AAA server, which allow us to add multiple resources lets say behind a CSW vServer where we only use one VIP.

Highlander there can be only one - There can be only one VIP

This is part of the latest feature release from Citrix (build 11. 63 from October) which has this feature.
It can either be setup using CLI or using the GUI.

User-added image

So when setting up the AAA vServer we can then use the option non-adressable


Note that when biding it to the CS vServer you need to specify that it needs to use 401-based authentication, since forms based requires an HTTP session externally to function


So from an enduser perspective a users tried to go to LB1, which resides on the CSW vServer, which will then trigger an AAA request to the non-adressable 401 based authentication and then the user will be authenticated.

#aaa, #netscaler, #vip

Testing SSL from Netscaler–Issues with SSL handshake

From time to time we need to setup load balancing to a SSL based service or when setting up connection to a secure Storefront (which is the default) there is one thing that alot of people are missing from the config when setting up, which results in wierd issues or getting SSL handshake errors from the monitors. In most cases it because of two things

  • Missing Root CA
  • Wrong Ciphers or not supported ciphers

So how can we verify from the Netscaler that it is missing the rootCA or that we have the right CA in place?

That is when we uses OpenSSL, which is a toolkit that is used on the Netscaler, which also has a commandline interface which allow us to test different parameters.

So if we enter Shell on the Netscaler and then do a CD to /nsconfig/ssl (This is where all the NS certificates are stored by default and from there we can use OpenSSL.

By using the command

openssl s_client –connect FQDN

First of this will show us, the certificate that is presented, and the certificate chain. It will also list out what kind of connection that is being used towards to FQDN (In this case below we are using TLS 1.2 against a Storefront server.

depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
   i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
   i:/C=US/O=DigiCert Inc/ Global Root CA

Server certificate
issuer=/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA

No client certificate CA names sent

SSL handshake has read 3034 bytes and written 479 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: EB46000017E1621AA1BB5491BDFD3EDB2C273F35E73DB2029651C5B00DEC62BC
    Master-Key: 65CA41A8B811869F0C005469E20578BB3C876AB7207AB5D2D42370B7779FD1EB                                 7F971DC3A0001EF9B54963D1D2B080BD
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1448336973
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)

What we can see here is that we have an error message at the tops stating that

 verify error:num=20:unable to get local issuer certificate

This error occurs if

  • The certificate chain for the certificate wasn’t provided by the other side or it doesn’t have one (it is self-signed).
  • The root certificate is not in the local database of trusted root certificates.
  • The local database of trusted root certificates was not given and thus not queried by OpenSSL.

In order to verify against a chain of certificates with a RootCA or Intermidiate with both, we can use the parameter –CAfile or –CAPath which we can specify behind the command to test a connection with a RootCA.

Now there are a bunch of different parameters that we can use with OpenSSL, for instance we can also test openSSL using different protocols such as -ssl3, -tls1, -no_ssl3, -no_tls1, -no_tls1_1, -no_tls1_2

Which allow us to test using SSL3 for instance. You can see the list of different options on the openSSL site here –>


Load balancing port ranges with Netscaler

So I got a question earlier today, if it was possible to load balance a specific port range within Netscaler. Now by default we cannot specify a port range when setting up a load balanced vServer or setting up services.

In order to ensure that traffic from a specific endpoint going to the same backend service we have some options.

1: Persistency Groups, in this case we need to define load balanced vServers and services for each port nr and then we need to define a persistency group afterwards. Which I have blogged about earlier

2: Another option we have is to define the port nr to * and service to ANY, which means that we can have a single vServer and single service objects, the issue is that it will intercept ANY traffic to ANY port, which is a bad thing.

What we can do with option 2 is define a listening policy. Listen policy will allow us to customize which ports a vServer should respond to. For instnace we can define a vServer with ANY service and port equals * but then we attach a listen policy and define for instance port range between 80 to 8080. Even thou the vServer is setup with ANY port and ANY service it will only listen to requests coming from within the port range in the listen policy.

Listen policies can be defined within the vServer, such as shown in the screenshot below


#listen-policy, #netscaler, #port-range

New Netscaler books available!

Just a small post about what I have been busy with lately Smilefjes For those who have been following me on Twitter/LinkedIn/Blog notice that I from time to time blog about Netscaler which kinda has become my little baby.
Anyhow… 2 years back I started working on my second book for Packt Publishing called Netscaler VPX, which was the first technical book (outside of Citrix Education) on Netscaler available on Amazon.

Now two years later, I have done a bit more.

Implementing Netscaler VPX Second Edition:

Which is a upgraded/polished version of my first book, which was created upon version 10, this book is based upon V11 and contains more content around security, troubleshooting, azure/amazon deployments and front-end optimization.
And my latest project Mastering Netscaler VPX

Mastering Netscaler VPX:

This is a book which I co-wrote with Rick Roetenberg (Note: He did most of the work) Which goes a bit more deep into the material. I did some chapters on Network optimization, troubleshooting, Content switching, GSLB, Datastream and security features.

So if you are unsure what to get for christmas, this might be a good idea Smilefjes

#books, #netscaler