Citrix NetScaler–TCP profiles

After having a huge number of questions regarding this topic of the last couple of weeks, I decided to write a blogpost about it, to clarify some of the misconceptions about this feature on NetScaler.

NOTE: TCP profiles can be found under System –> Profiles –> TCP

TCP profiles is a feature which allows us to customize TCP parameters on a NetScaler which we then can bind to a specific object. TCP profiles can be bound either globally, to a virtual server or to service (service groups). Important to note that TCP profiles can be bound to for instance at a global level, this will affect all TCP communication on the NetScaler, but we can for instance customize a TCP profile which we can bind to a virtual server, which will then override the TCP profile on the global level for that partciular virtual server.

Same goes for services, if we have a TCP profile bound globally, if we create a custom TCP profile which we then bind to a service, then it will override the global TCP settings that are defined.

So why should we customize TCP settings for different objects?

image

Our end-users access resources differently, for instnace on one hand we might have users using Citrix Receiver which is dependant on having a good experience wherever they are and on many different devices. On the other hand we might have mobile users working from their phones accessing resources using an app, and in most cases working wirelessly and roaming between 3G/4G and WiFi where it also often roams between access points, where you also have an high amount of packet loss.

Now in another of the puzzle are the internal resources that the NetScaler needs to talk to which are often connected to an high-speed ethernet 1GB/10GB connection, with no to little packet loss.

Think about it if you were to talk with a friend that sits right next to you, which is like internal traffic. No latency, little retransmission. On the other hand try talking to someone riding a bycicle far away, you would need to maybe repeat alot of word or sentences to that person and also you might tneed to speak slower as well to adjust and make sure that the other person receives what you are saying.

So TCP should also act differently depending on where the user is, and how their connection is. The default TCP profile on the NetScaler has not be adjusted for a long time, so it tries to communicate in the same way with internal resources and with external resources on the virtual server level, but of course it is there to ensure compability.

Another thing to remember is that there are many TCP settings that if enabled might impact the TCP performance badly as well. So when configuring TCP settings, if you are customizing on your own be sure that you test and validate TCP performance.
Now for most of us, it is alot simpler. Citrix NetScaler has pre created TCP profiles for different use cases.
Some of it, is use of features like SACK and DSACK, Nagle, MTCP and so on. Another important factor is the use of congestion algoritms and when to choose what.
This chart can be used as a guideline on which congestion algoritm to choose.

User-added image
 
source: http://support.citrix.com/article/CTX211877
Now as an important factor
⦁    NetScaler Gateway does not have the concept of Services, hence a TCP profile can only be bound to the Virtual Server. All other internal traffic will be using the default TCP profile.
⦁    Virtual Servers like Content Switching, Load balancing and so on, can have its own TCP profile attached to it. For instance if we have a virtual server that is used for serving mobile users content I would consider looking into using another congestion algoritm, and use of MTCP is the devices/application supports it
⦁    All services and service groups which communicate with internal resources can also have their own TCP profile, which is most cases nstcp_default_tcp_lan can be used for internal communication.

So hopefully you got a better understanding of TCP profiles Smilefjes

The future of VMware NSX–Transformers and licensing update

A couple of days ago, VMware announced the new release of NSX-MH (Multi-hypervisor) also known as Transformers – Codename Bumbelbee. Which supports now KVM and vSphere as hypervisor choices and being able to share the same transport zone. This picture displays how it looks like.

Also you can now have the Edge installed on a bare-metal, which then allows us to create our own hardware vTep.

Now also VMware introduced licensing models on NSX-V (which is still for VMware ESXi only) into three different editions.

  • Standard Edition: Automates IT workflows, bringing agility to the data center network and reducing network operating costs and complexity.
  • Advanced Edition: Standard Edition plus a fundamentally more secure data center with micro-segmentation. Helps secure the data center to the highest levels, while automating IT provisioning of security.
  • Enterprise Edition: Advanced Edition plus networking and security across multiple domains. Enables the data center network to extend across multiple sites and connect to high-throughput physical workloads.-
  • Also some interesting about the licensing: All three editions are available per socket on a perpetual basis. The advanced edition is available as a per-user offering (to align with virtual desktop deployments). The Enterprise edition is also available on a per-VM term basis.

    Existing customers will be moved into Enterprise edition when they move to the new licensing model. You can see the differences between the editions here –> http://www.vmware.com/products/nsx/compare.html?mid=1824&eid=CVMW2000000369654

    And more detailed information about the editions and feature matrix here –> https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145269

    Also according to: http://vcloud.sx/new-nsx-license-tier-thoughts-transformers/

    Where previously the list (USD) for NSX was $5,996 per Socket the new editions come in at $1,995, $4,995 and $6,995 per Socket. The Standard edition is well priced but taking a look through the Matrix in the official KB you are getting an extremely slimmed down version of NSX…short of the bells and whistles that make it the awesome SDN platform that it is, however I’m sure the feature set will be attractive for some.

    So one of the main reasons that people get NSX is the firewall parts, which is only part of the Advanced licensing which makes sense for VDI as well. Standard gives distributed switching and routing, which is also something that people should take a closer look at!

    NetScaler Security Insight

    With NetScaler build 11.65, Citrix introduces Security Insight as part of the Insight Appliance. This feature is used to give real-time insight into attacks, and recommendations on countermeasures.

    NOTE: Security Insight requires Application Firewall enabled, since it reports based upon AppFw violations and rules. You also need the same version of Insight and the NetScaler appliance for this to be supported and for it to work.

    To setup Security Insight there are basically three steps.

    1: Setup NetScaler Insight build 11.65 and NetScaler appliance with the same build number, then add the NetScaler appliance to Insight

    2: Alter the AppFlow paramters on the NetScaler appliance to send Security Insight data

    set appflow param  -SecurityInsightRecordInterval 60 -SecurityInsightTraffic ENABLED

    This enables it to send the IPFIX templates and data to Insight.

    3: Enable Application Firewall rules and bind it either globally or to a particular virtual server. The simplest way to setup it could be done like this.

    add appfw profile pr_appfw -defaults advanced

    set  appfw profile pr_appfw -startURLaction log stats learn

    add appfw policy pr_appfw_pol «HTTP.REQ.HEADER(\»Host\»).EXISTS»pr_appfw

    bind appfw global pr_appfw_pol 1

    This enabled the StartURL feature on the Application Firewall modul, which is set to learn which will learn the start URL on a virtual server, if someone tries to go into the virtual server on another start url for instance http://test.fqdn.no/test2.html they will be violating the StartURL rule and will then trigger an alert.

    This will then generate an AppFlow alert which will be sent to Insight Center and processed there.

    So in my case I have a simple virtual server, which represents a HTTP server. (This could also be Exchange, SharePoint, eCommerce sites for instance)

    After the configuration is done, you will see when logging into Insight and going into –> Security Insight that a device with Security Insight AppFlow will appear in the list

    image

    As of now I only have one Application called “test” which has an Application firewall policy attached to it.

    NOTE: Yeah I love descriptive application names.

    So after triggering some Appfw violations by accessing the virtual server on another URL which is not the start URL I get a bit more information. It will get some more  information about my Application Firewall policy and NetScaler System security policies, ill come back to that in a bit.

    image

    When clickcing on the virtual server form within Insight I get more information about what is configured for the virtual server in terms of AppFw signatures and security checks and so on. Also a threat level is generated based upon the violations that are created, te hmore critical the attacks on an application, the higher the threat index for that application. Values range from 1 through 7.

    So if I click on the threat index I get a detailed overview of what kinds of violations that has been triggered.

    image

    I can also from here click on the violation type and get information about client ip address, and if I have GeoIP database added it will draw a map of where it originated from

    image

    Now what we saw from the earlier pane, we noticed that my Application Firewall configuration was level one and that my NetScaler System Security was level 3, which means that Insight noticied that I haven’t done any changes beside the default and I should take a closer look at the system configuration to harden it more..

    So if I now go into the NetScaler system policy configuration I get feedback on what I should do, to ensure that the NetScaler is more locked down

    image

    The NetScaler System Security is built up of different categories.

    • Access
    • Monitoring
    • Logging
    • Cryptography
    • Others

    The different categories still show limited information and the crpytography pane just looks at if SSL/TLS is enabled and if it is using AES256 as the default encrpytion.

    So what’s missing?

    Even thou this is a good starting point, I would love for Citrix to go even deeper here in this feature, because so many have no clue about how secure their external services are. So some things I would love to see in the product moving forward

    • Cipher groups indexing
    • Certificates indexing
    • HSTS enabled?
    • DDoS attacks?
    • AAA bruteforcing ?

    Would love for them to incorporate features from what ssllabs are running and display a better SSL/TLS overview of each virtual servce.

    And also all the recommendations that are shown from Insight should be able to configure directly from Insight, instead of just show what you should do and then you need to log into the appliance and then configure it from there.

    New award, Aviator!

    So what is an Aviator? Well It’s now what you think… For those that have been following me, notice that  now that I from time to time have been working with Avi Networks. Why? Because they think differently from other ADC vendors in the market and they deliver a pretty cool product!

    If you for instance look at Google Load balancer,  Maglev –> http://research.google.com/pubs/pub44824.html or at Microsoft’s own load balancer Duet –>  http://research.microsoft.com/pubs/220640/sigcomm14-duet-final.pdf

    Both Microsoft and Google’s load balancer’s (which they use for their public clouds) share many of the same characteristics for cloud scale load balancing, to easily allow of scaling up and  down load balancing resources and having rich analytics built-in which is easily distributed across multiple servers. Which of course are running on plain x86 hardware with powerful automation features.

    This is the same vision that Avi Networks want to deliver to on-premises datacenters, providing the same feature set and analytics as Microsoft and Google does in their own datacenters. Also instead of having a single appliance which does all the features + management, which is typically the case for many ADC vendors which comes from a physical space and then turned to virtulization,  Avi networks is built for virtuliazation where management and packet handling is seperated into their own bits of virtual appliance and can be integrated into the virtulization layer.

    So back to the subject, what is an Aviator? Aviator is a new community program from Avi networks, where I am actually the first official member as of now!  I am honored that Avi choose me as their first member, and I look forward to participating in the program and trying out the new upcoming features in the upcoming beta bits! Smilefjes

    Troubleshooting a slow ICA-proxy session NetScaler

    This article is meant as an way to troubleshoot network issues on a NetScaler appliance, and of course ways to troubleshoot may differ, if you have any comments on what you typically do in this type of scenario please post a comment below!

    So the other day I was tasked to troubleshoot a NetScaler issue, where the customer had someissues with ICA sessions going slow and unreliable. A big problem was the file transfers were not working at all, where the bandwidth usage was going between 0KBps – 200 KBps. So when doing an initial assesment I noticed the following

    • Running NetScaler VPX 50
    • Running on VMware
    • Using LACP on the vDS on VMware
    • Firewall between the NetScaler and the external users, where they were using NAT for incoming requests

    First a couple things worth checking if ICA sessions are going slow

    • Amount of SSL transactions (Depending on the CPU performance and compute resources available to the NetScaler, it is going to affect the performance on the appliance) If this is pretty high, it could be that the resources available to the NetScaler was just saturaged.
    • Bandwidth use (Was it consuming to much resources so it couldn’t actually handle the amount of users trying to access this solution?)
    • Packet CPU usage (On NetScaler the packet CPU’s are responsinble for all the packet handling, and it also has one dedicated vCPU for management) on a VPX 50 you can only have 2vCPU (1 for management and 1 for packet management)

    So I noticed that the VPX had plenty of resources, the amount of SSL transactions were low (This could also be why they customer has issues with unreliable connections) the Packet CPU usage was low (I could see this by using stat cpu in CLI)

    Then after we noticed that there was nothing wrong with the VPX, we took a closer look at the virtual infrastructure. I checked if the NetScaler VMware host was sagurated, of if there was any performance issues on the virtuel network that the NetScaler was placed on.

    Since the issue was persistent and that it affected both client drive transfers and plain ICA proxy sessions, we guessed that this was issues with the external traffic and not the internal traffic which was causing the issue. We also checked that there were no bandwidth policies set on the XenApp farm which might affected the file transfer.

    Now since the bandwidth performance of the NetScaler was going up and down, I was thinking that this might be congestion somewhere. So the simplest way was to do a trace file from the NetScaler to see what kind of traffic is going back and forth and if there were any issues.

    After using WireShark for a while you get used to search for the most common parameters. If you have congestion somewhere you might get alot of RST or retransmits because of a full buffer. If you think about it, file transfer using client drive mapping will try to use as much bandwidth as possible. Another thing that was done before I did my test was to change the TCP profile to use nstcp_xa_xd_tcp_profile, which enables use of features like SACK and Nagle to reduce the amount of TCP segments and need for ACK messages in case of packet drops.

    NOTE: A good tip when doing starting trace files from NetScaler for SSL connections is to enable for “Decrypt SSL packets”

    User-added image

    From the trace file we noticed a couple of things.

    1: Alot of retransmissions from the XenApp server to the NetScaler SNIP

    2: TCP ZeroWindow

    Which are two symptoms which are often connected.

    image

    This meant that the NetScaler was not able to receive further information at the time, and the TCP transmission is halted until it can process the information in its receive buffer. So what I immediately assumed that the TCP buffer size was adjusted or somewhat altered. This was not the case since it was still using the default size.

    So why was this happening?

    A quick google search indicated that this was an issue in the NetScaler build, which has since then been resolved in the later build –> http://support.citrix.com/article/CTX205656

    So some quick tips when troubleshooting an NetScaler VPX

    • Check if the appliance has enough compute resources
    • Check if the hypervisor / virtualization layer it is running on has enough resources, or if it is a problem affecting other parts of the virtual network as well
    • Draw a topology map of the network and elimiate other possible components in the network path
    • Check TCP settings, remember that ICA-proxy is using TCP to the end-users
    • Check a Trace file, use filters in WireShark to easily filter out traffic (https://wiki.wireshark.org/DisplayFilters) (Even thou you can set filters in the NetScaler it can consume more resources on the NetScaler and you might not see the whole picture from a networking perspective, for instance the NetScaler might be flooded with Network traffic from another IP source which will then not be displayed in the trace file)
    • Last but not least, check if there are any known bugs in the current build and that the build is supported for the hypervisor that is being used. (http://support.citrix.com/en/products/netscaler)

    NOTE: You can read more about TCP Window Scaling in this article –> http://support.citrix.com/article/CTX113656

    Networking SIG on My CUGC

    So as of late I have been involved in setting up and creating a SIG (Special Interest Group) under MYCUG aimed at Networking (Can be found here –> https://www.mycugc.org/p/bl/bl/blogid=104) . Now you might think “err does that mean Citrix Networking?” no! our goal is to create a open community around all types of different networking content

    • Network virtualization
    • Network Security
    • Cloud networking
    • Datacenter networking

    And of course Citrix Networking is going to be a big part of it, since it is natural that many of the users there are familiar and used to working with Citrix networking.

    Now one of the objectives as well is to bridge the gap between Citrix and the community, so we will have Citrix employees in direct contact to help us create content, answer questions doing webinars and so on. For instance if someone asks a question are we aren’t able to answer it, we might get an official answer from Citrix on the forums.

    Now our hope is that this is going to be an “active” community where we have fresh content pusblished, and a bit exclusive compared to the other place where NetScaler content gets published.

    For my part is going to be organizing events, content and so on. If you have an idea about content or that you want to publish or stuff you would like to learn or hear more about, please reach out to me. Also I will start to publish more NetScaler related blogs there, so this blog might be a bit quiet on the NetScaler front Smilefjes

    Other then that I hope you join our community!

    Running RDS 2016 TP5 Connection Broker Database in Azure AD

    As part of the RDS 2012, Microsoft introduced the support for high-availability for the connection broker role, this on the other hand required an external SQL database for the connection brokers to share the connection tables between them.
    Yesterday Microsoft released a new Tech Preview for Windows Server 2016, which introduces a new feature for the RDS Connection Broker, the ability to use an Azure SQL Database as the DB source for the Connection Broker.

    To set this up we need to precreate a new SQL database in Azure, which can be easily located from the Marketplace

    image

    So when defining the parameters of the SQL database you can choose the Basic Pricing tier, which costs about 4$ each month.

    image

    Now we just have to wait for the deployment to finish from the Azure portal. After the deployment is done you can find the SQL database inside the created resource group

    image

    From here click on the “Show database connection strings” and save the ODBC line.
    Now when we want to create a high availabilty connection broker, we first need to create a deployment. From within server manager right click on the connection broker

    image

    And click on “configure high availability” From the wizard choose Shared Database

    image

    NOTE: You need to have the SQL native client installed on the connection broker server to make this feature work, you can download it from here –> http://go.microsoft.com/fwlink/?LinkID=239648&clcid=0x409 this needs to be installed before you can finish the wizard or else it will not connect.

    Also you might also need to configure firewall rules against the SQL database, to allow access from the connection brokers.

    image

    After this is done you will be able to complete the wizard

    image

    image

    If you are getting some issues, pay attention to the Terminal Servers Connection Broker event log

    image