NetScaler Security Insight

With NetScaler build 11.65, Citrix introduces Security Insight as part of the Insight Appliance. This feature is used to give real-time insight into attacks, and recommendations on countermeasures.

NOTE: Security Insight requires Application Firewall enabled, since it reports based upon AppFw violations and rules. You also need the same version of Insight and the NetScaler appliance for this to be supported and for it to work.

To setup Security Insight there are basically three steps.

1: Setup NetScaler Insight build 11.65 and NetScaler appliance with the same build number, then add the NetScaler appliance to Insight

2: Alter the AppFlow paramters on the NetScaler appliance to send Security Insight data

set appflow param  -SecurityInsightRecordInterval 60 -SecurityInsightTraffic ENABLED

This enables it to send the IPFIX templates and data to Insight.

3: Enable Application Firewall rules and bind it either globally or to a particular virtual server. The simplest way to setup it could be done like this.

add appfw profile pr_appfw -defaults advanced

set  appfw profile pr_appfw -startURLaction log stats learn

add appfw policy pr_appfw_pol «HTTP.REQ.HEADER(\»Host\»).EXISTS»pr_appfw

bind appfw global pr_appfw_pol 1

This enabled the StartURL feature on the Application Firewall modul, which is set to learn which will learn the start URL on a virtual server, if someone tries to go into the virtual server on another start url for instance http://test.fqdn.no/test2.html they will be violating the StartURL rule and will then trigger an alert.

This will then generate an AppFlow alert which will be sent to Insight Center and processed there.

So in my case I have a simple virtual server, which represents a HTTP server. (This could also be Exchange, SharePoint, eCommerce sites for instance)

After the configuration is done, you will see when logging into Insight and going into –> Security Insight that a device with Security Insight AppFlow will appear in the list

image

As of now I only have one Application called “test” which has an Application firewall policy attached to it.

NOTE: Yeah I love descriptive application names.

So after triggering some Appfw violations by accessing the virtual server on another URL which is not the start URL I get a bit more information. It will get some more  information about my Application Firewall policy and NetScaler System security policies, ill come back to that in a bit.

image

When clickcing on the virtual server form within Insight I get more information about what is configured for the virtual server in terms of AppFw signatures and security checks and so on. Also a threat level is generated based upon the violations that are created, te hmore critical the attacks on an application, the higher the threat index for that application. Values range from 1 through 7.

So if I click on the threat index I get a detailed overview of what kinds of violations that has been triggered.

image

I can also from here click on the violation type and get information about client ip address, and if I have GeoIP database added it will draw a map of where it originated from

image

Now what we saw from the earlier pane, we noticed that my Application Firewall configuration was level one and that my NetScaler System Security was level 3, which means that Insight noticied that I haven’t done any changes beside the default and I should take a closer look at the system configuration to harden it more..

So if I now go into the NetScaler system policy configuration I get feedback on what I should do, to ensure that the NetScaler is more locked down

image

The NetScaler System Security is built up of different categories.

  • Access
  • Monitoring
  • Logging
  • Cryptography
  • Others

The different categories still show limited information and the crpytography pane just looks at if SSL/TLS is enabled and if it is using AES256 as the default encrpytion.

So what’s missing?

Even thou this is a good starting point, I would love for Citrix to go even deeper here in this feature, because so many have no clue about how secure their external services are. So some things I would love to see in the product moving forward

  • Cipher groups indexing
  • Certificates indexing
  • HSTS enabled?
  • DDoS attacks?
  • AAA bruteforcing ?

Would love for them to incorporate features from what ssllabs are running and display a better SSL/TLS overview of each virtual servce.

And also all the recommendations that are shown from Insight should be able to configure directly from Insight, instead of just show what you should do and then you need to log into the appliance and then configure it from there.

New award, Aviator!

So what is an Aviator? Well It’s now what you think… For those that have been following me, notice that  now that I from time to time have been working with Avi Networks. Why? Because they think differently from other ADC vendors in the market and they deliver a pretty cool product!

If you for instance look at Google Load balancer,  Maglev –> http://research.google.com/pubs/pub44824.html or at Microsoft’s own load balancer Duet –>  http://research.microsoft.com/pubs/220640/sigcomm14-duet-final.pdf

Both Microsoft and Google’s load balancer’s (which they use for their public clouds) share many of the same characteristics for cloud scale load balancing, to easily allow of scaling up and  down load balancing resources and having rich analytics built-in which is easily distributed across multiple servers. Which of course are running on plain x86 hardware with powerful automation features.

This is the same vision that Avi Networks want to deliver to on-premises datacenters, providing the same feature set and analytics as Microsoft and Google does in their own datacenters. Also instead of having a single appliance which does all the features + management, which is typically the case for many ADC vendors which comes from a physical space and then turned to virtulization,  Avi networks is built for virtuliazation where management and packet handling is seperated into their own bits of virtual appliance and can be integrated into the virtulization layer.

So back to the subject, what is an Aviator? Aviator is a new community program from Avi networks, where I am actually the first official member as of now!  I am honored that Avi choose me as their first member, and I look forward to participating in the program and trying out the new upcoming features in the upcoming beta bits! Smilefjes

Troubleshooting a slow ICA-proxy session NetScaler

This article is meant as an way to troubleshoot network issues on a NetScaler appliance, and of course ways to troubleshoot may differ, if you have any comments on what you typically do in this type of scenario please post a comment below!

So the other day I was tasked to troubleshoot a NetScaler issue, where the customer had someissues with ICA sessions going slow and unreliable. A big problem was the file transfers were not working at all, where the bandwidth usage was going between 0KBps – 200 KBps. So when doing an initial assesment I noticed the following

  • Running NetScaler VPX 50
  • Running on VMware
  • Using LACP on the vDS on VMware
  • Firewall between the NetScaler and the external users, where they were using NAT for incoming requests

First a couple things worth checking if ICA sessions are going slow

  • Amount of SSL transactions (Depending on the CPU performance and compute resources available to the NetScaler, it is going to affect the performance on the appliance) If this is pretty high, it could be that the resources available to the NetScaler was just saturaged.
  • Bandwidth use (Was it consuming to much resources so it couldn’t actually handle the amount of users trying to access this solution?)
  • Packet CPU usage (On NetScaler the packet CPU’s are responsinble for all the packet handling, and it also has one dedicated vCPU for management) on a VPX 50 you can only have 2vCPU (1 for management and 1 for packet management)

So I noticed that the VPX had plenty of resources, the amount of SSL transactions were low (This could also be why they customer has issues with unreliable connections) the Packet CPU usage was low (I could see this by using stat cpu in CLI)

Then after we noticed that there was nothing wrong with the VPX, we took a closer look at the virtual infrastructure. I checked if the NetScaler VMware host was sagurated, of if there was any performance issues on the virtuel network that the NetScaler was placed on.

Since the issue was persistent and that it affected both client drive transfers and plain ICA proxy sessions, we guessed that this was issues with the external traffic and not the internal traffic which was causing the issue. We also checked that there were no bandwidth policies set on the XenApp farm which might affected the file transfer.

Now since the bandwidth performance of the NetScaler was going up and down, I was thinking that this might be congestion somewhere. So the simplest way was to do a trace file from the NetScaler to see what kind of traffic is going back and forth and if there were any issues.

After using WireShark for a while you get used to search for the most common parameters. If you have congestion somewhere you might get alot of RST or retransmits because of a full buffer. If you think about it, file transfer using client drive mapping will try to use as much bandwidth as possible. Another thing that was done before I did my test was to change the TCP profile to use nstcp_xa_xd_tcp_profile, which enables use of features like SACK and Nagle to reduce the amount of TCP segments and need for ACK messages in case of packet drops.

NOTE: A good tip when doing starting trace files from NetScaler for SSL connections is to enable for “Decrypt SSL packets”

User-added image

From the trace file we noticed a couple of things.

1: Alot of retransmissions from the XenApp server to the NetScaler SNIP

2: TCP ZeroWindow

Which are two symptoms which are often connected.

image

This meant that the NetScaler was not able to receive further information at the time, and the TCP transmission is halted until it can process the information in its receive buffer. So what I immediately assumed that the TCP buffer size was adjusted or somewhat altered. This was not the case since it was still using the default size.

So why was this happening?

A quick google search indicated that this was an issue in the NetScaler build, which has since then been resolved in the later build –> http://support.citrix.com/article/CTX205656

So some quick tips when troubleshooting an NetScaler VPX

  • Check if the appliance has enough compute resources
  • Check if the hypervisor / virtualization layer it is running on has enough resources, or if it is a problem affecting other parts of the virtual network as well
  • Draw a topology map of the network and elimiate other possible components in the network path
  • Check TCP settings, remember that ICA-proxy is using TCP to the end-users
  • Check a Trace file, use filters in WireShark to easily filter out traffic (https://wiki.wireshark.org/DisplayFilters) (Even thou you can set filters in the NetScaler it can consume more resources on the NetScaler and you might not see the whole picture from a networking perspective, for instance the NetScaler might be flooded with Network traffic from another IP source which will then not be displayed in the trace file)
  • Last but not least, check if there are any known bugs in the current build and that the build is supported for the hypervisor that is being used. (http://support.citrix.com/en/products/netscaler)

NOTE: You can read more about TCP Window Scaling in this article –> http://support.citrix.com/article/CTX113656

Networking SIG on My CUGC

So as of late I have been involved in setting up and creating a SIG (Special Interest Group) under MYCUG aimed at Networking (Can be found here –> https://www.mycugc.org/p/bl/bl/blogid=104) . Now you might think “err does that mean Citrix Networking?” no! our goal is to create a open community around all types of different networking content

  • Network virtualization
  • Network Security
  • Cloud networking
  • Datacenter networking

And of course Citrix Networking is going to be a big part of it, since it is natural that many of the users there are familiar and used to working with Citrix networking.

Now one of the objectives as well is to bridge the gap between Citrix and the community, so we will have Citrix employees in direct contact to help us create content, answer questions doing webinars and so on. For instance if someone asks a question are we aren’t able to answer it, we might get an official answer from Citrix on the forums.

Now our hope is that this is going to be an “active” community where we have fresh content pusblished, and a bit exclusive compared to the other place where NetScaler content gets published.

For my part is going to be organizing events, content and so on. If you have an idea about content or that you want to publish or stuff you would like to learn or hear more about, please reach out to me. Also I will start to publish more NetScaler related blogs there, so this blog might be a bit quiet on the NetScaler front Smilefjes

Other then that I hope you join our community!

Running RDS 2016 TP5 Connection Broker Database in Azure AD

As part of the RDS 2012, Microsoft introduced the support for high-availability for the connection broker role, this on the other hand required an external SQL database for the connection brokers to share the connection tables between them.
Yesterday Microsoft released a new Tech Preview for Windows Server 2016, which introduces a new feature for the RDS Connection Broker, the ability to use an Azure SQL Database as the DB source for the Connection Broker.

To set this up we need to precreate a new SQL database in Azure, which can be easily located from the Marketplace

image

So when defining the parameters of the SQL database you can choose the Basic Pricing tier, which costs about 4$ each month.

image

Now we just have to wait for the deployment to finish from the Azure portal. After the deployment is done you can find the SQL database inside the created resource group

image

From here click on the “Show database connection strings” and save the ODBC line.
Now when we want to create a high availabilty connection broker, we first need to create a deployment. From within server manager right click on the connection broker

image

And click on “configure high availability” From the wizard choose Shared Database

image

NOTE: You need to have the SQL native client installed on the connection broker server to make this feature work, you can download it from here –> http://go.microsoft.com/fwlink/?LinkID=239648&clcid=0x409 this needs to be installed before you can finish the wizard or else it will not connect.

Also you might also need to configure firewall rules against the SQL database, to allow access from the connection brokers.

image

After this is done you will be able to complete the wizard

image

image

If you are getting some issues, pay attention to the Terminal Servers Connection Broker event log

image

Load balancing features in Azure

Deploying distributed applications and services within Azure can be cumbersome, atleast when you are not familiar with the different options/features you have available. Therefore I wanted to use to post to explain the different options we have for load balancing within Azure, both for internal services and external facing services.

Now there are three different load balancing features available directly in Azure

  • Azure Load Balancer
  • Application Gateway
  • Traffic Manager

There is also third party solutions from different vendors available in the marketplace which has alot more features but again, this in most cases requires additional licenses and compute resources, but ill come back to those later in the blogpost.

Now there are some distinct differerences between the different load balancing features in Azure and the third party vendors.

Service Azure Load Balancer Application Gateway Traffic Manager Third Party (NetScaler, etc)
Layer Layer 4 support Layer 7 DNS Layer 4 to Layer 7, DNS etc
Protocol support TCP, UDP (Any applications based upon these protocol) HTTP, HTTPS All services which use DNS Most Protocols
Endpoints Azure VMs and Cloud Services instances Azure VMs and Cloud Services instances Azure endpoints, On-premises endpoints Azure VM endpoints, On-premise, Internal Instancs
Internal & External support Can be used for internal and External communication Can be used for internal and external communication Externally Interal and External traffic
Health Monitoring Probes: TCP or HTTP Probes: HTTP or HTTPS Probes: HTTP or HTTPS Custom based, TCP, HTTP, HTTPS, Inline HTTP, PING, UDP etc
Pricing Free $0,07 per gateway-hour per INSTANCE (Medium size) which  is default size $0.54 per million queries (Cheaper above 1 billion queries) Depending on the Vendors, regular compute instance + license from the vendor
         

So depending on the requirements we have it is also possible to do a combination of the different services, and in some cases it might also be more cost effective to do a combination. For instance that we have Traffic Manager to do GEO based load balancing between different regions and that we have NetScaler HA-pair setup on each site to deliver local load balancing capabilities in each region.

Azure Load Balancing

NOTE: That Azure Load Balancing requires that we have an availability-set in place for our virtual machines that we want to load balance. If you have existing virtual machiens that you want to add to an availability set which are created using ARM you cannot do this yet.

Which as mentioned this can be created either for external or internal purposes. For this example I’m going to use an external load balancing service against a couple of web servers against an ARM deployment.

image

We have two IIS web servers, which are deployed in the same availability-set, which are deployed  to the same virtual network. They are deployed in different network security groups, which allows us to handle ACLs based upon each vNIC.

To create an Azure Load balancer, we can just select and create a new Azure Load balancer resources from within the UI.

First we need to specify a probe, which is used to check health of the backend pool. Here we just specify what kind of protocol we want to use for the health test and port and path to check on against the backend pool.

image

Also when setting up the Azure Load balancer we have to specify a backend pool which consists of the virtual machines we want to load balance.

image

Then we need to create the load balancing rules which ties all the different settings together. Here we specify the backend pool, probe, backend port (For Reverse proxy features we can for instance load balance services externally on port 8080, while the servers are listening to port 80 interally)

image

And we define the the session persistency. NOTE: We cannot define the load balancing technique her, for instanse based upon load or amount of connections. This can be done using PowerShell to alter the distribution mode –> https://azure.microsoft.com/nb-no/documentation/articles/load-balancer-distribution-mode/

SSL Offloading is also not supported using Azure Load balancing, it is also missing important features like Content Switching and rewrite settings.

Application Gateway

Application Gateway is a layer 7 HTTP load balancing solution, Important to note however is that this feature is built upon IIS/AAR in Windows Server

As of now it is only available using the latest Azure PowerShell, but moving forward it will become available in the portal. New-AzureApplicationGateway –Name AppGW –Subnets 10.0.0.0/24 –vNetname vNet01 or we can use an ARM template from here –> https://azure.microsoft.com/en-us/services/application-gateway/

And we can now see that the AppGW is created

NOTE: The default value for InstanceCount is 2, with a maximum value of 10. The default value for GatewaySize is Medium. You can choose between Small, Medium and Large.

image

Next we need to do the configuration, this is by using an XML file where the declare all the speicifcs like external ports, what kind of protocol and if for instance cooke based persistency should be enabled

The XML file should look like this

<?xml version=»1.0″ encoding=»utf-8″?>
<ApplicationGatewayConfiguration xmlns:i=»http://www.w3.org/2001/XMLSchema-instance« xmlns=»http://schemas.microsoft.com/windowsazure«>
    <FrontendPorts>
        <FrontendPort>
            <Name>FrontendPort1</Name>
            <Port>80</Port>
        </FrontendPort>
    </FrontendPorts>
    <BackendAddressPools>
        <BackendAddressPool>
            <Name>BackendServers1</Name>
            <IPAddresses>
                <IPAddress>10.0.0.5</IPAddress>
                <IPAddress>10.0.0.6</IPAddress>
            </IPAddresses>
        </BackendAddressPool>
    </BackendAddressPools>
    <BackendHttpSettingsList>
        <BackendHttpSettings>
            <Name>BackendSetting1</Name>
            <Port>80</Port>
            <Protocol>Http</Protocol>
            <CookieBasedAffinity>Enabled</CookieBasedAffinity>
        </BackendHttpSettings>
    </BackendHttpSettingsList>
    <HttpListeners>
        <HttpListener>
            <Name>HTTPListener1</Name>
            <FrontendPort>FrontendPort1</FrontendPort>
            <Protocol>Http</Protocol>
        </HttpListener>
    </HttpListeners>
    <HttpLoadBalancingRules>
        <HttpLoadBalancingRule>
            <Name>HttpLBRule1</Name>
            <Type>basic</Type>
            <BackendHttpSettings>BackendSetting1</BackendHttpSettings>
            <Listener>HTTPListener1</Listener>
            <BackendAddressPool>BackendPool1</BackendAddressPool>
        </HttpLoadBalancingRule>
    </HttpLoadBalancingRules>
</ApplicationGatewayConfiguration>

If we want to change the config file we can use the command Set-AzureApplicationGatewayConfig -Name AppGwTest -ConfigFile «D:\config.xml»

You can also create custom probes against each backend –> https://azure.microsoft.com/nb-no/documentation/articles/application-gateway-create-probe-classic-ps/

Traffic Manager

Traffic Manager is a DNS based load balancing solution. It is as simple as when a client requests a particular resource, Azure will respond with one or another backend resources. Even thou it is an Azure features it can also be used for on-premises load balancing.

The logic behind it is pretty simple we create an resource in Azure which is the traffic manager resources, which will be bound to an FQDN (resourcename.trafficmanager.net) when someone asks for this particular resource the traffic manager object will look at the available endspoints that it has to determine which are healthy and not and which is the preffered endpoint for this client. The endspoint are just other FQDN which could be a resource from an Azure datacenter on an on-premises web services which is available on an external solution.

image

Traffic Manager can be used to direct clients to their closest datacenter as well using the default performance based routing method

image

So how to we use this externally? Since we can’t actually use an traffic-manager.net URL for our external users. The way to do this is to point the traffic-manager.net as an CNAME alias for the official FQDN. So for instance www.msandbu.org would be a CNAME alias to iis.traffic-manager.net which would again resolve into our endpoints which might be another address entirely, dependong on the routing method and availability.

So now we have gone trough all the different load balancing features in Azure, can we combine and mix different load balancing features to achive even more uptime and better performance?  Yes!

A good example is using Traffic Manager in Combination with Azure Load Balancing, think about having an e-commerence solution where we have multiple instances deploying across different Azure regions.

image

Where we can for instance have Traffic Manager to load balance between different regions which will point the end-user to the closest location and from there we have Azure Load Balancing to load balance between resources inside each region. This is by using the combination of DNS + TCP based load balancing.

Now since we have all these nifty features in Azure why would we even need third party vendor load balancing features at all ?

  • Sure Connect (Meaning that the load balancing will queue incoming requests that the backend resources are unable to process)
  • Better granular load balancing (For instance load balancing with custom probes and monitors, SIP traffic, Database load balancing, group based load balancing on multiple ports. Different load balancing methods)
  • Content switching (Allows us to load balance requests based upon which URL for instance a end-user is going to)
  • Web Application Firewall capabilities (Now Azure only has Security Groups ACLs but it has no deep insight into what kind of traffic is hitting the external services, most ADCs have their own WAP feature which can be used to migiate attacks)
  • Rewrite and Responder policies (Being able to alter HTTP requests inline before hitting the services, when moving from one site to another for instance without needing to change the external FQDN, or changing HTTP headers) We can also use Responder policies to respond directly to blocked endpoints
  • HTTP Caching and Compression (Most ADC’s can cache often access data, it can also compress outbound data in order to optimize traffic flow)
  • Web Optimization (In most cases these solutions can also do optimization of the web traffic going outbound to further optimize the traffic)
  • GSLB (Most ADCs also have their own Global Server Load balancing features which does the same feature as traffic manager does)

Now most of these solution are available from the Azure marketplace, and most cases require their own license as well but they all run as a virtual machine instance and has the same limitations as they do. So how would they fit in?

In most cases it will be a combination of many elements. Since the ADC runs as virtual appliance they have the same limitations. They need to be placed in their own availbility set. We in most cases cannot use their built-in HA feature because of the Azure networking limitations for failover, therefore we need to use the Azure Load balancing do distribute traffic between them and use the probe to check if they are online or not.  Also in this scenario we have all web servers and NetScaler in the same virtual network, but only the NetScaler will be bound to the Azure Load balancing as the backend pool, since the NetScaler will handle internal communication with the web servers

image

Now we have a 3 way load balancing feature setup, so an example traffic flow would look like this.

1: User requests test.msandbu.org
2: Traffic Manager responds with a Public IP from the Azure region which is closest the end-user
3: User initiats connection with the IP closest in the Azure region
4: The Azure Load balancing feature which serves the VIP looks at the connection and fowards it to one of the active NetScalers
5: The NetScaler looks at the request and travers any policies for load balancing, content switching, url rewrites or WAP policies
6: The traffic is served from one of the web-servers back to the NetScaler and back to the client.

So now we have an combination of DNS, TCP, and HTTP / SSL load balancing to make sure that the content is optimized when deliverd.

AVINetworks – Architecture

I have previously blogged about AVINetworks –> https://msandbu.wordpress.com/2016/02/19/a-closer-look-at-avi-networks/
Where I wrote briefly on how their scale-out architecture differs from their competitors, where most ADC vendors have a single virtual appliance which contains all the features + management. Avi networks consists of a Cloud Controller which is the management layer where we do management either using the UI or using REST APIs.

This example below shows an VMware architecture. The Avi Controller can either be a stand-alone virtual appliance or be configured in a 3 nodes controller cluster.

image

The Controller nodes can be setup with an integration with VMware with either read or read/write privileges. With read/write priviliges the Avi Controller can read and discover VMs, data centers, networks, and hosts. This also allows for automatic deployment of the service engine.

When we configure a virtual service, for instance an load balanced service for our web servers AVI will automatically deploy two SEs (Service Engines) based upon a OVA file. After the deployment there will be one active service engine and the other is passive, which will be responsible for serving the requests on the VIP.

Now the Service Engines also report server health, client connection statistics, and client-request logs to the Controllers, which share the processing of the logs and analytics information.

The cool thing with Avi is the auto-scaling feature! Now let’s say for instance if our Service Engines run out of resources, this might be CPU, Memory or the amount of packets per second. Then AVI can for instance move the virtual service to an unused Service Engine or scale out the service across multiple service engines.

By default the primary service engines reponds to all ARP requests to the VIP, if the service needs to scale out. The primary SE will move the overflow traffic to a secondary SE on layer two, where the secondary Service engine will terminate the TCP connection and then process the connection and responds back to the end-client.

Now if for instance an primary SE will fail, a secondary SE will repond using GARP to take over existing connections. Any existing connections by the Primary SE will need to be recreated in the session table.

Now I can alter the default behaviour for the SE’s in a deployment

image

For instance the default memory for an Service Engine is 2048 MB, SSL connections consume alot more memory then regular L4 connections, and therefore we might even alter the memory capacity if we have available. NOTE: The memory limit can be between 1 GB and 128 GB per service engine.

I can also from the service window, migrate the service to another service engine or scale-out or scale-in

image

Doing Migrate or scale-in will handle all new connections, forwarding any older connections to the secondary SE.

Now we can easily see the benefits of this type of architecture, where we have a distributed management layer and a distributed data plane unlike the other ADCs which have effectivly moved their existing appliance model to a virtual appliance where they suffer from the limited resources they have available.