Stikkordarkiv: netscaler

Netscaler and AAA with CSW One VIP

As part of the latest release from Citrix Netscaler V11, there was an interesting feature added to the firmware. Which in essence allows ut to add a NO-IP Virtual AAA server, which allow us to add multiple resources lets say behind a CSW vServer where we only use one VIP.

Highlander there can be only one - There can be only one VIP

This is part of the latest feature release from Citrix (build 11. 63 from October) which has this feature.
It can either be setup using CLI or using the GUI.

User-added image

So when setting up the AAA vServer we can then use the option non-adressable

image

Note that when biding it to the CS vServer you need to specify that it needs to use 401-based authentication, since forms based requires an HTTP session externally to function

image

So from an enduser perspective a users tried to go to LB1, which resides on the CSW vServer, which will then trigger an AAA request to the non-adressable 401 based authentication and then the user will be authenticated.

Testing SSL from Netscaler–Issues with SSL handshake

From time to time we need to setup load balancing to a SSL based service or when setting up connection to a secure Storefront (which is the default) there is one thing that alot of people are missing from the config when setting up, which results in wierd issues or getting SSL handshake errors from the monitors. In most cases it because of two things

  • Missing Root CA
  • Wrong Ciphers or not supported ciphers

So how can we verify from the Netscaler that it is missing the rootCA or that we have the right CA in place?

That is when we uses OpenSSL, which is a toolkit that is used on the Netscaler, which also has a commandline interface which allow us to test different parameters.

So if we enter Shell on the Netscaler and then do a CD to /nsconfig/ssl (This is where all the NS certificates are stored by default and from there we can use OpenSSL.

By using the command

openssl s_client –connect FQDN

First of this will show us, the certificate that is presented, and the certificate chain. It will also list out what kind of connection that is being used towards to FQDN (In this case below we are using TLS 1.2 against a Storefront server.

CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
   i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

Server certificate
—–BEGIN CERTIFICATE—–
MIIFHzCCBAegAwIBAgIQCT7QmpMfh8Xq7pY2PnSiSjANBgkqhkiG9w0BAQsFADBN
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMTUwODI3MDAwMDAwWhcN
MTYwODMxMTIwMDAwWjBnMQswCQYDVQQGEwJOTzERMA8GA1UECBMIQWtlcnNodXMx
FjAUBgNVBAcTDVNrZWRzbW9rb3JzZXQxFjAUBgNVBAoTDU1hcml1cyBTYW5kYnUx
FTATBgNVBAMMDCouY29tbWF4eC5ubzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAMxHenWSyUtXpYakBSPOTZj98L6LZ8WPv7oFvCqcG0TxWN8KBYWABhcS
V0Z/KA8uWFf9hnSXVWBaI/CSX218Ccay4ulAGxavfPvW9M0f7tPn7sp19rBJsqz/
PMsPKXstQAdK4trF1Ag4mecqwQmMreSBJelSlhNtKHrMpXXkBm4cTyI2uCYFKFND
Gc9JpaqzJiyPn/3kGhZXR039q2lW4c0nMM5eZD3x9GpOEgMeZU1uabbVR8Pviflg
djfaZ7PW9tDV0UQd3HpG9sJaXsr99eCt8/0eQCqgRzxtCP2MS0vI/Wxtu9Vuoi7m
1T6H6VXuoaeW+cRMCVID0DUcdepXzxECAwEAAaOCAd8wggHbMB8GA1UdIwQYMBaA
FA+AYRyCMWHVLyjnjUY4tCzhxtniMB0GA1UdDgQWBBRLJltFnloNz4pStHnGYHWk
CHGcGDAjBgNVHREEHDAaggwqLmNvbW1heHgubm+CCmNvbW1heHgubm8wDgYDVR0P
AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBrBgNVHR8E
ZDBiMC+gLaArhilodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vc3NjYS1zaGEyLWc0
LmNybDAvoC2gK4YpaHR0cDovL2NybDQuZGlnaWNlcnQuY29tL3NzY2Etc2hhMi1n
NC5jcmwwTAYDVR0gBEUwQzA3BglghkgBhv1sAQEwKjAoBggrBgEFBQcCARYcaHR0
cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAIBgZngQwBAgIwfAYIKwYBBQUHAQEE
cDBuMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wRgYIKwYB
BQUHMAKGOmh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydFNIQTJT
ZWN1cmVTZXJ2ZXJDQS5jcnQwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOC
AQEAHikELfi9t3NMI90pKbxcPnaHufJivbqwEAU4Vu2JjB8Lr3WPOj5oqSvFKC9e
XigAtZvK08NmxKlLRFHDNUNauHCOUh4rRDuzJEaI6INngsBULRPAKx8Ynk9E/epT
0lE0BLSpoXbvYb1HG+7ngeTnh3XsELI6NWo1G86FMHlzzDrnEaCX0DYzgxMHYfM5
P41ZCOLX/Zw46RPUTj/Up50jnqcbIVNsYt3En13c7vu0Chg2Wqb5iRnOEC5nQ8TS
DqfLsGCXq0JFVbz7t4amskNv9VSn2bDauTy0u6aSbfLknn8DYY/q+ruj0Nh+ELsA
y0Tmkrvh8IaD/OiFRq+osToQ2w==
—–END CERTIFICATE—–
subject=/C=NO/ST=
issuer=/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA

No client certificate CA names sent

SSL handshake has read 3034 bytes and written 479 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: EB46000017E1621AA1BB5491BDFD3EDB2C273F35E73DB2029651C5B00DEC62BC
    Session-ID-ctx:
    Master-Key: 65CA41A8B811869F0C005469E20578BB3C876AB7207AB5D2D42370B7779FD1EB                                 7F971DC3A0001EF9B54963D1D2B080BD
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1448336973
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)

What we can see here is that we have an error message at the tops stating that

 verify error:num=20:unable to get local issuer certificate

This error occurs if

  • The certificate chain for the certificate wasn’t provided by the other side or it doesn’t have one (it is self-signed).
  • The root certificate is not in the local database of trusted root certificates.
  • The local database of trusted root certificates was not given and thus not queried by OpenSSL.

In order to verify against a chain of certificates with a RootCA or Intermidiate with both, we can use the parameter –CAfile or –CAPath which we can specify behind the command to test a connection with a RootCA.

Now there are a bunch of different parameters that we can use with OpenSSL, for instance we can also test openSSL using different protocols such as -ssl3, -tls1, -no_ssl3, -no_tls1, -no_tls1_1, -no_tls1_2

Which allow us to test using SSL3 for instance. You can see the list of different options on the openSSL site here –> https://www.openssl.org/docs/manmaster/apps/s_client.html

Load balancing port ranges with Netscaler

So I got a question earlier today, if it was possible to load balance a specific port range within Netscaler. Now by default we cannot specify a port range when setting up a load balanced vServer or setting up services.

In order to ensure that traffic from a specific endpoint going to the same backend service we have some options.

1: Persistency Groups, in this case we need to define load balanced vServers and services for each port nr and then we need to define a persistency group afterwards. Which I have blogged about earlier https://msandbu.wordpress.com/2015/02/09/netscaler-and-persistency-groups/

2: Another option we have is to define the port nr to * and service to ANY, which means that we can have a single vServer and single service objects, the issue is that it will intercept ANY traffic to ANY port, which is a bad thing.

What we can do with option 2 is define a listening policy. Listen policy will allow us to customize which ports a vServer should respond to. For instnace we can define a vServer with ANY service and port equals * but then we attach a listen policy and define for instance port range between 80 to 8080. Even thou the vServer is setup with ANY port and ANY service it will only listen to requests coming from within the port range in the listen policy.

Listen policies can be defined within the vServer, such as shown in the screenshot below

image

New Netscaler books available!

Just a small post about what I have been busy with lately Smilefjes For those who have been following me on Twitter/LinkedIn/Blog notice that I from time to time blog about Netscaler which kinda has become my little baby.
Anyhow… 2 years back I started working on my second book for Packt Publishing called Netscaler VPX, which was the first technical book (outside of Citrix Education) on Netscaler available on Amazon.

Now two years later, I have done a bit more.

Implementing Netscaler VPX Second Edition: http://amzn.to/20R8M2R

Which is a upgraded/polished version of my first book, which was created upon version 10, this book is based upon V11 and contains more content around security, troubleshooting, azure/amazon deployments and front-end optimization.
And my latest project Mastering Netscaler VPX

Mastering Netscaler VPX: http://bit.ly/1HFzmQK

This is a book which I co-wrote with Rick Roetenberg (Note: He did most of the work) Which goes a bit more deep into the material. I did some chapters on Network optimization, troubleshooting, Content switching, GSLB, Datastream and security features.

So if you are unsure what to get for christmas, this might be a good idea Smilefjes

Putting ThinWire and Framehawk to the test!

Framehawk and Thinwire – It’s all about the numbers

Recently me and Mikael @mikael_modin attended a Citrix User Group Conference in Norway, where Mikael held a session regarding when and when to use Framehawk, you can read his entire blogpost here –> http://bit.ly/1PV3104 and I have already done some details regarding Framehawk from a networking perspective.

The main point in Mikael’s presentation was that although using Framehawk in situations when packet loss is tremendously better, Thinwire Advance will often be “enough” or even more useful when there is only latency involved. This is because of the use of CPU, RAM and most of all bandwidth.
Another thing he pointed out was that Framehawk needs “a lot” of bandwidth to be at its best.
The recommendations for Thinwire is a minimum of 1,5MBps + 150kbps per user while recommendations for Framehawk is a minimum of 4-5Mbps + 150kbps per user.

There is a lot of naming conventions when it comes to Thinwire. Although we can see Thinwire as one protocol, there are different versions of it.
Thinwire is all about compressing data before sending it. The methods for this are:

· Legacy Thinwire (Pre win8 / Server 2012R2)

· Thinwire Compatibility Mode (New with FP3, also known as Thinwire +, Win8 / Server 2012R2 and later. This version takes advantage of how new operating systems constructs the graphics.
For more info read the following blog post written by Muhammad Dawood http://bit.ly/WEnSDN

· Thinwire Advance (uses H.264 to compress the data)

For a more detailed overview when to use each technology, you can refer to the following table:

clip_image002

When we came back home we decided to take a closer look at what impact had on CPU, RAM and bandwidth Thinwire or Framehawk had and we have found some very interesting data.

Our tests includes the following user workload;

· Logging in and waiting 1 minute for the uberagent to gather data and getting the session up and ready.

· Open a PDF file, scrolling up and down for 1 minute. (The PDF is located locally on the VM to exclude network I/O)

· Connect to a webpage www.vg.no, which is a Norwegian newspaper which contains a lot of different objects and high graphics, and scrolling up and down for a 1 minute. 

· We then open Microsoft Word and type randomly for 1 minute.

· Last but not least our favorite opening of the Avengers trailer in fullscreen using Chrome for the full duration of 2 minutes.

This allows us to see which workloads generate how much bandwidth, CPU- and RAM usage with each of the different protocols.

To collect and analyze the data we were using the following tools

· Splunk – Uberagent (Get info we didn’t even think was possible!)

· Netbalancer (Show bandwidth, set packet loss, define bandwidth limits and define latency)

· Citrix Director

– Displaystatus (to verify the protocol status)

The sample video below shows how the tests is being run. This allows us to closer analyze the sample data from Netbalancer as well.

NOTE: During the testing there might be some slight alterations from test to test since this not an automated test but running as an typical enduser experience, but these were so minor that we can conclude that the numbers are within +/-5%

We had two Windows 10 VDI running the latest release of XenDesktop 7.6 FP3 during the testing phase.

· MCS1002 is for the test02 user, which is not using Framehawk

· MCS1003 is for the test01 user, which has Framehawk, enabled using policies

· Use of Codec were deactivated through policy to ensure that Thinwire was used

The internett connection is a solid 100 MBps, the average connection to the Citrix enviroment is about 10 – 20 MS latency.

The sample video in this URL https://www.youtube.com/watch?v=F89eQPd7shs shows how the tests is being run. This allows us to closer analyze the sample data from Netbalancer as well.

Some notes so far: Some Framehawk sessions get stuck on the Netscaler, we can see existing connections not being dropped correctly, we can see this in the Netscaler GUI under Gateway –> DTLS sessions

After we changed the TCP profiles on the Netscaler we were unable to use Framehawk.
We then needed to reconfigure the DTLS and Certificate settings on the vServer and setup a new connection and Framehawk worked again as expected.

So after the initial run, we can note the following from the Netbalancer data;

We begin with looking at how Framehawk handles bandwidth.

We can see that the total session, which was about 7 minutes, Framehawk uses about 240 MBs of bandwidth to be able to deliver the graphics.
However, it was during the PDF and Webpage part of the test which really pushed it in terms of bandwidth, not the Youtube trailer.

clip_image003

Thinwire on the other hand, used only 47 MBs of bandwidth, and like we would expect more data was being used when showing the trailer than the PDF- and webpage section.

clip_image004

Using Splunk we care able to get a closer look at the Framehawk numbers.
Average CPU usage for the VDA agent was close up to 16% on average.

clip_image005

While using ThinWire the CPU usage was only 6% on average.

clip_image006

But the maximum amount of CPU usage came from Framehawk, which was close to 50% CPU usage at one point.

clip_image007

While ThinWire on the other hand, was only up to 18%

clip_image008

We can conclude that Framehawk uses much more CPU cycles in order to process the bandwidth, but from our testing we could see that the PDF part which generated a lot more traffic, allowed for a much more smooth experience. Not just from scrolling the document but also zooming in.

On the other side we can also see that Framehawk uses a bit more RAM then ThinWire does, about 400 MB was the maximum number

clip_image009

While Thinwire was about 300 MB

clip_image010

So this was the initial test, which shows that Thinwire uses less bandwidth, less memory and less CPU, but we can see that Framehawk on applications like PDF deliver a better user experience. So now, let us see how they fare when taking into account of latency and packet loss.

2% Packet loss
Framehawk

We started by testing Framehawk at 2% packet loss.
Looking at the bandwidth test we could see that is uses about 16 MB of bandwidth less with the packet loss. It’s still the PDF and Webpage that consumes the most resources, and now it is down to 224 MBs of bandwidth usage

The Maximum CPU usage peaked at 45%

And the average CPU usage was 19%

The amount of RAM used was a slight increase with 4MB

clip_image011

clip_image012

clip_image013

clip_image014


ThinWire

Now here comes the interesting part, using Thinwire at 2% packet loss, (up and down) will trigger a lot of TCP retransmissions because of the packet drops

clip_image015

(Remember that this is using an optimized Netscaler) we can see that ThinWire uses only 12 MBs of bandwidth! This is because of the TCP retransmissions, it will never be able to send large enough packets before the packet loss occurs.

So with Thinwire and 2% packet loss we could see that the bandwidth usage dropped with about 59 MB when we had the packet loss. The maximum bandwidth used in this session was 12Mbps

The maximum was also 50% lower than the reference test and showed only 3%

The average CPU usage was now only 3% (that is 50% of the reference test)

The RAM usage was about 30MB more than earlier

clip_image016

clip_image017

clip_image018

clip_image019

5% Packet loss
Framehawk

At 5% packet loss we can see that is uses about 50 MB of bandwidth extra. It’s still the PDF and Webpage that consumes the most resources, but now it is up to 300 MBs of bandwidth

We can also see that from a resource perspective, it still uses almost the same amount of max CPU %, but this might vary from test to test, but it is close to the 50%)

On average CPU usage we can see that it went up 4% from the initial testing, which makes sense since it needs to send more network packets which uses CPU cycles.

The RAM usage is the same as with 2% packet loss

clip_image020

clip_image021

clip_image023

clip_image024

5% Packet loss
ThinWire

Looking at the bandwidth usage with 5% packet loss and use of Thinwire the number is slightly lower and now uses 11MB

This can also be seen in the CPU usage of the protocol, since the packet loss occurs, the VDA does not need to send so much packets and hence the CPU usage is lower and stops at 7%

Average CPU usage is now just under 3%

RAM however is a bit larger with 330MB

clip_image025

clip_image026

clip_image027

clip_image028

End-user perspective
From an end-user perspective we can safely say that Framehawk delivered a much better experience, if we tried to follow the test from minute to minute, the ThinWire test took about 40 seconds longer just because of the delay from a mouse click to occur and doing things like zooming into a PDF file took so much time that it caused the test to take a longer time to complete.

Winner: Framehawk!

10% Packet loss
Framehawk

clip_image029

With 10% packet loss, we could see that the bandwidth usage went down a bit. That might again be that the packet loss was so high that it was unable to process all the data and hence the total bandwidth usage was lower than it was with 5%, and with the decrease in bandwidth, we can also see the CPU usage go down as well.

The max CPU usage was about the same with 47%

The average CPU usage was 19%

The RAM usage is the same at 404 MB

clip_image030

clip_image031

clip_image032

10% Packet loss
ThinWire

With 10% packet loss Thinwire was down to 6MB and the CPU usage also reflected this by only use 4% at peak and 1.6 % at average
RAM usage was still about the same as earlier and peaked at 326MB

clip_image033

clip_image034

clip_image035

clip_image036

End-user perspective
What we noticed here is that most of the different graphic intensive testing became unresponsive and that the ICA connection froze. The only thing that was really workable was using Word. Opening the PDF, Webpage and youtube became so unresponsive that is was not really workable.

Winner: Framehawk!

CPU Stats on Framehawk and Thinwire
NOTE: We have taken multiple samples of the CPU statistics on the Netscaler so this screenshots represent the average number we saw.
What we can see is that a framehawk which uses more bandwidth also will increase the CPU usage on the packet engines. The Netscaler from an idle state uses about 0 – 1,5 % CPU, which can be seen here à

clip_image037

NOTE: This is a VPX 1000 with 2 vCPU (Where we have only 1 packet engine) starting an ICA proxy session with the defaults over thin wire and starting the process that generates the most bandwidth (PDF scrolling and zooming) the packet CPU rises to about <1%

clip_image038

So it’s a minor increase which is expected since ThinWire uses a small amount of bandwidth. Now Framehawk on the other hand will use about 4% of the packet engine CPU. Note again that this was when we kept working with the PDF documentet.
We can conclude that using Framehawk will put a lot more strain on the Netscaler packet engine and therefore we cannot have as many users on the Netscaler.

clip_image039

RDP usage:
We also wanted to give RDP a test under different scenarios. We have some issues fetching out CPU and memory usage since RDP uses DWM and MSTSC which can appear as a sub-process of svchost
We therefore skipped that part and only focused on the bandwidth usage and end-user experience.

First we started out with a test where we have no limitations in form of latency and packet loss (This was using regular RDP against a Windows 10 with TCP/UDP

The initial test shows as we expected, RDP uses 53 MB of bandwidth

clip_image041

We also noticed that under the YouTube part that the Progressive rendering engine kicked in order to ensure optimal delivery but the graphics was ok.

RDP, 2% Packet loss

With 2% Packet loss the bandwidth usage was basically half 26MB of bandwidth

clip_image043

Keystrokes and some operations was a bit delayed, but still workable, on the other hand the progressive rendering engine on the youtube part made the graphics nearly impossible to see what actually happened, even thou audio worked fine.

RDP 5% Packet loss

RDP used about 17MB of bandwidth PDF scrolling and zooming made a huge delay in how the end-user could work. Surfing on the webpage which has a huge amount of graphics, freezed up for a couple of seconds. Youtube itself, well it didn’t work very well.

clip_image045

We can conlude that RDP uses more bandwidth that Thinwire under normal circumstances, but when coming to packet loss it does not deal with that pretty well.

So what does all these data tell us?
We can clearly see that Framehawk and Thinwire has its own use cases.
While Thinwire is the preferred method of delivering graphics, even with high latency, as soon as we experience packet loss off 3% or higher, Framehawk will definitively give a better use experience. Just remember to keep an eye on the resource usage on the VDI.
Especially when using it with XenApp since a spike in the CPU usage will have a great impact on the users who are logged on and will decrease the numenbr of users you can have on each server.

Office365 on Terminal Servers done right

So this is a blogpost based upon a session I had at NIC conference, where I spoke about how to optimize the delivery of Office365 in a VDI/RSDH enviroment.

There are multiple stuff we need to think / worry about. Might seem a bit negative, but that is not the idea just being realistic Smilefjes

So this blogpost will cover the following subjects

  • Federation and sync
  • Installing and managing updates
  • Optimizing Office ProPlus for VDI/RDS
  • Office ProPlus optimal delivery
  • Shared Computer Support
  • Skype for Buisness
  • Outlook
  • OneDrive
  • Troubleshooting and general tips for tuning
  • Remote display protocols and when to use when.

So what is the main issue with using Terminal Servers and Office365? The Distance….

This is the headline for a blogpost on Citrix blogs about XenApp best pratices

image

So how to fix this when we have our clients on one side, the infrastructure in another and the Office365 in a different region ? Seperated with long miles and still try to deliver the best experience for the end-user, so In some case we need to compromise to be able to deliver the best user experience. Because that should be our end goal Deliver the best user experience

image

User Access

First of is, do we need to have federation or just plain password sync in place? Using password sync is easy and simple to setup and does not require any extra infrastructure. We can also configure it to use Password hash sync which will allow Azure AD to do the authentication process. Problem with doing this is that we lose a lot of stuff which we might use on an on-premises solution

  • Audit policies
  • Existing MFA (If we use Azure AD as authentication point we need to use Azure MFA)
  • Delegated Access via Intune
  • Lockdown and password changes (Since we need change to be synced to Azure AD before the user changes will be taken into effect)

NOTE: Now since I am above average interested in Netscaler I wanted to include another sentence here, for those that don’t know is that Netscaler with AAA can in essence replace ADFS since Netscaler now supports SAML iDP. Some important issues to note is that Netscaler does not support • Single Logout profile; • Identity Provider Discovery profile from the SAML profiles. We can also use Netscaler Unified Gateway with SSO to Office365 with SAML. The setup guide can be found here

https://msandbu.wordpress.com/2015/04/01/netscaler-and-office365-saml-idp-setup/

NOTE: We can also use Vmware Identity manager as an replacement to deliver SSO.

Using ADFS gives alot of advantages that password hash does not.

  • True SSO (While password hash gives Same Sign-on)
  • If we have Audit policies in place
  • Disabled users get locked out immidietly instead of 3 hours wait time until the Azure AD connect syng engine starts replicating, and 5 minutes for password changes.
  • If we have on-premises two-factor authentication we can most likely integrate it with ADFS but not if we have only password hash sync
  • Other security policies, like time of the day restrictions and so on.
  • Some licensing stuff requires federation

So to sum it up, please use federation

Initial Office configuration setup

Secondly, using the Office suite from Office365 uses something called Click-to-run, which is kinda an app-v wrapped Office package from Microsoft, which allows for easy updates from Microsoft directly instead of dabbling with the MSI installer.

In order to customize this installer we need to use the Office deployment toolkit which basically allows us to customize the deployment using an XML file.

The deployment tool has three switches that we can use.

setup.exe /download configuration.xml

setup.exe /configure configuration.xml

setup.exe /packager configuration.xml

NOTE: Using the /packager creates an App-V package of Office365 Click-To-run and requires a clean VM like we do when doing sequencing on App-V, which can then be distributed using existing App-V infrastructure or using other tools. But remember to enable scripting on the App-V client and do not alter the package using sequencing tool it is not supported.

The download part downloads Office based upon the configuration file here we can specify bit editions, versions number, office applications to be included and update path and so on. The Configuration XML file looks like this.

<Configuration>

<Add OfficeClientEdition=»64″ Branch=»Current»>

<Product ID=»O365ProPlusRetail»>

<Language ID=»en-us»/>

</Product>

</Add>

<Updates Enabled=»TRUE» Branch=»Business» UpdatePath=»\\server1\office365″ TargetVersion=»16.0.6366.2036″/>

<Display Level=»None» AcceptEULA=»TRUE»/>

</Configuration>

Now if you are like me and don’t remember all the different XML parameters you can use this site to customize your own XML file –> http://officedev.github.io/Office-IT-Pro-Deployment-Scripts/XmlEditor.html

When you are done configuring the XML file you can choose the export button to have the XML file downloaded.

If we have specified a specific Office version as part of the configuration.xml it will be downloaded to a seperate folder and storaged locally when we run the command setup.exe /download configuration.xml

NOTE: The different build numbers are available here –> http://support2.microsoft.com/gp/office-2013-365-update?

When we are done with the download of the click-to-run installer. We can change the configuration file to reflect the path of the office download

<Configuration> <Add SourcePath=»\\share\office» OfficeClientEdition=»32″ Branch=»Business»>

When we do the setup.exe /configure configuration.xml path

Deployment of Office

The main deployment is done using the setup.exe /configure configuration.xml file on the RSDH host. After the installation is complete

Shared Computer Support

<Display Level="None" AcceptEULA="True" /> 
<Property Name="SharedComputerLicensing" Value="1" />

In the configuration file we need to remember to enable SharedComputerSupport licensing or else we get this error message.

image

If you forgot you can also enable is using this registry key (just store it as an .reg file)

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\15.0\ClickToRun\Configuration]
«InstallationPath»=»C:\\Program Files\\Microsoft Office 15»
«SharedComputerLicensing»=»1

Now we are actually done with the golden image setup, don’t start the application yet if you want to use it for an image. Also make sure that there are no licenses installed on the host, which can be done using this tool.

cd ‘C:\Program Files (x86)\Microsoft Office\Office15’
cscript.exe .\OSPP.VBS /dstatus

image

This should be blank!

Another issue with this is that when a user starts an office app for the first time he/she needs to authenticate once, then a token will be stored locally on the %localappdata%\Microsoft\Office\15.0\Licensing folder, and will expire within a couple of days if the user is not active on the terminalserver. Think about it, if we have a large farm with many servers that might be the case and if a user is redirected to another server he/she will need to authenticate again. If the user is going against one server, the token will automatically refresh.
NOTE: This requires Internet access to work.

And important to remember that the Shared Computer support token is bound to the machine, so we cannot roam that token around computers or using any profile management tool.

But a nice thing is that if we have ADFS setup, we can setup Office365 to automatically activate against Office365, this is enabled by default. So no pesky logon screens.

Just need to add the ADFS domain site to trusted sites on Internet Explorer and define this settings as well

Automatic logon only in Intranet Zone

image

Which allows us to basically resolve the token issue with Shared Computer Support Smilefjes

Optimizing Skype for Buisness

So in regards to Skype for Buisness what options do we have in order to deliver a good user experience for it ? We have four options that I want to explore upon.

  • VDI plugin
  • Native RDP with UDP
  • Natnix PCoIP
  • Native ICA (w or without audio over UDP)
  • Local app access
  • HDX Optimization Pack 2.0

Now the issue with the first one (which is a Microsoft plugin is that it does not support Office365, it requires on-premises Lync/Skype) another issue that you cannot use VDI plugin and optimization pack at the same time, so if users are using VDI plugin and you want to switch to optimization pack you need to remove the VDI plugin

ICA uses TCP protcol works with most endpoints, since its basically running everyone directly on the server/vdi so the issue here is that we get no server offloading. So if we have 100 users running a video conference we might have a issue Smilefjes If the two other options are not available try to setup HDX realtime using audio over UDP for better audio performance. Both RDP and PCoIP use UDP for Audio/Video and therefore do not require any other specific customization.

But the problems with all these are that they make a tromboning effect and consumes more bandwidth and eats up the resources on the session host

image

Local App from Citrix access might be a viable option, which in essence means that a local application will be dragged into the receiver session, but this requires that the enduser has Lync/Skype installed. This also requires platinum licenses so not everyone has that + at it only supports Windows endpoints…

The last and most important piece is the HDX optimization pack which allows the use of server offloading using HDX media engine on the end user device

And the optimization pack supports Office365 with federated user and cloud only users. It also supports the latest clients (Skype for buisness) and can work in conjunction with Netscaler Gateway and Lync edge server for on-premises deployments. So means that we can get Mac/Linux/Windows users using server offloading, and with the latest release it also supports Office click-to-run and works with the native Skype UI

So using this feature we can offload the RSDH/VDI instances from CPU/Memory and eventually GPU directly back to the client. And Audio/video traffic is going to the endpoint directly and not to the remote session

image

Here is a simple test showing the difference between running Skype for buisness on a terminal server with and without HDX Optimization Pack 2.0

Permalink til innebygd bilde

Here is a complete blogpost on setting up HDX Optimization Pack 2.0 https://msandbu.wordpress.com/2016/01/02/citrix-hdx-optimization-pack-2-0/

Now for more of the this part, we also have Outlook. Which for many is quite the headache…. and that is most because of the OST files that is dropped in the %localappdata% folder for each user. Office ProPlus has a setting called fast access which means that Outlook will in most cases try to contact Office365 directly, but if the latency is becoming to high, the connection will drop and it will go and search trough the OST files.

Optimizing Outlook

Now this is the big elefant in the room and causes the most headaches. Since Outlook against Office365 can be setup in two modes either using Cached mode and the other using Online mode. Online modes uses direct access to Office365 but users loose features like instant search and such. In order to deliver a good user experience we need to compromise, the general guideline here is to configure cached mode with 3 months, and define to store the OST file (Which contains the emails, calender, etc) and is typically 60-80% than the email folder) on a network share. Since these OST files are by default created in the local appdata profile and using streaming profile management solutions aren’t typically a good fit for the OST file.

. Important to note that Microsoft supports having OST files on a network share, IF! there is adequate bandwidth and low latency… and only if there is one OST file and the users have Outlook 2010 SP1

NOTE: We can use other alternatives such as FSLogix, Unidesk to fix the Profile management in a better way.

Ill come back to the configuration part later in the Policy bits. And important to remember is to use Office Outlook over 2013 SP1 which gives MAPI over HTTP, instead of RCP over HTTP which does not consume that much bandwidth.

OneDrive

In regards to OneDrive try to exclude that from RSDH/VDI instances since the sync engine basically doesnt work very well and now that each user has 1 TB of storagee space, it will flood the storage quicker then anything else, if users are allowed to use it. Also there is no central management capabilities and network shares are not supported.

There are some changes in the upcoming unified client, in terms of deployment and management but still not a good solution.

You can remove it from the Office365 deployment by adding  this in the configuration file.

<ExcludeApp ID=»Groove» />

Optimization and group policy tuning

Now something that should be noted is that before installing Office365 click-to-run you should optimize the RSDH sessions hosts or the VDI instance. A blogpost which was published by Citrix noted a 20% in performance after some simple RSDH optimization was done.

Both Vmware and Citrix have free tools which allow to do RSDH/VDI Optimization which should be looked at before doing anything else.

Now the rest is mostly doing Group Policy tuning. Firstly we need to download the ADMX templates from Microsoft (either 2013 or 2016) then we need to add them to the central store.

We can then use Group Policy to manage the specific applications and how they behave. Another thing to think about is using Target Version group policy to manage which specific build we want to be on so we don’t have a new build each time Microsoft rolls-out a new version, because from experience I can tell that some new builds include new bugs –> https://msandbu.wordpress.com/2015/03/09/trouble-with-office365-shared-computer-support-on-february-and-december-builds/

image

Now the most important policies are stored in the computer configuration.

Computer Configuration –> Policies –> Administrative Templates –> Microsoft Office 2013 –> Updates

Here there are a few settings we should change to manage updates.

  • Enable Automatic Updates
  • Enable Automatic Upgrades
  • Hide Option to enable or disable updates
  • Update Path
  • Update Deadline
  • Target Version

These control how we do updates, we can specify enable automatic updates, without a update path and a target version, which will essentually make Office auto update to the latest version from Microsoft office. Or we can specify an update path (to a network share were we have downloaded a specific version) specify a target version) and do enable automatic updates and define a baseline) for a a specific OU for instance, this will trigger an update using a built-in task schedulerer which is added with Office, when the deadline is approaching Office has built in triggers to notify end users of the deployment. So using these policies we can have multiple deployment to specific users/computers. Some with the latest version and some using a specific version.

Next thing is for Remote Desktop Services only, if we are using pure RDS to make sure that we have an optimized setup.  NOTE: Do not touch if everything is working as intended.

Computer Policies –> Administrative Templates –> Windows Components –> Remote Desktop Services –> Remote Desktop Session Host –> Remote Session Enviroment

  • Limit maximum color depth (Set to16-bits) less data across the wire)
  • Configure compression for RemoteFX data (set to bandwidth optimized)
  • Configure RemoteFX Adaptive Graphics ( set to bandwidth optimized)

Next there are more Office specific policies to make sure that we disable all the stuff we don’t need.

User Configuration –> Administrative Templates –> Microsoft Office 2013 –> Miscellaneous

  • Do not use hardware graphics acceleration
  • Disable Office animations
  • Disable Office backgrounds
  • Disable the Office start screen
  • Supress the recommended settings dialog

User Configuration –> Administrative Templates  –>Microsoft Office 2013 –> Global Options –> Customizehide

  • Menu animations (disabled!)

Next is under

User Configuration –> Administrative Templates –> Microsoft Office 2013 –> First Run

  • Disable First Run Movie
  • Disable Office First Run Movie on application boot

User Configuration –> Administrative Templates –> Microsoft Office 2013 –> Subscription Activation

  • Automatically activate Office with federated organization credentials

Last but not least, define Cached mode for Outlook

User Configuration –> Administrative Templates –> Microsoft Outlook 2013 –> Account Settings –> Exchange –> Cached Exchange Modes

  • Cached Exchange Mode (File | Cached Exchange Mode)
  • Cached Exchange Mode Sync Settings (3 months)

Then specify the location of the OST files, which of course is somewhere else

User Configuration –> Administrative Templates –> Microsoft Outlook 2013 –> Miscellanous –> PST Settings

  • Default Location for OST files (Change this to a network share

Network and bandwidth tips

Something that you need to be aware of this the bandwidth usage of Office in a terminal server enviroment.

Average latency to Office is 50 – 70 MS

• 2000 «Heavy» users using Online mode in Outlook
About 20 mbps at peak

• 2000 «Heavy» users using Cached mode in Outlook
About 10 mbps at peak

• 2000 «Heavy» users using audio calls in Lync About 110 mbps at peak

• 2000 «Heavy» users working Office using RDP About 180 mbps at peak

Which means using for instance HDX optimization pack for 2000 users might “remove” 110 mbps of bandwidth usage.

Microsoft also has an application called Office365 client analyzer, which can give us a baseline to see how our network is against Office365, such as DNS, Latency to Office365 and such. And DNS is quite important in Office365 because Microsoft uses proximity based load balancing and if your DNS server is located elsewhere then your clients you might be sent in the wrong direction. The client analyzer can give you that information.

image

(We could however buy ExpressRoute from Microsoft which would give us low-latency connections directly to their datacenters, but this is only suiteable for LARGER enterprises, since it costs HIGH amounts of $$)

image

But this is for the larger enterprises which allows them to overcome the basic limitations of TCP stack which allow for limited amount of external connection to about 4000 connections at the same time. (One external NAT can support about 4,000 connections, given that Outlook consumes about 4 concurrent connections and Lync some as well)

Because Microsoft recommands that in a online scenario that the clients does not have more then 110 MS latency to Office365, and in my case I have about 60 – 70 MS latency. If we combine that with some packet loss or adjusted MTU well you get the picture Smilefjes 

Using Outlook Online mode, we should have a MAX latency of 110 MS above that will decline the user experience. Another thing is that using online mode disables instant search. We can use the exchange traffic excel calculator from Microsoft to calculate the amount of bandwidth requirements.

Some rule of thumbs, do some calculations! Use the bandwidth calculators for Lync/Exchange which might point you in the right direction. We can also use WAN accelerators (w/caching) for instance which might also lighten the burden on the bandwidth usage. You also need to think about the bandwidth usage if you are allow automatic updates enabled in your enviroment.

Troubleshooting tips

As the last part of this LOOONG post I have some general tips on using Office in a virtual enviroment. This is just gonna be a long list of different tips

  • For Hyper-V deployments, check VMQ and latest NIC drivers
  • 32-bits Office C2R typically works better then 64-bits
  • Antivirus ? Make Exceptions!
  • Remove Office products that you don’t need from the configuration, since this add extra traffic when doing downloads and more stuff added to the virtual machines
  • If you don’t use lync and audio service (disable the audio service! )
  • If using RDSH (Check the Group policy settings I recommended above)
  • If using Citrix or VMware (Make sure to tune the polices for an optimal experience, and using the RSDH/VDI optimization tools from the different vendors)
  • If Outlook is sluggish, check that you have adequate storage I/O to the network share (NO HIGH BANDWIDTH IS NOT ENOUGH IF STORED ON A SIMPLE RAID WITH 10k disks)
  • If all else failes on Outlook (Disable MAPI over HTTP) In some cases when getting new mail takes a long time try to disable this, used to be a known error)

Remote display protocols

Last but not least I want to mention this briefly, if you are setting up a new solution and thinking about choosing one vendor over the other. The first of is

  • Endpoint requirements (Thin clients, Windows, Mac, Linux)
  • Requirements in terms of GPU, Mobile workers etc)

Now we have done some tests, which shown the Citrix has the best feature across the different sub protocols

  • ThinWire (Best across high latency lines, using TCP works over 1800 MS Latency)
  • Framehawk (Work good at 20% packet loss lines)

While PcoIP performs a bit better then RDP, I have another blogpost on the subject here –> https://msandbu.wordpress.com/2015/11/06/putting-thinwire-and-framehawk-to-the-test/

Optimizing a crappy web application using Citrix Netscaler

So I have had the pleasure of setting up optimizing of a crappy web application over the last couple of days. This particular web application had the following properties

  • Bound to port 8080
  • 401 based Authentication enabled
  • URL Absolutes
  • Alot of jibber in the Code
  • Default page is hardcoded to a specific URL (Which we do not want to have as first page exposed externally)

So when first looking at this setup my first thought was……

But we wanted to setup this application using AAA module to have Forms-based authentication, redirect the mainpage to another URL, remove uneccesary code and make sure that the URL absolutes are taken care of.

First thing we needed to do is handle SSO and SSO against the application (after setting up the basic load balancing against the internal services)

1: Setup an AAA vServer and bind it to an LDAP policy (simple AD authenticaiton)

2: Setup an Authentication Profile (Which is used to handle the auth session and different authenticaiton levels) Important that we enter a domain name which will be bound to the session

image

When adding the AAA vServer to the LB vServer it is important to do not choose both Authentication Virtual Server and Authenticaiton Profile (The vServer will the default to Virtual Server and bypass the profile where the domain info is set)

image

And then set it to Form based Authentication as well, this will give the end-user a Netscaler based login image

Next we had to manage SSO logout for the application, since terminating the session within the application we wanted the endusers to be redirected back to the login page.
This can be done using a traffic policy and by setting initiate logout, first setup an expression which will trigger when the user click on the logout URL, in this application the logout URL was logout-currentuser

My expression looked like this HTTP.REQ.URL.CONTAINS(«logout-currentuser») make sure that the Initate logout button is enabled

image

Next we needed to handle the default URL to be redirected to another page. The simplest way to handle this was using responder policy, since we know that the default url was /config1 we could use the responder to redirect it to another custom page.

So we can use an expression HTTP.REQ.URL.CONTAINS(«/config1») then setup an action to redirect them to the URL we want to

image

Now another thing we were struggling with was that the application with absolut URLs were redirecting the user requetsts to internal URLs which made the connection fail. In order to change this we needed to use the URL transformer policy

image

Now everything was almost complete, one piece was missing…. When a user logged on the application it worked fine, redirect to the custom page, url transform rules were working and AAA signout was working… But if a user pressed F5 custom files outside the webapp were not loading (taken from Chrome)

image

WTF? So then I took a long coffee break and didn’t quite comprehend what was happening… Then I was going deep-dive

clip_image001

Now after some troubleshooting I found out that using CTRL+F5 on the browser made the page refresh and the page was loading as I wanted it to, when comparing the different requests I saw this.

image

The only difference in the requests using browser F5 (Refresh) and CTRL + F5 was that the HTTP header Cache-Control was set to no-cache in the Request header)

So what I needed to do was to use something to set the Cache-Control to no-cache (Which will basically say that the browser will not cache any content, and since this was a quite sensitive application that was fine)

So using a specific rewrite action I could insert a new HTTP header

Cache Control: No-store

image

and then bound it to response on the vServer, now if I looked at my new requests going to the virtual server, I can see that the response was containng the no-store HTTP header

 image

End result, application working as intended!