session recording deep dive and troubleshooting final version

© 2016 Citrix | Confidential

Learning Together to Deliver the Future18-21 July, 2016

Session Recording Deep Dive and Troubleshooting

[email protected] | 涂振凯Senior Technical Support Engineer2016/07/19


Agenda

• What is Session Recording

• Session Recording Architecture & Work Flow

• Components in Details

• Deployment Tips & Troubleshooting


What’s Session Recording


Product History

Editon Main Production Version RemarkSmartAuditor 1.1 Citrix Presentation Server 4.5/ 5 for

Windows 2003End of life since 31-Mar-2013

SmartAuditor 1.2 Citrix XenApp 5.0 for Windows 2008 End of life since 13-Jan-2015SmartAuditor 1.3 Citrix XenApp 6.0/6.5Session Recording 7.6 XenApp and XenDesktop 7.6 Released Q1 2015Session Recording 7.8 XenApp and XenDesktop 7.8 Provide the ability to record

the Desktop OS VDA

The SmartAuditor feature is abandoned from the XenApp and XenDesktop 7.0 to 7.5, since it’s very welcomed by Customers from the APAC region, it

returns since as a part of XenApp and XenDesktop 7.6


What is Session Recording

• Session Recording is like a Digital Video Recorder…– Session Recording allows you to record the on-screen activity of any user session hosted

from a VDA machine

• Session Recording leverages Citrix VDA to provide…– Policy-based recording of an ICA session & associated session information– Manages recording & logs ICA sessions to persistent storage– Ability to search a catalog of recorded ICA sessions– Playback of recorded ICA sessions


Session Recording Key Capabilities

• Records the entire ICA session

• Provides powerful policy-based recording capabilities

• Provides catalog search of recorded sessions

• Records user sessions very efficiently– A typical 8 hour outlook recording is about 20MB

• Digitally signs recorded session files to ensure data integrity

• Records session data centrally and stores it securely

• Automates session recording without requiring client-side software


Use Case – User behavior monitoring / audit

• Management team wish to Record / Monitor / Audit User Activity– High-value / sensitive transactions– Users with high-privilege access– Security, corporate and regulatory policy compliance– Guest/ third party employee access to corporate systems


Use Case – Collect feedback about the Product Design

• Collect feedback about product design from customer’s behavior – After deploying a new prototype

website or launching A/B test on User Experience or Navigation, Product Manager or Product Designer could check the recorded sessions to see if end user likes the new changes or feels hard to navigate


Use Case - Technical Support and Troubleshooting

• Accelerate problem troubleshooting– Record the one-time issue

– Record the hard to reproduce issue

– Problem fast review


Session Recording Architecture & Workflow


Session Recording Architecture and Workflow

Unsecure Network

Client Users

Secure DatacenterSession Recording Agent

Server VDA or Desktop VDA

Session Recording Policy Console

SR ServerSession Recording Player

3rd Party Archive Solution

Storage

1

2

2

3

4

5

Establish ICA Connection

Verify recording Policy

Send Session Data

Retrieve Session Data

Archive files

Log Session Data and Write to Storage

2

3

4

5

0

1

Delivery ControllerApplication Enum

0

0 Predefined configurationSet the recording policy


Components in Details


Session Recording Server Components• Session Recording Broker– Installed as an IIS /ASP.NET hosted web application – Responsible for communicating with the Session Recording Database to enforce

policy query decisions and communicating with the Session Recording Player to manage access to session recordings

• Session Recording Storage Manager– Installed as a Windows service– Main Function:

– Writing Session Recording data to disk– Writes session metadata to database– Generates digital signatures– Records performance data (use perfmon)

– Characteristics– Does NOT interact with Session Recording Broker

– Can detect missing and duplicate data

– Can be restarted without data loss


Session Recording Database• The Storage Manager writes session recording file metadata and

policies in the Session Recording Database– Can co-exist on SQL Server with other databases– Database schema installer will create appropriate login and user security settings– SQL Server can be clustered, mirrored and also support AlwaysOn for Sql Server

2012 – Availability features such as replication and mirroring are supported– Session metadata is approximately 1KB per recording

• Installation pre-requisites for Database:– SQL Server 2008 R2 SP2 Enterprise and Express editions– SQL Server 2012 SP1, Express, Standard, and Enterprise Editions– SQL Server 2014, Express, Standard, and Enterprise Editions.


Session Recording Agent• The Session Recording Agent is the component which will be installed on the VDA

• Responsible for recording session data

• The Session Recording Driver is installed as part of the agent and is responsible for gathering session recording data

• XenApp Platinum License is required

• Supported Windows operating systems:– Microsoft Windows Server 2012 R2– Microsoft Windows Server 2012– Microsoft Windows Server 2008 R2 with Service Pack 1

• Requirements:– Microsoft Message Queuing (MSMQ), with Active Directory integration disabled, and MSMQ HTTP support

enabled– .NET Framework Version 3.5 Service Pack 1

• Data collected– Screen updates– Mouse activity– Session information

• No keyboard logging currently


Session Recording Agent – Rollover

• Rollover prevents files from becoming– too large in size

– too long in duration

• Large files or files of too long duration– difficult to download

– poorer searching performance in Player

– can cause problems with “dormant” file detection and archiving

• Session Recording Agent also prevents rollover of– Too short sessions

– Too small files

Important Notes: The rollover setting does not apply to VDI desktop sessions for XenDesktop 7.8/7.9. In those cases, each recording file has a maximum size limit of 1GB and activities are not recorded after that limit is reached.


Recording Files – Rollover Data Packets

Header

Completed recording file with linked rollover file

Start MD Login MD End MD Signature

Header Start MD Login MD

Next File GUID

Previous File GUID

Start reason = Rollover

End reason = Rollover


Session Recording Policy Console

• The Session Recording Policy Console provides the ability to manage policies related to recording of ICA sessions

• The Session Recording Policy Console is implemented as an MMC Console Snapin

• Installation pre-requisites (verified before installation):– Supported Windows operating systems:

– Microsoft Windows Server 2012 R2– Microsoft Windows Server 2012– Microsoft Windows Server 2008 R2 with Service Pack 1

– Requirements:– .NET Framework Version 3.5 Service Pack 1 (Windows Server 2008 R2 only) or .NET Framework

Version 4.5.1 or 4.6.


Session Recording Player

• The application used to replay captured session recordings files

• Only interacts with Session Recording Broker component

• Search option for metadata ,E.g. date/time, user, application, server, etc

• Installation pre-requisites (verified before installation):– Supported Windows operating systems:

– Microsoft Windows 8 / 8.1; Microsoft Windows 7 with Service Pack 1(Stated in Citrix Edocs)– For optimal results, install Session Recording Player on a workstation with:

– Screen resolution of 1024 x 768– Color depth of at least 32-bit– Memory: 1GB RAM (minimum). Additional RAM and CPU/GPU resources can improve performance

when playing graphics intensive recordings; especially when there are a lot of animations in the recordings.


Data storage

• Session recording data is stored on a central file store in flat files– Multiple directories can be defined– Storage Manager will distribute files over directories

• Directories are created by year, month, date– E.g. E:\Recordings\2016\05\20


Other Components and Utilities• Session Recording Authorization Console– A utility that enables Session Recording Server administrators to add users to pre-determined user roles

• Session Recording Custom Event API– API for the Session Recording software which enables ISVs to inject custom data through a third-party

application into a session recording

• Session Recording Player SDK– An SDK for use by ISVs to write third-party Session Recording Player extensions which can display

custom event data injected into the recorded session using the Session Recording Custom Event API

• icldb– A command line utility that enables you to run queries and perform maintenance of the Session Recording

Database

• iclstat– A command line utility for the Session Recording Server that enables you to view metadata information

about a session recording file


Session Recording Database – icldb Utility

• Citrix Session Recording Database Utility– Perform maintenance operations and queries

• Located on Session Recording Server in

C:\Program Files\Citrix\Session Recording\Server\Bin


Session Recording Database – icldb Utility

• Archive older recordings, option to move physical files

• Remove older recordings, option to remove physical files

• Restore recorded files into database

• Import or rebuild data from set of physical recordings– this will overwrite existing record if present

• Locate recording on disk by file GUID

• Query database schema version

• Remove all records


Deployment Tips & Troubleshooting


Deployment Tips


Deployment Tips 1

• When Machine Creation Services (MCS) or Provisioning Services creates a VDA with configured master image and Microsoft Message Queuing (MSMQ) installed, the VDA has the same QMId as the MSMQ. This might cause various issues, such as:– Sessions mighty not be recorded even if the recording agreement is accepted.– The session logoff signal might not be received by the Session Recording server, which

leads to the session always in Live status.

• Tips– Use a powershell script to create a unique and persistent QMId for each VDA

• Reference:– http://

docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-whats-new/xad-xaxd76-knownissues.html

http://docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-whats-new/xad-xaxd76-knownissues.html




Deployment Tip 2

• When recording a session with a resolution higher than or equal to 4096 x 4096, there might be fragments in the recording appearance.

• When you change your XenApp or XenDesktop license type, the change does not take effect immediately for Session Recording. Tips: Restart the VDA machine.

• You might receive an Installation failed error in the following two cases. You can ignore the message, but to avoid receiving the message, restart the machine before reinstalling the Session Recording components. – Uninstalled the Session Recording components, and then reinstalled them without restarting

the machine.– Installation failed and rollback happened, and then you tried to reinstall the Session

Recording components without restarting the machine.


Deployment Tips 3

• Limitation for Session Recording to support the Pre-Launched application sessions [BUG0561109]Problem:– If the active policy tries to match the application name, the application launched in the pre-

launched session will not be matched, which results in the session not being recorded.– If the active policy records every application, when the user logs into the Windows Receiver

(at the same time the pre-launched session is established) a notification for recording will appear and the empty session and any applications that will be launched in this session later will be recorded.

– Workaround:– Publish the applications in separate Delivery Groups according to their recording policy. Do not use the

application name as the recording condition. This will ensure pre-launch sessions will be recorded. However, notifications will still appear.

• Session Recording does not support Framehawk display mode.


Deployment Tips 4 – for Desktop Agent

• You cannot record the Windows 7 desktop sessions correctly when Legacy Graphics Mode is enabled by XenDesktop site policy and Disk-based Caching is enabled by Citrix Receiver for Windows policy. Those recordings show a black screen.

• Tips:– Disable Disk-based Caching by deploying with GPO to the machines on which you

installed Citrix Receiver for Windows. For more information about disabling Disk-based Caching, see http://support.citrix.com/article/CTX123169

http://support.citrix.com/article/CTX123169



Troubleshooting


Troubleshooting Methodology

Define the issue clearly

Gather information

Analyze and list possible cause

Possible Cause Verifying

Root Cause Documentation

• Issue Behavior• Expected

Behavior

• Review logs• Reproduce the

problem• Enable logging

• Rank the possible causes

• Create Action Plan

• Verify the action plan for the possible cause

• Document resolution and root cause for future reference

Redefine problem

Get further information


General Troubleshooting Session Recording

• Session not recorded– Check server names and protocols in configurations– Check if the recording notification can be seen

– Yes: MSMQ or DB issue– Check Agent CDF Trace to see policy query results

– Should not record (actually should): Broker or Agent issue– Timeout: network or configuration issue

– Check Agent CDF Trace to see if metadata is obtained correctly– Correct: Broker issue– Incorrect: Agent issue

• Playback error or corruption– Check client type and version which launched the session

– Play recording file locally– Check player CDF trace


General Troubleshooting Session Recording

• Check system event log first– Most of the service errors/warnings are logged

• Troubleshooting MSMQ– Open “Server Management” or “Computer Management” “Message Queuing”– Check status of private queue named citrixsmauddata– Check incoming (Server) or outgoing (Agent) packets in

http[s]://<servername>/msmq/private$/CitrixSmAudData

• Troubleshooting IIS– Ensure protocol used for connect is correct (HTTP/HTTPS)– Ensure correct certificate is used for HTTPS– Access http[s]://<servername>/SessionRecordingBroker/<name>.rem?wsdl for testing– <name> can be RecordPolicy, Player or PolicyAdministration


Case Study 1Failed to record session


Case Study1 – Failed to Record SessionProblem Symptom:Cannot record after migrating of the recording data and restart the Session Recording Server

Information Gathering

• Session Recording once worked fine• The problem only happened after migrate the

recording data and a server restart• Event Log – Nothing found• Reproduce the issue • End-user can get the Recording notification when

launch the application It means agent can get the record policy correctly

Possible CauseMSMQ or DB

Get further information for MSMQ


Case Study1 – Failed to Record Session• Checked the MSMQ outbound queue in the

Agent Massive outbound message queue found

• Checked the MSMQ inbound queue in the Server No message queue found

• Checked the database connectivity OK

• Agent Side Message Queue status is waiting to connect

• MSDN tells me should check if the agent can telnet server side port 1801

• Verified that server side port 1801 cannot be Telnet

• The ntstat –ano shows that 1801 is listening on the loopback ip 127.0.0.1 which is different from my working environment


Case Study1 – Failed to Record Session• Problem Analysis• It obviously is a problem of the MSMQ

• https://support.microsoft.com/en-us/kb/2554746

• Fixed the issue• Document the root

cause


Case Study 2Records files gets deleted


Case Study2 – Recording file gets deletedProblem Symptom:1. This is a newly built

environment2. The user does not get the

record notification window when launch the application

3. When the application is launched , *.icl file is generated

4. Once we close the session, the .icl file gets deleted

• Checked the agent event log first Find the error log from the source “Citrix Session Recording Agent” with the event id 0 , the error is “Exception caught while obtaining session metadata or running record policy query.”

Information Gathering


Case Study2 – Failed to record the session

Another session recording log tells you , it’s using the fallback policy : No not record

Agent configuration check

Verified with customer if the SR Server is correctly configured and if they are using the SSL communication

The answer is no

So here is the misconfiguration issue

Note:If the answer is yes, we need to check if the certificate is installed correctly, and also verify if the certificate is trusted by the agent


Most frequently asked question


FAQ – Disk planning for Session Recording for Server VDA

• Q: How to plan the Session Recording disk?

• A: The example stated in KB CTX200869– Recording 5,000 Outlook sessions over an eight-hour work day consumes about 100GB of

storage space – One Outlook session will consume about 20 MB for an eight-hour work day

• Additionally information is:– As Session Recording is capturing screen update for the ICA Session, also depends on the

graphic content of the application, session resolution and color depth; we suggest to select a group of end-user to perform a pilot run for a week, then we can calculate the requirement of the disk based on the pilot run result.

– As a best practice for session recording, Windows Media Redirection and Flash Redirection are suggested to be enabled in the ICA Session.


Resources


Configuring Security Features of Session Recording


Building a Highly Scalable Session Recording System


Troubleshooting Recording Issues in Citrix SmartAuditor

http://archive.citrite.net/article/CTX114819

Troubleshooting Session Recording http://docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-monitor-article/xad-session-recording/xad-sr-trouble.html

Session Recording FP2 installation video

Server: https://youtu.be/Q47GwgsdW-IAgent: https://youtu.be/pLynm6S9gZk Player: https://youtu.be/ZTfKCuaYWVc

https://youtu.be/Q47GwgsdW-I

https://youtu.be/pLynm6S9gZk

https://youtu.be/ZTfKCuaYWVc


Q & A ?


Work better. Live better.Work better. Live better.

session recording deep dive and troubleshooting final version

Software