Inconsistencies with Citrix Cloud Remote PowerShell SDK

Automation is key when doing performance testing. This is required to ensure performance tests are running in a clean state and produces consistent results. For a ICT-R research, additional automation was required in the base automation script for Citrix Cloud. But the behavior of the script was inconsistent using the PowerShell SDK for Citrix Cloud. This blog post will cover the issues encountered during this research.

The goal and flow of the script

Let’s first start with providing some details of the goal of the script. When doing a performance test the environment needs to be in a clean state. This means the hypervisor where the workload will be executed needs to be restarted. Any running machines need to be stopped before the reboot on the hypervisor is initiated. The steps are as follows:

  1. Set the delivery group in maintenance;
  2. Collect all machine from the delivery group;
  3. Invoke power action to all machines in order to shutdown all the machines;
  4. Ensure all machines are down before continuing to the next step;
  5. Reboot the hypervisor;
  6. Ensure the hypervisor is back online;
  7. Disable the maintenance of the delivery group.

This script will be started from the management server where the Remote PowerShell SDK for Citrix Cloud is installed.

The inconsistencies

The script is executed from the management server which contains the latest Citrix Cloud Remote PowerShell SDK at that time which was around December 2018. Connecting to the Citrix Cloud instances and set the correct delivery group in maintenance mode works as expected. But for some reason, it was not able to collect all the machines. It produced the following error:

When creating the Citrix Cloud integration I had validated all the steps and have seen it working. Thanks to a support case with Citrix a working solution was provided. The script needs to be executed on one of the Cloud Connectors. This worked but for other reasons, it is required to run the script from the management server. Therefore an Invoke-Command is added to run the PowerShell commands remotely on one fo the Cloud Connectors which will collect all the machines and ensure they are powered off. Unfortunately, during the script, it got stuck in the Invoke-Command section.

It took a lot of time to troubleshoot this behavior but in the end, it had to do with the remote pipeline towards the Cloud Connector machine. After configuring the WINRM specifying a MaxMemoryPerShell it worked how it should be.

Feedback on the Citrix Cloud Remote PowerShell SDK

When installing a Remote PowerShell SDK you can expect to remotely connect to the environment, as the name implies. For some reason, Citrix requires and recommends to run the scripts from a Cloud Connector which doesn’t make sense. The statement from Citrix support is as follows:

Always run the PowerShell from the cloud connector where all the services are enabled and talking directly to cloud control plane.

Note: please see the update below.

There is also a downside at running the script at the Cloud Connector as it may update because Citrix manages updates to the Cloud Connector software, which may result in downtime. So taking that into account you need to ensure it is executed on the right available connector. While writing this blog post I have validated in the same environment and noticed it’s working. This shows the benefit of Cloud technology as there is a continuous process in improvements and fixes.

UPDATE: Interesting fact, as Martin Zugec pointed out this issue has to do with a mismatch of the Citrix Cloud Remote PowerShell SDK and Citrix Virtual Apps and Desktops service instance. Another important note is Citrix does not recommend to run the script on the Cloud Connectors.

Citrix recommends that you do not run this SDK’s cmdlets on Cloud Connectors. The SDK’s operation does not involve the Cloud Connectors.

Sourcehttps://docs.citrix.com/en-us/citrix-virtual-apps-desktops-service/sdk-api.html

Conclusion

Based on the feedback from Martin, if you encounter similar issues. ensure you are running the correct Citrix Cloud Remote PowerShell SDK. As stated it is not recommended to run the script on the Cloud Connector.

As always it is important to add proper error handling especially as Citrix is in control with pushing updates which may require updated SDK.

If you encounter issues using the Citrix Cloud Remote PowerShell SDK try to run it on a Cloud Connector. But keep in mind the Cloud connectors may update during your script. It is important to understand Citrix manages all the update to the Cloud Connector software, this may result in breaking your script. So, please ensure the script contains error handling and in our case sending a notification to avoid losing valuable time in the ICT-R lab.

Special thanks to Eltjo van Gulik for helping with troubleshooting these issues.

Photo by Jeremy Perkins on Unsplash