Tucson Test Stand

This section contains site specific variations for the Tucson test stand.

Resources

(1) Need to get kubeconfig file from here. File a Jira ticket with Tucson IT for access. Once able to log into Rancher:

  1. Select the pillan cluster.
  2. Click the Kubeconfig File button in top-right.
  3. Near bottom of dialog, click the download link.

Non-Production Systems

The Tucson test stand operates all CSCs and systems on the production domain.

Bare Metal Machines

  • LOVE: love1.tu.lsst.org
  • T&S CSCs: tel-hw1.tu.lsst.org
  • Kubernetes: Can be done from own machine, just need kubeconfig file and kubectl installed.
  • ATCamera (Tony Johnson): auxtel-mcm.tu.lsst.org
  • CCCamera (Tony Johnson): comcam-mcm.tu.lsst.org
  • Calibration Systems (Parker Fagrelius): loonie.tu.lsst.org

LOVE Summary View

The overall system summary state view is called Summary State.

Checking the Number of Federations

This uses a script in https://github.com/lsst-ts/k8s-admin. Run ./feds-check-k8s from a machine with kubectl and the proper kubeconfig file.

Shutdown Camera Services

  • Shutdown Camera OCS Bridges:
    • ATCamera: sudo systemctl stop ats-ocs-bridge.service
    • CCCamera: sudo systemctl stop comcam-ocs-bridge.service
  • Shutdown Camera Daemons
    • sudo systemctl stop opensplice.service
    • Command is the same everywhere.

Shutdown LOVE

This needs to be done from love1.

  • Uses the docker-compose-admin scripts in tucson-teststand/love1 directory, which are linked into the dco user home directory.
    • Become the dco user: sudo -iu dco
    • ./shutdown_love
    • ./shutdown_daemon

Shutdown T&S Bare Metal Services

Handle tel-hw1:

  • Uses the docker-compose-admin scripts in tucson-teststand/tel-hw1 directory, which are linked into the dco user home directory.
    • Become the dco user: sudo -iu dco
    • ./shutdown_atmcs_atp
    • ./shutdown_m1m3
    • ./shutdown_daemon

Handle calibration systems:

Log into the machines listed in that section then stop and remove all running containers.

Interacting with Kubernetes

Commands can be executed from your own machine with kubectl and the pillan.yaml kubeconfig file.

Download from https://rancher.tu.lsst.org/dashboard

Shutdown Main Daemon

The main daemon on TTS runs on Kubernetes. Shut it down by deleting the deployment under the ospl-main-daemon app on Argo CD.

Update Configuration

  • Gather the branch for the configurations and version number for ts_ddsconfig.
  • Uses the docker-compose-admin/tucson-teststand/update_repo script, which is linked into the dco user home directory.
  • Repos to update:
    • docker-compose-ops (love1, tel-hw1)
    • LOVE-integration-tools (love1)
    • ts_ddsconfig (love1, tel-hw1) NOTE: Only necessary if there are updates.
  • Become the dco user: sudo -iu dco
  • ./update_repo <repo path> <branch or version>

Startup Main Daemon

The main daemon on TTS runs on Kubernetes and will be handled by the sync_apps.py script. This will be detailed in the next section

Startup Minimal Kubernetes System

This replaces most of step 6.3 in the main document. Follow the first three bullet points in that step and then continue the process with the next steps.

  • python sync_apps.py -p -t
  • csc-cluster-config, ospl-config and ospl-main-daemon apps will be synced automatically.
  • Once the ospl-main-daemon app is synced, the script will pause.
  • Check the logs on Argo CD UI to see if daemon is ready.
  • Type go and enter to move onto syncing the ospl-daemon app
  • Once the ospl-daemon app is synced, the script will pause.
  • Check the logs on Argo CD UI to see if daemons are ready.
  • Type go and enter to move onto syncing the kafka-producers app.
  • Script will again pause once the kafka-producers are synced.
  • The kafka-producers use a startup probe, so once all of the pods show a green heart, type go and enter to move onto syncing the love app.
  • Once the love app is synced, stop here and return to step 6.4 in the main document.
  • Make sure you leave the script running.

Startup LOVE

This needs to be done from love1.

  • Uses the docker-compose-admin scripts in tucson-teststand/love1 directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding.
    • ./launch_love

Startup Camera Services

This needs to be done from auxtel-mcm and comcam-mcm.

  • Start Camera Daemons
    • sudo systemctl start opensplice.service
    • Command is the same on both machines.
    • Wait for the daemon to be ready before starting the bridge services.
  • Start Camera OCS Bridges:
    • ATCamera: sudo systemctl start ats-ocs-bridge.service
    • CCCamera: sudo systemctl start comcam-ocs-bridge.service
    • Ensure bridge services are running:
      • ATCamera: sudo systemctl status ats-ocs-bridge.service
      • CCCamera: sudo systemctl status comcam-ocs-bridge.service
  • Transition to OFFLINE_AVAILABLE:
    • ATCamera:
      • ccs-shell
      • ccs> set target ats-ocs-bridge
      • ccs> lock
      • ccs> setAvailable
      • ccs> unlock
      • ccs> exit
    • CCCamera:
      • ccs-shell
      • ccs> set target comcam-ocs-bridge
      • ccs> lock
      • ccs> setAvailable
      • ccs> unlock
      • ccs> exit

Startup T&S Bare Metal Services

Handle tel-hw1

  • Uses the docker-compose-admin scripts in tucson-teststand/tel-hw1 directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding.
    • ./launch_atmcs_atp
    • ./launch_m1m3

Enabled CSCs

If proceeding with integration testing, the CSCs will be brought to ENABLED state as part of that process. All of the startup processes maybe necessary for recovering the TTS from any maintenance. In this case, all of the CSCs must be returned to ENABLED state. The following components will automatically transition to ENABLED state when launched:

  • WeatherForecast
  • ScriptQueue:1
  • ScriptQueue:2
  • DSM:1
  • DSM:2

For the other components, leverage the following scripts. Required configurations will be given for each script execution.

Note

Both ATCamera and CCCamera must be in OFFLINE_AVAILABLE state before putting them into ENABLED state.

  • auxtel/enable_atcs.py

    atcamera: Normal

  • auxtel/enable_latiss.py

  • maintel/enable_mtcs.py

    mtm1m3: Default

  • maintel/enable_comcam.py

    cccamera: Normal

  • set_summary_state.py

    data:
  - [MTAirCompressor:1, ENABLED]
  - [MTAirCompressor:2, ENABLED]
  - [LaserTracker, ENABLED]
  - [GenericCamera:1, ENABLED]
  - [GCHeaderService:1 ENABLED]

  • set_summary_state.py

    data:
  - [ESS:1, ENABLED]
  - [ESS:101, ENABLED]
  - [ESS:102, ENABLED]
  - [ESS:103, ENABLED]
  - [ESS:104, ENABLED]
  - [ESS:105, ENABLED]
  - [ESS:201, ENABLED]
  - [ESS:202, ENABLED]
  - [ESS:203, ENABLED]
  - [ESS:204, ENABLED]
  - [ESS:205, ENABLED]
  - [ESS:301, ENABLED]

  • set_summary_state.py

    data:
  - [Scheduler:1, ENABLED]
  - [Scheduler:2, ENABLED]
  - [OCPS:1, ENABLED]
  - [OCPS:2, ENABLED]

  • set_summary_state.py

    data:
  - [Watcher, ENABLED]