Upgrades#

Use this page as the operational checklist for a control-system deployment or an OS/K8s maintenance window. The detailed Control System Upgrade guide remains the source of truth for builds, configurations, camera work, Kafka, and exceptional recovery.

An upgrade changes CSC interfaces provided by ts_xml. A full upgrade is a Cycle release; an incremental upgrade changes a limited, schema-compatible set of interfaces between Cycle releases. OS/K8s maintenance is not an interface upgrade, but it requires the same careful shutdown and recovery.

Before you begin#

  1. Confirm the deployment schedule, scope, affected CSCs, and system owners.

  2. Ensure local k8s-admin and vanward checkouts are current.

  3. Verify access to the site’s Kubernetes cluster, LOVE, Argo CD, Argo Workflows, Chronograf, and Slack channel.

  4. For a release, confirm configuration changes are merged. For an incremental XML release, confirm schema compatibility.

  5. Announce in the site Slack channel with the appropriate vanward helper: release_announcement for a full release or incremental_release_announcement for an incremental release. Announce the day before, one hour before, and at deployment start.

  6. Select the site, then follow exactly one checklist below.

BTS uses the Base test stand (manke Kubernetes context). Notify #base-teststand before starting.

Full upgrade

  1. Shut down the control system in LOVE by running system_wide_shutdown.py from a ScriptQueue. Bring the Watcher down first and ScriptQueues down last. Verify CSC states and VM heartbeats in LOVE and Chronograf.

  2. Stop the camera bridges:

    # On auxtel-mcm.ls.lsst.org
sudo systemctl stop ats-ocs-bridge.service

# On lsstcam-mcm.ls.lsst.org
sudo systemctl stop ocs-bridge.service

  3. Drain LOVE-producer and Telegraf consumer lag and clean up deployments. Set the BTS Kubernetes context, run cleanup_all from k8s-admin, clean up Nublado if required, and scale Telegraf connectors to zero:

    ./cleanup_all
./cleanup_nublado  # only when required
kubectl scale deploy -n sasquatch \
  --selector app.kubernetes.io/name=sasquatch-telegraf --replicas=0

  4. Coordinate configuration and camera updates with the responsible teams. Use the detailed guide for schema-registry compatibility or camera-stage changes.

  5. In Argo CD, sync LOVE first to start the LOVE producers required for CSCs to appear in LOVE. Then sync science-platform, nublado, and sasquatch if needed, then telescope applications, starting with a small application.

  6. Start the camera bridges:

    sudo systemctl start ats-ocs-bridge.service
sudo systemctl start ocs-bridge.service

  7. Confirm CSCs reach STANDBY (except expected enabled CSCs), restore required enabled CSCs scale Telegraf connectors back to one, and verify EFD ingestion in Chronograf.

  8. Run minimal testing for AuxTel and SimonyiTel when applicable. Return each tested telescope to its documented safe end state.

Incremental upgrade

  1. Send the Watcher, every affected CSC, and the ScriptQueues to OFFLINE with set_summary_state.py in LOVE.

  2. Set the BTS Kubernetes context and delete jobs only for affected CSCs:

    kubectl delete job -n <namespace> -l csc-class=<csc-class>

  3. Update the necessary configuration, then sync LOVE first, followed by the ScriptQueues and affected applications in Argo CD.

  4. Confirm new topics, if any, were created; cycle affected CSCs through their states; and confirm expected topic traffic with no FAULT.

  5. If a telescope CSC changed, verify it can track and take images and that the images are ingested.

OS/K8s maintenance

  1. Bring down the control system as for a full upgrade, including camera bridges, stopped-job cleanup, drained consumer lag, and Telegraf at zero replicas.

  2. From k8s-admin, run ./cleanup_all and then ./shutdown_kafka. Monitor the shutdown, tell IT when the system is ready, and wait for maintenance to finish.

  3. From k8s-admin, run ./start_kafka. Verify Kafka brokers, controllers, schema registry, and Telegraf are healthy.

  4. Sync LOVE first, then telescope namespaces in Argo CD; restart camera bridges; and confirm CSC and EFD health.

  5. Run BTS minimal testing for both telescopes.

TTS uses the Tucson test stand (pillan Kubernetes context). Notify #tucson-teststand before starting.

Full upgrade

  1. Shut down the control system in LOVE by running system_wide_shutdown.py from a ScriptQueue. Bring the Watcher down first and ScriptQueues down last. Verify CSC states and VM heartbeats in LOVE and Chronograf.

  2. Stop the camera bridges:

    # On auxtel-mcm.tu.lsst.org
sudo systemctl stop ats-ocs-bridge.service

# On comcam-mcm.tu.lsst.org
sudo systemctl stop comcam-ocs-bridge.service

  3. Drain LOVE-producer and Telegraf consumer lag and clean up deployments. Set the TTS Kubernetes context, run ./cleanup_all from k8s-admin, clean up Nublado if required, and scale Telegraf connectors to zero.

  4. Coordinate configuration and camera updates with responsible teams. Use the detailed guide for schema-registry compatibility or camera-stage changes.

  5. In Argo CD, sync LOVE first to start the LOVE producers required for CSCs to appear in LOVE. Then sync science-platform, nublado, and sasquatch if needed, then telescope applications, starting with a small application.

  6. Start the camera bridges:

    sudo systemctl start ats-ocs-bridge.service
sudo systemctl start comcam-ocs-bridge.service

  7. Confirm CSCs reach expected states, restore required enabled CSCs, scale Telegraf connectors back to one, and verify EFD ingestion.

  8. Perform TTS minimal testing. Current documentation states that neither camera can take images at TTS, so verify tracking only.

Incremental upgrade

  1. Send the Watcher, every affected CSC, and the ScriptQueues to OFFLINE with set_summary_state.py in LOVE.

  2. Set the TTS Kubernetes context and delete jobs only for affected CSCs:

    kubectl delete job -n <namespace> -l csc-class=<csc-class>

  3. Update the necessary configuration, then sync LOVE first, followed by the ScriptQueues and affected applications in Argo CD.

  4. Confirm new topics, if any, were created; cycle affected CSCs through their states; and confirm expected topic traffic with no FAULT.

  5. If a telescope CSC changed, verify it can track. Do not require imaging until the TTS camera limitation is resolved.

OS/K8s maintenance

  1. Bring down the control system as for a full upgrade, including camera bridges, stopped-job cleanup, drained consumer lag, and Telegraf at zero replicas.

  2. From k8s-admin, run ./cleanup_all and then ./shutdown_kafka. Monitor the shutdown, tell IT when the system is ready, and wait for maintenance to finish.

  3. From k8s-admin, run ./start_kafka. Verify Kafka brokers, controllers, schema registry, and Telegraf are healthy.

  4. Sync LOVE first, then telescope namespaces in Argo CD; restart camera bridges; and confirm CSC and EFD health.

  5. Verify tracking for both TTS telescopes.

The Summit uses the yagan Kubernetes context. Notify #summit-announce before starting.

Important

Before a full upgrade or OS/K8s maintenance, coordinate with the hardware teams: M2 must be in closed-loop control, MTM1M3TS must remain ENABLED, and MTM1M3 must remain DISABLED.

Full upgrade

  1. Shut down the control system in LOVE by running system_wide_shutdown.py from a ScriptQueue, using the documented Summit exclusions for MTM1M3 and MTM1M3TS. Bring the Watcher down first and ScriptQueues down last. Verify CSC states and VM heartbeats in LOVE and Chronograf.

  2. Stop the ATCamera and MTCamera bridges. Stop additional Summit bare-metal T&S services as required.

  3. Drain LOVE-producer and Telegraf consumer lag and clean up deployments. Set the Summit Kubernetes context, run ./cleanup_all from k8s-admin, clean up Nublado if required, and scale Telegraf connectors to zero.

  4. Coordinate camera, bare-metal, ESS-controller, and cRIO updates with their owners.

  5. In Argo CD, sync LOVE first to start the LOVE producers required for CSCs to appear in LOVE. Then sync science-platform, nublado, and sasquatch if needed, then telescope applications, starting with a small application.

  6. Start camera and required bare-metal services. Confirm CSCs reach expected states, restore required enabled CSCs, restore Telegraf to one replica, and verify EFD ingestion.

  7. Run minimal testing for affected telescopes, including image-ingest checks where imaging is performed. Return each telescope to its documented safe end state.

Incremental upgrade

  1. Send the Watcher, every affected CSC, and the ScriptQueues to OFFLINE with set_summary_state.py in LOVE.

  2. Set the Summit Kubernetes context and delete jobs only for affected CSCs:

    kubectl delete job -n <namespace> -l csc-class=<csc-class>

  3. Update the necessary configuration, then sync LOVE first, followed by the ScriptQueues and affected applications in Argo CD.

  4. Confirm new topics, if any, were created; cycle affected CSCs through their states; and confirm expected topic traffic with no FAULT.

  5. If a telescope CSC changed, verify it can track and take images and that the images are ingested.

OS/K8s maintenance

  1. Bring down the control system as for a full upgrade, preserving the M2 and M1M3 states above. Do not shut down Summit ESS controllers for routine OS/K8s work.

  2. From k8s-admin, run ./cleanup_all and then ./shutdown_kafka. Monitor the shutdown, notify IT, and wait for maintenance to finish.

  3. From k8s-admin, run ./start_kafka. Verify Kafka brokers, controllers, schema registry, and Telegraf are healthy.

  4. Sync LOVE first, then telescope namespaces in Argo CD. Start documented Summit cRIO services and restart CSCs on azar03.cp.lsst.org.

  5. Restore Telegraf to one replica, verify CSC and EFD health, and run Summit minimal testing for both telescopes.

Further detail#

Use the detailed deployment activities and its site-specific sections for recovery steps, Kubernetes access, camera coordination, and minimal-test configurations. Report any CSC that cannot be restored promptly in the appropriate site Slack channel before ending the deployment.