Upgrading the Interface (XML)#

  1. Coordinate with major stakeholders at the Commissioning Activities Planning (CAP) meeting to discuss the upcoming release.

  2. Determine the summit deployment date and work backwards to set the XML work closure deadline.

    • Consider observing runs and special overnight work at the observatory.

    • Typically allow about two weeks between work closure and deployment.

    • This period may be extended if extra testing or other circumstances require it.

    • See Setting a Schedule for a full Cycle upgrade for further instructions.

    • Once the deployment date is set, run the create_summit_upgrade_ticket script in the vanward package to create the appropriate Jira ticket.

  3. Create a cycle build Confluence page (Software Upgrades) listing core package versions and any operational changes.

  4. Present the schedule at the CAP meeting at least one week before the work closure deadline.

  5. Review the release in the CAP Jira project, identified by the label ts_xml X.Y.

    • Use the release_tickets script from the vanward package.

  6. Confirm all merged work in the ts_xml repository is linked to a Jira ticket in the release.

    • Folks should be following the XML Work Reporting procedure.

    • Use the find_merges_without_release_tickets script in the vanward package.

  7. Send reminders about the work closure deadline at least one day prior AND on the day of the deadline.

  8. Ensure all work tickets are closed when the deadline passes (use the Step 5 script).

  9. Work with the Telescope and Site Build Engineer on the day of the artifact build to go over any potentially open work and sign off on all software versions being used.

Incremental Upgrades to the Interface (XML)#

Incremental upgrades to the interface (XML) are handled similarly to full upgrades, but with some key differences:

  1. They can be requested through Slack in the #cap-software channel as the need arises. The request will then proceed to be discussed in the next CAP meeting.

  2. Since an incremental upgrade does not require a full Cycle build, nor the whole suite of integration tests, the time between the closure of XML work and deployment to the Summit will be shorter than that of a full Cycle upgrade. The schedule will thus depend on the amount of components updated.

  3. Notes detailing the components affected, XML changes and software versions will be added to the current Cycle’s confluence page.

  4. The Jira tickets tracking the relevant changes for the affected components need to be identified and kept in the current release in the Jira CAP project. The people requesting the incremental upgrade are responsible for providing the Jira ticket keys for the work that needs to be included in the upgrade. It is advisable to double check with folks that they have specified all necessary tickets, as it has been the case before that people forget a relevant one.

  5. Once the relevant tickets have been identified, the commit SHAs associated with them need to be found so they can be included in the new release. The The collect_ticket_commits in the vanward can help with that.

  6. All XML changes not included in the incremental upgrade need to be moved to the next release in the CAP project. The move_bucket_ticket_links script in the vanward package can be used to take care of that.

  7. Since an incremental upgrade changes the XML interface only partially, it is important to check that the changes do not break schema compatibility. See Checking Schema compatibility for an Incremental upgrade for further instructions.

Upgrading SAL#

While upgrading SAL usually coincides with an upgrade to the XML, it does not have to be the case. An upgrade for SAL may use the previous cycle’s XML version in order to limit the potential for surprises. The primary developers (Valerie Becker, Igor Suarez-Sola) for SAL are responsible for ensuring the necessary work is completed and the new version is ready.

Upgrading Kafka#

Upgrading the communication backplane via updating the Kafka version requires care and extra lead time. The Kafka oversight committee (Tiago Ribeiro, Michael Reuter) will make the determination if a new version of Kafka is ready for incorporation into a new cycle. This determination requires dedicated testing from the main members of committee to ensure readiness. Cycle builds upgrading Kafka have longer testing periods split into two phases. The first phase builds a smaller section of the control system components and deploys them for testing on the TTS. Work is done to ensure that this small system is operating within the normal parameters. The second phase happens when the full system is built as part of the standard deployment operations.

Setting a Schedule for a full Cycle upgrade#

While below is an example, use your best judgment to set dates and make sure the major stakeholders are informed of the schedule by the CAP meeting.

  • Day 1: Close of release work.

  • Days 2–3: Build artifacts (RPMs/JARs).

  • Days 4–10: Build conda packages and deployment artifacts.

  • Day 13: Initial deployment to BTS.

  • By Day 15 (9 AM PT): All CSCs available on BTS.

  • Days 15–17: Integration testing.

  • Day 20: Summit deployment.

It takes roughly one week from work closure to finishing the deployment artifacts, allowing time to identify and resolve problems. After t

he initial BTS deployment the CSC developers have about 2 days to react to changes in the interface. While it’s ideal to notify folks of these changes in advance, this may not always be possible. Integration testing is limited to three days. Summit deployments always occur at 9 AM summit time on the Tuesday following the BTS deployment.

Checking Schema compatibility for an Incremental upgrade#

  • To check that the changes made to the XML interface for the incremental upgrade are compatible, you will need to start a local Kafka Server and schema registry. You can do so by running:

    docker compose -f {path/to/ts_salobj}/docker-compose.yaml up -d

    This will run the docker-compose.yaml file found in the ts_salobj repo.

  • Create a conda environment and make sure that the branches for the repos ts_xml_ and ts_salobj are installed:

    conda create -n schema-checker-dev pip -y
conda activate schema-checker-dev
cd {/path/to/ts_xml}
pip install -e .
cd {/path/to/ts_salobj}
pip install -e .

  • Now you will need to build the topic_registrar container. You can do by running:

    docker build . --tag ts-dockerhub.lsst.org/topic_registrar:c00{NN}

    Where NN corresponds to the cycle number. The docker file for the container is:

    ARG cycle=c00{NN}
ARG hub=ts-dockerhub.lsst.org
FROM ${hub}/deploy-env:${cycle}
LABEL maintainer="Michael Reuter <mareuter@lsst.org>"
WORKDIR /home/saluser
RUN source /home/saluser/.setup_sal_env.sh && \
    conda install -c lsstts ts-xml={xml_version}
COPY startup.sh /home/saluser/.startup.sh
USER root
RUN chown saluser:saluser /home/saluser/.startup.sh && \
    chmod a+x /home/saluser/.startup.sh
USER saluser

    And startup.sh:

    #!/usr/bin/env bash
source $HOME/.setup_sal_env.sh
create_topics --all

  • Run the following script to register the topics:

    docker run --rm --name topic_registrar \
--env LSST_TOPIC_SUBNAME=chk \
--env LSST_SCHEMA_REGISTRY_URL=http://schema-registry:8081 \
--env LSST_KAFKA_BROKER_ADDR=broker:29092 \
--network kafka \
--platform linux/amd64 \
ts-dockerhub.lsst.org/topic_registrar:c0041

  • To generate the report of all differences found, run the following:

    #!/usr/bin/env bash
export LSST_TOPIC_SUBNAME=chk

    Notice that the scripts will only report when a new topic is created or an old one removed. It will not produce results for variables added to topics.