CCPEDIAby Unity Nodes
Mailing Lists/Reminder: MAJOR UPGRADE ON MAINNET WEDNESDAY JUNE 25 9:00 AM US EASTERN (UTC -4)Source on lists.sync.global ↗

Reminder: MAJOR UPGRADE ON MAINNET WEDNESDAY JUNE 25 9:00 AM US EASTERN (UTC -4)

validator-announce1 messagesstarted 24-06-2025
  1. #1Wayne Collier24-06-2025source ↗

    COUNTDOWN TO MAJOR UPGRADE WITH DOWNTIME ON THE GLOBAL SYNCHRONIZER MAINNET

    WHEN:  2025-06-25 13:00 UTC  ( check your local time: https://dateful.com/convert/utc?d=2025-06-25&t=13 )

    AUDIENCE: Validator Operators

    TERMINOLOGY:

    • HDM (Hard Domain Migration) - “Major Upgrade of Global Synchronizer with Downtime”

    ACTIONS REQUIRED:

    • Prior to the date of HDM:

      • Backup the Validator identities

    • On the date of HDM:

      • Wait for the Slack announcement stating that the Global Synchronizer has paused

      • Make sure the Validator has caught up

      • If you operate your own application, pause your app automation / app backend

      • Make sure the migration dump has been automatically created

      • Stop the Validator (v0.3.21)

      • Take backups of the Validator databases (validator db and participant db)

      • Wait for the Slack announcement stating that the Global Synchronizer migration has been completed

      • Update the Validator version (to v0.4.1), configuration (if needed), migration id and set the flag “migrating” to true if you are using helm charts or pass -M if you are using the docker compose deployment

        • Make sure to also adjust the migration id suffix in the DB name of the participant node

        • This means you are keeping the validator database but creating a fresh new database for the participant

      • Verify that the Validator works

      • Set “migrating” flag back to false

      • If you operate your own application with a separate database and automation, reset any off-ledger contract stores, let them reingest from the ACS, and then bring up your application backend again

    • Follow the link if you need more details for each step: https://docs.sync.global/validator_operator/validator_major_upgrades.html

    DEPLOYMENT DETAILS:

    • MIGRATION_ID: 3

    • VERSION: 0.4.1

    DEPLOYMENT SPECIFIC HINTS:

    • Kubernetes deployment hints (Helm-chart)

      • Making sure Validator has caught up and there are no “Ingested transaction” messages generated within the last 10 minutes

    kubectl -n "$VALIDATOR_NAMESPACE" logs deployment/validator-app --since=60m | grep '^{' | jq -rs '[.[] | select(.message | startswith("Ingested transaction"))][-5:][] | ."@timestamp" + " " + (.message[:50] + "..." | tojson) + " " + (."@timestamp"[0:19] + "Z" | fromdate as $ts | now - $ts | . / 60 | . * 10 | round / 10 | tostring + "m ago")'

    • Making sure the migration dump has been created

    kubectl -n "$VALIDATOR_NAMESPACE" logs deployment/validator-app | grep "Wrote domain migration dump"

    • Docker Compose deployment hints

      • Making sure the migration dump has been created

    docker compose logs validator | grep "Wrote domain migration dump"

    FAQ:

    • What should I do if I missed setting the flag “migrating” to true when starting the Validator?

      • Scale down the validator and participant.

      • Delete the participant database for the new migration id, usually named participant_MIGRATION_ID. 

      • Restart the validator and participant with migrating: true

    • What should I do if Validator keeps restarting after migration?

      • check the logs before a crash for warnings and errors to identify the cause

    GENERAL INFORMATION ABOUT MAJOR UPGRADES WITH DOWNTIME

    1. This is a forking upgrade. All Super Validators will adopt a new Canton protocol version on their synchronizer nodes. The result will be a new version of the Global Synchronizer, identified by a new Migration ID: 3. 

    2. This will be the third major upgrade completed on MainNet. Given projections from the Splice and Canton teams, the Super Validators expect to complete two more of these major upgrades with downtime before transitioning to a rolling upgrade process.  

    3. The Super Validators have successfully rehearsed this procedure on DevNet and TestNet earlier this year, giving all Validator node operators an opportunity to also practice.

    4. This upgrade has been successfully completed on DevNet and TestNet in the past three weeks. All Validator nodes on DevNet and TestNet have also had the opportunity to practice it.  

    5. During a Major Upgrade with Downtime, all transactions on the old version of the Global Synchronizer, using the prior Migration_ID, will stop at or just after the time of the Major Upgrade. 

    6. You can find the current Migration_ID, posted by the GSF’s SV node as a reference, by looking here:

    https://sync.global/sv-network/

    1. All transaction processing, including all reward minting, on all Validator nodes will halt at the time of the upgrade. You will not receive rewards for any of the time your node is not communicating. 

    2. Prepare thoroughly. Detailed Technical Docs here:

      1. https://docs.sync.global/validator_operator/validator_major_upgrades.html#validator-upgrades

    3. If you have not practiced this upgrade, there’s a chance you’ll make an error. So please study the docs carefully, and refresh your understanding of disaster recovery as well. 

      1. Disaster recovery docs for Validators

    https://docs.sync.global/validator_operator/validator_disaster_recovery.html

    1. Errors may cause:

      1. A long delay in resuming reward minting 

      2. Loss of access to your data

      3. Failure to recover assets on your node