Environment

• Network: Mainnet

• Splice: 0.6.7

• Canton: 3.5.1

• Participant: Solv-Mainnet-01

• Validator backend: validator_backend

---

Background

We missed the Mainnet LSU on June 27.

Our timeline is:

• Before June 27:

  • Running Splice 0.5.10

  • No LSU upgrade performed.

• June 29:

  • A CC transfer was submitted from the main account while still running 0.5.10.

  • The transfer failed / never completed.

  • We later realized Mainnet had already completed LSU.

• Afterwards:

  • Upgraded to 0.6.6.

  • Encountered multiple upgrade issues.

  • Upgraded again to 0.6.7.

  • Most issues disappeared.

  • Only validator backend can no longer complete initialization.

Participant itself remains healthy.

---

Current behavior

Participant starts successfully.

Validator backend starts successfully until the Vet Packages phase.

Then it loops forever during:

check_establish_topology_mapping

Eventually initialization never finishes.

---

Important observations

1. Participant is healthy

Participant logs show:

• Admin API healthy

• Ledger API healthy

• PackageService healthy

• Sequencer connection healthy

• No database issues

• No topology write errors

---

2. Current topology state

Participant console:

sandbox.topology.vetted_packages.list(

store = Some(TopologyStoreId.Synchronizer(...)),

filterParticipant = "Solv-Mainnet-01"

)

returns

serial = 29

operation = Replace

packages = …160 more

There is no serial 30.

---

3. No pending topology proposals

sandbox.topology.vetted_packages.list(

proposals = true

)

returns

Vector()

Likewise,

sandbox.topology.transactions.list(

proposals = true

)

returns empty.

Therefore there are no pending topology proposals.

---

4. Authorized store

sandbox.topology.vetted_packages.list(

store = Some(TopologyStoreId.Authorized),

filterParticipant = "Solv-Mainnet-01"

)

returns empty.

So serial 30 never appears in the Authorized store either.

---

5. Package store

Participant package store contains

194 uploaded packages

Current VettedPackages contains

serial 29

about 165 vetted packages

Validator appears to expect

about 190 packages

(validator log prints “…185 more”, which corresponds to roughly 190 total packages).

---

Validator log

Validator repeatedly retries:

check_establish_topology_mapping

Failure:

FAILED_PRECONDITION

Condition is not yet observed

Validator compares:

Proposed:

VettedPackages

participant = Solv-Mainnet-01

packages = …185 more

against

Found:

serial = 29

packages = …160 more

and retries forever.

The retry is specifically:

check established dars … are vetted

using

TopologyManagerReadService/ListVettedPackages

It keeps polling and never progresses.

---

What we did NOT observe

We expected to see something like:

TopologyManagerWriteService

Authorize

ReplaceVettedPackages

TOPOLOGY_TOO_MANY_PENDING_TOPOLOGY_TRANSACTIONS

However, during the retry loop we only see repeated:

ListVettedPackages

calls.

We no longer observe any write request during this phase.

---

Questions

1. Why is validator waiting for a VettedPackages state (serial 30) that does not exist in either:

   • Authorized store

   • Synchronizer store

2. Should validator submit a new ReplaceVettedPackages request at this point, or is it expected to only wait?

3. Is there a supported recovery procedure to force validator to restart the Vet Packages process?

4. Is there a supported way to clear or reset the validator-side Vet Packages / app initialization state?

5. Could this state have been caused by upgrading after LSU while a CC transfer submitted on 0.5.10 was still incomplete?

6. Is there any supported procedure to reconstruct serial 30 from the current package store without manually editing topology?

Any guidance would be greatly appreciated.