Environment

• Network: Mainnet

• Splice: 0.6.7

• Canton: 3.5.1

• Participant: Solv-Mainnet-01

• Validator backend: validator_backend

---

Background

We missed the Mainnet LSU on June 27.

Our timeline is:

• Before June 27:

  • Running Splice 0.5.10

  • No LSU upgrade performed.

• June 29:

  • A CC transfer was submitted from the main account while still running 0.5.10.

  • The transfer failed / never completed.

  • We later realized Mainnet had already completed LSU.

• Afterwards:

  • Upgraded to 0.6.6.

  • Encountered multiple upgrade issues.

  • Upgraded again to 0.6.7.

  • Most issues disappeared.

  • Only validator backend can no longer complete initialization.

Participant itself remains healthy.

---

Current behavior

Participant starts successfully.

Validator backend starts successfully until the Vet Packages phase.

Then it loops forever during:

check_establish_topology_mapping

Eventually initialization never finishes.

---

Important observations

1. Participant is healthy

Participant logs show:

• Admin API healthy

• Ledger API healthy

• PackageService healthy

• Sequencer connection healthy

• No database issues

• No topology write errors

---

2. Current topology state

Participant console:

sandbox.topology.vetted_packages.list(

store = Some(TopologyStoreId.Synchronizer(...)),

filterParticipant = "Solv-Mainnet-01"

)

returns

serial = 29

operation = Replace

packages = …160 more

There is no serial 30.

---

3. No pending topology proposals

sandbox.topology.vetted_packages.list(

proposals = true

)

returns

Vector()

Likewise,

sandbox.topology.transactions.list(

proposals = true

)

returns empty.

Therefore there are no pending topology proposals.

---

4. Authorized store

sandbox.topology.vetted_packages.list(

store = Some(TopologyStoreId.Authorized),

filterParticipant = "Solv-Mainnet-01"

)

returns empty.

So serial 30 never appears in the Authorized store either.

---

5. Package store

Participant package store contains

194 uploaded packages

Current VettedPackages contains

serial 29

about 165 vetted packages

Validator appears to expect

about 190 packages

(validator log prints “…185 more”, which corresponds to roughly 190 total packages).

---

Validator log

Validator repeatedly retries:

check_establish_topology_mapping

Failure:

FAILED_PRECONDITION

Condition is not yet observed

Validator compares:

Proposed:

VettedPackages

participant = Solv-Mainnet-01

packages = …185 more

against

Found:

serial = 29

packages = …160 more

and retries forever.

The retry is specifically:

check established dars … are vetted

using

TopologyManagerReadService/ListVettedPackages

It keeps polling and never progresses.

---

What we did NOT observe

We expected to see something like:

TopologyManagerWriteService

Authorize

ReplaceVettedPackages

TOPOLOGY_TOO_MANY_PENDING_TOPOLOGY_TRANSACTIONS

However, during the retry loop we only see repeated:

ListVettedPackages

calls.

We no longer observe any write request during this phase.

---

Questions

1. Why is validator waiting for a VettedPackages state (serial 30) that does not exist in either:

   • Authorized store

   • Synchronizer store

2. Should validator submit a new ReplaceVettedPackages request at this point, or is it expected to only wait?

3. Is there a supported recovery procedure to force validator to restart the Vet Packages process?

4. Is there a supported way to clear or reset the validator-side Vet Packages / app initialization state?

5. Could this state have been caused by upgrading after LSU while a CC transfer submitted on 0.5.10 was still incomplete?

6. Is there any supported procedure to reconstruct serial 30 from the current package store without manually editing topology?

Any guidance would be greatly appreciated.

I have not reproduced this on Mainnet, so please treat this as a possible diagnosis rather than a confirmed recovery procedure.

From the Splice 0.6.7 code, my understanding is that the validator should generate the next VettedPackages mapping itself. If the current mapping is serial 29, the validator would attempt to submit serial 30 and then poll until that mapping becomes visible in the authorised synchronizer topology state.

One detail that may be relevant is that this mapping appears to be submitted as an authorised transaction rather than as a proposal. Because of that, checking only with proposals=true may not show the transaction you are looking for.

The earlier TOPOLOGY_TOO_MANY_PENDING_TOPOLOGY_TRANSACTIONS error also seems significant. It may indicate that the serial-30 submission was rejected before it could enter the topology outbox, or that the outbox was already full or not draining. I would therefore check:

• the original logs from the first vetting attempt, especially around the topology submission;
• authorised topology transactions with proposals disabled;
• the synchronizer topology outbox and whether it is draining;
• whether the outbox queue limit was reached at the time of the failure.

After confirming that the outbox problem is resolved, restarting the validator may allow it to submit the mapping again. I would avoid manually creating serial 30 or modifying topology database state without guidance from Canton support.

Are there any pending topology-outbox entries, or related rejection errors in the participant logs, around the time the first TOPOLOGY_TOO_MANY_PENDING_TOPOLOGY_TRANSACTIONS error occurred?

Also, any repo where this issue is reproducible could also be helpful ^^