Validator missed June 27 LSU, manual roll-forward done, now sequencers reject with MemberDisabled
Hi all,
Our MainNet validator (party hint: aurpay-validator-1) was running Splice 0.5.12 and missed the June 27 LSU. It lost sync at the topology freeze and we only noticed recently.
What we did so far, following the recovery instructions that were shared for this LSU:
- Upgraded the node to 0.6.14 (docker compose deployment, migration id 4).
- Connected a Canton console and ran perform_manual_lsu from global-domain::…::34-0 to global-domain::…::35-5.
- This worked. The participant now runs on ::35-5 and connects to the serial-5 BFT sequencer pool, the connection pool reaches Ok() with trust threshold 5.
But then we hit a wall that the recovery doc does not cover. Every sequencer rejects our subscription with:
FAILED_PRECONDITION / MemberDisabled(PAR::aurpay-validator-1::122027d4cc22b12ccc9cfcf620692d630bd2eddd34ce93fe40697d71f6e1fbfa30c3)
After that the participant crashes with exitOnFatalError (SYNC_SERVICE_SYNCHRONIZER_DISCONNECTED) and docker restarts it in a loop. It looks like our participant was disabled on the new synchronizer after being absent for about 7 weeks.
Our questions:
- Is there any way to re-enable a disabled member on the global synchronizer, or is this permanent?
- If it is permanent, is the correct path the documented “Recovery from an identities backup” flow (new participant ID, restore identities dump, recover balances)? Anything special to watch out for in our case?
- For that flow we would need a new MainNet onboarding secret. What is the right way to request one?
We have full backups (participant database and identities dump). Happy to share logs. Thank you.