Multi-party actAs across participants failure?

All, I could use some real help, I’m getting a strange error message and cannot seem to figure this out.

Subject: Multi-party actAs across participants fails NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT even after fixing party multi-hosting – what are we missing?

Setup

Canton 3.5.5 (open-source dpm distribution, dpm 1.0.21, JDK 17), one
Canton process running three participants via docker-compose:

• bank – admin-api :5001, ledger-api :5002, JSON API :7575
• depositor – admin-api :5011, ledger-api :5012, JSON API :7576
• beneficiary – admin-api :5021, ledger-api :5022, JSON API :7577

One local synchronizer, escrow-domain, bootstrapped via
bootstrap.synchronizer_local("escrow-domain"). All three participants
connect via connect_local. Four parties: PrimaryIssuer and
EscrowMediator hosted on bank, Depositor on depositor,
Beneficiary on beneficiary.

Directly confirmed all three participants are on the same synchronizer:

DEBUG-CONN [bank] connected synchronizers: Vector(Synchronizer 'escrow-domain')
DEBUG-CONN [depositor] connected synchronizers: Vector(Synchronizer 'escrow-domain')
DEBUG-CONN [beneficiary] connected synchronizers: Vector(Synchronizer 'escrow-domain')

Minimal reproducer

A JSON Ledger API v2 CreateCommand for a template whose signatory
clause is owner :: issuer :: beneficiaries (three parties, one per
participant), submitted to depositor’s JSON API (:7576) with actAs
listing all three:

POST http://localhost:7576/v2/commands/submit-and-wait-for-transaction
{
  "commands": {
    "commandId": "create-multi-1787926458995020000",
    "actAs": [
      "Depositor::1220d98fd877599dda57f6322895568120162bdb41443784a3c9c092c609a8a1a0ba",
      "PrimaryIssuer::12201a7bb5b49510309a0cdb9b747b1308df42b2b32d1ce471098cea211ab903fa99",
      "Beneficiary::1220316661edbf3df988e2be03f8a2039729f2ab0c4dda1c18357097c6a2c0d8c1bb"
    ],
    "userId": "Depositor",
    "commands": [
      {
        "CreateCommand": {
          "templateId": "<pkgid>:Test.StablecoinEscrowTest:MockHolding",
          "createArguments": {
            "owner": "Depositor::1220d98fd877...",
            "amount": "100.0000000000",
            "issuer": "PrimaryIssuer::12201a7bb5b4...",
            "assetId": "PROBE-1787926458995020000",
            "beneficiaries": ["Beneficiary::1220316661ed..."]
          }
        }
      }
    ]
  }
}

Response:

HTTP 404
{
  "code": "NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT",
  "cause": "Not connected to a synchronizer on which this participant can submit for all submitters",
  "context": {
    "participant": "depositor",
    "unknownSubmitters": "List(bank::12201a7bb5b4..., beneficiary::1220316661ed..., depositor::1220d98fd877...)"
  },
  "errorCategory": 11,
  "grpcCodeValue": 5
}

Note unknownSubmitters lists all three actAs parties, including
depositor::... itself – the party that IS locally hosted on the
participant this request was sent to.

Two-party case that works fine, same topology, submitted to bank’s
JSON API (:7575):

{
  "commands": {
    "actAs": [
      "PrimaryIssuer::12201a7bb5b4...",
      "Depositor::1220d98fd877..."
    ],
    "userId": "PrimaryIssuer",
    "commands": [{ "ExerciseCommand": { "templateId": "<InterfacePkg>:Escrow.Interface:Escrow", "contractId": "...", "choice": "Activate", "choiceArgument": {} } }]
  }
}

This one routes and executes fine (progresses to a different, unrelated
error further downstream) every time. Any combination involving
beneficiary fails identically to the above, every time, regardless of
which participant’s JSON API receives the request or the order of parties
in actAs.

What we’ve tried

1. Confirmed synchronizer connectivity is not the issue (see debug
output above – all three participants connected to the same synchronizer).

2. Found and fixed a real bug in our own bootstrap script. The
original topology-authorization step had each of the three participants
try to unilaterally add itself to every party’s PartyToParticipant
mapping:

// ORIGINAL (broken) -- every cross-participant attempt here silently failed
allMappings.foreach { case (partyId, host) =>
  Seq(bank, depositor, beneficiary).foreach { target =>
    ignore(target.topology.party_to_participant_mappings.propose(
      party = partyId,
      newParticipants = Seq((host.id, ParticipantPermission.Submission)),
      store = TopologyStoreId.Synchronizer(syncId),
      mustFullyAuthorize = true
    ))
  }
}

Canton logged (visible once we stopped swallowing the exception):

TOPOLOGY_UNAUTHORIZED_TRANSACTION(5,...): Topology transaction is missing
authorizations by namespaces=Set(<depositor-namespace>, <beneficiary-namespace>)
and keys=Set()

i.e. bank has no authority to make a topology statement about a party
whose namespace key it doesn’t hold. Makes sense in hindsight – we fixed
this with the real two-sided flow: each party’s home participant
proposes a mapping listing all three desired hosts (mustFullyAuthorize = false), then each additional host independently submits the identical
proposal to add its own countersignature:

val newParticipants = Seq(
  (bank.id, ParticipantPermission.Submission),
  (depositor.id, ParticipantPermission.Submission),
  (beneficiary.id, ParticipantPermission.Submission)
)
ignore(host.topology.party_to_participant_mappings.propose(
  party = partyId, newParticipants = newParticipants,
  store = TopologyStoreId.Synchronizer(syncId), mustFullyAuthorize = false
))
allParticipants.filter(_ != host).foreach { cosigner =>
  ignore(cosigner.topology.party_to_participant_mappings.propose(
    party = partyId, newParticipants = newParticipants,
    store = TopologyStoreId.Synchronizer(syncId), mustFullyAuthorize = false
  ))
}

Verified this actually worked by querying the resulting mappings directly
on all three participants afterward:

DEBUG [bank] party=Beneficiary::1220316661ed... participants=Vector(
  (PAR::bank::12201a7bb5b4...,Submission),
  (PAR::depositor::1220d98fd877...,Submission),
  (PAR::beneficiary::1220316661ed...,Submission))
DEBUG [depositor] party=Beneficiary::1220316661ed... participants=Vector(...)  // identical
DEBUG [beneficiary] party=Beneficiary::1220316661ed... participants=Vector(...)  // identical

Every party, queried from every participant, shows all three participants
as hosts with Submission permission – fully merged, no partial/pending
state, no errors logged anywhere in this step.

3. Despite that, the exact same NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT
persists, byte-for-byte identical error, for any combination involving
beneficiary.

4. Also granted ledger-api user rights in case this was a separate
authorization layer:

depositor.ledger_api.users.rights.grant(id = "Depositor", actAs = Set(depId, cbId, benId))
beneficiary.ledger_api.users.rights.grant(id = "Beneficiary", actAs = Set(benId, cbId, depId))

No change in behavior.

5. Tried settle time – up to 30s between the topology fix taking
effect (confirmed via query) and the actual command submission, in case of
async propagation lag between the topology store and whatever the command
service’s routing check reads. No change.

6. Party order in actAs doesn’t matter. depositor-first vs
bank-first vs beneficiary-first: same error whenever beneficiary is
present.

Question

• Is there a further topology/permission layer beyond PartyToParticipant
  hosting + ledger-api user actAs rights that governs whether a
  participant can be the submitting participant for a multi-party
  actAs command spanning other participants – something at the
  mediator, sequencer, or synchronizer-parameter level we haven’t touched?
• Is NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT’s routing check
  based on something other than PartyToParticipant topology at all? If
  so, what console command lets us inspect/set it?
• Why would bank+depositor succeed as a submitter combination while
  depositor+beneficiary or all three fail identically, when
  PartyToParticipant topology (per direct query, shown above) looks
  completely symmetric across all four parties? Is there something
  specific to beneficiary’s participant/party allocation order (it was
  allocated and topology-authorized last in our bootstrap script) that
  could leave it in a different state despite the query showing otherwise?

Happy to share the complete devnet_init.canton bootstrap script or full
request/response logs if useful.

Versions: Canton 3.5.5, dpm 1.0.21, JDK 17, docker-compose (not a managed
network), single Canton process hosting all three participants + one local
synchronizer.

I spun your setup up locally (three participants, one local synchronizer, Canton 3.5.12 open source) and I’m fairly confident your Beneficiary mapping never actually became authorized. It’s still sitting there as a proposal, and everything else you’re seeing falls out of that.

First thing I’d check is what each participant thinks it hosts:

curl -s localhost:7575/v2/parties | jq '.partyDetails[] | {party, isLocal}'   # bank
curl -s localhost:7576/v2/parties | jq '.partyDetails[] | {party, isLocal}'   # depositor
curl -s localhost:7577/v2/parties | jq '.partyDetails[] | {party, isLocal}'   # beneficiary

isLocal is read off the authorized topology, so on a healthy three-way hosting every party is true on all three ports. In my broken run :7576 gave me:

Beneficiary      isLocal=false     <-- there it is
Depositor        isLocal=true
EscrowMediator   isLocal=true
PrimaryIssuer    isLocal=true

If you see that, that’s your bug.

That would explain the asymmetry too. Routing only checks whether the submitting participant hosts every party in actAs. If PrimaryIssuer and Depositor merged and Beneficiary didn’t, bank hosts the first two, so that command routes and then dies at interpretation on something else (probably your “different error further downstream”). Nothing but the beneficiary participant hosts Beneficiary, so anything naming it fails from bank and depositor both.

Sanity check: send the three-party create to :7577. In my run it goes through.

Order matters, but only for timing. propose(mustFullyAuthorize = false) returns as soon as the local node accepts the proposal. It doesn’t wait for the other two signatures, so the party you authorize last has the least time to aggregate. Beneficiary being last isn’t a coincidence. Took about 600 ms in my run.

From there it’s one of two things. Either the signatures are still in flight, which clears in a second or two. Or it’s stuck for good, because your cosigners proposed payloads that weren’t byte identical, so the signatures land on two different hashes and neither gets enough.

I reproduced the second one by flipping a single countersignature from Submission to Confirmation. Still stuck 60 seconds later. Since you already waited 30s, that’s my bet.

To tell them apart (same API you’re already calling in devnet_init.canton, or TopologyManagerReadService.ListPartyToParticipant over grpcurl on the admin port):

val store = TopologyStoreId.Synchronizer(syncId)

// authorized state only. is serial still 1?
depositor.topology.party_to_participant_mappings.list(
  synchronizerId = store, filterParty = benId.filterString)

// anything still waiting on signatures?
depositor.topology.party_to_participant_mappings.list(
  synchronizerId = store, proposals = true)

One pending row with fewer signatures than hosts means someone didn’t countersign. Two pending rows for the same party at the same serial means rival payloads, which is the permanent version. Either way, sign the existing hash instead of proposing again, otherwise you just add a third rival:

depositor.topology.transactions.authorize(syncId, <tx-hash>)

On your question about extra layers: there’s one, though for your case it’s one command to rule out.

The whole routing check is AdmissibleSynchronizersComputation.canUseSynchronizer. It wants the party hosted on the submitting participant, effective permission at least Submission, and threshold exactly 1. Nothing at the mediator or sequencer level.

The extra layer is ParticipantSynchronizerPermission. BaseTopologySnapshot takes lowerOf(partyPermission, participantAttributes.permission), so a participant the synchronizer capped at Confirmation can’t submit for anyone even though PartyToParticipant still reads Submission. isLocal stays true in that case, so the curl above won’t catch it. Check with:

sequencer1.topology.participant_synchronizer_permissions.list(store)

Empty output means everyone defaults to Submission. I reproduced that cap too and it fails the same routing check with the same error code, so you can’t tell the two cases apart from the message. What does rule it out for you: bank+depositor working from bank means bank itself isn’t capped, and a cap on another participant doesn’t affect what bank can submit. So check it, but hosting is still my bet.

Ledger API user rights are a separate check entirely, they only gate which parties a user is allowed to name in actAs, so step 4 was never going to change anything.

For the bootstrap script itself, drop the settle time and gate on the authorized mapping instead:

def fullyHosted(p: ParticipantReference, party: PartyId): Boolean =
  p.topology.party_to_participant_mappings
    .list(synchronizerId = store, filterParty = party.filterString)
    .exists { r =>
      r.item.threshold == PositiveInt.one &&
      hosts.map(_.id).toSet.subsetOf(
        r.item.participants
          .filter(_.permission == ParticipantPermission.Submission)
          .map(_.participantId).toSet)
    }

hosts.foreach(p => parties.foreach(pty => utils.retry_until_true(fullyHosted(p, pty))))

Cleared in 628 ms for me and all three participants could submit after that. Worth passing an explicit serial to every propose in a cosigning group too, otherwise each node picks its own and a node that’s a step behind signs something nobody else sees.

Last thing, the error message itself is misleading. unknownSubmitters isn’t the list of broken parties, NoSuitableSynchronizer is built from submitters.toSeq so it just echoes your whole actAs. Seeing depositor in there says nothing about depositor’s hosting, which might be what threw you off.

And the names in that list are bank::, beneficiary::, depositor::, which are the participant admin parties, not the PrimaryIssuer:: / Depositor:: / Beneficiary:: you put in actAs. If you didn’t rename those by hand when you pasted, then whatever threw that error isn’t the request you posted, and that’s worth chasing on its own.

---

Source for all of the above is the Canton source at GitHub - digital-asset/canton: Global Workflow Composition that is Scalable, Secure, and GDPR-compliant · GitHub, everything I named is in there if you want to read it yourself.

Setup I reproduced on: Canton 3.5.12 open source (dpm install 3.5.5, dpm 1.0.21), three participants plus one local synchronizer in a single process, in-memory storage, JDK 17. Not your exact 3.5.5, so caveat there, though the routing code is the same at 3.5.7 and on current main so I’d expect 3.5.5 to match.

@crisog thanks so much for your insights and for taking the time to spin up our setup locally and dig into this. Really appreciated, and useful even though it turned out our specific case had a little different history than the isLocal/mapping-authorization theory.

A quick update on where we landed, since I know you were curious about the CREATE-vs-EXERCISE asymmetry too: we’d actually already root-caused and fixed the underlying NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT issue a few days before your reply (2026-08-28). It turned out to be a template-design anti-pattern on our side, not a topology-hosting or timing problem.

One of our test fixtures had a 3-way co-equal signatory owner :: issuer :: beneficiaries clause with no prior joint history between the parties, which requires an atomic multi-actAs CREATE across genuinely disjoint home participants, and as you can imagine, that really doesn’t route on real Canton, confirmed for 2- and 3-party combinations alike, CREATE only (the same multi-party actAs on an EXERCISE against an already-existing contract routes fine). Switching to signatory owner, observer issuer :: beneficiaries (since those parties never actually needed choice authority, just visibility) resolved it completely.

So when your reply arrived, our bootstrap’s own live topology query had already confirmed full multi-hosting with Submission permission on all three participants for every party which was why the specific “Beneficiary never got authorized” diagnosis didn’t match what we were seeing on our side by then.

That said, your diagnostic writeup was still genuinely useful, and we applied two pieces of it as real hardening to our devnet_init.canton bootstrap script:

1. Explicit serial pinning on the two-sided propose/countersign topology calls in our multi-hosting step. We were relying on Canton to auto-negotiate serials across three near-simultaneous submissions, which is exactly the rival-payload race you described! a node a step behind signing a payload nobody else converges on.
   1. We now compute one serial per party (existing + 1, or 1 if new) and share it across the home participant’s proposal and every cosigner’s countersign.
2. A real fullyHosted convergence check, replacing a check that only verified a party’s ID existed somewhere in the topology store (true well before full multi-hosting actually merges) plus a blind “settle time” sleep as a band-aid for that gap. It now checks threshold==1 and all three participants present at Submission permission closes exactly the class of “reports synchronized mid-race” bug you were describing.

Live-verified both these “future issues” on a clean run: the new poll actually caught a real transient non-convergence mid-run (one “waiting for 1 parties to fully converge” cycle) before resolving correctly, and the whole bootstrap completed clean with the explicit serials in place, no compile or routing errors.

Appreciate you writing up the Canton source-level detail (the AdmissibleSynchronizersComputation/ParticipantSynchronizerPermission layering especially), that’s exactly the kind of thing that’s hard to find without reading the engine itself, and it’ll save us time on any future topology issue in this repo even though it wasn’t the cause this time around.

I really do appreciate your help! 5stars!