PrepareSubmission fails with "unknown package … while looking for template without contract key" for upgraded keyed contracts after a participant restart

Hi all,

We’ve hit what looks like it might be a bug in the interactive submission (external signing) prepare path when using smart contract upgrades with keyed templates. I’d like to know if anyone else has seen it, or knows of a fix or workaround.

Environment

• Canton 3.6.4 (Splice 0.9.4 canton image), LocalNet
• External parties, using the prepare/execute flow with HASHING_SCHEME_VERSION_V3
• Templates with contract keys

Steps to reproduce

1. Upload package my-pkg v1.0.0 and create some keyed contracts with externally signed parties.
2. Upload my-pkg v1.0.1, an SCU-compatible upgrade. Both versions are vetted.
3. Restart the participant.
4. Prepare an exercise on one of the v1.0.0 contracts, using a package-name template ID (#my-pkg:Module:Template) so it resolves to v1.0.1.

Before the restart, everything works. After the restart, every prepare that touches a pre-upgrade keyed contract fails with a 500 / INTERNAL:

Request InteractiveSubmissionService/PrepareSubmission ... failed with INTERNAL/unknown package
6af726c5... while looking for template without contract key 6af726c5...:Module:Template.

java.lang.RuntimeException: unknown package 6af726c5... while looking for template without contract key ...
  at com.digitalasset.canton.interactive.InteractiveSubmissionEnricher.drive$1(InteractiveSubmissionEnricher.scala:67)
  at com.digitalasset.canton.interactive.InteractiveSubmissionEnricher.$anonfun$consumeEnricherResult$1(InteractiveSubmissionEnricher.scala:73)
  ...

Here 6af726c5… is the old (v1.0.0) package. It is stored on the participant (par_daml_packages / par_dars) and still vetted, so the package isn’t actually missing.

What we think is happening

Looking at the 3.6.4 source:

• LedgerApiServer builds a separate Engine for InteractiveSubmissionEnricher. Its package cache is empty after a restart, and it loads packages on demand through packageResolver.
• ExternalTransactionProcessor.lookupAndEnrichInputContracts calls enricher.enrichContract(original, targetPackageIds), where targetPackageIds comes from the transaction nodes. That’s v1.0.1 only.
• EnricherImpl.enrichContractWithPackages pulls only targetPackageIds before running unsafeEnrichCreateWithPackages.
• unsafeEnrichCreateWithPackages enriches the argument against the target package (v1.0.1). The key, however, goes through unsafeEnrichContractKey(create.keyOpt), which uses the original contract’s key. Its globalKey.templateId still points at v1.0.0.
• pkgInterface.lookupTemplateKey(<v1.0.0 template>) then fails, because v1.0.0 was never pulled into this engine. Hence LookupError with Reference.TemplateKey, which renders as “template without contract key …”.

Before a restart, the enricher engine usually already has the old package loaded (from creating those contracts), which hides the problem. As far as I can see, main has the same code.

Questions

1. Has anyone else run into this? Is it a known issue, or already fixed?
2. Is there a recommended workaround?

Happy to share more logs or a minimal reproduction if useful. Thanks!