Load & performance testing for Canton apps — early results benchmarking a Token Standard registry
grants-discuss0 messages
- Hi all,
Canton gets sold into regulated finance, where throughput and latency end up in contracts. But when an application team asks *"can we settle N transactions a second at p99 under X ms, and what breaks first?"*, there isn't really a way to find out. Generic tools (k6, JMeter, Gatling) can generate HTTP load, but they have no idea what contention is, or the difference between a transaction that was rejected and one that was merely slow.
I've been building a load/performance harness for Canton applications - explicitly app-level, driving load at an app's DARs through the Ledger API. Not protocol or synchronizer performance; that's Digital Asset's domain and I'm
staying well clear of it.
It works, and I've used it on real code. Sharing early results because I'd rather find out now whether this is useful to anyone.
Benchmarking OpenZeppelin's Canton token template
Measured through the real CIP-0056 interface choice `TransferFactory_Transfer`:
- ~15 transfers/second, p50 ~400 ms, p99 ~1.2 s
- Stops scaling around 22 transfers/s offered
- Their DvP allocation path runs at the same throughput as a plain transfer,
despite doing strictly more work per operation
The interesting part isn't the rate, it's what limits it. Every failure under concurrency is `CONTRACT_NOT_FOUND` - a second transfer reaching an input holding that another had already archived. The bottleneck is input-holding
selection, not the factory contract. Their implementation archives inputs first, deliberately, for a contention guarantee; this measures what that costs when things run in parallel.
For a wallet author that's actionable: your achievable transfer rate is governed by your UTXO selection strategy, not by the registry you're talking to.
One more result that surprised me
Reaching a registry's factory requires explicit disclosure (the factory is signed by the admin, so a wallet can't see it). Attaching that created-event blob to every submission costs about 3% throughput but raises p99 by ~45%
(803 ms -> 1172 ms). The blob is ~576 bytes and rides on every transfer. It shows up in the tail, not the median.
I haven't seen that number published anywhere.
Caveats, up front
All of this is a single-participant sandbox on one laptop. They're floor numbers. What transfers to a real deployment is the *shape* - where the bottleneck is and how the system behaves past it - not the absolute rate. Real
figures need a proper deployment, which is exactly what I'd want funding for.
Also: CIP-0104 traffic cost is plumbed and returns the standard's shape, but a sandbox has no traffic control, so it reports zero. Cost-per-transaction is still an open question.
What I'm actually asking
1. Does anyone have this problem right now? Specifically: a team with a launch or an SLA who needs a capacity answer and doesn't have one.
2. What would you want measured on your app that isn't in the list above?
3. Is anyone already doing this internally? I'd rather join in than duplicate.
Happy to run it against your app and share the results - the workloads are JSON files and the reports are self-contained HTML.
Thanks,dfrnw - Ran it against OpenZeppelin's Canton token template - third-party code... not mine. Numbers in the image
- Hi dfrnw,Thanks for publishing those numbers. As you rightly point out, the limiting factor is usually UTXO selection and contension on the involved wallets. What would be valuable to share with your numbers is the load profile. How many wallets were sending/receiving in parallel, with how much overlap/contention? What's the UTXO selection logic you use on submission? How do the numbers scale with more wallets?Do you intend to publish the load runner you wrote in any way?Kind regards,Bernhard
- Thanks Dr. Bernhard Elsner, you put your finger on exactly what was wrong with what I posted...so I went and measured it.
1. How many wallets, and how much overlapOne sending wallet and one receiving wallet. All 400 holdings belonged to that single sender, so every concurrent submission drew from the same pool. The overlap wasn't partial, it was total: the maximum-contention case, and not a profile anyone would call representative. Concurrency was 8 in the steady-state run, and the ramp offered 4 to 30 submissions/s
Which means the 15/s I published is not the registry's capacity. It is what one wallet sustains while contending with itself, and I should have framed it thatway.
2. UTXO selectionDeliberately naive: a uniform random pick of a single holding from the sender's pool. Each holding is 1000.0 and each transfer is 1.0, so one input always covers the amount - no multi-input selection, no coalescing,
no merge step. I chose that so the number reflected the registry rather than my selection algorithm, but the consequence is the one you'd predict: random selection over a shared pool is close to the worst case for collisions, and CONTRACT_NOT_FOUND dominates every failure at every load level.
3.How it scales with more walletsI hadn't measured this. I have now. Each sender gets its own pool of 150 holdings and its own receiver, so wallets never touch each other's holdings - the only variable is how many independent wallets the load is spread across.
I ran it two ways, because they answer different questions.
Holding offered load constant, more wallets just convert failures into successes, contention goes to zero and nothing is lost, but throughput barely moves, because the offered rate never rose. Scaling offered load with wallet count is the more useful one, and three things fall out of it.
Contention is a property of wallet count, not of the registry: 48.8% down to 6.2%, falling roughly with the number of independent pools.
The same code does 3x the throughput at 8 wallets, 14.6/s to 44.0/s. So ~15/s was never a registry ceiling. I was measuring my own load profile.
But scaling is sublinear and latency rises throughout - p50 triples, p99 nearly doubles - so there is a second ceiling sitting above wallet contention. On a single in-memory participant on one laptop that is almost certainly the participant rather than the registry, and I can't separate the two on this hardware.
What I still haven't varied: pool depth per wallet, transfer size relative to holding size so multi-input selection actually engages, partial overlap between sneders rather than fully disjoint pools, and selection strategy - random versus least-recently-used versus a per-submission reservation. My guess is that last one moves the number more than the choice of registry does, but it is a guess.
So if there is a load profile you'd consider representative of real wallet traffic - wallet count, holdings per wallet, overlap between senders, transfer size distribution... I would much rather measure that than one I invented.
4. Intend to publishYes. I'd planned to open-source it under Apache-2.0. What I'm less sure about is whether the right home is a standalone tool or something closer to the SDK or the Token Standard's conformance tooling - you'd have a far better view on that than I do.
Where I'd most value your steer is direction. I can see three, and they imply quite different amounts of work: a standalone tool teams run in CI against their own app; a benchmark suite publishing comparable numbers across Token Standard implementations, so figures from two teams mean the same thing; or something closer to the SDK, where capacity testing is part of the normal development loop rather than a separate thing people have to go and find.
And if you think it's worth doing properly rather than as a side project, I'd welcome a steer on whether a Development Fund proposal is the right route, and which SIG would be the natural home.
Kind regardsdfrnw - Going deeper on input selection, and it turned out to be more predictable than I expected.
A random pick fails when it lands on a holding the wallet has already spent. Over a run the spent fraction of the pool grows from zero to f, so the average failure rate should be about half of that - and it should not depend on how many submissions are in flight.
contention ~= f / 2 where f = holdings spent / pool size
I put that prediction in the workload generator before running anything, so what follows is a test of it rather than a curve fitted afterwards.Fixing operations and concurrency, varying only pool depth:
Contention halves every time the pool doubles, mean error about two points across an eight-fold range.
The result that changed how I think about this
Holding pool depth constant and raising concurrency from 8 to 32 - four times as many submissions in flight - moved contention from 12.5% to 11.7%.
Essentially nothing. If contention were concurrent submissions colliding with each other, that should have moved sharply. It didn't, because for the most part they are not colliding with each other. They are reaching for coins the wallet has already spent.
That is a different mental model from the one I started with, and I suspect a different one from the one a lot of wallet code is written against.
The law also covers the earlier numbers
The single-wallet figure I posted before - 48.8% - comes from a much deeper turnover regime: that run spent about 80% of its pool, where the model predicts around 40%. The 12.5% here comes from a run that spent 22%, where it predicts 11%.
Same law, four-fold different pool conditions, both inside a couple of points. Which is the useful part: those two numbers looked contradictory and aren't. Contention wasn't varying because of the wallet count or the concurrency, it was varying because of how much of the pool each run burned through.
Extending that: the wallet-count sweep I posted earlier moved two things at once, since spreading the same load across more wallets also reduces how far each wallet's pool is drawn down. I'm re-running it with each arm's pool sized to its own consumption, which separates the two cleanly and should leave wallet count as the only variable.
Contention is also avoidable outright
Same registry, same wallet, same load. The only change is that each in-flight submission gets a distinct holding instead of picking at random:
Reservation takes contention to zero at every concurrency tested, commits all 240 of 240 offered transfers against 208 with random selection, and costs nothing in latency. At concurrency 32 it is actually faster (p99 1902ms down to 1763ms), and the throughput gain widens under pressure: about 5% at concurrency 8, 15% at 32.
What this gives a wallet author
You can estimate your own contention before writing a line of test code:
contention ~= (transfers / holdings) / 2
And you can drive it to zero by reserving an input per in-flight submission rather than picking at random. Pool depth and selection strategy are the two levers, and neither of them is the registry you picked.
Usual caveats: single participant, in-memory, one laptop. The model under-predicts at high turnover, which is where genuine concurrent collisions begin adding on top - so it is a good estimate and a floor, not an identity.
Thanks to @Bernhard Elsner
And still the same open question... if anyone has a load profile they would consider representative of real wallet traffic - wallet count, holdings per wallet, transfer size distribution, overlap between senders - I would rather measure that than keep inventing my own.Kind regardsdfrwn - Two things have changed since I last posted and one of them is a correction I
owe this list.
First, two numbers I published are wrong.
Kevin at K2F Labs, who runs a self-custodial wallet provider and a MainNet validator, went through about twelve months of their data — roughly 1.97M committed transactions and checked my results against production. He was right about two of them.
The claim that explicit disclosure raises p99 latency by ~45% (803ms to 1172ms) does not survive scrutiny. That run was 60 transfers. At n=60 the "p99" is the single worst observation, so what I published was one garbage-collection pause against another, not a measurement of disclosure. The medians were identical at about 400ms and that was the honest result. The rule I now apply throughout is roughly 10 x 100/(100-p) samples before a percentile carries information, about 1,000 for a p90, 10,000 for a p99.
The claim that throughput stops scaling at ~22 transfers/s is also withdrawn but for a different reason: that figure came from a ramp using random input selection and what it measured was the wallet shedding load by colliding with holdings it had already spent. On the same laptop and the same registry, giving each in-flight submission a distinct input commits 240 of 240 at 42.1/s. So ~22/s was never a ceiling of anything and calling it "where scaling stops" understated the registry by roughly half.
Both retractions are documented in the repository rather than tidied away and the card from the original post has been redrawn with both numbers removed
For completeness on your other questions, which I answered on the Canton forum thread: the original run used one sending wallet and one receiver with all holdings in a single pool, so it was one wallet contending with itself rather than registry capacity; selection was a uniform random pick of a single holding and spreading the same load across eight independent wallets took throughput from 14.6 to 44 transfers/s while contention fell from 48.8% to 6.2%.
Second... the work moved from performance to cost and that is the part I think matters
Kevin's point was that at current traffic prices, envelope size dominates the economics of a registry long before latency does — his median is about 35.5 KB of paid traffic per transaction, roughly $2.13 at $60/MB, against application payloads of about 146 bytes.
I had been treating traffic cost as unmeasurable without a metered synchronizer, because CIP-0104 cost estimation reads zero on a sandbox. That was too narrow. The interactive-submission prepare endpoint interprets a command without committing it and hands back the prepared transaction and its size is a real measurement on any participant. I had been discarding it.
Comparing two independent CIP-0056 registries executing the same standard transfer
minimal reference registry, direct completion 11,733 bytes $0.6714
OpenZeppelin template, direct completion 11,446 bytes $0.6549
OpenZeppelin template, two-step propose/accept 13,955 bytes $0.7985
My first reading was "one registry is 19% more expensive" and that was wrong in the same way as before, I was comparing a completed transfer against half of a two-step one. Isolating each registry's direct-completion path separates the two effects and the result inverts:
The settlement model costs roughly ten times more than the data model
Propose/accept versus completing directly is +2,509 bytes, about $0.14 a transfer — and that is a floor, because a two-step transfer still owes the acceptance, which I have not counted. The two registries' data models differ by 287 bytes, in favour of the one carrying more fields.
Stakeholder count is priced linearly on top of that175.5 bytes per party on a create and 437.8 on a transfer both fitting to within about ten bytes across a sixteenfold range. So the rate is a property of the transaction rather than a constant, which is precisely why it needs measuring rather than estimating.
Unlike everything else I have posted here, these figures are exactly reproducible, three runs per configuration, byte-identical every time. Envelope size is not a sample from a noisy distribution; it is a property of the
transaction's shape.
Caveats, since they are load-bearingThese are prepared-transaction sizes, which are a lower bound: the sequenced confirmation request adds encrypted views per informee and mine sit 2.6 to 3.1 times below Kevin's MainNet median, which
is the right direction but is consistency rather than confirmation. Everything is single participant, in-memory, one machine. And $60/MB is a quoted price, not one I measured — the tool takes it as a parameter and records it beside any figure it produces.
I gave OpenZeppelin advance notice before publishing the comparison, since it names their code; the finding is favourable to them and their template is not doing anything wrong — two-step transfer is correct behaviour absent a preapproval. It simply has a price nobody could see.
Anyone willing to point it at their own registry and tell me what came out... particularly if the number looks wrong. Every figure I have is from one laptop against two implementations and the fastest way to find where
the model breaks is for it to break on somebody else's code. It takes a DAR and about three minutes, with no configuration for anything implementing the Token Standard.
Kind regards,
dfrnw