Skip to content

Dump also protobuf encoding to bulk storage

OPENIssue
by isegall-da21-07-2026

Before we start pruning Scan, we want to persist in bulk storage also the data in its raw format as was in the DB. This is useful also for safety, to make sure we don't lose any data by encoding it in the API encoding, as well as for backfilling from bulk storage as part of onboarding (see pruning design doc).

I believe the easiest way to do this is to refactor SingleAcsSnapshotBulkStorage and UpdateHistorySegmentBulkStorage, which dump a single snapshot/updates-segment respectively.

  • In the refactor, make them take the encoding as the argument (implementation details might vary a bit between the two), as well as the format for the object name (currently, it's just ACS_$i.zstd and updates_$i.zstd, should be instead ACS_${encoding}_$i.zstd and updates_${encoding}_$i.zstd), and whenever we instantiate one of those to dump a snapshot/segment, instantiate two instead, one per encoding.
  • Make sure they don't compete on metrics etc, might need to split some of the metrics as well
  • Also make sure that following steps (updating last-dumped, other metrics etc.) wait correctly until both complete

Other changes in order to to align with that:

  • Make sure the APIs read the objects containing the compact encoding (the existing one)
  • Add internal APIs to read the protobuf encoded objects (could wait until we actually need those when we implement DB backfilling)
  • Adjusts tests to assert that both encodings exist and are correct
  • Make sure the bft check considers all 4 sets of objects (acs+updates x 2 encodings). Hopefully this is a no-op, but needs checking and testing