Historywork Subsystem Technical Summary

The historywork subsystem implements the concrete work units (tasks) for stellar-core's history archive interactions. It provides the building blocks for publishing ledger history to archives and downloading/verifying history during catchup. All classes inherit from the Work/BasicWork/BatchWork framework defined in src/work/.

---

Base Infrastructure

RunCommandWork (RunCommandWork.h/cpp)

Inherits: BasicWork

Base class for all work units that execute external shell commands via ProcessManager. Subclasses override getCommand() to return a CommandInfo (command string + optional output file path). The work spawns a process, enters WORK_WAITING, and wakes up via an async callback on ProcessExitEvent when the process completes.

Key functions:

• onRun(): If not done, calls getCommand(), spawns a process via mApp.getProcessManager().runProcess(), and installs an async callback that sets mDone/mEc and calls wakeUp().
• onReset(): Clears done state, error code, and exit event.
• onAbort(): Attempts tryProcessShutdown() on the running process.
• getCommand(): Pure virtual — returns CommandInfo{command, outFile}.

Key data:

• mDone (bool): Whether the process has exited.
• mEc (asio::error_code): Exit status of the process.
• mExitEvent (weak_ptr<ProcessExitEvent>): Handle to the running process.

CommandInfo (RunCommandWork.h)

Simple struct holding mCommand (shell command string) and mOutFile (optional output file path for redirected output).

Progress (Progress.h/cpp)

Utility function fmtProgress(app, task, range, curr) that formats a human-readable progress string like "downloading ledger files 5/10 (50%)" based on checkpoint frequency and a LedgerRange.

---

File Transfer Operations (Low-level)

GetRemoteFileWork (GetRemoteFileWork.h/cpp)

Inherits: RunCommandWork

Downloads a single file from a history archive. If no specific archive is provided (mArchive == nullptr), selects a random readable archive on each run/retry via HistoryArchiveManager::selectRandomReadableHistoryArchive().

Key functions:

• getCommand(): Resolves the archive (random or specified), calls mCurrentArchive->getFileCmd(remote, local) to get the shell download command.
• onSuccess(): Records bytes downloaded to metrics.
• onFailureRaise(): Records failure metric and logs a warning identifying the archive.

Key data:

• mRemote, mLocal: Source and destination paths.
• mArchive: Fixed archive (or null for random selection).
• mCurrentArchive: The archive actually used for the current attempt.
• mFailuresPerSecond, mBytesPerSecond: Medida metrics.

PutRemoteFileWork (PutRemoteFileWork.h/cpp)

Inherits: RunCommandWork

Uploads a single file to a history archive using mArchive->putFileCmd(local, remote). Requires a non-null archive with put capability. Retries RETRY_A_LOT.

MakeRemoteDirWork (MakeRemoteDirWork.h/cpp)

Inherits: RunCommandWork

Creates a directory on a remote archive via mArchive->mkdirCmd(dir). If the archive has no mkdir command, the command string is empty and the work succeeds immediately. Retries RETRY_A_LOT.

GzipFileWork (GzipFileWork.h/cpp)

Inherits: RunCommandWork

Compresses a local file using gzip. Supports a keepExisting mode that uses gzip -c and redirects to an output file. On reset, removes the .gz file.

GunzipFileWork (GunzipFileWork.h/cpp)

Inherits: RunCommandWork

Decompresses a .gz file using gzip -d. Supports keepExisting mode. Defaults to RETRY_NEVER. On reset, removes the decompressed file.

---

Composite Download Operations

GetAndUnzipRemoteFileWork (GetAndUnzipRemoteFileWork.h/cpp)

Inherits: Work

Two-phase work: downloads a gzipped file from a history archive then gunzips it locally. Orchestrates GetRemoteFileWork → file validation (rename .gz.tmp to .gz) → GunzipFileWork.

Key functions:

• doWork(): Three-state machine: (1) spawn GetRemoteFileWork, (2) on download success, validate file and spawn GunzipFileWork, (3) check gunzip result and verify .nogz file exists.
• validateFile(): Renames .gz.tmp → .gz, checking existence at each step.
• doReset(): Removes all local file variants (.nogz, .gz, .gz.tmp).
• onSuccess(): Notifies LedgerApplyManager::fileDownloaded().
• onFailureRaise(): Logs potential archive corruption.
• getArchive(): Returns the archive used if download succeeded.

Key data:

• mFt (FileTransferInfo): Describes the file being transferred (type, paths, checkpoint).
• mArchive: Optional fixed archive.
• mGetRemoteFileWork, mGunzipFileWork: Child work references.

BatchDownloadWork (BatchDownloadWork.h/cpp)

Inherits: BatchWork

Downloads a range of checkpoint files of a given FileType (ledger headers, transactions, results, SCP messages). Iterates over a CheckpointRange, yielding one GetAndUnzipRemoteFileWork per checkpoint. BatchWork manages parallelism.

Key functions:

• yieldMoreWork(): Creates a GetAndUnzipRemoteFileWork for the next checkpoint in range, advances mNext.
• hasNext(): Returns true if mNext < mRange.limit().
• resetIter(): Resets mNext to mRange.mFirst.

Key data:

• mRange (CheckpointRange): The range of checkpoints to download.
• mNext (uint32_t): Next checkpoint to yield.
• mFileType (FileType): Type of history files to download.
• mDownloadDir (TmpDir ref): Local temp directory for downloads.

---

Bucket Download & Verification

DownloadBucketsWork (DownloadBucketsWork.h/cpp)

Inherits: BatchWork

Downloads, verifies, and adopts all bucket files needed for catchup. Handles both LiveBucket and HotArchiveBucket types via a templated inner BucketState<BucketT> struct. Each bucket goes through a three-step sequence: download → verify+index → adopt.

Key functions:

• yieldMoreWork(): For each bucket hash, creates a WorkSequence of: GetAndUnzipRemoteFileWork → VerifyBucketWork<BucketT> → WorkWithCallback (adopt). Iterates live buckets first, then hot archive buckets.
• prepareWorkForBucketType<BucketT>(): Template helper that creates the verify work and the adopt callback, managing index storage and mutex locking.
• onSuccessCb<BucketT>(): Static callback that extracts the verified index, calls BucketManager::adoptFileAsBucket, and stores the result in the output map.

Key data:

• BucketState<BucketT>: Inner template struct containing:
  • buckets: Reference to output map of hash→Bucket.
  • hashes: Vector of bucket hashes to download.
  • nextIter: Iterator tracking progress.
  • indexMap: Map of ID→index pointer, used for ownership transfer between verify and adopt steps.
  • mutex: Protects concurrent access to buckets and indexMap.
  • indexId: Monotonic counter for indexMap keys.
• mLiveBucketsState, mHotBucketsState: Separate state for each bucket type.

VerifyBucketWork<BucketT> (VerifyBucketWork.h/cpp)

Inherits: BasicWork (template class)

Verifies a bucket file's SHA-256 hash and builds its index, running on a background thread. Template instantiated for LiveBucket and HotArchiveBucket.

Key functions:

• onRun(): If not done, calls spawnVerifier() and returns WORK_WAITING.
• spawnVerifier(): Checks bucket size against MAX_HISTORY_ARCHIVE_BUCKET_SIZE, then posts work to background thread. Background thread calls createIndex<BucketT>() (which also computes the hash via a SHA256 hasher), then posts result back to main thread setting mIndex, mEc, mDone.
• onFailureRaise(): Calls mOnFailure callback if set.

Key data:

• mBucketFile (string): Path to the bucket file.
• mHash (uint256): Expected hash.
• mIndex (shared_ptr ref): Output index pointer, written by the background verifier.
• mOnFailure (OnFailureCallback): Called on verification failure for logging.
• mDone (bool), mEc (error_code): Completion status.

---

Transaction Result Verification

VerifyTxResultsWork (VerifyTxResultsWork.h/cpp)

Inherits: BasicWork

Verifies transaction results for a single checkpoint by comparing txSetResultHash in ledger headers against computed SHA-256 hashes of transaction result sets. Runs verification on a background thread.

Key functions:

• onRun(): Posts verifyTxResultsOfCheckpoint() to background thread. On completion, posts result back to main thread.
• verifyTxResultsOfCheckpoint(): Opens ledger header and result XDR files, iterates through all headers in the checkpoint, loads corresponding result sets, and verifies each hash matches.
• getCurrentTxResultSet(): Reads from the result XDR stream, validates ledger is within checkpoint range and monotonically increasing.

Key data:

• mDownloadDir (TmpDir ref): Directory containing downloaded files.
• mCheckpoint (uint32_t): The checkpoint being verified.
• mHdrIn, mResIn (XDRInputFileStream): Streams for header and result files.
• mLastSeenLedger (uint32_t): Tracks monotonic ordering of result entries.

DownloadVerifyTxResultsWork (DownloadVerifyTxResultsWork.h/cpp)

Inherits: BatchWork

Batch work that downloads and verifies transaction results for a range of checkpoints. Each checkpoint yields a WorkSequence of GetAndUnzipRemoteFileWork (results) → VerifyTxResultsWork.

---

History Archive State

GetHistoryArchiveStateWork (GetHistoryArchiveStateWork.h/cpp)

Inherits: Work

Downloads and parses a HistoryArchiveState (HAS) JSON file from an archive. The HAS describes the current state of an archive including its latest ledger and bucket list references.

Key functions:

• doWork(): Spawns GetRemoteFileWork to download the HAS file; on success, calls mState.load(mLocalFilename) to parse the JSON.
• getHistoryArchiveState(): Accessor (only valid after WORK_SUCCESS).
• getRemoteName(): Returns either the well-known path (seq==0) or a ledger-specific path.
• onSuccess(): Optionally reports metrics via LedgerApplyMananger::historyArchiveStatesDownloaded().

Key data:

• mState (HistoryArchiveState): Parsed result.
• mSeq (uint32_t): Target ledger sequence (0 = latest/well-known).
• mArchive: Archive to fetch from (null = random).
• mLocalFilename (string): Temp local file path (random hex name).

PutHistoryArchiveStateWork (PutHistoryArchiveStateWork.h/cpp)

Inherits: Work

Serializes and uploads a HistoryArchiveState to an archive. Validates that the HAS contains valid buckets before publishing. Uploads to both the ledger-specific path and the well-known path (/.well-known/stellar-history.json).

Key functions:

• doWork(): Saves HAS to local file, then calls spawnPublishWork().
• spawnPublishWork(): Creates two parallel WorkSequences: one for the seq-specific path and one for the well-known path. Each sequence is MakeRemoteDirWork → PutRemoteFileWork.

---

Publishing Pipeline

ResolveSnapshotWork (ResolveSnapshotWork.h/cpp)

Inherits: BasicWork

Waits for a StateSnapshot's bucket futures to resolve. Delays one ledger past the snapshot ledger (unless standalone) to guard against publishing divergent data.

Key functions:

• onRun(): Calls prepareForPublish() and resolveAnyReadyFutures() on the snapshot. If all futures are resolved and we're past the conservative delay, returns WORK_SUCCESS. Otherwise sets up a 1-second polling wait.

WriteSnapshotWork (WriteSnapshotWork.h/cpp)

Inherits: BasicWork

Writes SCP messages from a StateSnapshot to local files. Runs on a background thread if DB connection pooling is available, otherwise on the main thread via postOnMainThread.

Key functions:

• onRun(): Posts a lambda that calls mSnapshot->writeSCPMessages(). On completion, posts back to main thread setting mDone and mSuccess.

PutSnapshotFilesWork (PutSnapshotFilesWork.h/cpp)

Inherits: Work

Three-phase orchestrator for uploading a snapshot to all writable archives:

1. Get archive states: Spawns GetHistoryArchiveStateWork for each writable archive to learn what files they already have.
2. Gzip files: Compresses only the files that differ between the snapshot and each archive's current state (avoids redundant uploads). Uses StateSnapshot::differingHASFiles().
3. Upload: For each archive, spawns a WorkSequence of PutFilesWork → PutHistoryArchiveStateWork.

Key data:

• mGetStateWorks: List of archive state download works.
• mGzipFilesWorks: List of gzip works for differing files.
• mUploadSeqs: List of upload work sequences.
• mFilesToUpload: Map of local path → FileTransferInfo (deduplicates across archives).

PutFilesWork (PutFilesWork.h/cpp)

Inherits: Work

Uploads all differing files for a single archive. For each file from mSnapshot->differingHASFiles(remoteState), creates a WorkSequence of MakeRemoteDirWork → PutRemoteFileWork.

PublishWork (PublishWork.h/cpp)

Inherits: WorkSequence

Top-level publish work that wraps a sequence of publish steps. On success or failure, notifies HistoryManager::historyPublished() with the ledger number and bucket hashes. Stores mOriginalBuckets separately because the snapshot's bucket list may change during async execution.

---

Verification & Integrity Checking

CheckSingleLedgerHeaderWork (CheckSingleLedgerHeaderWork.h/cpp)

Inherits: Work

Offline self-check: downloads the checkpoint file containing a given LedgerHeaderHistoryEntry, scans it, and verifies the archive copy matches the expected local copy. Used by the offline self-check command.

Key functions:

• doWork(): Downloads checkpoint via GetAndUnzipRemoteFileWork, then synchronously scans the XDR file comparing each header against mExpected.

Key data:

• mExpected (LedgerHeaderHistoryEntry): The expected header to verify.
• mArchive: The archive to check against.
• mCheckSuccess, mCheckFailed: Medida metrics.

WriteVerifiedCheckpointHashesWork (WriteVerifiedCheckpointHashesWork.h/cpp)

Inherits: BatchWork

Produces a JSON file of verified [ledger_seq, hash] pairs by downloading ledger header files and running VerifyLedgerChainWork on them in a chained fashion. Works backwards from a trusted mRangeEnd toward genesis (or a fromLedger/latestTrustedHashPair if specified).

Key functions:

• yieldMoreWork(): For each batch, creates a WorkSequence of BatchDownloadWork (ledger headers) → ConditionalWork wrapping VerifyLedgerChainWork. Each VerifyLedgerChainWork depends on the previous one's verified hash output via a shared_future<LedgerNumHashPair>.
• startOutputFile() / endOutputFile(): Manage the JSON output file lifecycle. If a trustedHashFile is provided, its content is appended to the output.
• loadHashFromJsonOutput() / loadLatestHashPairFromJsonOutput(): Static helpers to read back hashes from the JSON output.

Key data:

• mRangeEnd (LedgerNumHashPair): The trusted endpoint (highest ledger).
• mRangeEndPromise / mRangeEndFuture: Promise/future pair providing the trusted hash to the first link in the verification chain.
• mCurrCheckpoint (uint32_t): Current iteration point, decreasing toward genesis.
• mPrevVerifyWork: Previous VerifyLedgerChainWork, whose output future feeds the next batch.
• mNestedBatchSize: Controls inner parallelism (default 64 checkpoints per batch).
• mTmpDirs: Vector of (WorkSequence, TmpDir) pairs; TmpDirs are cleaned up as sequences complete.
• mOutputFile: Shared output stream written by VerifyLedgerChainWork instances.
• mTrustedHashPath, mLatestTrustedHashPair, mFromLedger: Optional parameters for incremental verification.

---

SCP / Quorum Set Fetching

FetchRecentQsetsWork (FetchRecentQsetsWork.h/cpp)

Inherits: Work

Three-phase work for downloading and scanning recent SCP messages to discover active quorum sets:

1. Fetches the latest archive state via GetHistoryArchiveStateWork.
2. Downloads SCP message files for the last ~100 checkpoints (~9 hours) via BatchDownloadWork.
3. Scans downloaded XDR files to extract SCPHistoryEntry records.

---

Key Data Flows

Publish Flow

ResolveSnapshotWork (wait for bucket futures)
  → WriteSnapshotWork (write SCP messages to local files)
    → PutSnapshotFilesWork
      → GetHistoryArchiveStateWork (per archive, get current state)
      → GzipFileWork (gzip only differing files)
      → PutFilesWork (per archive: MakeRemoteDirWork → PutRemoteFileWork per file)
      → PutHistoryArchiveStateWork (upload HAS JSON to seq path + well-known path)

All wrapped in PublishWork (a WorkSequence) which notifies HistoryManager on completion.

Download/Catchup Flow

BatchDownloadWork (download checkpoint files of a given type: ledgers, txs, results, SCP)
  → GetAndUnzipRemoteFileWork (per checkpoint)
    → GetRemoteFileWork (download .gz)
    → GunzipFileWork (decompress)

DownloadBucketsWork (download+verify+adopt all buckets)
  → per bucket: GetAndUnzipRemoteFileWork → VerifyBucketWork → adopt callback

DownloadVerifyTxResultsWork (download+verify tx results)
  → per checkpoint: GetAndUnzipRemoteFileWork → VerifyTxResultsWork

Verified Checkpoint Hash Chain

WriteVerifiedCheckpointHashesWork (iterates backwards from trusted endpoint)
  → per batch: BatchDownloadWork (ledger headers)
    → ConditionalWork(predicate: prev batch succeeded)
      → VerifyLedgerChainWork (verifies hash chain, writes to shared output file)
         (chained via shared_future<LedgerNumHashPair> from previous batch)

---

Threading Model

• Main thread: All Work state machine transitions, scheduling, and doWork()/onRun() calls.
• Background threads (via postOnBackgroundThread):
  • VerifyBucketWork::spawnVerifier(): SHA-256 hashing and index creation.
  • VerifyTxResultsWork::onRun(): Transaction result verification.
  • WriteSnapshotWork::onRun(): SCP message writing (if DB pooling available).
• External processes (via ProcessManager::runProcess): All RunCommandWork subclasses (gzip, gunzip, get/put remote files, mkdir). These spawn shell commands and use async ProcessExitEvent callbacks.
• Synchronization: DownloadBucketsWork::BucketState uses std::mutex to protect buckets and indexMap maps accessed from both main and background threads. Background workers always post results back to main thread via postOnMainThread before modifying BasicWork state.

---

Ownership & Lifetime

• Work objects form a tree: parent works own child works via addWork<T>(). The work scheduler drives the tree.
• StateSnapshot is shared across the publish pipeline via shared_ptr.
• TmpDir objects own temporary directories; their destructors clean up files. WriteVerifiedCheckpointHashesWork explicitly manages TmpDir lifetime per batch.
• HistoryArchive is shared via shared_ptr and may be null (meaning "pick randomly").
• FileTransferInfo is a value type describing file paths and types; not heap-allocated.
• BatchWork (parent class) manages the pool of active child works and controls parallelism.