SHA-256 Deduplication

Hash file bytes with SHA-256 before upload to reuse existing transaction IDs, cut duplicate AR spend, and preserve submission history.

Before every upload, a SHA-256 hash of the content is checked against the registry hash index in PostgreSQL. Identical payloads return the existing Arweave TX immediately — zero AR spent.

Stop paying permanent-storage costs for content you already have.

Deduplication is one of the easiest ROI stories in the platform. It avoids avoidable writes, reduces repeat storage spend, and speeds up preflight checks.

Value

Cut repeat upload spend

Hash checks prevent duplicate payloads from consuming AR, confirmation time, and unnecessary downstream processing.

Improve migration economics

Large backfills and archive imports benefit most when duplicate content appears across batches or systems.

Keep storage strategy defensible

Finance teams can see that permanent storage is being used deliberately rather than as a blind sink.

Use cases

Document Repositories

Legal and compliance repositories frequently receive duplicate submissions. Dedup eliminates redundant AR spend while preserving the complete upload history.

Media Pipelines

CDN ingest pipelines often receive the same asset from multiple sources. SHA-256 dedup collapses duplicates to a single permanent TX.

Enterprise Backups

Incremental backup strategies generate high duplication across snapshots. Content-addressed dedup ensures only net-new data consumes AR balance.

For operators

  • Run single-file or batch dedup checks before upload paths that carry meaningful cost.
  • Invalidate cache intentionally when you need to reconcile state or force recomputation.
  • Pair with batch operations to reduce waste during imports and bulk publishing jobs.

For buyers

  • Good fit for customers with reissued files, replicated archives, or heavy operational exports.
  • Useful in platform demos because the savings story is immediate and easy to understand.
  • Strengthens conversations around cost governance rather than raw storage volume alone.

What if two files have the same content but different metadata?

Deduplication is content-addressed — it checks the SHA-256 hash of the raw file bytes only. Files with identical bytes but different filenames or tags are considered duplicates. The returned txId points to the original upload.

Can I opt out of deduplication?

Yes. Pass dedup: false in your upload request body. The file will be uploaded as a new transaction regardless of whether an identical hash already exists.

Does deduplication work across API keys on the same account?

By default, dedup scope is per-account. Enterprise plans can configure cross-account dedup for multi-team environments.