Sydney's Duplicate Image Problem: How the City Stacks Up Against London, Singapore and Toronto

Sydney's public institutions are sitting on millions of duplicate digital images — redundant files clogging servers, distorting search results and wasting storage budgets — and the strategies being deployed to fix the problem vary wildly depending on whether you work in Macquarie Street or Parramatta Square.

The issue has sharpened this year as government agencies across New South Wales accelerate the digitisation programs launched under the Digital.NSW strategy, pushing enormous volumes of scanned documents, heritage photographs and planning maps into centralised repositories. When the same image gets ingested twice, or three times, or a dozen times across different agency silos, the downstream problems compound quickly: staff retrieving wrong file versions, duplicate records appearing in public-facing portals, and cloud storage bills climbing for no useful reason.

It matters now because NSW is in the middle of the most ambitious public-sector digitisation push in a generation. The State Archives and Records Authority of NSW, based on Globe Street in The Rocks, is processing historical collections at a scale that was not operationally possible five years ago. Parramatta City Council's digital records team, handling planning documents and heritage imagery for one of Australia's fastest-growing local government areas, has flagged internally that duplicate detection workflows have not kept pace with ingestion volumes — a structural gap familiar to archivists in cities far larger than Sydney.

What Sydney Is Doing — and Where It Falls Short

The State Library of New South Wales, on Macquarie Street in the CBD, uses a perceptual hashing system to flag near-duplicate images within its digitised collections, a technique that compares pixel-pattern fingerprints rather than file metadata alone. That approach is broadly consistent with what the British Library in London deployed after a 2023 internal audit found tens of thousands of redundant image files in its digital newspaper archive. The difference is resourcing: the British Library dedicated a named remediation program with a defined budget cycle to the cleanup. Sydney's equivalent work is absorbed into existing operational budgets without a standalone program or published timeline.

Singapore's National Archives completed a deduplication sweep of its entire photographic holdings in 2024, using AI-assisted clustering tools developed in partnership with Nanyang Technological University. The project processed roughly 2.3 million images and reduced active storage requirements by an estimated 18 percent, according to figures the National Archives of Singapore published on its website. Toronto Public Library, which manages one of North America's largest municipal digital collections, contracted a specialist vendor in late 2024 to audit its image repositories ahead of a planned migration to a new content management platform.

Sydney has no comparable published benchmark. Neither the State Library nor State Archives has released deduplication audit figures to date, which makes it difficult to assess the true scale of the problem or compare it directly against the Singapore or Toronto results. What archivists and records managers in the sector say privately — though none would speak on the record — is that the gap between ingestion speed and quality-control capacity has widened noticeably since 2023.

The Practical Cost and What Comes Next

Cloud storage is not free. AWS S3 standard storage, the tier used by many NSW government workloads under the NSW Government's whole-of-government cloud procurement arrangement, carries costs that scale directly with volume. Every gigabyte of redundant image data retained indefinitely represents real expenditure against agency IT budgets that are already under pressure heading into the 2026-27 state budget cycle.

The City of Sydney's open data portal, which hosts planning maps, heritage photographs and infrastructure imagery for the LGA stretching from Redfern to Pyrmont, updated its data governance policy in March 2026 to include a deduplication checkpoint before new image datasets are published. That is a meaningful procedural change, though it applies only prospectively — the backlog of existing records remains unaudited.

For institutions wanting to close the gap on Singapore and London, the path is straightforward in principle: commission a baseline audit using perceptual hashing tools, set a published remediation target, and ring-fence a budget line for the work rather than absorbing it into general operations. The technology is not exotic. The constraint, here as in most of the cities Sydney is being compared against, is organisational will and the willingness to declare a problem solved only when the numbers actually show it.