Sydney's Duplicate Image Problem: How the City Compares to London, Singapore and New York

Sydney's public institutions are sitting on millions of duplicate digital images — redundant scans, repeat uploads, and legacy file mismatches accumulated over two decades of rushed digitisation — and the tools being deployed to fix the problem are drawing comparisons, not always flattering, with approaches taken in London, Singapore and New York.

The issue matters now because several large infrastructure and urban planning projects are accelerating the pace at which councils and state agencies must process and publish visual records. Metro West construction through Parramatta and the inner west has generated a significant volume of site documentation photography, engineering imagery and heritage survey records. When those files move between agencies — say, from Transport for NSW to the City of Sydney Council's development portal — duplicate detection becomes a practical bottleneck, not an abstract archival concern.

What Sydney Institutions Are Actually Doing

The State Library of New South Wales, on Macquarie Street in the CBD, has been running a deduplication review across its digital collections since late 2024, targeting its photographic holdings spanning colonial-era glass plates through to born-digital acquisitions. The Library uses a combination of perceptual hashing algorithms and metadata cross-referencing to flag likely duplicates before human reviewers make final calls. The City of Parramatta Council has a separate but related challenge: its heritage and development image registers have expanded sharply as Western Sydney's growth corridor attracts more development applications, each arriving with attached site photography.

Neither institution has published detailed outcome data from those programs. But the structural approach — algorithmic pre-screening followed by manual review — is consistent with what larger municipal archives in comparable cities have adopted.

In London, the Wellcome Collection completed a publicly documented deduplication project across roughly 250,000 digitised images in 2023, publishing its methodology openly. Singapore's National Archives completed a similar exercise for its built-environment photography holdings, and has since embedded automated duplicate checking into its ingest pipeline, meaning new uploads are screened before they enter the main catalogue. New York's Municipal Archives operates under a different model: it relies heavily on contributor metadata and does not yet run systematic retrospective deduplication across all collections.

Where Sydney Lags — and Where It Leads

Sydney's position in this comparison is mixed. The State Library's Macquarie Street operation benefits from relatively strong resourcing by Australian standards, and its technology partnerships with the UNSW-affiliated digital humanities research community give it access to current computer vision tools. But the fragmentation of records across local government areas — the 33 councils within Greater Sydney each maintain separate image repositories with no mandated interoperability standard — creates duplication at the system level that no single deduplication algorithm can easily resolve.

A review of publicly available tender documents from the NSW Department of Customer Service, dated March 2026, showed that at least three separate agencies had independently procured image management software with deduplication features in an 18-month window, without apparent coordination. That parallel procurement pattern is precisely what Singapore's National Archives avoided by centralising ingest protocols from 2021 onward.

The practical stakes are higher than they might appear. Development applications lodged with councils in suburbs like Blacktown, Campbelltown and Liverpool routinely include hundreds of photographic attachments. When the same images appear multiple times under different file names, planning officers must manually reconcile records before decisions can be progressed. That adds time to a DA process already under scrutiny amid the state's housing supply crisis.

For organisations managing large image libraries in Sydney right now, the most actionable step is auditing ingest pipelines before the next major data migration — not after. The State Library's open-access documentation of its own methodology, available through its website, offers a workable template for smaller councils that lack dedicated digital preservation staff. London's Wellcome Collection published its full technical specification under a Creative Commons licence in 2023, and several Australian memory institutions have drawn on it directly. The tools exist. The remaining gap, at least in Sydney's case, is coordination across the 33 councils — a governance problem more than a technical one.