Sydney's Duplicate Image Problem: How the City Stacks Up Against London, Singapore and New York

Sydney's government agencies and cultural institutions hold millions of digitised images in fragmented databases — and a significant share of those files are duplicates, orphaned copies, or mislabelled assets that waste storage, mislead researchers, and cost taxpayers money to maintain. The problem is not new, but pressure to resolve it has sharpened in 2026 as agencies consolidate onto shared cloud infrastructure under the NSW Government's GovConnect program.

The timing matters for a specific reason. The State Archives and Records Authority of NSW is midway through a five-year digitisation drive that has accelerated the volume of image assets entering public repositories. More files flowing in means more duplicates compounding, and the institutions tasked with managing them — including the State Library of NSW on Macquarie Street and the Museum of Applied Arts and Sciences at Ultimo — are under pressure to demonstrate they are not simply building larger, messier haystacks.

What Sydney Is Actually Doing

The City of Sydney Council's open-data portal, which covers archives running back to historical surveys of the Rocks and Pyrmont, adopted a deduplication protocol in late 2024 under its Digital Infrastructure Review. Parramatta City Council, which manages one of the fastest-growing local government areas in the country, has been piloting an AI-assisted asset management tool across its planning and heritage image libraries since early 2026. Neither council publicly releases figures on what percentage of its image holdings are redundant, making independent assessment difficult.

The NSW State Library, for its part, runs its digital collections through the IIIF — International Image Interoperability Framework — standard, which helps external researchers identify matching images across institutions. That framework is now used by more than 300 libraries and archives globally, including the British Library and the Smithsonian Institution in Washington DC. Adoption of a shared standard is a meaningful step, but standardisation alone does not delete redundant files from underlying storage.

How Sydney Compares to London, Singapore and New York

London's approach offers a useful benchmark. The UK National Archives at Kew completed a deduplication audit of its photographic holdings in 2023, reducing its active image catalogue by roughly 18 percent and cutting associated cloud storage costs. That audit was part of a broader Digitisation Strategy published in 2022. Sydney's equivalent bodies have not publicly completed a comparable end-to-end audit, though the State Archives has flagged deduplication as a priority in budget submissions since at least 2024.

Singapore's National Heritage Board went further. It integrated deduplication directly into its ingest pipeline in 2021, meaning duplicate images are flagged automatically before they enter the primary catalogue at the National Archives of Singapore on Canning Rise. New York's Metropolitan Museum of Art, which made its Open Access collection available in 2017, runs regular integrity checks and publishes correction logs — a transparency measure Sydney's cultural institutions have not formally adopted.

The financial dimension is not trivial. Cloud storage costs for large image libraries — files running into tens of gigabytes per high-resolution scan — can reach six figures annually for a mid-sized institution. A 2024 report by the Australian Library and Information Association noted that digital storage costs across the sector had risen by more than 30 percent over the preceding three years, driven by both volume growth and the shift away from on-premises servers.

For institutions on Macquarie Street or Bridge Street dealing with heritage collections, the practical risk is not just cost. Duplicate images with different metadata tags can appear as separate historical records, creating false impressions of volume or leading researchers to draw conclusions from what is effectively a single source file counted twice.

What happens next depends heavily on whether the NSW Government's forthcoming Digital Strategy update — expected in the second half of 2026 — mandates deduplication standards across all agencies rather than leaving it to individual institutions to self-regulate. Organisations such as the Australian Society of Archivists have pushed for a uniform state-level policy. Until that policy lands, Sydney will continue managing its image problem institution by institution, a patchwork approach that cities like Singapore largely moved past half a decade ago.