Sydney's Duplicate Image Problem: What Happens Next and the Key Decisions Ahead

Thousands of duplicate images are clogging the digital archives of Sydney's public institutions, and the organisations responsible for managing them are running out of road to delay the hard decisions. From the City of Sydney Council's civic photo library to the State Library of NSW on Macquarie Street, collections managers are under mounting pressure to audit, deduplicate and reclassify holdings that have ballooned over two decades of uncoordinated digital uploads.

The problem is not abstract. When agencies digitise records without a consistent naming protocol — a known gap in the State Records NSW framework, which last updated its digital preservation guidelines in 2022 — the same image can exist under four or five different file names across multiple servers. Staff waste time, storage costs climb, and public-facing search tools return cluttered, unreliable results. With the NSW government already stretched across major infrastructure commitments including Metro West and the ongoing housing policy overhaul, finding budget for what looks like a back-office IT problem has proved politically difficult.

Where the Backlog Is Worst

The City of Sydney's own image library — which documents everything from building approvals in Redfern to public art installations along Bourke Street in Surry Hills — is believed to hold a significant proportion of redundant files, though the council has not publicly released a precise duplication rate. The broader challenge mirrors what archivists at the Powerhouse Museum in Ultimo flagged internally during its controversial Parramatta relocation process: moving a collection forces a reckoning with what has been stored carelessly for years.

State Library NSW, which holds more than five million photographs, maps and documents in its Mitchell and Dixson collections, has been piloting automated deduplication software since late 2024. Librarians there are working through a prioritised list, starting with the most-searched collections. The Mitchell Reading Room on Macquarie Street gives researchers direct access to some of these holdings, and cataloguing errors — including duplicate entries for the same image under different donor names — have frustrated academic users for years.

At the federal level, the National Archives of Australia, which maintains a Sydney regional reading room at Villawood, operates under the Archives Act 1983. That legislation sets mandatory retention schedules, which means agencies cannot simply delete duplicates without clearance. Getting that clearance requires a formal Disposal Authority application — a process that can take months and demands senior sign-off. For cash-strapped state agencies watching the NSW government's $1.3 billion housing acceleration fund dominate budget conversations, dedicating compliance staff to that paperwork is a tough sell to Treasury.

The Decisions That Cannot Wait

Three choices will define how this plays out over the next 12 to 18 months. First, institutions need to decide whether to run deduplication algorithmically — fast but prone to error, particularly with historical photographs where two near-identical images may record genuinely different moments — or manually, which is accurate but slow and expensive. Second, they must settle on a common metadata standard. The Dublin Core standard remains the most widely adopted framework in Australian public collections, but uptake is inconsistent across NSW government agencies. Third, and most politically fraught, they must determine who pays.

The NSW Department of Customer Service, which oversees digital government policy through its Service NSW division, has indicated it is reviewing whole-of-government data storage contracts. Those contracts, currently structured around per-gigabyte cloud storage pricing with vendors including major hyperscalers operating out of data centres in Western Sydney's Eastern Creek, are up for renegotiation in the second half of 2026. That renewal window is arguably the best leverage point available: agencies that can demonstrate a reduction in redundant storage will have a concrete case for lower costs.

For Sydney's cultural institutions in particular, the practical next step is a collection audit with a hard deadline. The State Library model — prioritise by search frequency, automate first-pass deduplication, then apply human review — is the closest thing to a tested local blueprint. Institutions that delay past the current contract cycle risk locking in another three to five years of inflated storage costs and degraded search results. The window to act cheaply is now.