Councils, archivists and tech experts weigh in on Sydney's duplicate image problem

Sydney's government agencies and cultural institutions are sitting on mountains of duplicated digital images — and the bill for storing, managing and misidentifying them is climbing. That is the consensus emerging from conversations across councils, libraries and urban-planning bodies this week, as data managers and records specialists push for a coordinated approach to what has quietly become a significant administrative headache.

The issue has sharpened focus in 2026 for a practical reason: the NSW Government's ongoing digitisation push, which accelerated after the State Archives and Records Authority of New South Wales flagged backlogs across multiple agencies in its most recent compliance reporting period. When physical records are scanned and uploaded in bulk — sometimes across different departments at different times — the same image can end up duplicated dozens of times across separate folders, portals and cloud environments. Nobody disputes it happens. The argument is over how much it costs and who is responsible for fixing it.

What officials and experts are saying

At the City of Parramatta Council, data governance has become a recurring agenda item as the council manages imagery tied to its development assessment portal — a system processing hundreds of DA applications along the Church Street corridor and surrounding suburbs each year. Council staff responsible for records management have described the deduplication challenge as a workflow problem as much as a storage one: when duplicate images carry different metadata tags, they can surface as apparently distinct records during planning reviews, slowing assessment times.

At the State Library of New South Wales on Macquarie Street, the digitisation of historical photographic collections presents a related but distinct version of the same problem. Collections digitised in multiple tranches — sometimes years apart, using different scanning standards — can produce near-identical files that are technically unique but functionally redundant. Archivists working on the Mitchell and Dixson collections have long grappled with how to flag these without inadvertently deleting images that differ only in resolution or colour calibration.

Technology specialists working with local government clients across Greater Sydney point to a 2025 report by the Australian Digital Alliance, which found that duplicate and near-duplicate digital assets represent a measurable share of total cloud storage costs for public-sector organisations. The report did not publish agency-specific figures, but its broader finding — that deduplication tools could reduce redundant storage in large public collections by a meaningful margin — has circulated widely among records managers in NSW.

The Metropolitan Local Aboriginal Land Council, which manages significant photographic and documentary records relating to Country across the Sydney basin, has separately raised concerns about the cultural risks of deduplication done badly. Automated tools that flag images as duplicates based on pixel similarity can fail to distinguish between photographs that look alike but carry distinct ceremonial or provenance significance. Getting that wrong is not a minor administrative error.

What happens next — and what agencies are being told

The NSW Department of Customer Service, which oversees the Service NSW digital infrastructure and coordinates with agency data teams, has not yet announced a centralised deduplication policy. But specialists who advise state government clients say pressure is building ahead of the next State Archives compliance cycle, due in the first quarter of 2027.

For councils managing planning imagery — including the rapidly growing local government areas of Camden and the Hills Shire, both processing high volumes of subdivision and development records — the practical advice from records management consultants is consistent: audit before you automate. Running a deduplication algorithm across an unvetted image library without human review checkpoints is the single most common source of accidental record loss.

The cost of doing nothing is not abstract. Cloud storage pricing for government contracts in NSW has risen alongside broader infrastructure costs, and agencies carrying years of unprocessed digitisation backlogs are paying to store files — including duplicates — at scale. The State Archives and Records Authority has published guidance on digital disposal, but records managers say implementation varies enormously from one agency to the next.

A coordinated framework, specialists argue, would need to cover not just the technical deduplication process but the human review layer that follows — and the cultural consultation required before any photographic records with community significance are touched. That means time, budget and inter-agency agreement. All three, at present, remain works in progress.