Sydney's Duplicate Image Problem: The Numbers Driving a Digital Cleanup Across NSW

NSW government departments collectively hold an estimated tens of millions of digital image files across shared drives, content management systems and cloud storage — and a significant portion of those files are exact or near-exact duplicates. That's the core finding driving a quiet but accelerating push among Sydney-based public sector IT teams to audit and replace redundant visual assets before the 2026-27 budget cycle locks in another year of inflated storage contracts.

The timing matters. With the Minns government under pressure on public spending and the NSW Department of Customer Service mid-way through a broader digital transformation program, agencies are being asked to justify every line of their cloud infrastructure costs. Duplicate image files — historically treated as a minor nuisance — have emerged as a surprisingly large contributor to avoidable expenditure.

The Scale of the Problem in Sydney's Public Sector

Digital asset management specialists working with Sydney councils say the duplication rate inside large unmanaged repositories typically sits between 30 and 40 per cent by file count. A single communications team cycling through stock photography, campaign imagery and event photography over five years can accumulate upwards of 200,000 files, with a third of those sharing pixel-identical or near-identical content with another file already in the system.

The City of Sydney Council's digital asset library, maintained through its communications and engagement directorate at Town Hall House on George Street, is understood to be among the larger municipal image repositories in the state. Councils in Western Sydney — including Cumberland Council, which serves suburbs from Merrylands to Wentworthville — have faced similar challenges as amalgamation in 2016 merged multiple incompatible legacy systems into a single environment without a full deduplication pass.

The financial case for cleaning up those repositories is straightforward. Cloud storage pricing from major Australian providers currently runs at roughly $0.023 per gigabyte per month for standard-tier object storage. A repository carrying 2 terabytes of duplicate imagery — not unusual for a mid-sized council — costs around $552 a year in raw storage alone, before factoring in backup, redundancy and egress charges that can multiply that figure three to four times. Across dozens of agencies and 33 metropolitan councils, the aggregate waste adds up quickly.

Deduplication Tools and What Comes Next

The replacement process — identifying duplicates, selecting the canonical version of an image, and purging or archiving the rest — is now largely automated through software using perceptual hashing algorithms. These tools compare images not just byte-for-byte but visually, catching resized, recompressed or lightly edited versions of the same original photograph. Sydney-based digital archiving firm Arkive Systems, which has worked with cultural institutions including the State Library of NSW on Macquarie Street, has been pitching this class of tooling to local government clients since early 2025.

State Library NSW itself underwent a partial digital collection audit in 2024 as part of its broader digitisation program, which has been running since 2021. The library holds millions of photographic items, many digitised from physical originals, and duplication across different scanning batches has been an acknowledged complication in its cataloguing workflow.

For organisations still running manual reviews, the arithmetic is punishing. A human reviewer assessing image duplicates at a rate of 500 files per hour would need roughly 400 hours to check a 200,000-file repository — equivalent to ten full working weeks for a single staff member. Automated deduplication tools marketed to the NSW government sector advertise processing speeds of 50,000 to 100,000 images per hour, reducing that workload to a matter of hours with a review queue rather than a full audit.

Agencies yet to begin this work should treat the current financial year as the window to act. The NSW Government's Whole of Government ICT Strategic Plan flags cloud cost optimisation as a priority through to 2027. Departments that can demonstrate reduced storage overhead before their next budget submission will be better placed to argue for discretionary technology investment elsewhere. For councils in the city's west still carrying the legacy burden of 2016 amalgamations, a deduplication pass is overdue — and the numbers now make the case more clearly than any policy directive has managed to.