More than 500 graves have been swept away by the floods in the past week alone.
The chief imam of the town, Alhassan Musa Na’ibi, told the BBC about 1,000 decomposed bodies had been reburied.
The floods followed days of torrential rains in the area.
The imam said the cemetery had never experienced such devastation since its establishment 500 years ago.
The cemetery is located near a river.
Residents say recent gold mining activities near the cemetery have also made it vulnerable – as the ground became weakened.
Nigeria is experiencing its worst wave of flooding in a decade – affecting 29 of its 36 states.
Since the end of July, more than 300 people have been killed and more than 100,000 others displaced from their communities.
Bridges and large swathes of farms have also been destroyed.
The authorities say more floods are expected in the coming weeks as torrential rains continue.