Three bodies were discovered on Sunday after two houses were burned to the ground in a secluded hamlet in western Yamaguchi prefecture.
Then on Monday a further two victims were found. Investigators said they had been battered to death.
All five victims were in their 70s or 80s. The number of dead represents a third of the local population, Ninemsn reports.
Police say they want to speak to a 63-year-old villager in connection with the killings after they found a 'haiku' poem stuck to the window of his home.
The haiku poem follows a traditional Japanese form of three-line verses of 17 syllables. They are arranged in a five-seven-five formula.
The note read: 'Setting on fire- smoke gives delight - to a country fellow.'
Police said the two houses which were destroyed by fire were 60 metres apart, Fuji TV reported.
Within them they discovered the remains of a 79-year-old woman and the bodies of a couple in their 70s.
The last two victims were a 70-year-old man and a 73-year-old woman found in two other houses 200 meters away.
Initial autopsies have reportedly shown that all the five people died instantly after they were beaten in the head with a blunt instrument.