Use an offensive tsuba ability like Suzaku's flame or Genbu's ground slam on it then use punishing strike (the finisher/execution move) while it's staggered. Might not be one hit kill depending on your build and damage, but most the HP should be gone and you can melee attack the rest of it yourself no problem.
Shiden dance style and combo-ing into a charged attack does a lot of damage as it's getting up after the punishing strike is over.

Relying on villagers will take a considerable amount of people and time, I don't recommend it.
I'm sure you know already but just for good measure the method is position one to be bait and make it turn, Sumo is best due to taunt, then focus everyone else on the back for max DPS.

Side note, ranged units might not be consistent due to their aim variation and target selection if any other enemies are nearby. Melee units are preferable.

Easy... his weak point is his back... surround him either with the archer or with the infantry, keeping his distance and manage until you kill him... Use a peasant to distract him and make him turn around, exposing his back.

He's pretty slow and easy to dodge, unlock the final move of one of the sword styles so you can stagger him and use punish strike, it takes a huge chunk of his HP.

thanks everyone but yeah it seems there is no other way besides attacking him from behind.

I was struggling at the shrine stage when you cant do anything besides relying on the villager and i almost lose 5 my villager to the giant and good thing somehow the manage to deal a lot of damage to the giants back.

Other than the sumo taunt for direction locking you can also try the ascetic's slow debuff to buy more time for your villagers to attack.

The kamakura shrine map in particular doesn't have other enemies swarming that bridge at the same time so using ranged units is A okay.
That bridge has a villager buff lantern/post in the middle so you can position your archers or marksmen there.

Although I'm glad it sounds like you got it already.

Yeah there's no super-easy way to do it unfortunately, if you wanna deal with them the fastest it's a tsuba guard use you want.

For stages where you are stuck as a spirit, sumos are invaluable for being able to force aggro from them, leaving other villagers free to attack their back.