Could be multiple things. I personally haven't noticed huge spikes though.
Have you tried the standard router reset and GPU update?

I have a decent gaming computer, hard wired NBN internet and are on the Australian East Coast but prefer to connect to USA to play Infinite through a VPN for stability and it massively reduces my game wait time. I suspect the wait times outside of USA and EU are ridiculous, and who has all day to wait for a dumb game to say when you can play, right?

Anyway, playing expanded search mode, I start at an average of about 250ms in the morning and it slowly lowers as the day goes on to about 160ms. I do get blowout games at 300+ and can't hit a thing, but never get below 160ms. The randomness of the latency spoils any muscle memory on mnk play.

Without the VPN on, I used to pick up the odd game connected to the Australian server. It was a glimpse at what the game actually was meant to be played at. I was playing at 4ms. The hit reg was beautiful, movement crisp, no glitching or shot around corners etc. Those games became few and fewer with the player numbers dropping.

Unfortunately mate, the tyranny of distance has relegated Aussie fans of Infinite to a sub-par experience.
Lucky it's free!

I think there was an issue with the server. My GPU is current, Router is reset with a brand new TP Link Router. I had VPN set to USA.

Today all of a sudden I could not control my movement and ended up dying 4 times due to it. I messaged in the chat and all 3 other team mates experienced the same thing. Heck even my Weapon disappeared from my hand and got randomly replaced even though the game was NOT random loadout.