TTFB and caching

Hi Everyone,

I am struggling with the speed performance on my VPS.
We do not have cheap hosting and specs are 2 cores and 4gb ram dedicated.

But still we have very slow TTFB and mainly when you try GTmetrix first time it takes 4-7 seconds. Second time it is below 1 second but other websites on same server 2-3 seconds.

So somehow something is not correct. My Structure is 88% and I even asked my host to move to different hardware but still no performance improvement. Also all DNS is done by Cloudflare.

Followed every guide on here but nothing seems to help.