Why don't you attack people above you? They're not fully portalled. If you just sit there and allow others who've been aggressive to gain the advantage, then obviously you're not going to win.
They took a chance to capture infra when they could and benefitted. Now you should do the same.
Those people around you, are free planets. Free planets = more space for your infra to be more efficient. Planets also prevent you from getting killed off.
The incentive of attacking people is to win. If you don't attack, you'll either be killed off, or knocked out cos you can't keep up with the planet counts.
If you're happy with just infrawhoring, then feel free to do so. But at the end of the day, if you're not attacking, you're not gonna surpass the others in pcount and hence get eliminated. It's up to you.