[aws][ebs] research/benchmark the new volume type gp3 to be used on IPI

*USER STORY:*

As a SPLAT engineer I would like to review if new EBS type gp3 can generate a better benefit for our customers when deploying IPI in default installation, considering performance and reliability on the control plane.

*DESCRIPTION:*

AWS recently launched gp3 EBS type without relation between capacity and IOPS. That option can bring us a benefit to adjust the control plane default installation using IPI, as we are using a gp2 with 128GiB (384 IOPS) o root volume.

The default installation could bring a risk on control plane due to low IOPS available by default when we are comparing with our recommendations[1] and default installation in other providers[2], ex Azure. We also had a recent outage on vSphere CI cluster (build1)[3] running on AWS with defaults that one volume was throttling after a spikes of IO when running a update with a chain of events that happened meanwhile.

The study case should consider the default installation of Azure and performance tests done on etcd that generates good outcomes to Azure IPI[4].

*ACCEPTANCE CRITERIA:*
 
create a doc including: * tests done with gp2, gp3, io1 with etcd on root and isolated partition;

• cost benefits between volume types
• mapping of changes to be done ( if applicable based on the results )

*ENGINEERING DETAILS:*

  [1] Recommended etcd practices

  [2] Additional Azure configuration parameters

  [3] vSphere CI cluster outage

  [4] Separate Disk for etcd: Azure Performance Comparison

[5] Useful references to monitor the EBS burst balance