I am not an expert, and am pursuing my own issues in respect of a wrongly pitched propeller behind a Yanmar 3HM.
If you can over-rev your engine for a sustained period that suggests to me your prop may be too small or of insufficient pitch for the engine. As I understand it (see disclaimer at the beginning), ideally the prop should be such that the engine can reach its max continuous RPM and for short periods its max RPM and the boat should reach close to hull speed in the latter case (although your fuel consumption will be high) - unless the engine is underpowered for the boat. I'm not sure what the water line length and beam of your boat is.
As I understand it, to some extent, and within limits, propeller pitch and diameter can trade-off - more diameter less pitch and vice versa (because less slip (more efficiency) with larger prop - all else being equal it shifts more water per revolution (and so is also harder to turn at a given rpm)). Clearance from prop to hull can be a determining factor in choosing diameter. If you're lucky the pitch may be stamped on the prop. If you need to measure your prop to work out its pitch, this site may help:
goneoutdoors.com/measure-boat-propeller-6101713.html I think max continuous power for the 2ym15 is 9.1 kW (12.4 hp metric) at 3489 rpm. Max power (for limited periods) is 10.0 kW (13.6 hp metric) at 3600 rpm. All at the crank shaft. At the prop the power is about 3% less due to transmission losses. These taken from p84 of the
Yanmar Operation Manual for Marine Engines 2YM15, 3YM20, 3YM30E, 3YM30. Also the rpm of the shaft will be less due to transmission ratio. If you have the KM2P-1 (S) then the forward ratio is 2.21 and the reverse ratio is 3.06 (p88 of same). This matters for calculating prop rpm at a given engine rpm. If you have another variant of the KM2P-1 or a different transmission, then ratios will likely be different.
Of itself, unless you have a bent shaft or prop (even slightly so), I wouldn't have thought the prop would, of itself, cause vibration. The 2ym15 is a two cylinder engine of 570cc displacement. As I understand it, two cylinder engines are not inherently balanced (has to be a six for that, if cylinders are in-line), and so would vibrate at all RPM, and that would be more or less apparent at different revs depending on the damping characteristics of the engine mounts (and the state of the engine mounts). I'd expect another possibility is the alignment of the engine to the shaft. If that is slightly out of whack that may also cause vibration, but I have no specific knowledge about that.
A useful site for calculating what prop diameter and pitch should be is
vicprop.com/displacement_size_new.php/?m=1 Another is
www.seahawk.com.au/propcalc.html Both require some assumptions about slip (to what extent the propeller moves less far forward through the water per revolution than its pitch.)