Turns out, we should have not removed the small segments of the contours. It, in turn, removed the valid contours that properly enclosed the keys.
...
/* filter out contours that are too small or too big */
// if the arc length of the contour is smaller than the 1% of the image's width
if(arcLength(piano_contours_1[i] , false) < 0.01*img_width) continue;
// if the arc length of the contour is bigger than the sum of image's width and height
if(arcLength(piano_contours_1[i] , false) >= img_width+img_height) continue;
...This code actually removes the useful contours rather than removing the noises. I never knew that those small contours will be helpful for detection..! Very good news!
Now it works like a charm!
The problem was the rotation. If we adjust the height of the detected rectangle, the rectangle needs to be rotated to proper angle. In this rotation, the pivot of the rotation should not be the center of mass; it should be a bit lower side of the rectangle, as shown in the image below.
To do this, we need to simultaneously change the rectangle's angle and displace the rectangle's center of mass.
We can see that we need to rotate the center of mass by the pivot, thus we need to use Rotation Matrix to move the point! Here's the basic rotation matrix :
\[ \begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} cos \theta & -sin \theta \\ sin \theta & cos \theta \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} \]If we multiply those two matrices, we get this :
\[ \begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} xcos \theta -ysin \theta \\ xsin \theta+ycos \theta \end{pmatrix} \]This matrix rotates the point (x,y) by the pivot (0,0). Since we want to rotate a point by a certain pivot, we change the formula a bit :
\[ \begin{pmatrix} x'-x_0 \\ y'-y_0 \end{pmatrix} = \begin{pmatrix} cos \theta & -sin \theta \\ sin \theta & cos \theta \end{pmatrix} \begin{pmatrix} x-x_0 \\ y-y_0 \end{pmatrix} \] \[ \begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} (x-x_0)cos \theta-(y-y_0)sin \theta +x_0\\ (x-x_0)sin \theta+(y-y_0)cos \theta+y_0 \end{pmatrix} \]We can use this rotational matrix formula to rotate the center of mass!
For new value for adjustment, we can just use median value of the angles of rectangles. (The reason why we don't use mean is that outliers tend to affect mean more than median.)