Today I figured out a better way to determine a pivot of the rectangle. Previously, we used the midpoint of the rectangle's vertices as the pivot, but that made the rotation quite ugly, which we don't like.

Also, the flip detection system didn't seem to work properly, so we need to fix that first. (We need to figure out whether the image is flipped before determing the pivots of the rectangles, because the pivot also needs to flip when the image is upside down.)

Previously, we calculated the average y value of the center of mass of rectangles and compare it with the image's height divided by 2. Most of the time this worked well, but it won't work well if the image has lens distortion or got heavily tilted.

We're gonna do this in more inefficient but accurate way. Simply, we horizontally split the image into halves and check which side of the image has the black keys. Since the black keys will always be at the top side of the image, we can determine whether the image is flipped with this method.

void PianoRecognition::detect_black_keys(Mat piano_image , struct piano_keys_info &keys_info) {
    ...
    // check whether the image should be flipped
    bool flipped = false;
    std::vector<double>black_cm_y_list;
    int black_median_cm_y = 0;
    Mat test_image;
    cvtColor(piano_image_padding , test_image , COLOR_GRAY2BGR);
    
    RotatedRect piano_rotated_rect = minAreaRect(hull[i_max_arc_length]);
    if(piano_rotated_rect.size.height > piano_rotated_rect.size.width) {
        std::swap(piano_rotated_rect.size.height , piano_rotated_rect.size.width);
        piano_rotated_rect.angle -= 90.0f;
    }

    // split the bounding rectangle into upper half and lower half
    RotatedRect piano_rotated_rect_upper(piano_rotated_rect) , piano_rotated_rect_lower(piano_rotated_rect);
    adjust_rotated_rect_height(piano_rotated_rect_upper , piano_rotated_rect.size.height/2 , false);
    adjust_rotated_rect_height(piano_rotated_rect_lower , piano_rotated_rect.size.height/2 , true);
    std::vector<Point>piano_rr_contour_upper , piano_rr_contour_lower;
    rotated_rect_to_contour(piano_rotated_rect_upper , piano_rr_contour_upper);
    rotated_rect_to_contour(piano_rotated_rect_lower , piano_rr_contour_lower);
    
    drawContours(test_image , std::vector<std::vector<Point>>({piano_rr_contour_upper}) , -1 , Scalar(0x00 , 0xff , 0x00) , 1);
    drawContours(test_image , std::vector<std::vector<Point>>({piano_rr_contour_lower}) , -1 , Scalar(0xff , 0x00 , 0x00) , 1);

    // check which side contains more rectangles
    int upper_hit_count = 0;
    int lower_hit_count = 0;
    for(RotatedRect rr : keys_rect_list_1) {
        if(pointPolygonTest(piano_rr_contour_upper , rr.center , false) > 0) upper_hit_count++;
        if(pointPolygonTest(piano_rr_contour_lower , rr.center , false) > 0) lower_hit_count++;
    }
    flipped = upper_hit_count < lower_hit_count;
    keys_info.flipped = flipped;
    ...

The codes are quite self-explanatory! Just create upper and lower halves of the bounding rectangle, perform the polygon test with the center of mass of all the detected black rectangles, and see which side has more of them.

Now that's covered, we now have to fix the pivot system.

I tried my best...

The new system will set the pivot in the method presented on the image. The pivot of the rectangle will be located on