Today I tried to implement the note detection system. The idea is to circulate each white keys and find what black keys overlap with it. By finding the white key's side in which the black keys occupy, we can identify the shape of the white key.

Types of white keys

Now we need to figure out how to detect what sides the black key overlaps. Simplest method is to make a small segment of rectangle that encompasses the rectangle's edges and compare whether the small rectangles intersect with the black keys, allowing to determine what side does the black key intersects.

To make the small rectangle, we just need to create rectangle with width of 1 located at the edge :

The following code is an excerpt from the piano note detection system which creates that small segment. left_side_contour and right_side_contour respectively encompasses the left and right side.

    ...
    bool left = false;
    bool right = false;

    std::vector<std::vector<Point>>overlapping_contours;
    Point2f white_rect_pts[4];
    white_rect.points(white_rect_pts);
    std::vector<Point>left_side_contour = {white_rect_pts[0] , white_rect_pts[0] , white_rect_pts[1] , white_rect_pts[1]};
    std::vector<Point>right_side_contour = {white_rect_pts[2] , white_rect_pts[2] , white_rect_pts[3] , white_rect_pts[3]};

The code below now compares the two contours(rectangles) with all the black keys. First, it checks whether the black key intersects with white keys and detect what side does it intersect. To detect whether the two contours intersect, we create an empty Matrix area, perform the AND operation with the edge contour and black key contour, draw the contour that went through the AND operation on the new matrix and finally calculate the area of the matrix. Very dumb way to determine the intersection, but it works.

...
for(RotatedRect black_rect : black_keys_info.keys_rectangle_list) {
    std::vector<Point>intersect;
    if(rotatedRectangleIntersection(white_rect , black_rect , intersect) == INTERSECT_NONE) continue;
    
    Mat overlap_check_1 = Mat::zeros(white_keys_info.piano_image.size() , CV_8UC1);
    Mat overlap_check_2 = Mat::zeros(white_keys_info.piano_image.size() , CV_8UC1);
    Mat overlap_check_3 = Mat::zeros(white_keys_info.piano_image.size() , CV_8UC1);
    std::vector<Point>black_rect_contour;
    rotated_rect_to_contour(black_rect , black_rect_contour);
    drawContours(overlap_check_1 , std::vector<std::vector<Point>>({black_rect_contour}) , -1 , 0xff , -1);
    drawContours(overlap_check_2 , std::vector<std::vector<Point>>({left_side_contour}) , -1 , 0xff , -1);
    drawContours(overlap_check_3 , std::vector<std::vector<Point>>({right_side_contour}) , -1 , 0xff , -1);

    int overlap_left_check = countNonZero(overlap_check_1 & overlap_check_2);  // check the area of the overlapping contour
    int overlap_right_check = countNonZero(overlap_check_1 & overlap_check_3);
    if(overlap_left_check >= 1 && overlap_right_check >= 1) {
        std::cout << "error!" << "\n";
        if(white_rect.center.x > black_rect.center.x) {
            left = true;
        }
        else {
            right = true;
        }
        error_index_list.push_back(white_key_index);
    }
    else {
        if(overlap_left_check >= 1) { left = true; }
        if(overlap_right_check >= 1) { right = true; }
    }
}
...

Now we just use the acquired information to determine what shape does the white key have.

    ...
    Scalar color;
    int key_type;
    switch((left << 1)|right) {
        case 0:    key_type = KEY_NO_BLACK;    break;
        case 0b01: key_type = KEY_BLACK_RIGHT; break; // right
        case 0b11: key_type = KEY_BLACK_BOTH;  break; // both
        case 0b10: key_type = KEY_BLACK_LEFT;  break; // left
    }
    white_keys_info.white_key_shapes.push_back(key_type);
    white_key_index++;
    ...

It's bit inefficient, but I don't think there is another that can do better.