Today I will implement the detection system for black keys.

Black keys are very easy to detect, unlike the white keys. For white keys we need to filter the image to make the edges more magnified, but for black keys we just need to remove the white key's details and only focus on the black keys. What we gotta do is just dilate the image or blur the image to remove the white key's contours, perform the thresholding and finally get the contours of the image.

The only problem is that the black keys don't have the concrete outline to individually detect the keys, because the black keys blends to the background.

So, we need to figure out the way to enclose the image to make the black keys separated from the background. At first, I thought of detecting the corners points and connecting the corners with the other closest corner point. I used Harris Corner Detection Algorithm to test my idea.

First, I detected all the corners and just connected each of them with their closest corners. Although that did result in enclosing the black keys' outline, it was not pretty reliable, as it often failed to enclose some keys.

Then I just thought of really simple solution! We just need to use the convex hull algorithm and overlay the resulted contours! Convex Hull Algorithm creates a polygon that encompasses all the line segments of a contour. (According to Wikipedia, it's like a rubber band encapsulating all the points)

Source : Convex Hull, wikipedia

What we had to do is just get the contours from the thresholded image and simply use the contour to get the contour of Convex Hull :

std::vector<std::vector<Point>>contours;
only_black_keys.copyTo(black_keys_enclosed);
findContours(only_black_keys , contours , RETR_EXTERNAL , CHAIN_APPROX_SIMPLE);
std::vector<std::vector<Point>>hull(contours.size());
for(int i = 0; i < contours.size(); i++) {
    convexHull(contours[i] , hull[i]);
    drawContours(black_keys_enclosed , hull , i , 0xff , 10);
}
morphologyEx(black_keys_enclosed , black_keys_enclosed , MORPH_CLOSE , getStructuringElement(MORPH_RECT , Size(3 , 3)));

Isolated black keys using the convex hull algorithm

Now if we just find the contours of the black keys and approximate them to rotated rectangles, we can finally get the clean bounding boxes of the black keys.

Isolated black keys (finally!)

Nice! Now what we gotta do is detecting the hand using AI model(somehow) and matching the hand positions with the contours of the piano keys.