How CamChess Works

CamChess uses OpenCV and Numpy to analyse images of the chess board. It uses python-chess for its chess related processing.

After taking its first picture of the board, CamChess needs to recognise the board and find its four corners. OpenCV has a very helpful function called findChessboardCorners. Its main use appears to be for identifying checker board patterns for calibrating lenses, but it serves our purpose. It finds the inner corners of the chess board. Unfortunately, it does not return them in a consistent order for different images, but this problem is easily side stepped, as we shall see.

The 49 inner corners are shown in red and blue above. We are particularly interested in the red corners. If we have the coordinates of the red corners, we can easily find approximate coordinates for the outer corners of the chess board (shown in green). We can do this by simple proportion along the two long diagonals.

We can find the coordinates of the top-left and bottom right inner corners by adding the x and y values for each inner corner. The top-left corner will have the smallest sum, and the bottom-right corner the largest. Similarly, we can find the coordinates of the top-right and bottom left corners by subtracting the y value from the x value for each inner corner. The top-right corner will have the smallest difference, and the bottom-left corner the largest.

With my set up, the approximate coordinates of the four outer corners are rarely more than two pixels out. Inaccuracy can arise by perspective effects, and as a result of lens aberrations. Both of these are minimised by having the camera high above the centre of the board.

CamChess improves on the approximate coordinates by constructing Region Of Interest (ROI) around each of the approximate corner locations. (A fifth of a square in each direction). It then uses another OpenCV function goodFeaturesToTrack to find more accurate corner locations. CamChess repeats this every time an image is captured. This allows for small movements of the board or camera between moves of up to a tenth of a square.

CamChess uses the coordinates of the four outer corners to do a perspective transformation on the image of the chess board using another two OpenCV functions,  getPerspectiveTransform and warpPerspective. You can see the effect of this transformation by looking at the first two diagrams in How to Use CamChess above. The image is transformed to give a cropped square image of the chess board.

Having obtained a cropped square image of the board, CamChess's next task is to identify pieces on the board. For each square on the board, CamChess needs to find out whether that square is empty, has a Black piece standing on it, or has a White piece standing on it. The most difficult case is distinguishing a black piece from an empty dark green square. When the image is split into its blue, green and red channels, the dark green background is lightest in the green channel. Nonetheless, the contrast with the black pieces is not great, particularly where there are shadows. Here a grey scale image of the green channel:

CamChess distinguishes a black piece from the background by setting a threshold value. If the brightness of a green pixel is less than that threshold it is counted as a pixel belonging to the piece. If it is greater than the threshold value, it is counted as a pixel belonging to the background. If the number of pixels counted for the piece exceeds a preset minimum, a black piece is detected. (The black pieces are shiny. The bright reflections are counted as belonging to the background rather than the piece. This is not a significant problem because the brightly reflecting areas are relatively small.)

CamChess uses Otsu's Binarization to determine threshold values. Otsu works very well when finding a threshold value for a black piece on a dark green square. If you apply it to an empty square, it usually finds a threshold value there too, as a result of shadows and random noise. At first sight that makes Otsu useless for determining whether a piece is present on the square, but there is a way round this problem. CamChess uses the Open CV function threshold to calculate Otsu threshold values for each of the eight black pieces on dark green squares in the start position. In an early version of CamChess, I used the average value of these thresholds to find black pieces on dark green squares. That worked reliably when I stepped well back from the board before hitting the space bar to take a picture. It did not work reliably when I sat in front of the board. Phantom black pieces sometimes appeared on the dark green squares.

It soon became evident that the cause was my shadow on the board. It also became clear that there were significant variations in the brightness of the squares across the board, even when I was well away from it. (I investigated this for an image of an empty board. I calculated the mean and standard deviation of the green channel value for each of the dark green squares. I then calculated the standard deviation of the mean green values, and the mean of the standard deviations. The variability between squares was just over three times that within individual squares.) I needed to correct for those variations.

CamChess divides each square into three regions. The outer region is a thin margin to filter out noise from neighbouring squares. The ROI for the square is the largest square within this margin. The area for finding pieces is the largest circle that fits within the ROI. The remainder of the ROI (shown in yellow) is used to estimate the brightness of the ROI as a whole. CamChess calculates the mean value of the green channel in this region. When threshold values are calculated, they are divided by the mean value in the yellow area for the square concerned. When threshold values are used, they are multiplied by the mean value in the yellow area for the square.

The "light wood" coloured pieces are much brighter than the dark green squares in the red channel. They are found easily using another threshold value. Finding White pieces on White squares is more difficult, but they are clearly darker than the background in the blue channel. For White squares, CamChess looks for a piece in the blue channel, with a threshold value optimised for finding White pieces. That threshold value will also find Black pieces. CamChess distinguishes between White and Black pieces by finding the mean red channel value of the pixels that are below the threshold value in the blue channel.