Fenify - Josh Kraft

Project Overview

Fenify is a mobile chess analysis app that uses computer vision to convert 2D & 3D chess board photos into interactive positions for analysis with the Stockfish engine. The app works offline and allows users to screenshot chess positions and analyze them in real-time.

I created Fenify after finding myself frequently hoping to analyze chess positions from youtube videos, instagram posts, and primarily, my own online games.

Key Features

Computer Vision

Board detection with automatic perspective correction and cropping.

Piece Recognition

Custom TensorFlow Lite model with 13 classes for accurate on-device piece classification.

Stockfish Integration

Full Stockfish 17 engine running in isolated threads with real-time position evaluation.

Offline Functionality

Reliability offline with all processing happening locally.

Position Editing

Interactive board editor with tap-and-drop piece placement and comprehensive validation.

Analysis Tools

Multi-PV analysis showing top moves with evaluations and move navigation through game history.

Machine Learning Pipeline

In order for Fenify to achieve its objective, I had to build a machine learning model to classify each square on the board into one of 13 classes. This included:

Synthetic Data Generation: I dynamically rendered over 10,000 realistic chess positions with various board styles, piece sets, and augmentations.
Dataset Preparation: From there, I extracted and labeled 64 squares per board, balancing class sizes with custom parameters to avoid skewing training data.
Model Training: I fine-tuned a pretrained MobileNetV2 on chess-piece data with a custom classifier,
Inference Optimization: Added batch processing and confidence scoring for reduced load and clear debugging,
Mobile Deployment: and lastly, integrated the TFLite model with the iOS build through Swift platform channels.

Technical Challenges & Solutions

Stockfish Crashes: Since the engine lacks any inherent FEN validation, and Dart isolates can't catch native C++ assertions, I encountered Stockfish crashes on any invalid FEN inputs. I had to fix this by implementing comprehensive FEN validation before intializing the engine. As I failed to find a package that provided an exhaustive solution without API calls, I used the dartchess library clases and built a custom FEN validation layer to prevent crashes.
Board Detection Issues: Image borders were causing the model to cut board squares incorrectly. I solved this issue by improving edge detection and square extraction in openCV.
Piece Orientation: Mapping pieces arbitrarily from photo to board, the tflite model often initialized each color moving the wrong way. While there is no guaranteed way to determine the correct direction pieces are moving from just a photo, some ad-hoc Coordinate mapping and board orientation logic improvements have drastically reduced mismaps.
Real-time Analysis: If the move navigation or position updates happened too quickly, the stockfish engine would lose synchronization with the board state. Adding debouncing with timer-based rate limiting fixed this.
Async Race Conditions: Fixed UI updates happening out of order by implementing explicit sequencing: Update UI -> Notify -> Delay -> Start background work

|

Project Overview

Technologies & Tools

How It Works

Capture

Key Features

Machine Learning Pipeline

Technical Challenges & Solutions

Future Development

Planned Features

Still in progress