Projects

Independent projects

Open-source systems I designed, built and evaluated myself.

1. 

   1. Overcooked AAR — evaluating video-language models

   GitHub repository ↗

   I had Video LLMs (Qwen2.5-VL and Qwen3-VL) write after-action reviews of real two-player Overcooked play twice, once from the event log and once from the gameplay video, and I checked every timestamped claim against the log. On 15 episodes, claims from the log were 67–93% precise against 22–43% from video, and the video timestamps were no better than a random moment in the same minute. A perception probe I built (1,646 items over 76 episodes) shows why: the models read the clock and the score perfectly but can't tell what a chef is holding (24–26% against 25% by chance for Qwen2.5-VL). I'm running the other 61 episodes and a blind human rating next.

   Qwen2.5-VLQwen3-VLClaim verificationRandom-time controlPerception probe
2. 

   2. DocuVision — key–value extraction from scanned documents

   GitHub repository ↗Model on Hugging Face ↗

   I fine-tuned LayoutLMv3-large on FUNSD and built a learned question→answer linker that turns scans, PDFs and phone photos into structured JSON. With ground-truth words, pair F1 rose from 0.23 to 0.74; end to end on the raw scans with OCR it rose from about 0.15–0.17 to 0.43–0.46, because OCR noise dominates. The ablations taught me the most: even with perfect tags, the old rule-based linker capped at 0.72, and grouping words in reading order was silently costing 25 F1 points on two-column forms.

   LayoutLMv3DETR layoutTable TransformerFastAPI
3. 

   3. Sentinel — tracking-based video anomaly detection

   GitHub repository ↗

   I combined YOLOv8 detection and BoT-SORT tracking with a rule engine for falls, loitering, abandoned objects, crowding and wrong-way motion, with deduplicated alerts and a dashboard. It runs at 19.7 FPS end to end with a pretrained YOLOv8n on an RTX A5000 (one 768×576 test clip). That is a throughput number only; I haven't evaluated detection accuracy yet.

   YOLOv8BoT-SORTOpenCVPrometheus
4. 

   4. DocuRoute — routing between retrieval and text-to-SQL

   GitHub repository ↗

   I built an LLM router that sends each question about SEC 10-K filings either to hybrid retrieval or to sandboxed text-to-SQL, and every answer carries citations. I also wrote the evaluation harness (context precision and recall, faithfulness, answer relevancy). On my 15-question golden set from one company's filings, a general-domain reranker lowered faithfulness from 0.93 to 0.77, so I kept plain hybrid retrieval as the default.

   RAGHybrid retrievalText-to-SQLEvaluation harness
5. 

   5. DepthCraft — measurable 3D from a single photo

   GitHub repository ↗

   I turn one photo into measurable 3D: Depth Anything V2 predicts metric depth, which I lift into a cleaned point cloud with a virtual tape measure that reports each distance with its uncertainty (for example 18.63 ± 0.29 m on one demo image). So far this is a smoke test; I haven't checked it against ground-truth distances yet. It has a FastAPI backend and a Three.js viewer, and takes about 330 ms per image on an RTX A5000.

   Depth Anything V2Open3DFastAPIThree.js
6. 

   6. Speech Translator — speech-to-speech translation in your own voice

   GitHub repository ↗

   I built a translator that takes spoken English and says it in Spanish, German or Japanese in the speaker's own cloned voice. I chose every component by testing it on 300 FLEURS sentences: a 100 MB CPU model beat NLLB-200 3.3B on German and Japanese, so I route each language to the model that does it best. Named-entity masking, which I expected to help, cost 3–6 BLEU, so I turned it off. When NLLB-3.3B suddenly scored 6 BLEU, I traced it to a library upgrade that made the tokenizer ignore the source-language token; with that fixed it scores 28.2.

   WhisperNLLB-200ChatterboxFLEURS evaluation
7. 

   7. Jarvis — a local voice agent with sandboxed tools

   GitHub repository ↗

   I built a voice assistant that runs entirely on my own hardware: wake word, Whisper, a tool-calling Qwen2.5-7B, memory and Piper, with 1.5 s median latency from the end of speech to the first audio. After tuning the prompt, I wrote a 20-case held-out test set, and tool-calling accuracy on it rose from 65% to 90%. The sandbox I put around the tools blocked every unsafe command the model tried.

   faster-whisperOllamaTool callingPiper TTS
8. 

   8. SentinelRAG — self-correcting retrieval with hallucination auditing

   GitHub repository ↗

   I built a self-correcting RAG pipeline as a LangGraph workflow: input guardrails, hybrid dense and BM25 retrieval, a relevance grader, query rewriting or a web fallback when retrieval falls short, cross-encoder reranking, and a final pass that audits the answer for claims the sources don't support.

   LangGraphQdrant + BM25BGE rerankerHallucination auditing
9. 

   9. Research→Code — multi-agent program synthesis with human approval

   GitHub repository ↗

   I built researcher, coder and reviewer agents in LangGraph, with their tools served over MCP. The reviewer runs the generated code in a sandbox, the coder gets a limited number of retries, and nothing is written to disk until a person approves it.

   LangGraphMCPSandboxed executionHuman-in-the-loop
10. 

    10. voice-dub — dubbing in the original speaker's voice

    GitHub repository ↗

    I built a pipeline that dubs a video into another language in the original speaker's cloned voice: optional pyannote diarization, faster-whisper transcription, NLLB-200 translation and XTTS-v2 voice cloning, then time-stretching so the new speech fits the original timing before I remux it into the video.

    WhisperpyannoteNLLBXTTS-v2
11. 

    11. Phone Support Agent — real-time voice AI for appointments

    GitHub repository ↗

    I built a voice agent for a fictional dental clinic that books, reschedules and cancels appointments: Twilio and Deepgram in, a tool-calling state machine, ElevenLabs out, with barge-in handling and escalation to a human. I verified it offline with scripted calls (6 of 6 automated personas with a stub LLM; 2 of 4 scenarios with a local Llama 3.1 8B). I haven't tested it on live phone calls yet.

    TwilioDeepgramTool callingElevenLabs
12. 

    12. Semantic segmentation on CamVid — comparing FCN, attention U-Net and DeepLabV3+

    GitHub repository ↗

    I compared three ResNet-50-based models on CamVid's 32 classes: an FCN with my own transposed-convolution decoder, an attention U-Net, and DeepLabV3+, all trained with a combined Dice–Jaccard loss and augmentation. The attention U-Net reached the best mIoU at 45.72%, against 38.13% for DeepLabV3+ and 34.14% for the FCN. Limited GPU memory forced a small batch size, and I've left hyperparameter search and CRF post-processing for future work.

    PyTorchFCNAttention U-NetDeepLabV3+
13. 

    13. Pneumonia detection — an attention ensemble for chest X-rays

    GitHub repository ↗

    Following An et al. (2024), I combined EfficientNetB0 and DenseNet121 features with multi-head self-attention, channel attention and a residual block to classify chest X-rays as pneumonia or normal. After tuning the learning rate, my best model reached 92% F1 and 97% AUC on the test set. I used LIME to check which lung regions drove each prediction.

    TensorFlow/KerasDenseNet121EfficientNetB0LIME