AI Engineer
Innodata · May 2025 - Present
· Canada, Remote
Worked across 7+ production ML workflows (Ads, Search, Monetization, Responsible AI,
Recommendation), owning different stages of the lifecycle on each: labelling guidelines,
gold sets and sampling design, prompt and model development, fine tuning, evaluation,
deployment and monitoring.
The following projects are the most notable:
-
Pushed for and led the migration of a generative AI ad enhancement job from the
OpenAI API to a self hosted, fine tuned open source model, running offline
batch inference on 100M+ items/day and saving $1.1M/month.
-
Distilled the production API model into Qwen3-4B via LoRA distributed
data parallel (DDP) supervised fine tuning (SFT) without any performance regression in key metrics.
-
Built an evaluation pipeline using LLM-as-a-judge for factual faithfulness
(0.90 F1 against human labels) and deterministic checks for verifiable regressions to
monitor performance during training.
-
Raised throughput (req/s) by 2.14x by tuning vLLM scheduler hyperparameters and
exploring quantization techniques (AWQ, FP8, NVFP4) without causing performance regressions.
-
Deployed to production using an Airflow DAG that triggers inference on a Ray cluster.
-
Simulated job completion times (p50, p95, p99) using historical volume data to choose
optimal GPU instance type and sizes for production deployment to satisfy a 1 day SLA.
-
Built Apache Superset dashboards monitoring batch job health and drift in model
inputs and output quality.
Responsible AI
-
Built an LLM classifier to detect representational harms across multiple upstream
multimodal workflows. Increased precision from 0.30 to 0.52 and cut false positive rate from
0.25 to 0.07.
-
Built a synthetic data generator (predefined harm taxonomy, meta-prompts, production
seeds chosen by embedding clustering) to produce demographic counterfactuals and ensure equal
treatment of sparsely represented groups.
Technologies: Python, PyTorch, Hugging Face, LoRA, vLLM,
Ray, Airflow, SQL, AWS
