Abhishek Verma
AI Engineer · RL environments and LLM evaluation · Gurugram, India
About
I build the environments that LLM coding agents train and get graded in. My day job is turning real merged GitHub pull requests into reproducible Docker tasks with hidden tests, then checking that agents actually solve them instead of gaming the grader. I've shipped 2,000+ of these environments and 2,000+ agent trajectories for SFT and RL post-training.
Outside work I read the source of the eval and interpretability libraries I use, find bugs nobody has reported yet, and send fixes with regression tests. I'm open to AI engineer, LLM evaluation and Python backend roles, remote or with relocation.
Open source contributions
| Project | Pull request | What I fixed | Status |
|---|---|---|---|
| virtualenv | #3514 | The csh activator left PYTHONHOME set, so the activated interpreter could not start at all. | Merged |
| tox-uv | #418 | Changing system_site_packages never recreated the environment, so the setting had no effect either way. | Merged |
| tox | #4393 | The same stale-environment gap upstream, in VirtualEnv.python_cache. | Open |
| TransformerLens | #1965 | Eval helpers hardcoded "cuda", so they crashed on CPU and Apple MPS machines. | Merged |
| SAELens | #776 | Unseeded calls to random_firing_probabilities returned identical values every time. | Merged |
| nnsight | #800 | Batched .skip() lost the output's container type, breaking namedtuple and ModelOutput results. | Merged |
| EleutherAI delphi | #192 | split_quantiles dropped the remainder, so small requests scored a latent on zero examples. | Merged |
| tox | #4371 | A re.VERBOSE regex dropped a literal space, so commented-out PEP 723 lines were read as live metadata. | Merged |
| pyproject-api | #354 | ensure_empty_dir followed symlinks: a dangling link raised, and a link to a directory emptied its target. | Merged |
| filelock | #908 | Read-write locks took blocking=0 as blocking and waited the full timeout. | Merged |
| platformdirs | #686 | Media and fonts functions couldn't take use_site_for_root like their properties. | Merged |
| TransformerLens | #1976 | stack_head_results silently returned the wrong residual stream for a negative layer. | Merged |
| Hugging Face lighteval | #1465 | The LiveCodeBench sandbox's fake stdout had no binary buffer, so correct solutions were graded wrong. | In review |
| SWE-bench | #711 | A bare FAIL: line crashed the Django log parser and killed grading for that instance. | In review |
| Gymnasium | #1827 | MaxAndSkipObservation returned the previous step's frames when an episode ended mid-skip. | In review |
| circuit-tracer | #124 | A bare squeeze() broke single-token prompts and let real batches through unchecked. | In review |
| BrowserGym | #421 | hide_bid_if_invisible crashed with an AttributeError instead of the clear error its sibling flags give. | In review |
| pluggy (pytest) | #827 | A 1.7.0 regression let subset_hook_caller call plugins it was told to exclude. | In review |
| Gymnasium-Robotics | #362 | get_joint_qvel read 4 velocities for a MuJoCo ball joint instead of 3. | In review |
| Shimmy | #169 | OpenSpiel reset() dropped a game's parameters, crashing or silently changing the game. | In review |
| Minigrid | #551 | MissionSpace rejected valid missions when two placeholders shared one character. | In review |
| PettingZoo | #1538 | Random rollout helpers ignored an action mask passed in info and took illegal actions. | In review |
| MO-Gymnasium | #172 | breakable-bottles returned rewards and observations outside its declared spaces. | In review |
| more-itertools | #1346 | Type stubs for concurrent_tee and serialize disagreed with the runtime code. | In review |
| EleutherAI sparsify | #146 | isnumeric() turned -5 into a float and crashed on Unicode numerals. | In review |
| iniconfig (pytest) | #89 | str.splitlines() also split on form feeds and Unicode separators, breaking values and shifting reported line numbers. | In review |
| pytest-mock | #661 | mocker.spy() on an inherited classmethod or staticmethod raised TypeError when called through an instance. | In review |
| pytest-randomly | #755 | With xdist, --randomly-seed=last --cache-clear gave every worker its own seed and crashed collection. | In review |
Gauntlet: leak-free RL environments from fresh GitHub PRs
My main open source project. It mines PRs merged after a model's training cutoff, rebuilds each repo at the base commit in Docker, and grades agents the way SWE-bench style benchmarks do. Then it tries hard to cheat its own grader.
- gauntlet-harness: PR mining, F2P/P2P test classification, a leak scan, rubric judging, a Gymnasium env and exports for SWE-bench, SFT, GRPO and DPO
- gauntlet-cheatcatcher: reward-hacking detectors for coding agents
- gauntlet-repo2env: turns a repo into Docker env images and SWE-style tasks
- gauntlet-sandbox and gauntlet-timemachine: record, replay and diff agent trajectories
- gauntlet-arena: a FastAPI app where a human and an AI agent race on the same task
Backend work: a policy and claims REST API
policy-claims-api is an insurance back office built with FastAPI, PostgreSQL and Redis. Agents write policies and file claims, assessors decide them, admins see everything.
- JWT authentication with three roles. A record the caller cannot read returns 404 rather than 403, so the API never confirms that someone else's policy exists, and the role is re-checked against the database row on every request instead of being trusted from the token.
- The money rules are the point: approved claims cannot add up past the sum insured even when each one is individually valid, a rejection frees its cover again, and a claim cannot be decided twice. Amounts are stored in paise as integers, never floats.
- Claim filing takes an
Idempotency-Key, so a retried request returns the claim it already created and two racing retries resolve to one. Redis caches reads and degrades to a cache miss when it is down rather than failing the request. - 35 tests at 93% line coverage, running on SQLite so CI needs no services. GitHub Actions runs ruff, the suite, and a job that builds the image, boots the container and curls the health endpoint.
Experience
- Ethara AI (via Quess Corp), Software Engineer, RL Environments and Evaluation, 2026 to now
- NavAdr Technologies, Software Engineer (Data), 2025 to 2026. Python ETL into Snowflake on AWS
- GlobalLogic, Associate Analyst, 2024 to 2025. CI/CD with Docker and GitHub Actions on AWS
Skills
- Python
- Docker
- AWS
- PyTest
- LLM-as-judge
- RL environments
- SWE-bench
- Terminal-Bench / Harbor
- Hugging Face
- PyTorch
- FastAPI
- SQL
- Snowflake
- GitHub Actions