Applied AI Researcher, Benchmarking at Distyl AI

You are a research-focused engineer with a track record of building evaluation frameworks for complex AI systems. You prioritize practical, agentic workflows an

Work type: hybrid

Location: San Francisco | New York

Salary: $150,000 – $250,000/yr

Type: Full-time

Summary

You are a research-focused engineer with a track record of building evaluation frameworks for complex AI systems. You prioritize practical, agentic workflows and have a background in designing reproducible experiments to test model performance. **What makes it worth a look...** Distyl AI offers a base salary between $150,000 and $250,000 plus equity for this hybrid role based in San Francisco or New York. Beyond the competitive pay, they provide full medical, dental, and vision coverage for you and your dependents. **You might be a good fit if you...** * Have experience designing and running benchmarks or test suites for intelligent systems. * Possess strong skills in data analysis and prototype building using compound AI techniques like ReAct or graph-of-thoughts. * Use tools such as Cursor and Perplexity daily to accelerate your research workflow. * Can demonstrate a history of empirical research results through publications or public projects.

Job Description

# About Distyl AI

Distyl is an applied AI technology company partnering with the world’s most ambitious institutions to rearchitect critical operations for the frontier of AI. Our customers include the largest companies in telecom, healthcare, insurance, manufacturing, consumer goods, and global social organizations.

We research and deploy technologies that power AI-native operations — both for our partners and for Distyl itself. Our work spans research into self-constructing systems, the development of the most reliable execution of AI systems, and products that transform mission-critical workflows. As a result, Distyl's technologies affect some of the world's largest operations — from hundreds of millions of consumer interactions to tens of millions of supply chain transactions and millions of patient journeys.

Distyl is backed by leading investors including Lightspeed Venture Partners, Khosla Ventures, Coatue, DST Global, and the board-members of 20+ F500s. The results reflect this approach: a 100% production deployment success rate for our customers and one of the few enterprise AI companies to run a profitable business.

# What We Are Looking For

At Distyl we’re pushing the envelope of AI utilization in enterprise. This requires creative researchers who don’t just want to drive incremental improvements on benchmarks or optimize an existing process but instead are looking to creatively redefine how software is used.

Our researchers come from many academic backgrounds but have strong research track records, operate in an AI-native way, and would be bored staying on the rails of a traditional research org.

# Key Responsibilities

• The Benchmarking team defines how progress is measured. Researchers design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. They construct benchmarks that reflect real-world complexity. Their systems become the standard by which new architectures, techniques, and releases are judged.

• Researchers in Benchmarking explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment. They investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability. Their insights drive both Distyl’s internal research priorities and industry-wide standards.

• Experience Designing and Running Evaluations: You’ve built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance

• Statistical and Analytical Rigor: You design fair, reproducible experiments and can extract signal from noisy empirical results

• Experience Building with Models, Not Just Building Models: We develop intelligent systems using models rather than training or fine-tuning them. Ideal candidates have expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.)

• Proven Track Record of Research Results: Whether you’ve published in top journals, posted amazing work on twitter, or somewhere else we want to see what you've done

• Uses AI Every Day: Before you can revolutionize someone else’s workflow, you need to revolutionize yours. You should be using tools like ChatGPT, Cursor, and Perplexity to accelerate your workflow

• Strong Programming and Data Analysis Skills: While you might not consider yourself a software engineer you need to be able to build prototypes of your ideas and then perform the experiments to prove the effectiveness to a F500 Head of AI

• Biases Towards Showing vs Telling: Our customers want to see the power of AI today vs discuss the most elegant idea that will take 5 years to realize

• The base salary range for this role is $150K – $250K, depending on experience, location, and level. In addition to base compensation, this role is eligible for meaningful equity, along with a comprehensive benefits package

• 100% covered medical, dental, and vision for employees and dependents

• 401(k) with additional perks (e.g., commuter benefits, in‑office lunch)

• Access to state‑of‑the‑art models, generous usage of modern AI tools, and real‑world business problems

• Ownership of high‑impact projects across top enterprises

• A mission‑driven, fast‑moving culture that prizes curiosity, pragmatism, and excellence

View this job on nocollar jobs