The Eval Harness Caught a Real Bug on Day One In my previous post, I built the agent and the safety framework around it which included contracts, guardrails, metrics. The agent ran. It returned structured JSON every time. It failed safely…
John Dorlus
TPCAbout Me
I'm a staff-level engineer building intelligent test systems at the intersection of QA and AI. Over 15+ years I've helped organizations such as Elastic, GitHub, and Mozilla ship with confidence by designing automation frameworks, quality toolchains, and CI/CD pipelines that scale with distributed systems.
Outside of QA, my experience spans into data engineering with a focus on building data pipelines and data quality, creating large-scale data warehouses and data integrations with tools like Airflow, dbt, and Snowflake.
I have a passion for open source and have contributed to projects across testing, data engineering, and AI. This includes tools like Playwright, Airflow, and various LLM-based tools. Today I focus on applying LLMs to quality engineering: intelligent test generation, defect prediction, and AI-assisted code review as well as AI Agent quality and resilience. When I'm not writing tests or building pipelines, you'll find me in the gym or adding to my cuff link collection.
My Skills
Languages & Frameworks
- Python
- TypeScript / JavaScript
- Java
- Go
- SQL
- Playwright / Selenium / Cypress
- pytest / JUnit / TestNG
- React / Next.js
Tools & Platforms
- Elasticsearch / Kibana
- GitHub Actions / Jenkins / CircleCI
- Docker / Kubernetes
- Apache Kafka / Spark
- OpenAI API / Claude API / LangChain
- PostgreSQL / BigQuery
- Grafana / Prometheus
- Git / GitHub / Jira
What I Do
Test Architecture & Automation
Scalable test frameworks for distributed systems — from unit to end-to-end, built to survive organizational growth.
AI-Assisted Quality Engineering
Applying LLMs to test generation, intelligent test selection, and defect prediction to make quality proactive.
CI/CD & Developer Experience
Pipeline optimization, shift-left testing strategies, and tooling that keeps teams shipping fast with confidence.
Data Engineering & Observability
Data pipelines, telemetry validation, and analytics infrastructure that make system behavior legible.
Where I've Worked
- QA Lead for Kibana Management features, building and testing apps
- Modernized test automation by rearchitecting functional test runner migrating from Leadfoot to Selenium
- 424 pull requests merged across Elastic open source projects including Kibana and Elasticsearch
- Owned quality for IAM & Marketplace across millions of daily developers
- Developed test strategy and automation in Rails and Watir
- Designed and implemented custom automation framework for testing telemetry across Firefox release train
- Onboarded and mentored open source contributors
- Contributed to Firefox Browser and Firefox OS productivity projects as QA lead
- Provided support for Airflow users and contributed bug fixes and features to the open source project
- Built internal tools to help troubleshoot and debug Airflow deployments for enterprise customers
- Contributed to open source Airflow and related ecosystem projects
- Found 67 critical bugs in 7 weeks as lead QA
- Implemented test automation across API, web, and mobile platforms
- Drove shift-left quality practices in a fast-paced startup environment
- Lead Quality Engineer for HR software suite used by 20k+ companies
- Modernized test automation migrating from Cypress to Playwright
- Led adoption of Qase test management system and integrated with CI pipelines using TeamCity
- First data engineer working with dbt, BigQuery, Airbyte, and Fivetran.
- Proposed and built out data pipelines using Airflow and Great Expectations to validate data quality at each step.
- Integrated with SAP and OData for sales enablement.
- 426 bugs tested and closed for PDI data integration product
- Selenium automation for Pentaho 5.0 web-based admin console
- Backend engineer for Wolfram Programming Cloud
- Spring MVC, JUnit, and Selenium-based test automation for a web-based IDE product
- Caught Sev 1 Bug in production that revealed sensitive customer information
- Led 6 teams to implement QA practices across the organization
- Implemented Test Ops process to elevate tests to gatekeep builds
Writing
March 3, 2026Building an AI Agent - Reliability Framework Part: 1
ai-agentspydanticllmai
I Built an AI Agent. Then I Built a Framework to Make Sure It Actually Works. There’s no shortage of tutorials showing you how to wire up an LLM and call it an AI agent. Most of them stop at “look, it returned something.” I wanted to take…
August 28, 2019How Can I Land A Software Job If I Have No Experience
software-developmentportfoliointerviewingsoft-skills
This is everyone’s story, right? 1.) Graduated valedictorian from an A Ranked School and being captain of the Programming Club at 17 years of age. 2.) Graduated Summa Cum Laude from MIT with a Bachelors in Computer Science, minor in…
August 8, 2019Building A Review App Using ReactJS, Elasticsearch, App Search and Grommet — (Part 1: Data)
software-developmentjavascriptreactgrommet
I have been wanting to build an app for a while now. As a test automation engineer, I get to work with a lot of really cool tech and web apps but as a former web developer, I find myself needing to scratch the itch to build a web…
December 17, 2016Simple Single Page Applicaiton with Flask, AngularJS and Bower
software-developmentjavascriptangularjsflask
With the whole host of new Angular goodness making its way to the mainstream, I asked myself whether it was worth it to learn the entire new Angular2 workflow and set of tools for my small site. For this project, the answer is no but I did…
July 12, 2015Quality IS a Dev Task
software-developmentquality-engineeringtest-engineeringQE
“ I am not a tester. ”, “ QA will figure it out. ”, “ Works for me. Should be good in production. ”: These words are uttered in development organizations across the globe. I can’t lie. I even uttered it once during a production push as a…