What Is an AI Product Engineer? The Role, the Skills and What the Work Actually Looks Like

TL;DR: An AI Product Engineer builds products where a model does part of the work, and owns them end to end: the user problem, the model and prompt choices, the evals, the application around the model, the rollout, and the quality and cost numbers after launch. It isn't ML research (you rarely train models) and it isn't plain software engineering (your core component is probabilistic). The job is to turn a capable model into a feature people trust and a business can afford.

I'm Harsh Rastogi, AI Product Engineer and Business AI Head at Modelia, where we build AI for the fashion ecosystem: DAM and PIM, catalogue management, product imagery and visual generators with bulk generation. This post is the description of the role I'd have wanted when I started doing it.

A Working Definition

An AI Product Engineer is the person accountable for whether an AI feature works for users, not just whether the code runs. In practice that covers five things:

1. 01Problem framing. Deciding which part of a workflow a model should do, and which part should stay deterministic code or a human.
2. 02Model and prompt design. Choosing the model tier for each task, writing prompts and tool definitions, and structuring outputs so the rest of the system can rely on them.
3. 03Evaluation. Building the test sets and metrics that tell you whether a change made the feature better or worse.
4. 04Product engineering. The application, APIs, queues, storage, UI and integrations around the model, which is usually most of the code.
5. 05Operating it. Rollouts, guardrails, cost control and monitoring once real users and real data hit it.

How It Differs From Neighbouring Roles

The big difference from ordinary software engineering is that your most important component gives different answers to the same input. A passing unit test tells you very little. You need evals, staged rollouts and monitoring that measure quality, not just uptime.

The big difference from ML engineering is where the effort goes. With today's foundation models, most of the leverage is in context, tools, product design and evaluation, not in training. You pick models, you don't build them.

The Skills That Matter

1. Solid full-stack engineering

Most of an AI product is not the model call. It's auth, data models, background jobs, file storage, webhooks, retries and a UI people want to use. I work mostly in TypeScript, Node.js, React and PostgreSQL. A model wrapped in a fragile app is a fragile product.

2. Knowing what models are good and bad at

You need a working sense of which tasks a small fast model handles, where a frontier model is worth its price, and when a task should not be given to a model at all. That comes from running real workloads and reading the outputs, not from leaderboards.

3. Evals

The most underrated skill. A good eval set is a few dozen to a few hundred real examples with a clear pass/fail or scoring rule. Without one, every prompt change is a guess and every model upgrade is a gamble.

4. Agent and tool design

More features now involve a model calling tools: reading a catalogue, querying a database, editing an image, opening a pull request. Designing narrow tools, clear permissions and recoverable failures is core work. I've written about this in Building Agentic AI Systems: Real-World Patterns and Agentic AI Error Recovery and Observability.

5. Cost and latency thinking

Every call has a price and a wait. Choosing the right model tier per task, caching repeated context and keeping prompts tight are product decisions, because they decide whether a feature can be offered at scale. See How to Run an LLM Cost Audit.

6. Product judgement

Knowing when "good enough 90% of the time" is fine (a draft the user edits) and when it isn't (anything that sends, charges or deletes). That judgement drives the whole design: human review, fallbacks, and how much autonomy the feature gets.

What the Work Looks Like Day to Day

A typical week mixes all of the above:

• Reading outputs. Going through real generations or agent traces, tagging failures, and turning the patterns into eval cases.
• Changing one thing at a time. A prompt edit, a model swap, a new tool. Each one is run against the eval set before it ships.
• Building the product around the model. Bulk-generation queues, review screens, integrations with the systems customers already use.
• Rolling out carefully. Feature flags, internal testing, small cohorts, then a ramp with a kill switch ready. (Full playbook: How to Ship an AI Feature Safely.)
• Talking to users and the business. What failed for them, what it cost, and what to build next.

How to Become One

If you're a software engineer, you're most of the way there. The gap is usually evals, model judgement and agent design. The fastest way to close it:

1. 01Ship one small AI feature to real users, with a feature flag and a way to see every output.
2. 02Build an eval set from its failures and use it before every change.
3. 03Track tokens and cost per feature from day one.
4. 04Build one agent with real tools, and make it fail safely.
5. 05Write about what broke. It forces clarity and it's how people find your work.

I've packaged some of my own checklists as Claude Code agent skills you can use directly: LLM cost audit, AI feature rollout and agentic error recovery.

The Short Version

An AI Product Engineer is measured by one question: does the AI feature do the job for users, reliably, at a cost the business can carry? Everything in the role, from evals to rollouts to cost audits, serves that question.

Harsh Rastogi is an AI Product Engineer and Business AI Head at Modelia, building enterprise-grade AI products and agents. He writes about shipping AI to production at harshrastogi.tech.