# Agent Engineer: A Practical Career and Portfolio Guide

> Understand agent engineering across tools, state, evaluation and execution, with a portfolio exercise and scoped employer examples.

- Source: https://www.zarifautomates.com/blog/agent-engineer-career-guide
- Published: 2026-09-17
- Updated: 2026-09-17
- Pillar: AI Careers
- Tags: agent-engineering, careers, tools, reliability
- Author: Zarif

---

Agent engineering is the work of building systems in which a model can choose actions, observe results and continue toward a goal. It includes the software around the model: tools, permissions, state, stopping rules, evaluations and operations. “Agent Engineer” is a useful description of this work, but employers use several titles for it.

## Application work and infrastructure work differ

[Clay's Software Engineer, Applied AI posting](https://jobs.ashbyhq.com/claylabs/5e07db20-d96a-4dff-b7d3-3bf1cdde6fc1) describes both agent products and the harness, memory, tools and evaluation infrastructure beneath them. [OpenAI's Software Engineer, Agent Infrastructure posting](https://jobs.ashbyhq.com/openai/c1316397-25bb-4add-9e9d-0e3ea8ba929a) focuses on execution and training infrastructure at scale. Neither listing is literally titled “Agent Engineer.” Read them as examples of where agent-related work sits, not proof of a standardized occupation.

| Layer | Main design question | Evidence to show |
| --- | --- | --- |
| Task and orchestration | What decisions does the model need to make? | A workflow baseline and a reason for adding autonomy |
| Tools and permissions | What can an action change? | Narrow schemas, authorization and failure behavior |
| State and recovery | What survives a crash or retry? | Durable run state and a replay demonstration |
| Evaluation | Did the task succeed correctly? | Outcome checks plus a review of traces |
| Runtime | Where does untrusted execution happen? | Isolation, resource limits and operational ownership |

## Learn the execution model before adding autonomy

Start with ordinary backend engineering: HTTP interfaces, input validation, databases, transactions, tests and observability. An agent's bad choice often becomes a familiar software problem at the point where it changes something. You need to understand that boundary without relying on a prompt.

**Build a fixed workflow first.** Give it an explicit input, a sequence of steps and a terminal result. Draw the states that survive between requests. Then explain which choice genuinely requires a model: selecting relevant evidence, classifying an ambiguous ticket, or proposing the next tool. If the task is always the same three steps, keep those steps in code.

**Add one narrow tool.** Read [what MCP standardizes](/blog/what-is-model-context-protocol-mcp), then define a tool's arguments, result and errors. Keep identity and permission checks in the application. A model choosing a correctly shaped record ID does not establish that the user may access it. Demonstrate one allowed call and one rejection before adding tools from the [MCP directory](/blog/mcp-servers-for-ai-builders).

**Make interruption routine.** Persist the proposal, approval and action result, then stop and restart the process between steps. Add deadlines, a step limit and an explicit path for unresolved outcomes. The useful question is not whether the agent can keep trying; it is whether another worker can determine what happened and what is safe to do next.

**Evaluate outcomes, then inspect traces.** A run can make valid tool calls and still accomplish the wrong task. Check the final record, permissions, evidence and completion state. Use [monitoring and debugging](/blog/how-to-monitor-and-debug-ai-agents) to design traces that reveal the failing boundary without exposing secrets or requiring hidden model reasoning.

Only then inspect one [agent starter](/blog/best-ai-agent-repos-and-starter-templates) and map its abstractions to your working example. Use the [environment guide](/blog/best-ai-agent-development-environments) to distinguish a coding assistant, execution runtime and training environment. The [RL directory](/blog/rl-environments-for-coding-agents) is useful for understanding task resets and verifiers; training a model is not a prerequisite for building an agent application.

## Build a recoverable agent exercise

Create a local support-triage system with synthetic tickets and durable SQLite state. It can read a ticket, retrieve an applicable policy, propose one label change and wait for a person to decide. A deterministic executor applies an approved proposal. Start with authored proposals so the execution behavior can be tested without a model subscription.

Use separate records for tickets, runs, proposals, decisions and action receipts. A ticket has an owner, ID, label and version. A proposal has immutable contents, a target version, evidence IDs and an expiry. A decision identifies the exact proposal the reviewer saw. An action receipt connects that approved proposal to the resulting ticket version.

For a worked example, ticket `T-104` belongs to `team-a`, has label `untriaged` and version 7. A policy-backed proposal requests `billing-review`. Approval must refer to that exact target and change. If another process changes the ticket to version 8 before execution, applying the old proposal should fail visibly. A new proposal requires another review; approval is not a reusable permission to change the ticket however the model later chooses.

Implement this progression:

1. **Prepare:** validate the ticket owner, allowed label and cited policy; store the complete proposal.
2. **Wait:** display the stored change and evidence to the reviewer. A process restart must not reconstruct a different proposal from new data.
3. **Decide:** store approval or rejection for that proposal. Reject missing, expired or mismatched decisions.
4. **Apply locally:** in one database transaction, recheck the ticket version, change the label and write the receipt. Repeating the same approved action returns its receipt.
5. **Recover:** after a restart, read durable state and report whether the run is waiting, rejected, stale or applied. Do not infer success from the last log line.

That transaction covers a local ticket and receipt in the same database. If you extend the project to an external CRM, the remote write and local receipt are no longer one transaction. Use a supported idempotency key or a reliable status lookup to reconcile an unknown outcome. If neither exists, stop for investigation instead of blindly repeating a potentially completed action.

## The failure demonstrations belong in the portfolio

Run the following cases and save their actual traces. They are requirements for the exercise, not claimed results for an implementation supplied by this page.

| Scenario | Required observation |
| --- | --- |
| Valid proposal, approval and unchanged ticket | One label change and a matching receipt |
| No approval, rejection or expired approval | No ticket change; specific terminal or waiting state |
| Ticket changes after review | Version conflict; old approval is not reused |
| Same action requested after process restart | Existing receipt returned; no extra mutation |
| Failure inside the local action transaction | Ticket and receipt both roll back |
| Run stops after proposal storage | A new process loads the same proposal for review |
| Valid tool arguments name another owner's ticket | Authorization rejection before a read or mutation |
| Unknown tool or malformed arguments | Rejection without executing a substitute |
| Tool times out or the step budget is exhausted | Bounded stop with an explicit unresolved result |
| External action outcome is unknown | Reconciliation or human investigation before any repeat |

Use two independent processes for the restart demonstration. Reusing a Python object in one test does not prove persistence. State exactly how you induced an interruption: raising an exception in a transaction is different from terminating a process or losing power. Do not claim tests you did not perform.

If you add a model, let it propose actions through the same validated interface. Record model and prompt versions, observed tool choices and task outcomes. Keep authored fixtures as regression tests and run a separate, clearly labeled live-model evaluation. Passing the deterministic tests says the executor enforces its rules; it does not establish that the model will choose good proposals.

## How to present the work

Give a reviewer setup commands, synthetic fixtures, a state diagram, a schema, a test report and three short traces: success, stale approval and recovery. Include a threat/permission note naming what the local demonstration does and does not authenticate. A hard-coded test owner is not a production identity system.

Use a five-minute demonstration: create a proposal, inspect its evidence, restart, approve, change the underlying ticket, and show the stale rejection. Then run a fresh approved case twice and show the same receipt. Explain one unresolved tradeoff, such as reconciliation with an external service that lacks idempotency support.

The review criteria are concrete: authority stays outside the model, state survives the promised interruption, repeated requests do not duplicate the local action, failures are inspectable, and the report distinguishes executor checks from model quality. Adding a second agent is useful only if you can show which part of the task it improves and how you assess that improvement.

## Compensation in related employer postings

Checked September 17, 2026 on the employers' public boards. These are annual USD salary examples from related roles, not an “Agent Engineer” market average. Both separately list equity. [Ashby's date](https://developers.ashbyhq.com/docs/public-job-posting-api) records last publication, not our observation date or necessarily the latest text edit.

| Employer and exact title | Listed location context | Posted salary | Source last published |
| --- | --- | --- | --- |
| [Clay: Software Engineer, Applied AI](https://jobs.ashbyhq.com/claylabs/5e07db20-d96a-4dff-b7d3-3bf1cdde6fc1) | New York; hybrid | $170,000–$300,000 plus equity | August 14, 2026 |
| [OpenAI: Software Engineer, Agent Infrastructure](https://jobs.ashbyhq.com/openai/c1316397-25bb-4add-9e9d-0e3ea8ba929a) | San Francisco primary; New York and London also listed | $230,000–$385,000 plus equity; confirm location-specific terms | May 30, 2025 |

Clay asks for production LLM/agent work, backend fundamentals and evaluations, with team placement determined during its process. OpenAI emphasizes deep experience with large-scale AI infrastructure, distributed systems and research collaboration. The small portfolio above demonstrates application reliability; it does not reproduce that infrastructure scale. OpenAI's old publication date does not mean the posting was closed: it was still listed when checked. Its USD salary field does not verify the London package.

## Interview practice

**“The external action succeeded, but the worker crashed before saving a receipt.”** Draw both systems and the failure window. Explain what an idempotency key protects, how long the remote service retains it, and how you check an unknown outcome. A local transaction cannot make an unrelated remote service commit atomically.

**“The model supplies a valid schema with an unauthorized record ID.”** Trace identity from the caller to the service, then show where ownership is checked. Include reads as well as writes. A prompt saying “only use this user's data” is not an authorization mechanism.

**“Two agents edit version 7 of the same record.”** Show the version condition at the write boundary and what the losing request receives. Explain why it must reconsider the change rather than replay a stale decision against version 8. A retry policy without a conflict policy can repeat the wrong action faster.

**“A judge model says the task passed, but the requested label never changed.”** Check observable state and the receipt first. Separate task completion from the quality of an explanation. Then inspect the judge's input and scoring rule; a persuasive summary is not evidence that the action occurred.

These are original practice prompts. Compare the [AI Engineer path](/blog/ai-engineer-career-guide) and the customer implementation focus of the [AI Careers hub](/blog/pillar/ai-careers) before deciding which role fits your existing strengths.

Use the [MCP server directory](/blog/mcp-servers-for-ai-builders) to inspect integrations and the [benchmark directory](/blog/ai-agent-benchmarks-and-leaderboards) to choose an external evaluation reference.

## Related Guides

- [GTM Engineer: A Practical Career and Portfolio Guide](/blog/gtm-engineer-career-guide)
- [How to Transition Into an AI Career: Complete Guide](/blog/how-to-transition-into-an-ai-career-complete-guide)
- [Forward Deployed Engineer Interview Guide](/blog/forward-deployed-engineer-interview-guide)