- Step 1 of 10 in the Agent Engineer path
Agent Engineer: A Practical Career and Portfolio Guide
Zarif
A support agent reads a ticket, decides it's a billing question, and changes the ticket's label. That's one decision by the model and one change to a database.
Now the program crashes right after the change. Did the label actually change? Should it try again? And was this user even allowed to touch that ticket in the first place?
Agent engineering is the job of answering those questions before they come up. You build software where a model picks actions, sees what happened, and keeps going toward a goal. Most of the work is the software around the model: the tools it can call, what it's allowed to change, what gets saved, when it stops, how you test it, and how you keep it running.
"Agent Engineer" is a handy name for that work, but employers use several different titles for it.
What the work looks like in real postings
There are two broad flavors: building agent products, and building the systems those products run on.
Clay's Software Engineer, Applied AI posting covers both the agent products and what sits underneath them: the harness, memory, tools and evaluation setup. OpenAI's Software Engineer, Agent Infrastructure posting is about the systems that run and train agents at very large scale.
Neither one is actually titled "Agent Engineer." Treat them as examples of where this work shows up, not proof that it's a settled job title.
Either way, the work breaks into five parts:
| Part of the system | The question you answer | What to show |
|---|---|---|
| Task and flow | Which decisions actually need a model? | A plain fixed workflow first, and a reason for each choice you hand to the model |
| Tools and permissions | What can each action change? | Small, strict tool inputs, permission checks, and what happens when a call fails |
| Saved state and recovery | What survives a crash or a retry? | Run state saved to a database, and a demo of picking back up after a restart |
| Testing | Did the task actually get done right? | Checks on the final result, plus a read through the step-by-step log |
| Where it runs | Where does code you don't trust get executed? | A sealed-off environment, resource limits, and who's on call for it |
Why it's harder than it looks
When a model makes a bad choice, the damage usually happens at the moment it changes something: a database row, an email, a ticket. At that point it's an ordinary software problem. You need to understand that moment without relying on a prompt to keep things safe.
That's why the path starts with regular backend skills: web APIs, checking inputs, databases, transactions, tests, and logging. Then you add the model one piece at a time.
Build a fixed workflow first
Give it a clear input, a set of steps, and a final result. Draw what gets saved between requests.
Then point to the one choice that really needs a model: picking the relevant evidence, sorting an unclear ticket, or suggesting the next tool. If the task is always the same three steps, keep those steps in code.
Add one narrow tool
Read what MCP standardizes, then define one tool's inputs, output and errors. Keep identity and permission checks in your application, not in the model.
A model can produce a perfectly formatted record ID for a record the user isn't allowed to see. The format being right says nothing about permission. Show one allowed call and one rejected call before you add tools from the MCP directory.
Make stopping and restarting normal
Save the proposal, the approval and the result of the action. Then stop the program between steps and start it again. Add deadlines, a step limit, and a clear path for runs that don't finish.
The useful question isn't whether the agent can keep trying. It's whether a different worker could pick up the run, figure out what happened, and know what's safe to do next.
Check results, then read the logs
A run can make perfectly valid tool calls and still do the wrong job. Check the final record, the permissions, the evidence, and whether it actually finished.
Use the monitoring and debugging section of the deployment guide to design step-by-step logs (traces) that show where things broke, without exposing secrets or depending on the model's hidden reasoning.
Only after that, open one agent starter and match its pieces to your working example. The environment guide explains the difference between a coding assistant, a place to run agent code, and a training environment. The RL directory helps with how tasks get reset and checked, but you don't need to train a model to build an agent application.
A worked example: one ticket, one stale approval
Here's the kind of situation the portfolio project below has to handle.
Ticket T-104 belongs to team-a, has the label untriaged, and is at version 7. Based on a policy, the agent proposes changing the label to billing-review. A person approves that exact change to that exact ticket.
Before the change runs, another process edits the ticket, and it moves to version 8.
Now the old approval is out of date. Applying it should fail, and fail where someone can see it. A new proposal needs a new review. An approval isn't a permission slip for the model to change the ticket however it likes later.
Build a recoverable agent project
Build a small support-triage system on your own machine, with made-up tickets and a SQLite database. It reads a ticket, finds the matching policy, proposes one label change, and waits for a person to decide. A plain piece of code, not the model, applies the change once it's approved.
Start with proposals you write by hand. That way you can test how the system behaves without paying for a model.
Keep separate records for tickets, runs, proposals, decisions, and action receipts:
- A ticket has an owner, an ID, a label, and a version number.
- A proposal can't be edited once saved. It has the target version, the IDs of its evidence, and an expiry time.
- A decision points to the exact proposal the reviewer saw.
- A receipt links the approved proposal to the ticket version it produced.
Build it in this order:
- Prepare: check the ticket owner, the allowed label, and the cited policy. Save the whole proposal.
- Wait: show the reviewer the saved change and its evidence. A restart must not rebuild a different proposal from newer data.
- Decide: save the approval or rejection for that proposal. Refuse decisions that are missing, expired, or point to the wrong proposal.
- Apply locally: in one database transaction, recheck the ticket version, change the label, and write the receipt. Asking for the same approved action again just returns the receipt.
- Recover: after a restart, read the saved state and report whether the run is waiting, rejected, stale, or applied. Don't guess success from the last log line.
That one transaction works because the ticket and the receipt live in the same database. If you extend the project to an outside CRM, they don't anymore.
Picture it: the CRM accepts the change, then your worker crashes before writing the receipt. You don't know if it worked. Use the CRM's idempotency key (a unique ID that lets it recognize a repeat request) or a reliable way to look up the status, and settle the question before you do anything else. If neither exists, stop and have a person look, instead of blindly repeating something that may already have happened.
Show the failures, not just the success
Run each of these cases and save the actual logs. They're requirements for your project, not results this page is claiming for any code it supplies.
| Scenario | Required observation |
|---|---|
| Valid proposal, approval and unchanged ticket | One label change and a matching receipt |
| No approval, rejection or expired approval | No ticket change; specific terminal or waiting state |
| Ticket changes after review | Version conflict; old approval is not reused |
| Same action requested after process restart | Existing receipt returned; no extra mutation |
| Failure inside the local action transaction | Ticket and receipt both roll back |
| Run stops after proposal storage | A new process loads the same proposal for review |
| Valid tool arguments name another owner's ticket | Authorization rejection before a read or mutation |
| Unknown tool or malformed arguments | Rejection without executing a substitute |
| Tool times out or the step budget is exhausted | Bounded stop with an explicit unresolved result |
| External action outcome is unknown | Reconciliation or human investigation before any repeat |
Use two separate processes for the restart test. Reusing the same Python object inside one test doesn't prove anything was saved.
Say exactly how you caused each interruption. Throwing an error inside a transaction is different from killing the process or losing power. Don't claim tests you didn't run.
If you add a model, have it propose actions through the same checked interface. Record the model and prompt versions, which tools it picked, and how each task turned out. Keep the hand-written proposals as tests that run every time, and run a separate, clearly labeled test with the live model.
Passing the hand-written tests shows your code enforces its rules. It doesn't show the model will propose good changes.
How to present the work
Give a reviewer:
- setup commands and the made-up test data
- a diagram of the states a run can be in
- the database schema
- a test report
- three short logs: a success, a stale approval, and a recovery
- a short note on what this local demo does and doesn't check about who's calling it
A hard-coded test owner is not a real login system, so say so.
For a five-minute demo: create a proposal, show its evidence, restart, approve, change the ticket underneath it, and show the stale rejection. Then run a fresh approved case twice and show the same receipt both times. End with one problem you haven't solved, such as settling an unknown result with an outside service that has no idempotency support.
A reviewer will look for five things. The model never holds the authority. Saved state survives the interruption you said it would. Repeating a request doesn't repeat the action. Failures are easy to inspect. And the report keeps "my code enforces its rules" separate from "the model makes good choices."
Adding a second agent only helps if you can show which part of the task it improves and how you measured that.
Compensation in related employer postings
Checked September 17, 2026 on the employers' public boards. These are annual USD salary examples from related roles, not an "Agent Engineer" market average. Both separately list equity. Ashby's date records when the posting was last published, not when this page checked it, and not necessarily the latest text edit.
| Employer and exact title | Listed location context | Posted salary | Source last published |
|---|---|---|---|
| Clay: Software Engineer, Applied AI | New York; hybrid | $170,000–$300,000 plus equity | August 14, 2026 |
| OpenAI: Software Engineer, Agent Infrastructure | San Francisco primary; New York and London also listed | $230,000–$385,000 plus equity; confirm location-specific terms | May 30, 2025 |
Clay asks for experience shipping LLM or agent work to real users, solid backend skills, and evaluation work. Which team you land on gets decided during its hiring process.
OpenAI puts the weight on deep experience with very large AI systems, distributed systems, and working with researchers. The small project above shows you can make an application reliable. It doesn't recreate that kind of scale.
OpenAI's old publication date doesn't mean the posting was closed. It was still listed when checked. Its USD salary field doesn't confirm what the London role pays.
Interview practice
These are practice prompts written for this guide, not questions reported from real interviews.
"The outside action worked, but the worker crashed before saving a receipt"
Draw both systems and the moment the crash happened. Explain what an idempotency key protects, how long the outside service keeps it, and how you check an unknown result. A transaction in your own database can't force someone else's service to commit at the same time.
"The model sent a correctly formatted ID for a record the user can't access"
Follow the user's identity from the request all the way to the service, and show where ownership gets checked. Cover reads as well as writes. A prompt that says "only use this user's data" is not a permission system.
"Two agents edit version 7 of the same record"
Show the version check at the moment of writing, and what the losing request gets back. Explain why it has to rethink the change instead of replaying an out-of-date decision against version 8. Retrying without handling conflicts just repeats the wrong action faster.
"A judge model says the task passed, but the label never changed"
Check the actual saved state and the receipt first. Keep "did the task happen" separate from "was the explanation good." Then look at what the judge was given and how it scored. A convincing summary isn't proof the action happened.
Where to go next
Compare this with the AI Engineer path and the customer-facing roles in the AI Careers hub before you decide which fits what you're already good at.
When you start building, use the MCP server directory to look at real integrations and the benchmark directory to pick an outside test to compare against.
