For much of the last decade, working in Conversational AI meant designing what a chatbot or voice assistant should say.
Conversation designers mapped flows. Developers integrated channels. NLP specialists trained intent models. Product managers tracked containment, completion and customer satisfaction.
Those skills still matter. However, the centre of gravity is shifting.
On 22 July, OpenAI introduced OpenAI Presence, an enterprise product for deploying voice and chat agents across customer-facing and internal workflows. The most interesting part of the announcement was not a new model. It was the operational layer surrounding the model: policies, permissions, standard operating procedures, approved actions, simulations, evaluations, guardrails, escalation rules and controlled improvement after launch.
This is an important career signal.
Employers increasingly need people who can do more than make an AI agent sound convincing. They need people who can make it useful, measurable, governable and dependable in production.
Welcome to the age of AI agent operations.
The Unit of Value Has Changed
A traditional chatbot was commonly judged at the level of the conversation:
- Did it recognise the correct intent?
- Did it give an appropriate answer?
- Did the user reach the end of the flow?
- Did it successfully contain the enquiry?
A production AI agent must be judged at the level of the outcome:
- Did it understand what the user needed?
- Did it retrieve the correct information?
- Did it use the right tool?
- Was it permitted to take that action?
- Did it follow company policy?
- Did it recognise uncertainty?
- Did it escalate at the right moment?
- Can the organisation explain what happened afterwards?
That distinction changes the work.
Fluency is no longer enough. An agent can sound natural while still applying the wrong policy, calling the wrong system or confidently completing an action it should never have attempted.
The valuable professional is therefore not simply the person who writes the best dialogue. It is the person who can help turn an unpredictable model into a controlled service.
“AI agent operator” may not become the standard job title. The work will appear across Conversational AI engineering, conversation design, product management, service design, AI operations, quality assurance and customer experience roles. But the underlying capabilities are becoming easier to identify.
1. Translating Business Processes into Agent Behaviour
Production agents need a clearly defined job.
That starts with understanding the underlying service: the user’s objective, the organisation’s process, the information required, the available actions and the conditions under which the agent must stop.
This is more than converting a flowchart into a prompt. It involves turning policies and standard operating procedures into explicit behavioural rules.
For example, a billing agent might be allowed to:
- Retrieve an account balance.
- Explain a charge.
- Correct contact information after verification.
- Offer an approved payment plan.
- Escalate a disputed transaction.
It might not be allowed to waive a fee, alter a contract or make a financial recommendation without approval.
Professionals who can uncover and document these boundaries will be valuable because they connect operational reality with agent behaviour. Experience in service design, business analysis, customer operations or process mapping can transfer particularly well here.
2. Tool Use, Integrations and Permission Design
The difference between a chatbot and an agent is often its ability to act.
That action might involve checking an order, updating a CRM record, rescheduling an appointment, creating a support ticket or initiating a refund. Each action requires an integration and an appropriate level of access.
Knowing how APIs, tools and enterprise systems fit together is therefore becoming useful well beyond engineering roles.
Conversation designers and product managers do not necessarily need to build every integration themselves. They should, however, understand:
- What information the agent needs.
- Which system provides it.
- What the agent can read or change.
- How identity should be verified.
- Which actions need human approval.
- What should happen if a tool is unavailable.
- How an action can be reversed or corrected.
Meta’s latest AI update provides another example of this direction. The company is introducing recurring tasks and daily briefings that draw on user context, with expansion to platforms including WhatsApp planned. As assistants begin to operate across time rather than respond to one isolated message, skills in state management, consent, permissions and failure recovery become more important.
3. Evaluation Design
A good demonstration shows that an agent can work once. A good evaluation shows how reliably it works across many situations.
This is becoming one of the most valuable skills in Conversational AI.
Evaluating an agent requires more than reading a few transcripts. Teams need representative test scenarios, clear success criteria and repeatable ways to measure performance.
An evaluation set might test whether the agent:
- Reaches the correct outcome.
- Selects the right tool.
- Follows the relevant policy.
- Handles missing or contradictory information.
- Responds appropriately to interruptions.
- Recognises an unsupported request.
- Escalates higher-risk cases.
- Maintains an appropriate tone.
- Avoids inventing information.
The OpenAI Realtime Eval Guide describes this in terms of datasets, graders, an evaluation harness and a production feedback loop. The platform may change, but the underlying discipline is transferable.
Candidates who can discuss test coverage, failure taxonomies, evaluation metrics and regression testing will stand out from those who can only demonstrate prompt writing.
4. Human Handoffs and Failure Recovery
The best production agent is not necessarily the one with the highest containment rate.
It is the one that resolves appropriate requests while recognising when a person should take over.
A handoff is part of the service, not evidence that the design has failed. The agent needs to identify the trigger, gather the right context, explain what is happening and transfer the interaction without forcing the user to begin again.
Designers should consider:
- Low-confidence understanding.
- Policy exceptions.
- User distress or frustration.
- Authentication failures.
- Repeated unsuccessful attempts.
- Requests outside the agent’s authority.
- Tool and data failures.
- Safety, legal or financial risk.
OpenAI says Presence deployments define when an agent needs approval and when a person should take over. Its simulations then test whether the agent escalates in the right circumstances.
That is conversation design, service design, risk design and operational design working together.
5. Real-Time and Multimodal Experience Design
Voice agents introduce problems that do not appear in text-only experiences.
Latency, interruptions, turn-taking, background noise, pronunciation, pacing and silence can all affect whether the interaction works. A technically correct response delivered several seconds late may still feel broken.
Multimodal agents create further questions. Should an answer be spoken, displayed or both? What information is too complex to communicate through voice alone? What happens if the user switches channel halfway through the task?
These are not cosmetic details. They affect understanding, trust and completion.
Current Bot Jobs vacancies show that employers still value specialists who understand the interaction itself. A Senior Conversational Designer at Autodesk, for example, is focused on clear and trustworthy AI experiences across support, commerce, customer engagement and multimodal interactions.
The model may supply the words, but professionals still need to design the experience.
6. Observability and Continuous Improvement
Launching an agent is the start of the work.
Teams need to understand what happens in production:
- Which tasks are being attempted?
- Where are users abandoning conversations?
- Which actions fail most often?
- What causes human escalation?
- Which policies are misunderstood?
- Are changes improving one group of interactions while damaging another?
- Has a model or system update altered behaviour?
Production conversations can reveal new language, new user needs and failure cases that were not anticipated during testing.
OpenAI reports that the improvement process behind Presence reduced human handoffs in its own English-language phone support by 15 percentage points in ten days. That is a company-reported result, but it demonstrates the operating model: observe, diagnose, propose a change, test it and roll it out under human control.
Experience with conversation analytics, quality assurance, experimentation and incident review can all transfer into this work.
7. Governance and Change Control
An agent connected to business systems can create more harm than a chatbot that only gives a poor answer.
That means teams need to know:
- Who owns the agent.
- Who approves its instructions and tools.
- What evidence is required before launch.
- How access is restricted.
- How changes are tested.
- Which interactions are logged.
- How incidents are reported.
- How an earlier version can be restored.
Governance is not separate from product quality. It is part of making the agent dependable.
Bank of America’s latest EricaAssist update illustrates this connection. More than 18,000 customer service employees use the human-assisted agent during client calls. The company says it provides contextual guidance in under three seconds while keeping employees responsible for the interaction and operating with human oversight, transparency and accountability.
That combination of speed, assistance and control is the real production challenge.
What Current Vacancies Are Telling Us
The current Bot Jobs market reflects the cross-functional nature of this work. At the time of writing, the platform displays 40 Engineering roles, 15 Product Management roles, 14 Conversation Design roles and 13 Generative AI roles. Categories may overlap, so these figures should be treated as a directional snapshot rather than a count of unique vacancies.
Individual listings make the signal clearer:
- Sovos is hiring a Senior Conversation Designer to own the content and conversational experience of customer-support agents, with an emphasis on accuracy, brand and resolution.
- Autodesk is hiring a Senior Conversational Designer for trustworthy AI-powered support, commerce and customer engagement.
- Maven AGI is hiring a Group Product Manager for its Voice Agent, combining voice product strategy with cross-functional delivery.
- Telenet is hiring a Conversational AI Specialist, reflecting continued demand for dedicated practitioners inside established businesses.
The titles differ. The shared theme is responsibility for an agent’s performance in a real service, not simply its ability to produce plausible language.
Build a Portfolio That Shows More Than a Demo
If you are updating your Conversational AI portfolio, another simple FAQ bot may not show enough.
Instead, create one tightly scoped agent case study and document the production thinking around it.
For example, design an appointment-rescheduling agent and include:
1. The job and its boundaries
Define what the agent can complete, what it cannot do and when approval is required.
2. The service and conversation design
Show the main journey, ambiguity handling, interruptions, confirmation, recovery and human handoff.
3. The tool and permission map
Explain which systems the agent uses, what data it accesses and which actions are restricted.
4. The evaluation set
Create 15–20 scenarios covering normal requests, missing information, edge cases, policy conflicts and escalation.
5. The measurement plan
Define outcome completion, tool accuracy, policy adherence, handoff quality and user-experience measures.
6. The improvement record
Show one failure, how you diagnosed it, what you changed and how you checked that the change did not cause another problem.
This is far more persuasive than saying you understand agentic AI. It provides evidence that you can think beyond the prototype.
How to Position Your Existing Experience
You may already have many of the required skills under a different label.
- Conversation designers: Emphasise behavioural rules, evaluation, recovery, multimodal interaction and handoff design alongside dialogue writing.
- Engineers: Show production integrations, latency management, observability, permissions and reliable tool use.
- Product managers: Demonstrate how you define outcomes, manage risk, prioritise workflows and coordinate design, engineering, operations and governance.
- CX and operations professionals: Highlight your knowledge of customer needs, service processes, exceptions, quality assurance and escalation.
- NLP and data specialists: Connect model performance to representative evaluation data, monitoring and measurable service outcomes.
- Career changers: Use your sector knowledge. Understanding how banking, healthcare, retail or recruitment processes actually work can be as important as knowing the latest platform.
“Prompt engineering” alone is unlikely to communicate this breadth. Where the evidence supports it, more useful descriptions might include agent behaviour design, conversation evaluation, workflow orchestration, human–AI service design or AI agent operations.
The Next Conversational AI Career Is Built Around Reliability
Conversational AI is not becoming less human-centred. It is becoming more operational.
As agents gain access to data, tools and business processes, organisations need professionals who can decide what those agents should do, test whether they do it correctly and improve them without losing control.
The tools will continue to change. The durable skills are clearer:
- Define what a successful outcome looks like.
- Set appropriate boundaries.
- Design the interaction and the action.
- Test normal and abnormal behaviour.
- Build effective human escalation.
- Measure real performance.
- Control changes after launch.
If you can demonstrate those capabilities, you are no longer presenting yourself as someone who can build a convincing chatbot.
You are showing that you can help operate a dependable AI service.
Explore the latest Conversational AI, voice AI, product and agent roles on Bot Jobs or create your candidate profile to be discovered by specialist employers.
