Becoming an AI forward deployed engineer: why most AI projects never reach production — and who gets them there
More than 80% of AI projects fail before production, the speaker said — and the people who get them there spend most of their time on people and old systems, not models. Our notes from Dr Yu Yong Poh’s free AITraining2U webinar, with a live demo of a dispute agent built the naive way and the hardened way.
- Talk
- Becoming an AI Forward Deployed Engineer
- Speaker
- Dr Yu Yong Poh — Deputy Director of CBIEV at TAR UMT, former head of the data science laboratory at a public listed bank
- Host
- Warren, founder of AITraining2U
- When
- Thursday 8 October 2026, listed for 9:00–10:30 pm Malaysia time; billed as a 90-minute masterclass
- Where
- Online, one of AITraining2U’s free Thursday webinars
- Material
- 38 slides with seven “Thinking Corners”, and a live web demo of a dispute agent
Becoming an AI Forward Deployed Engineer was a free Thursday-night webinar by AITraining2U on 8 October 2026, given by Dr Yu Yong Poh and hosted by the company’s founder, Warren. The invitation promised a live example of “why naive AI scripts crash in production and how Forward Deployed Engineers harden enterprise agents”. Three weeks later the speaker gave a hands-on follow-up on building the agent itself; together the two talks make this two-part series.
The infographic sums up the talk on one page. The sections after it go through each idea with the speaker’s own examples.
Terms used in this part
- Forward deployed engineer (FDE)
- An engineer who works inside a client’s company to build, harden and connect AI systems that run the business — part consultant, part developer, part system integrator.
- Proof of concept (PoC)
- A quick demo that shows an idea can work. It is not built to survive real users, real data or real attacks.
- Production hardening
- The work that turns a demo into a system people can rely on: input checks, limits, monitoring, approvals and a way to roll back.
- System integration
- Connecting the new system to the company’s existing ones — databases, ERP, CRM, old mainframes — so data flows between them.
- Agentic AI
- AI that does not only answer but acts: it plans steps, calls APIs, queries databases and changes stored data.
- Prompt injection
- A trick in which a user’s text tells the AI to ignore its rules, for example to approve a refund it should refuse.
- Maker–checker
- A control in which one party makes a change and another, with authority, checks and approves it before it takes effect.
Who is Dr Yu Yong Poh?
Dr Yu Yong Poh is Deputy Director of CBIEV at Tunku Abdul Rahman University of Management and Technology (TAR UMT), where the work is linking the university with industry and turning research into commercial projects. By the slides’ account, the speaker previously headed the data science laboratory at a public listed bank, put more than 100 machine learning models into live banking systems, designed a large all-in-one recommender system and managed prediction models behind more than RM1 billion in revenue. The host added that the speaker had also worked on AI for an airline.
What is a forward deployed engineer?
A forward deployed engineer (FDE) is an engineer who works inside a client’s company to build, harden and connect the AI systems that run its business. Unlike a solutions engineer, who builds demos before a sale, an FDE ships production software into the client’s live systems and is judged by uptime and the client’s return on investment.
The slides define an FDE as an engineer embedded inside client environments to build, harden and integrate operational AI systems. The role comes from technology companies such as Palantir. An FDE does not write throwaway proofs of concept; an FDE ships production software into the client’s live systems, and then turns what was built for one client into features of the main product.
In the speaker’s words, coding is no longer rocket science. What makes the job hard is everything around the code. The client sees the FDE as a consultant, a developer and a system integrator at once, so the FDE must speak the client’s language without jargon, turn business needs into technical work, and turn technical results back into business terms. The host added that clients often cannot say what they want: the FDE has to draw out the need, confirm it, and explain the steps, the trade-offs, the real cost and how the return on investment will be measured.
| Role | Main focus | What the role delivers | How success is measured |
|---|---|---|---|
| Machine learning researcher | Model design and the maths of the loss function | Notebooks and model checkpoints | Accuracy on benchmarks |
| Solutions engineer | Pre-sales and demos | Scripted proofs of concept and slides | Technical deals won |
| Platform engineer | Internal tools and computing | Cluster set-up files and SDKs | How fast the company’s own developers work |
| Forward deployed engineer | Integration inside the client’s live systems | Production microservices | Uptime and the client’s return on investment |
The four stages of an engagement
The four stages of an FDE engagement
Discovery and scoping
Audit the client’s firewalls and how clean the data is, and agree strict response-time targets. Talk to everyone involved: the boss and the staff may want different things.
Fast prototype
Deliver a working end-to-end pipeline with sample data within 14 days. A working prototype explains more than a thousand words.
Production hardening
Add encrypted connections between services (mTLS), schema guards, Prometheus metrics, an approval process and automatic rollback.
Platform feedback
Turn the adapters built for this client into shared parts of the main product, so the next client starts further ahead.
User acceptance testing with the client, the speaker stressed, comes before anything goes live.
The host’s view was that many people can now do the first two stages with simple tools such as n8n and LangGraph, and start offering a little AI consulting. The real shortage in Malaysia, the host said, is people who can do stages 3 and 4 for large companies.
Why do most AI projects fail before production?
Most AI projects fail between the demo and production because a demo is built to impress, not to survive: the data is missing or messy, the notebook it runs in has no contracts or monitoring, the client’s network blocks it, and nobody planned the running cost. The slides open with three research figures.
- >80%of AI projects fail before productionRAND Corporation, 2024
- 30%of generative AI projects abandoned after the proof of conceptGartner, July 2024: poor data, weak risk controls, high running costs
- 40%of agentic AI projects at risk of cancellationA Gartner forecast for 2025–2027: agent loops without limits, writes with no rollback
The notebook is the usual culprit. Jupyter notebooks such as Google Colab are excellent for exploring, but in a client’s company they fail in three ways.
Hidden global state
Cells can run in any order, so variables hide and the same run does not give the same result twice.
No API contracts
No input validation, rate limits, token budgets or health checks — nothing another system can rely on.
Zero telemetry
Nobody can see response times, changes in the data, or an agent loop going wrong.
The speaker added reasons from experience. Sometimes the data for the planned model does not exist: a property company wanted a lead-scoring agent but had no history to learn from, so a data pipeline had to be built first. Company networks often block the tunnels and demo tools, such as Streamlit, that a prototype relies on. And cost surprises people: model APIs charge by the token, and some agent platforms charge for every message — even “hello”.
SME vs enterprise: what each kind of client needs from AI
A small business usually needs simple automation on top of the spreadsheets it already has; a large enterprise needs careful integration with old systems, which the speaker put at 70–80% of the work. The speaker treated small and medium enterprises (SMEs) and large enterprises as different clients with different problems. For an SME, the advice was to keep it simple; for an enterprise, to expect most of the work to be connecting systems.
| Item | SME client | Enterprise client |
|---|---|---|
| Main data problem | Data emptiness: little data | Data messiness: a lot of data in many tables with no clear links |
| Systems | Spreadsheets and simple tools | Old systems such as COBOL, SAP and SAS, often tied to Microsoft or Google tools |
| Rules | Few | Strict: no USB drives, only approved Python libraries, limits on cloud use |
| Suitable solution | Simple automation: link Google Sheets or Excel into a small database, add a dashboard and simple agents (n8n, Gemini Gems, Copilot Studio) | System integration (70–80% of the work), APIs, robotic process automation for systems with no API, and agents with approval steps |
| Do they need AI agents? | Mostly not: clean scripts and single-step AI tools; agent loops without limits can cost more than a human worker | Yes, to reach data locked in SAP, Salesforce and SQL databases — but only with state machines, rollback, rate limits and audit checkpoints |
| Whom to hire | An FDE, who connects ready-made AI APIs to QuickBooks, Shopify or HubSpot with token limits and caching | AI engineers to build many agents, working with FDEs who connect them to existing systems |
What is agentic AI, and why does it need an FDE?
Agentic AI is AI that acts rather than only answers: it plans a task in several steps, calls outside APIs, queries SQL databases, handles errors and changes the data the business keeps. Because a mistake now changes real records, the slides say the engineer’s role moves from “prompt designer” to “distributed systems architect”. They draw the flow in four phases.
How an agentic system runs a task, with its guardrails
- Task perception: understand the request and check it against a schema
- a plan, not a free chatState-graph planner: break the task into steps with retries (LangGraph)
- each step calls a toolTool dispatch through MCP: call authenticated company APIs in a standard way
- before data changesHuman guard gate: a person confirms any write before it happens
What skills does a forward deployed engineer need?
An FDE needs four groups of skills, according to the slides: solid software engineering, handling messy enterprise data, applied AI and LLMOps, and system integration — which the slides call “the real 70% of the job”. Security, deployment and talking to the business sit on top.
Software engineering
Strict typing and validation with Pydantic v2 and MyPy, asynchronous code so model calls do not block, and tests with mocked APIs.
Messy enterprise data
Fluent SQL, streaming and queues, and automated data cleaning.
Applied AI and LLMOps
Hybrid retrieval, fixed (deterministic) guardrails and continuous evaluation.
System integration
An API gateway, connectors to old systems, fallbacks when something fails, and telemetry — “the real 70% of the job”.
Security comes on top: OAuth2 and mutual TLS (mTLS), role- and attribute-based access control, and a cryptographic audit trail. Enterprise AI runs in one of three places — the client’s virtual private cloud, the client’s own servers with no public internet traffic, or an air-gapped network for defence and critical infrastructure. The toolchain on the slides runs from FastAPI, Pydantic and vLLM for serving models, through LangGraph, the Model Context Protocol (MCP), pgvector, Redis and Kafka for agents and data, to Ragas, Langfuse, DeepEval and OpenTelemetry for evaluation, and Docker, Kubernetes, Keycloak and HashiCorp Vault for infrastructure.
How to become a forward deployed engineer: a 12-month roadmap
The slides lay out a year in four quarters: first build a tested production service, then connect it to enterprise systems, then run and evaluate agents, and finally lead discovery and a deployment at a client.
- Months 1–3
Production core
Docker containers, asynchronous FastAPI, strict Pydantic and Git with continuous integration. Target: a containerised service with tests.
- Months 4–6
Enterprise plumbing
SQL window functions, OAuth2 sign-in, Redis queues and webhook buses. Target: a working bridge to an old CRM system.
- Months 7–9
Agent operations and evaluation
LangGraph state machines, tool discovery through MCP, and Ragas evaluation harnesses. Target: a fully evaluated local agent.
- Months 10–12
Field leadership
Discovery interviews, scoping a statement of work and architecture reviews with executives. Target: lead an embedded deployment at a client.
Three field cases
The slides walk through three cases, one in each industry. The speaker said they are built from real kinds of client request and adapted as examples; the results are the slides’ figures.
| Case | Problem and common mistake | What the FDE builds — and the result |
|---|---|---|
| Manufacturing: tracing where components come from | Air-gapped factory networks, messy serial data, machine protocols with no public SDK. Mistake: feeding static CSV files into a basic RAG store, which makes up batch IDs | An agent that walks the bill-of-materials graph: QA camera alert, search across supplier ERP systems, a safety gate above 98.5% confidence, then a batch quarantine in SAP. Result: recall of a bad batch cut from 72 hours to 4 minutes |
| Retail banking: product advice and customer care | COBOL mainframes about 30 years old, answers needed in under 200 ms, and privacy law. Mistake: letting an open-ended model approve fee refunds with no upper limit | An agent that reads spending patterns to suggest products and handles fee disputes, with a hard limit under $100 and a ledger that cannot be changed. Result: 62% of questions solved at first contact and 28% more sign-ups for tailored loans |
| E-commerce: complaints and disputes | Fake returns, gaps in third-party tracking, database lock problems. Mistake: an apology bot that sounds kind but has no tools to change the database | A care agent that checks the photo and claim, cross-checks the carrier’s GPS and weight data, writes to PostgreSQL and triggers a refund. Result: resolution time cut from 5 business days to 45 seconds, and escalations down 74% |
Talking to the business: “a cat talking to a dog”
One slide asks whether the technical team and the business team are “a cat talking to a dog”. Engineers report that a loss score fell from one number to another; the business hears only that six months and $300,000 are gone, and still does not know whether fewer customers are leaving. The result is cancelled budgets and resentment on both sides.
The FDE is the translator. Maths becomes money — a faster response is expressed as dollars saved per transaction. Executive goals are clarified before a line of code is written. And both sides look at one dashboard with joint measures, such as how fast disputes are resolved next to how often the API is up. A vague request must become a target with clear inputs, limits and a service level.
The client asks
Vague, unbounded, impossible to test
- “Can we plug an AI agent into our supply chain to make it smart?”
- Leads to scope creep and a project that never ends
The FDE translates
Clear inputs, limits and a service level
- “Predict component stock-outs 48 hours early with more than 92% precision.”
- Something both sides can measure and sign off
In updates to executives, the bottom line comes first: lead with time or money saved, flag blockers early, and keep technical depth in an appendix.
The researcher’s pitch — avoid
- “We fine-tuned a 7B-parameter model with LoRA rank 16 and reduced cross-entropy loss by 0.14.”
The FDE’s pitch — adopt
- “We cut dispute resolution from 5 days to 45 seconds, saving $380,000 a year with 99.4% audit accuracy.”
Each stakeholder worries about something different, and each needs something different from the FDE.
| Stakeholder | Worries about | What the FDE gives them |
|---|---|---|
| Business sponsor | Speed, revenue and a return on investment they can defend | A measurable demo every two weeks |
| Client IT lead | Downtime and software nobody will support | Automated set-up files (Helm charts) and runbooks |
| Security officer | Personal data leaks, unapproved database writes, compliance | Clean software bills of materials and penetration test reports |
When production breaks: the incident war room
Things go wrong in production, and the slides set out what the FDE does when they do.
Contain it
Trip a circuit breaker so the system falls back to fixed rules instead of taking the client’s apps down.
Say what is happening
Give every affected customer a status update every 30 minutes while the investigation runs.
Find the cause, without blame
Write a blameless root-cause analysis, with automated regression tests so it cannot happen again.
Fix the platform
Patch the edge case in the shared code, so the other clients’ deployments are protected too.
Seven Thinking Corners
The slides pause seven times for a short debate. These are the questions, and the answers on the slides.
| Question | Answer on the slides |
|---|---|
| Is the traditional data scientist still relevant? | Manual model tuning and analysis of static CSV files are now common goods. Statistics still matters for designing evaluations, and good data scientists move into FDE work, where the advantage is connecting the whole system |
| Open-source or proprietary AI models? | Proprietary models (OpenAI, Anthropic) reason best and need no servers, but bring lock-in, rate limits and data rules. Open-weight models (Llama, Mistral, Qwen) keep data in-house and run offline, but need GPUs; they suit defence, banking and healthcare |
| Do we really need AI agents? | Small firms mostly need simple scripts and single-step AI tools. Large enterprises do need agents, but only with fixed software guardrails |
| The more agents, the better? | No. Five agents that are each 90% reliable give only about 59% success end to end; agent chatter adds 30 seconds or more; free debates cannot be replayed to find a bug. Never use an agent where a 10-line Python function works; one strong model with strict JSON tools beats a swarm |
| Does an SME need an AI engineer or an FDE? | An FDE, who wires ready-made AI APIs into the systems the SME already has, keeps the monthly bill small with token limits and caching, and delivers something that pays for itself in weeks |
| Technical team and business team: a cat talking to a dog? | The FDE translates both ways — maths into money, goals into code — with one shared dashboard |
| Is coding still relevant in the era of AI? | Boilerplate and copy-paste coding are now done by AI, but reviewing code, designing systems and checking security matter more. The FDE’s strength is directing AI coding tools precisely and verifying what they produce |
Trust nobody: safety and accountability
The speaker’s example was a commission dispute. A property agent says this month’s pay should be 20K, not 17K. If an AI agent can change the database, who approves the change — and was the 17K already paid, so that only 3K is owed? The speaker’s rule is to “trust nobody”, human or agent. One person with authority approves the change; human errors go back to people and system bugs to the system; and user acceptance testing (UAT) with the client is essential before anything goes live.
“I practise trust nobody, be it agent or human.”
The other protections were need-to-know access to tables (one reason for many one-to-one tables), limits written into the system, maker–checker review, and above all defence against prompt injection, where a user tricks the AI into doing what it should not.
What does hardening an AI agent look like? A live demo
Hardening means writing the safeguards into the system rather than the prompt: mask personal data before it reaches the model, check claims with the source system, stop large payouts for a person, update the database in one locked transaction and keep a tamper-evident log. The speaker’s web demo shows each one.
To close, the speaker shared a web demo, the Agentic Dispute & Rectification Engine, and invited everyone to try it. It is not a full solution, the speaker said: it shows the trouble that comes when an AI engineer builds a dispute service the naive way, and what a professional FDE system adds — including a maker and a checker. The page has two modes, Naive Prototype and Hardened System, four test cases, and five steps. For each step it shows the common trap, the FDE’s fix and a short code sample.
| Step | The naive trap | The FDE fix |
|---|---|---|
| 1 · Take in the message | The raw message, with an identity number, goes straight to a cloud model | Personal data is masked first (Presidio), and the input is checked against a Pydantic schema |
| 2 · Check with the carrier | The agent believes the customer’s words | An authenticated MCP tool asks the carrier for GPS, scale-weight and photo data |
| 3 · Policy gate | The model decides the refund, with no upper limit | Hard limits in code: over $150, or a high fraud score, goes to a person |
| 4 · Update the database | A raw SQL update that can clash with another one | One ACID transaction with a row lock (SELECT … FOR UPDATE) |
| 5 · Keep a record | A console log that disappears when the container restarts | A SHA-256 hash in an append-only ledger, and a payment webhook with retries |
We ran all four cases in both modes. The difference is starkest in the third: the naive agent obeys a message that begins “SYSTEM OVERRIDE” and approves a $5,000 wire refund, while the hardened one spots the injection, locks the session and logs the incident.
| Test case | Naive prototype | Hardened system |
|---|---|---|
| A · A valid damaged parcel, $89.50 | Refunds, but sends the customer’s identity number to the outside model | Masks the number, confirms delivery with the carrier, passes the $150 check, records the fix in one transaction |
| B · A $1,850 laptop “never delivered” | Approves a large payout with no human approval | Stops at the $150 limit and sends the case, with its evidence, to a person |
| C · “SYSTEM OVERRIDE” prompt injection | Obeys the message and approves a $5,000 wire refund | Detects the injection, locks the session, stops and logs the incident |
| D · An empty box, $140 | Apologises and promises a refund, but updates nothing | Finds the parcel was 2.3 kg light at the hub scan and flags a carrier problem |
One small mismatch on the page itself: its text gives the fraud-score limit as 0.40, while its code sample uses 0.35. The run also shows fixed demo values for the cost of a run — 2,850 tokens naive, 640 hardened.
Practical advice from the questions
- Almost no company needs to train its own AI model. Frontier and open models are good enough, and training one costs far more than most companies can pay; when a client says “train the AI”, it usually means a knowledge base or better prompts.
- Open-source models are not free to run: they need powerful GPUs.
- Fewer agents are better, but an agent given too many instructions forgets them, so split agents by function.
- A vague goal such as “make it smarter” must become a measurable target before work starts.
- Coding still matters — for reviewing AI-written code, designing systems and checking security — even while the AI writes most of it.
- Give the client something useful within one to three months, and a roadmap for twelve.
The speaker also introduced a three-day AI engineering course in Kuala Lumpur, run with the organiser on 11 to 13 November 2026, for people with basic Python and a basic understanding of AI. Part 2 of this series covers the free follow-up webinar on 29 October, where the speaker built a production agent with Google ADK, RAG and MCP.
What this means if you run a community page
Most readers of this blog run a confession or community page, and the same rules carry over to any AI you let near it. Put limits in the system, not in a prompt the AI can be talked out of. Keep a maker and a checker on anything that is published or deleted. Mask personal details before text goes to an outside AI service. And trust nobody, human or agent, with a change that cannot be taken back.
Common questions
What is a forward deployed engineer?
A forward deployed engineer (FDE) is an engineer who works inside a client’s company to build, harden and integrate AI systems that run the business. The role comes from companies such as Palantir. An FDE ships production software rather than throwaway demos, and turns what was built for one client into features of the main product.
How is an FDE different from a solutions engineer?
According to the talk, a solutions engineer works before the sale, on demos and scripted proofs of concept, and is measured by deals won. An FDE works after the sale, inside the client’s live systems, delivering production microservices, and is measured by uptime and the client’s return on investment.
Why do most AI projects fail before production?
The speaker cited RAND Corporation research that more than 80% of AI projects fail before production. Common reasons given were poor or missing data, notebook demos with hidden state and no API contracts or monitoring, company networks that block demo tools, and running costs that nobody planned for.
Does a small business need an AI engineer or an FDE?
The slides argue for an FDE. Small businesses rarely have enough data to justify fine-tuning models or the budget for GPU servers. An FDE connects ready-made AI APIs to the systems the business already uses, such as QuickBooks, Shopify or HubSpot, with token limits and caching that keep the monthly bill small.
Are more AI agents better?
No, according to the talk. Errors compound: five agents that are each 90% reliable succeed together only about 59% of the time, agent-to-agent chatter adds delay, and free debates between agents are hard to replay when something breaks. The rule on the slides is never to use an agent where a short Python function works reliably.
What does hardening an AI agent mean?
Turning a working demo into a system that survives real use. In the talk’s demo that meant masking personal data before it reaches the model, checking claims against the source system, writing hard limits into code so a person reviews large refunds, updating the database in one locked transaction, and keeping a tamper-evident audit log.
Sources
- Free AI webinars — AITraining2U
- Agentic Dispute & Rectification Engine — web demo by Yu Yong Poh
- The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed — RAND Corporation, 2024
- Model Context Protocol — modelcontextprotocol.io