HOW TO BUILD IN THE AI AGENT ERA

AI is moving from answering prompts to completing complex work.

The opportunity is no longer just building a better model. It is designing the system around it: tools, context, memory, evaluators, hardware, and clear objectives.

THE NEW AI REALITY

AI HAS REACHED JUNIOR ENGINEER LEVEL

Jeff Dean believes modern coding agents are now capable enough to match many definitions of a junior engineer.

Their progress is also moving faster than expected. The next advantage comes from giving them longer tasks, better tools, and clearer operating rules.

AGENTS WILL WORK FOR DAYS

Most people still imagine an AI task lasting minutes or hours.

The bigger shift is agents that can work for days or weeks: rebuilding software in another language, testing alternatives, improving performance, and completing projects that once required an entire team.

THE NEXT LEAP: AI IMPROVING AI

By 2027, more ML systems may improve themselves through automated loops.

The system breaks a large problem into subproblems, runs many experiments, evaluates the results, combines the strongest solutions, and produces a better system with less human intervention.

INFERENCE IS THE NEW BOTTLENECK

THE PRODUCT CHANGES WHEN WAITING DISAPPEARS

Agent products feel limited when every step requires waiting.

Jeff Dean asks founders to imagine inference with dramatically lower latency. A system that responds 50× faster would not feel like a chatbot. It could operate continuously, interact naturally, and execute in real time.

AI PROBLEMS ARE OFTEN SYSTEMS PROBLEMS

Many teams call a limitation a model problem when the real bottleneck is hardware, memory bandwidth, data movement, or network communication.

Understanding the whole stack helps founders avoid expensive model changes when the better answer is a faster system design.

MOVING DATA COSTS MORE THAN COMPUTING

A calculation can be cheap. Moving the data required for that calculation can consume roughly 1,000× more energy.

This is why AI systems use batching. They process many tokens together to spread the data-movement cost, even though batching can reduce responsiveness.

INFERENCE NEEDS SPECIALIZED HARDWARE

Training can tolerate delay. Inference cannot.

Future hardware may minimize data movement, use extremely low-precision operations, and support fewer unnecessary features. The goal is not a machine that does everything. It is a machine that serves intelligence quickly and efficiently.

CONTEXT ENGINEERING BECOMES THE PRODUCT

THE MODEL IS ONLY ONE COMPONENT

A powerful AI product combines a model with retrieval, memory, tools, workflows, and evaluation.

The model must know what information to fetch, which tool to use, how to divide the task, when to try another approach, and how to judge whether the result is good.

CONTEXT IS CLEARER THAN TRAINING DATA

Training data is compressed into billions of parameters. The model does not recall every detail clearly.

Information placed directly in the current context is easier to use. This gives small teams a major advantage: they can improve results without training a frontier model.

TURN HUMAN METHODS INTO AI SKILLS

Jeff Dean and Sanjay Ghemawat encoded their performance-optimization process as an AI skill.

The agent learned to run benchmarks, change code, measure improvements, inspect trade-offs, and iterate. The core method was simple: give the model the workflow an expert would follow.

IMPROVE THE SYSTEM BY STUDYING FAILURES

Use agents on real problems and watch where they fail.

The fix may be a better instruction, a missing example, a new tool, a clearer evaluation rule, or a reusable skill. Context engineering improves when every failure becomes a permanent upgrade to the system.

RELIABLE AGENTS NEED STRUCTURE

AGENTS DRIFT OUTSIDE THEIR COMFORT ZONE

Agents often work well for the first few steps, then become less reliable.

As the task moves beyond patterns the model knows, performance can fall quickly. Long-running systems need guidance that keeps the agent on a clear, well-lit path.

USE MULTIPLE AGENTS TO SEARCH SOLUTIONS

A stronger system can send several agents to try different approaches.

Another agent or evaluator compares the results, keeps promising paths, and discards weak ones. This turns inference into a search process and makes complex workflows more reliable.

SKILLS KEEP AGENTS INSIDE THE WORKFLOW

A general model may not know your internal software, process, or quality standard.

Skills can teach it how to fetch logs, review code, run tests, access data, follow approval rules, and complete work in the correct order. This is how a general model becomes useful inside a specific company.

WHERE SMALL STARTUPS CAN WIN

WIN THROUGH DOMAIN DEPTH

Large AI companies build systems that work across many domains.

A small team can win by focusing on one painful workflow and designing a better interface, better data, better skills, stronger evaluation, and higher accuracy for that exact user.

LOOK FOR 0–1% MODEL SUCCESS

Test frontier models on the problem before building.

If they already succeed 20% of the time, the capability may improve quickly and absorb your product. A stronger opportunity is a task where general models currently succeed almost never.

PRIVATE DATA CREATES DEFENSIBILITY

A general model may organize public information well. It cannot automatically understand a user’s private history, company systems, customer records, or operating decisions.

Products that securely organize and activate this unique data can create durable value.

SPECIALIZED MODELS STILL MATTER

Some problems need a focused model trained for one scientific or technical domain.

AlphaFold is the pattern: a specialized system can outperform general intelligence on a narrow, valuable problem. Similar opportunities may exist in materials, chip design, biology, and other expert fields.

BECOME AN AI-NATIVE FOUNDER

CLEAR SPECS MULTIPLY AGENT OUTPUT

Agents perform better when the target is explicit.

Define the objective, inputs, constraints, expected behavior, tests, edge cases, and acceptance criteria. The clearer the specification, the less the agent must guess and the more work you can delegate safely.

EXISTING SOFTWARE IS A PERFECT SPEC

Agents are strong at translating software between programming languages because the original system is a detailed specification.

The code, behavior, and tests define the target. The agent can rebuild it, compare outputs, fix differences, and continue until the new version behaves correctly.

THE SCARCE SKILL IS TASTE

When agents can write most of the code, execution becomes abundant.

The scarce skill is choosing the right problem, defining the right product, recognizing quality, and knowing what not to build. Excellent execution on an unimportant problem still creates little value.

BUILD BETTER TASTE

RECORD WHAT YOU THINK WILL MATTER

Write down several problems or technologies you believe will become important in the next 12 months.

Later, review what actually happened. Which ideas became valuable? Which were ignored? Which assumptions were wrong? Repeated prediction and review builds practical taste.

QUESTION ASSUMPTIONS FROM FIRST PRINCIPLES

Do not stay anchored to how a problem is solved today.

Ask what would happen if speed improved 10×, cost fell 100×, errors became acceptable, or a constraint disappeared. Most thought experiments fail, but the successful ones can create new system architectures.

MAPREDUCE CAME FROM REMOVING NOISE

Google’s large-scale jobs mixed simple computation with complex code for parallelism, failures, and checkpointing.

Jeff Dean and Sanjay Ghemawat separated the repeated infrastructure from the actual task. That abstraction became MapReduce and made distributed computing easier to use.

TPU CAME FROM SIMPLE NAPKIN MATH

Better speech recognition meant people would use it more. Serving that demand on CPUs could require a dramatically larger server fleet.

That calculation exposed the bottleneck before it arrived. The response was specialized hardware that later became foundational for modern AI.

AUTOMATED SCIENCE CHANGES EVERYTHING

THE SCIENTIFIC METHOD CAN BECOME SOFTWARE

AI can automate the experimental loop: propose an idea, implement it, run the experiment, measure the result, and choose the next direction.

When thousands of experiments can run automatically, progress is measured by useful discoveries per unit of compute.

MAKE THE EVALUATOR FASTER

Some scientific simulations take hours for one answer.

Researchers can train a neural approximation using inputs and outputs from the expensive simulator. Jeff Dean described one case that became about 300,000× faster while remaining nearly as accurate.

AI WILL ORCHESTRATE SUBPROBLEMS

The most powerful systems will accept a high-level objective, divide it into subproblems, run automated experiments for each one, and combine the results.

This pattern could accelerate machine learning, science, engineering, molecule discovery, and chip design.

BUILD FOR POSITIVE IMPACT

REJECTION DOES NOT DEFINE IMPORTANCE

Jeff Dean’s work on model distillation was rejected by a conference reviewer who doubted its impact.

The idea later became central to creating smaller, faster, capable models. Good work can be early, misunderstood, or evaluated through the wrong lens.

CHOOSE WORK THAT CHANGES THE OUTCOME

Before committing years to a problem, ask one question:

If we succeed completely, will the world become meaningfully better?

A difficult project should create more than a technically impressive demo. It should unlock real capability, value, or progress.

BUILD WITH LOW-EGO PEOPLE

Choose teammates who are excellent in areas you are not, enjoy solving hard problems, and care more about the result than personal credit.

A strong team combines different tools, learns from each other, and becomes capable of building what no individual could create alone.

YOUR FOUNDER PLAYBOOK

1. Find a problem general models cannot solve.

2. Add private context, tools, skills, and evaluators.

3. Write a precise specification.

4. Run multiple agents and measurable loops.

5. Optimize the full system for speed, reliability, and impact.

THE NEXT ADVANTAGE IS SYSTEM DESIGN

The winning product will not be the one with the longest prompt.

It will combine the right problem, proprietary context, expert workflows, reliable evaluation, low-latency execution, and strong founder taste into a system users can trust.

PHYSICAL AI THAT SHIPS SAFELY

A working demo can impress people.

A real physical AI product must keep working safely through millions of unpredictable situations.

This is the playbook Waymo learned while turning autonomous driving from a prototype into a service operating at scale.

WHY PHYSICAL AI IS DIFFERENT

THE BEST AI MOMENT LOOKS LIKE NOTHING

When another driver suddenly cuts in, the best system reacts safely and smoothly.

Passengers may not even notice.

For physical AI, success is often invisible: no crash, no panic, no interruption. The task simply gets done.

MOVE FAST. SHIP SAFELY.

“Move fast and break things” works poorly when mistakes can damage property or harm people.

Physical AI needs speed with discipline.

Safety cannot be added later. It must shape the model, hardware, training, validation, and deployment from day one.

FOUR GAPS TO CROSS

Physical AI faces four structural gaps:

1. Cost of error
2. Latency
3. Data
4. Validation

Any serious robot, vehicle, or real-world agent must cross all four before it can operate safely at scale.

1. COST OF ERROR

A chatbot mistake may cost one retry.

A physical AI mistake can cost a human life.

The system must avoid dangerous actions before they happen because the real world offers no reset, rollback, or second attempt.

2. LATENCY

Digital assistants may take seconds to answer.

A car at freeway speed moves roughly 100 feet in one second.

Physical AI must sense, reason, and act in milliseconds using onboard compute that fits inside the machine.

3. DATA

Digital AI learned from the internet: a massive collection of human knowledge and labeled examples.

Physical AI has no equivalent dataset.

Teams must collect, generate, simulate, label, and continuously improve their own real-world data.

4. VALIDATION

Digital products can launch early and let users reveal edge cases.

Physical AI needs strong safety and confidence before the first public deployment.

The operating conditions must be defined clearly, tested rigorously, and expanded responsibly.

LESSON 1 : A DEMO IS ONLY 1% OF THE WORK

THE DEMO CAN FEEL LIKE THE FINISH LINE

Waymo reached major autonomous-driving demo goals in about 18 months.

It could drive day and night, through traffic, construction, highways, and city streets.

By demo standards, the problem looked solved. The product journey had barely started.

PRODUCT MEANS REPEATING IT RELIABLY

Driving ten difficult routes once is impressive.

Operating millions of miles without a human driver is a different engineering problem.

Waymo needed roughly 15 years to move from the first strong demo to a service operating at large scale.

RELIABILITY IS AN EXPONENTIAL LADDER

Reaching 90% or 99% performance is often the easy part.

Each additional nine of reliability can demand roughly ten times more effort.

The architecture, testing, redundancy, and operations must evolve at every level.

THE LONG TAIL BECOMES DAILY REALITY

An event that happens once in a million miles sounds negligible.

When the fleet drives millions of miles every week, it becomes a routine problem.

At scale, edge cases are not the edge of the product. They are the product.

COUNT YOUR NINES FIRST

A demo may need one nine.

An assistive tool may need several.

A fully autonomous public system needs many more.

Define the required reliability before choosing the architecture, budget, timeline, and launch strategy.

LESSON 2 : CHOOSE TECHNOLOGY FOR THE REQUIRED CEILING

THE FASTEST START CAN BE A TRAP

Many technologies improve quickly at first, then flatten.

Teams often choose the path with the fastest early demo and assume the curve will continue.

The danger appears when that approach reaches its ceiling below the product’s required performance.

BUILD FOR THE PRODUCT, NOT THE PROTOTYPE

A prototype can optimize for speed of learning.

A production system must optimize for the reliability ceiling it eventually needs.

Be honest about whether each technical choice supports a demo, an experiment, an assistive tool, or full autonomy.

USE COMPLEMENTARY SENSING

Cameras provide color and detail.

LiDAR measures 3D structure, even in darkness.

Radar handles weather and measures velocity.

Fusing multiple sensing modes creates a stronger view of the world than relying on one sensor alone.

REDUNDANCY MUST BE DESIGNED IN

A leaf, glare, darkness, dust, rain, or physical damage can weaken a sensor.

Physical AI needs fallback paths and redundant perception.

The goal is not only to detect failure, but to continue safely when part of the system degrades.

DESIGN FOR FUTURE HARDWARE ECONOMICS

Hardware becomes cheaper, smaller, and more capable across generations.

Do not lock the company into assumptions based only on current prices.

Design the system so sensors, compute, and vehicle platforms can be upgraded as the technology matures.

LESSON 3 : RIDE EVERY MAJOR TECHNOLOGY WAVE

ONE BREAKTHROUGH IS NOT ENOUGH

Waymo repeatedly rebuilt its driver around major advances: convolutional networks, transformers, vision-language models, and world models.

The advantage is not adopting one new technology.

It is learning how to absorb the next one without stopping deployment.

RESEARCH MUST HAVE A PRODUCTION PATH

Before launching a new technical project, define what happens if it succeeds.

How will it enter the main product?

How will it be validated?

How will it replace or simplify existing systems?

Innovation without an integration path creates expensive fragmentation.

DEMAND PERFORMANCE AND SIMPLIFICATION

A breakthrough that adds capability but creates another isolated stack can slow the company down.

The launch bar should demand two outcomes:

1. Meaningful performance gains
2. Radical simplification or unification

THE FOUNDATION MODEL BECOMES THE CORE

Waymo’s foundation model combines multimodal sensing, world understanding, action prediction, and language-aligned knowledge.

A large shared foundation moves complexity upstream, while smaller specialized models run efficiently on different vehicles and hardware.

THINK FAST. THINK SLOW. GENERATE.

The fast path handles split-second geometric danger.

The slow path reasons about deeper semantics, such as a burning vehicle or unusual scene.

The generative component predicts how others may behave and selects the vehicle’s next action.

LESSON 4 : USE STRUCTURE TO IMPROVE SCALING

GENERAL METHODS WIN WITH SCALE

The bitter lesson of AI is consistent: methods that scale with compute and data eventually outperform systems dominated by human-written rules.

High-capacity models learn richer patterns and can later be distilled into smaller models for real-time deployment.

STRUCTURE MUST CHANNEL SCALE

Structure that fights scale will lose.

Structure that channels scale can win.

The right representation should not shrink the solution space. It should make training, evaluation, and safety checks more efficient without removing the model’s ability to learn.

STRUCTURE-AUGMENTED END-TO-END

Waymo combines end-to-end learning with structured representations of the physical world.

The model still learns rich internal embeddings, while explicit structure supports validation, physics, road rules, object behavior, and measurable safety checks.

WHY THE STRUCTURE MATTERS

Structured intermediate representations enable:

1. Real-time correctness and safety checks
2. More efficient large-scale training and evaluation
3. Stronger feedback signals for metrics, loss functions, and reinforcement learning

LESSON 5 : BUILD A HIGH-FIDELITY SIMULATOR

OPEN LOOP IS NOT ENOUGH

Open-loop training asks: “In this situation, what would you do?”

Closed-loop training asks the agent to act, observe the consequence, update its world view, and act again.

Safety-critical systems must understand sequences, feedback, and counterfactual outcomes.

THE SIMULATOR IS ANOTHER AI SYSTEM

A serious simulator is not lightweight test software.

It is a world model that understands physics, traffic, weather, objects, behavior, and sensor outputs.

Its realism must be high enough to train and evaluate decisions that will later affect real people.

TRAIN ON EVENTS THAT RARELY HAPPEN

Simulation can create difficult situations that may be unsafe, expensive, or nearly impossible to collect repeatedly in reality.

A stopped car on a freeway, a plane landing ahead, an animal in traffic, or unusual weather can be generated, controlled, and tested at scale.

CLOSED LOOP CREATES CONFIDENCE

The goal is not only a realistic-looking video.

The agent must interact with a simulated world that responds to its decisions.

This allows teams to test whether one action creates a safe or dangerous sequence several steps later.

LESSON 6 : BUILD THE AGENT, SIMULATOR, AND CRITIC

YOU NEED THREE AIS

The agent acts in the real world.

The simulator creates realistic worlds and difficult scenarios.

The critic evaluates performance and explains where improvement is needed.

Together, they form the learning ecosystem behind a scalable physical AI product.

CREATE THE IMPROVEMENT FLYWHEEL

Real-world deployment generates data.

The data makes simulation more realistic.

The simulator produces harder edge cases.

The critic scores them.

The agent learns, improves, deploys again, and generates better data.

A FLYWHEEL NEEDS DIRECTION

A flywheel can spin quickly in the wrong direction.

Clear metrics determine which failures matter, which scenarios deserve more data, and which model changes create genuine improvement.

Without metrics, more data and compute can still produce confusion.

LESSON 7 : EVALUATION AND METRICS ARE THE MOAT

BUILD THE EVAL BEFORE THE PRODUCT

If the team cannot measure what “good enough” means, it is not building a product.

It is iterating on a demo.

Evaluation should define the target, expose weaknesses, prioritize data, measure progress, and determine whether the system is ready to deploy.

EVALUATE THE ENTIRE SYSTEM

Physical AI validation must cover sensors, compute, models, fallback systems, behavior, operations, maintenance, remote support, deployment rules, and real-world readiness.

Safety depends on the full system, not one benchmark score.

TRUST IS EARNED WITH EVIDENCE

Customers, communities, and regulators trust systems that repeatedly prove they work.

Models can be copied. Algorithms can leak.

Years of real-world operation, rigorous evaluation, safety data, and transparent evidence are much harder to replicate.

SAFETY BECOMES A BUSINESS ADVANTAGE

When a physical AI product demonstrates safer outcomes across hundreds of millions of autonomous miles, safety is no longer only a technical metric.

It becomes brand trust, regulatory confidence, customer adoption, operational learning, and a durable competitive advantage.

THE COMPLETE PLAYBOOK : SEVEN LESSONS THAT REINFORCE EACH OTHER

THE SYSTEM COMPOUNDS

Required nines define the technology ceiling.

The right architecture enables repeated upgrades.

Structure improves scaling and validation.

Simulation exposes rare events.

The agent, simulator, and critic create a flywheel.

Metrics keep the flywheel moving toward safety.

PHYSICAL AI IS ENTERING ITS DECADE

Digital AI transformed screens, software, and knowledge work.

The next major wave will increasingly move through vehicles, robots, factories, logistics, healthcare, and infrastructure.

The opportunity is massive, but the standard must remain: move fast and ship safely.

REMEMBER WHO YOU ARE BUILDING FOR

The goal is not the most impressive demo.

The goal is a system people can trust with real tasks, real environments, and real consequences.

Build for the customer, the mission, and the human life affected by every decision.