System Design Skills for FDEs Building Agent Memory and RAG Pipelines
System design skills for FDEs get tested the moment you're handed a customer requirement that sounds simple: "add memory to the agent." The first week you spend choosing a vector database, deciding how to chunk documents, and setting retrieval thresholds determines whether the demo impresses or the production system fails at 3 a.m. A forward deployed engineer sits between engineering and customer success, translating messy enterprise data into working pipelines under real latency and cost constraints. The honest answer is that most generic FDE skill lists won't help you here, because agent memory and RAG pipelines demand a specific kind of system thinking. GigaRAG helps with the agent memory side, but you still need to understand the architecture underneath. This guide covers the core system design skills for FDEs building agent memory and RAG pipelines, concrete architecture patterns with trade-offs, two worked scenarios, and an honest list of what you cannot do.
| At a glance | Details |
|---|---|
| Core skill | Designing retrieval and memory architectures |
| Key decision | Vector DB choice and embedding strategy |
| Memory design | Hierarchical: working, episodic, semantic |
| Evaluation | Retrieval metrics before generation quality |
| FDE constraint | Customer constraints trump ideal architecture |
| Biggest risk | Over-engineering before understanding data |
In This Guide
- What Is a Forward Deployed Engineer?
- Vector Database Selection: Managed vs Self-Hosted for FDE Deployments
- Why System Design Skills for FDEs Matter More Than Generic Coding
- System Design Skills For FDEs: A Step-by-Step Guide
- Core System Design Skills for FDEs Building Agent Memory
- RAG Pipeline Design: The System Design Skills That Actually Matter
- Worked Example: Designing a Retrieval Pipeline for Enterprise Documents
- Worked Example: Scaling an Agent Memory Store
- What FDEs Cannot Do: Honest Limitations and Misconceptions
- How to Build These System Design Skills as an FDE
- Final Thoughts on System Design Skills for FDEs
What Is a Forward Deployed Engineer?
A forward deployed engineer (FDE) is a software engineer who works directly with customers to deploy, customize, and debug a product in their environment. They write code, but they also sit in customer meetings, read error logs from production systems, and ship fixes the same week.
The FDE role in one paragraph
An FDE takes a product that works in the lab and makes it work in a customer's data center or cloud account. That means reading the customer's code, understanding their data, and building integrations the core product team never anticipated. The role blends engineering with deployment. You're not just writing features. You're making them work somewhere real, under constraints you didn't choose.
Why FDEs sit between engineering and customer success
Customer success teams explain the product. Core engineers build it. FDEs do the work in between: they find out what the customer actually needs, then build or configure the product to meet that need. The job requires enough engineering depth to change how a system behaves, and enough customer context to know which changes matter.
[!note] FDEs cannot unilaterally change a customer's data governance, security posture, or core infrastructure; system design must work within those constraints, not around them.
Vector Database Selection: Managed vs Self-Hosted for FDE Deployments
| Factor | Managed (e.g., Pinecone) | Self-Hosted (e.g., pgvector) |
|---|---|---|
| Setup speed | Minutes to first index | Hours to days, depends on infra |
| Operational burden | Vendor handles scaling and uptime | Team owns backups, tuning, upgrades |
| Data residency | Limited to vendor regions | Full control over data location |
| Cost model | Usage-based, scales with queries | Fixed infra cost, predictable at scale |
| Best for | Fast demos and small teams | Regulated or high-volume production |
Why System Design Skills for FDEs Matter More Than Generic Coding
Generic coding gets a feature working on your laptop. System design gets it working for a customer with 10,000 documents, a latency budget, and a production outage at 2 a.m. The difference is not theoretical. It shows up in the first week of a deployment.
The difference between building a demo and shipping a production RAG system
A demo RAG pipeline can run entirely in memory. You load a few PDFs, embed them, stuff the top three chunks into a prompt, and it works. A production system has to handle ingestion failures, stale embeddings, concurrent queries, and retrieval quality that degrades as the corpus grows. The demo hides every trade-off. Production forces you to name them.
Why FDEs need system design more than pure software engineers
Pure software engineers work inside a codebase with defined boundaries. FDEs work across boundaries: the customer's infrastructure, the product's architecture, and the integration layer between them. When retrieval latency spikes, you can't just file a ticket. You have to trace the request path, find the bottleneck, and decide whether to change the chunking strategy, the index, or the query pattern. That's system design, not coding.
[!tip] For FDEs deploying RAG pipelines, start with a simple baseline (e.g., single vector index with cosine similarity) and only add complexity like reranking or hybrid search when evaluation metrics show a clear gap.
System Design Skills For FDEs: A Step-by-Step Guide
- Map the customer's data sources and access patterns before choosing any technology.
- Define retrieval evaluation metrics (e.g., recall@k, MRR) and a golden query set with the customer.
- Choose an embedding model based on domain vocabulary, language, and latency budget.
- Select a vector database that fits data residency, scale, and team operational capacity.
- Design the memory hierarchy: working memory for the session, episodic for past interactions, semantic for long-term facts.
- Implement retrieval with hybrid search (keyword + vector) and reranking for precision.
- Instrument logging and feedback loops to measure retrieval quality in production.
Core System Design Skills for FDEs Building Agent Memory
Agent memory is not one database. It's a hierarchy, and the FDE who treats it as a single store will hit a wall the first time a customer asks why the agent forgot something from last week.
Memory hierarchy design: short-term vs long-term agent memory
Short-term memory holds the current conversation: the last few turns, the active context, the working state. It lives in the request itself or in a fast store like Redis. Long-term memory holds what should persist across sessions: user preferences, past decisions, facts about the customer's environment. It lives in a vector database or a structured store, or both. The skill is knowing what belongs where. Put too much in short-term memory and you blow the context window. Put too little in long-term memory and the agent repeats itself every session. The FDE decides the boundary, and the boundary is different for every customer.
Context window management and token budgeting
Every model has a hard limit on tokens. You can't change it. You can only decide what earns a spot inside it. Token budgeting means assigning a cost to every piece of context: system prompt, retrieved chunks, conversation history, tool outputs. When the budget runs out, something gets cut. The FDE who can't do this math will ship an agent that works in the demo and truncates mid-conversation in production.
Cache invalidation for agent memory stores
Cached embeddings and cached retrieval results go stale. A document changes, a user updates a preference, and the cache still serves the old answer. The FDE needs to know when to invalidate, and how: TTLs, version keys, or event-driven invalidation. There are only two hard problems in computer science, and this is one of them.
Latency budgets and real-time constraints
A retrieval call that takes 800ms is fine for a batch job and fatal for a chat interface. The FDE sets the latency budget per operation, then designs to it: which calls are parallel, which are cached, which are cut entirely. The budget is the constraint. Everything else is negotiation.
RAG Pipeline Design: The System Design Skills That Actually Matter
A RAG pipeline is five stages, and each stage has a trade-off that will bite you in production if you skip it. The FDE who can name the trade-off at each stage is the one who ships.
Ingestion: chunking strategy and document preprocessing
Chunk size is the first decision. Too small, and you lose context. Too large, and retrieval returns noise. For enterprise documents, 500 to 1,000 tokens per chunk with 10 to 15 percent overlap works for most cases. Preprocessing matters more than the model: strip headers, footers, and navigation. Clean text beats clever chunking.
Embedding strategy: model selection and dimensionality
You don't need the newest embedding model. You need one that matches your domain. General models like text-embedding-3-small handle most enterprise text. Higher dimensionality means more precision and more cost. 1,536 dimensions is the default. Don't pay for 3,072 unless retrieval quality tests show you need it.
Vector database selection: Pinecone, Weaviate, pgvector, or managed options
It depends on scale and ops budget. pgvector is free and fine under 1 million vectors. Pinecone and Weaviate are managed, which means you trade money for not running infrastructure. The FDE picks based on the customer's existing stack, not the benchmark leaderboard.
Retrieval: hybrid search, reranking, and recall vs precision
Keyword search catches exact terms. Vector search catches meaning. Hybrid search combines both, and reranking fixes the order. Recall is how many relevant chunks you found. Precision is how many of the found chunks were relevant. You tune for the customer's failure mode: missing answers or wrong answers.
Generation: prompt assembly and context injection
The retrieved chunks go into the prompt, but order matters. Most relevant first, capped by the token budget. The FDE writes the assembly logic: what gets injected, in what order, and what gets dropped when the budget runs out. This is where the demo either holds up or falls apart.
Worked Example: Designing a Retrieval Pipeline for Enterprise Documents
Here's a concrete scenario. You're an FDE at a company that sells an AI assistant to law firms. A customer has 10,000 legal documents, mostly PDFs and Word files, and wants their staff to ask questions and get answers with citations in under a second. You have one week to design the retrieval pipeline before the pilot starts.
The scenario: 10,000 enterprise documents, sub-second retrieval
10,000 documents is small. That's roughly 50,000 to 100,000 chunks after splitting. You don't need a distributed system. A single pgvector instance on one machine handles this easily. The sub-second requirement is the real constraint: embedding generation, vector search, and reranking all have to fit inside 1,000 milliseconds. That budget forces every decision downstream.
Choosing the chunking strategy
Legal documents have structure: sections, subsections, clauses. You split on headings first, not arbitrary token counts. Each clause becomes a chunk of 300 to 600 tokens. Overlap of 50 tokens preserves cross-references between clauses. Don't split mid-sentence. The chunking script is maybe 100 lines of Python. The hard part is testing it against real queries from the firm's paralegals.
Selecting the embedding model and vector database
Start with text-embedding-3-small at 1,536 dimensions. It's cheap and good enough for legal text. Don't fine-tune. For the vector store, pgvector is the honest default at this scale: free, runs inside the customer's existing Postgres, and the ops team already knows how to back it up. Pinecone would be overkill for 100,000 vectors.
Evaluating retrieval quality and iterating
Build a test set of 50 real questions from the firm. Run retrieval, check if the correct clause appears in the top 5 results. That's recall@5. If it's below 80 percent, adjust chunking before touching the model. Most retrieval problems at this scale are chunking problems, not embedding problems. Iterate on the test set, not on vibes.
Worked Example: Scaling an Agent Memory Store
The retrieval pipeline example was a single customer with a fixed document set. Agent memory is different. Memory grows with every conversation, every user action, every tool call. The system you design has to keep working as that memory compounds.
The scenario: from single-user demo to multi-tenant production
You built a demo where an agent remembers a user's preferences across sessions. It stores memories in a JSON file on disk. It works great in the demo. Then the customer says they want to roll it out to 2,000 employees, each with their own memory store, and the agent needs to recall relevant memories in under 200 milliseconds. The JSON file is now the bottleneck.
Partitioning and sharding the memory store
The first decision is how to split the data. Partition by tenant, not by memory type. Each customer gets their own logical store, which keeps one tenant's memory from leaking into another's retrieval results. In Postgres, that's a tenant_id column with a composite index on (tenant_id, memory_type). In a vector database, it's a namespace or collection per tenant. Don't put all tenants in one flat index. Retrieval quality drops as the index fills with irrelevant vectors.
Handling concurrent writes and consistency
Two thousand employees means concurrent writes. The agent writes a new memory every time it learns something. If two requests write to the same memory key at the same time, you need a conflict resolution rule. Last-write-wins is the honest default for most agent memory. Strong consistency is overkill here: a memory that's 50 milliseconds stale doesn't break the agent's behavior. Use an upsert, not a read-modify-write cycle. That avoids the race condition entirely.
Monitoring and debugging memory drift
Memory drift is the slow failure mode. The agent starts recalling memories that are no longer relevant, or it duplicates memories that mean the same thing. You need a dashboard that shows memory count per tenant, retrieval latency percentiles, and the most frequently retrieved memories. If retrieval latency creeps past 200 milliseconds, check the index size first. If recall quality drops, check for duplicate memories. Most memory problems are data problems, not model problems.
What FDEs Cannot Do: Honest Limitations and Misconceptions
The previous examples show what you can design. Now the honest part: what you can't.
The demo-to-production gap
A demo works because you control every variable. The document set is clean. The queries are predictable. The latency budget is generous. Production breaks all three. Real documents have broken tables, scanned PDFs, and duplicate content. Real queries come from users who don't know how to phrase what they want. Real latency budgets are set by a customer's SLA, not your laptop.
Closing that gap is your job. But you don't close it alone. You close it by escalating the parts you don't own.
Where FDE authority ends and core engineering begins
You can tune a chunking strategy. You can swap embedding models. You can add a reranker. You cannot rewrite the vector database's indexing algorithm. You cannot change how the customer's identity provider issues tokens. You cannot modify the core API gateway that sits in front of everything.
When the bottleneck is infrastructure you don't control, you write it up and hand it to the team that does. That's not a failure. That's the job.
Common misconceptions about FDE system design ownership
The biggest one: FDEs own the whole system. You don't. You own the integration layer and the configuration surface. Core architecture decisions, like which database the company standardizes on, are made above you. Your design work happens inside those constraints.
Another misconception: if the demo works, the design is done. The demo is the starting line, not the finish. Production is where the design gets tested.
How to Build These System Design Skills as an FDE
You don't build these skills by reading. You build them by designing, deploying, and breaking things.
Practice with real RAG and agent memory scenarios
Start with a public dataset: SEC filings, Wikipedia dumps, or a customer's old support tickets. Design the pipeline end to end. Pick a chunk size. Pick an embedding model. Pick a vector store. Write down why you chose each one. Then run real queries and measure recall. When recall is bad, figure out why. That debugging loop is the skill.
Learn by deploying: from local demo to production
A local demo teaches you the happy path. Production teaches you the failure modes. Deploy your pipeline to a cloud VM. Add a second user. Watch latency climb. Watch memory grow. Add monitoring. Fix the bottleneck. Repeat. The gap between demo and production is where FDE system design actually happens.
Use system design interview frameworks as a training tool
Frameworks like "requirements, constraints, trade-offs" force you to think before you build. For each scenario, write down the latency budget, the data size, and the consistency requirement. Then design against those numbers, not against vibes. It's slower. It's also how you avoid shipping a demo and calling it done.
Final Thoughts on System Design Skills for FDEs
The honest answer is that system design skills for FDEs are not a checklist. They're a set of trade-offs you learn by deploying real systems and watching them fail. The FDEs who stand out are the ones who can name their latency budget, defend their chunk size, and admit when a design won't scale.
You don't need deep ML expertise to design a RAG pipeline. You need to understand ingestion, embedding, retrieval, and generation well enough to make decisions and measure the results. That's the job.
If you're building agent memory or RAG pipelines, GigaRAG gives you a working memory store and retrieval layer to start from. It won't design the system for you. But it removes the boilerplate so you can spend your time on the trade-offs that matter.
Frequently Asked Questions
What skills are required for FDE?
FDEs need a blend of software engineering, system design, and customer-facing skills. Specifically for agent memory and RAG, they must understand retrieval architectures, embedding strategies, vector databases, and evaluation methods, plus the ability to translate customer requirements into technical designs.
What are the essential skills needed for a system design engineer?
Essential skills include requirements gathering, trade-off analysis, scalability and reliability design, and clear communication of architecture. For FDEs, add domain-specific knowledge of retrieval systems, memory hierarchies, and the ability to design under customer constraints.
What is the salary range for an FDE at Google?
Salary ranges for Forward Deployed Engineers at Google vary by level, location, and total compensation package. Public data is limited, and figures change; check current levels.fyi or Glassdoor data for the most accurate range at the time of your search.
What are the four pillars of system design?
The four commonly cited pillars are scalability, reliability, maintainability, and performance. In the context of agent memory and RAG, these translate to handling growing data volumes, ensuring retrieval uptime, keeping pipelines debuggable, and meeting latency budgets.
How do FDEs design agent memory for production?
FDEs typically design a hierarchical memory system: short-term working memory for the current session, episodic memory for past interactions, and semantic memory for long-term knowledge. Each layer has different storage, retrieval, and expiration policies, and the design must balance cost, latency, and relevance.
What should FDEs avoid when building RAG pipelines?
Avoid over-engineering before understanding the data and query patterns, skipping retrieval evaluation, and ignoring customer constraints like data residency. Also, do not assume a single embedding model or vector database fits all use cases; validate with real queries.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through system design skills for FDEs or something adjacent, we publish what we have actually tested, including where it falls short.
