Tech Week - Agentic AI in production: retrieval, drift, and what actually works
- Date
- 2026-05-28
- Location
- Boston, MA, USA
- Host
- You Know, for Search
About this event
Agentic AI looks impressive in demos. Production is where things get real: retrieval breaks in subtle ways, model behavior shifts over time, and systems that seemed solid in testing start showing edge cases the moment real users touch them. This session is built for people who want a clearer view of what actually holds up when agentic systems move from prototype to practice. About the Event This Tech Week gathering focuses on a practical question: what does it take to run agentic AI systems in production without getting buried by failure modes you did not plan for? The conversation centers on retrieval, drift, and the operational decisions that separate promising experiments from reliable products. Rather than treating autonomy as a magic layer, this event looks at the stack underneath it. Expect grounded discussion about how agents behave when knowledge sources are incomplete, when tools return messy results, and when model outputs change in ways that affect quality, safety, and user trust. The format is in person, which matters for a topic like this. Agentic AI in production is full of tradeoffs, and the most useful conversations tend to happen when people can compare notes directly: what they tried, what failed, what they had to simplify, and what they would do differently next time. Whether you are actively deploying AI features or still evaluating how much autonomy belongs in your product, this event is designed to help you think more clearly about the gap between concept and operational reality. What to Expect Expect a focused, practitioner-friendly session on the parts of agentic AI that create the most confusion once systems are live. The discussion will likely stay close to implementation reality rather than abstract predictions, with attention on how retrieval pipelines, orchestration choices, and monitoring practices affect performance over time. You can expect the conversation to touch on themes like: Retrieval quality: how context selection influences downstream behavior, where retrieval pipelines commonly fail, and what strong grounding actually looks like in practice Drift and change over time: what happens when models, data, prompts, user behavior, or tools shift after launch Operational reliability: where agent loops become brittle, how to think about fallback paths, and when less autonomy produces better outcomes Evaluation: what teams measure when they need more than anecdotal success stories Real-world tradeoffs: latency, cost, complexity, observability, and the tension between ambitious product goals and stable systems Because this is part of a community-driven Tech Week program, expect a room with people bringing different perspectives: builders, operators, founders, engineers, and AI-curious product teams. That mix tends to make the discussion sharper, especially on topics where the answer is often "it depends" and the details matter. There is also strong value in the informal layer of the event. Some of the most useful takeaways may come from side conversations with people who have already run into the exact issue you are currently trying to anticipate. Why Attend If you are tired of high-level AI talk that skips the hard parts, this event should feel refreshing. The value here is in getting closer to the real constraints of agentic systems: what breaks first, what is harder than expected, and what patterns seem to survive contact with production. You should come if you want a better mental model for where retrieval fits into agent behavior and why so many downstream problems start there. Strong agentic experiences depend on more than a capable model; they depend on the quality, freshness, and relevance of the information an agent can access, and on the controls around how it uses that information. This is also useful if you are thinking about drift not as a single failure, but as an ongoing operational condition. Production AI changes even when you do not explicitly redesign it. Inputs evolve, prompts get edited, upstream systems shift, and user expectations move. Understanding that dynamic is critical if you want durable performance instead of short-lived wins. You will leave with clearer questions to ask about your own systems, including: Where are the brittle points in your retrieval and orchestration flow? How are you detecting quality regressions before users report them? Which parts of your agent design truly need autonomy, and which should be constrained? What does "working" mean for your use case, and how are you evaluating it? Even if you are early in the build cycle, hearing what others have learned can save time, prevent avoidable mistakes, and help you design with production realities in mind from the start. Practical Details This event takes place in person in Boston, USA on Thursday, May 28 at 5:30 PM EDT. If you are based in Boston or around the broader local tech community, it is a strong opportunity to connect face to face with people thinking seriously about AI systems beyond the demo stage. Because the topic is technical and operational, coming prepared with a few concrete questions will help you get more out of the session. If you are currently building or evaluating an AI workflow, think about the areas where your confidence is lowest: retrieval quality, evaluation, model drift, tooling, guardrails, or reliability under real usage. This is a good fit for attendees who want signal, not spectacle. Expect a session that rewards curiosity, specificity, and honest discussion about what actually works when agentic AI leaves the lab and enters production. If that is the conversation you have been looking for, this is the room to be in.
Who should attend
This is for people who want a more honest, technically grounded conversation about deploying agentic AI in the real world. - You are an **engineer or ML practitioner** working on retrieval, orchestration, evaluation, or production AI systems and want sharper insight into failure modes and design tradeoffs. - You are a **product manager or technical lead** deciding how much autonomy belongs in your product and need a better framework for balancing usefulness, control, and reliability. - You are a **founder or startup builder** exploring AI-native features and want to learn where teams tend to overcomplicate systems, underestimate drift, or rely on weak retrieval. - You are part of a **platform, infrastructure, or applied AI team** responsible for monitoring quality over time, improving robustness, or supporting internal AI adoption. - You are an **AI-curious builder in the Boston tech community** who learns best by talking with people actively testing ideas in production rather than consuming theory from a distance. - You have already seen a prototype work once and are now asking the harder question: **will this still work consistently, safely, and efficiently at scale?**