We’re excited to share this insightful interview with Duncan Curtis, the Senior Vice President of Product and Technology at Sama, where he talks about the latest advancements in ethical AI, the impact of Agentic AI, and Sama’s innovative approach to data annotation and model evaluation.
About Sama: Sama is the global leader in ethical data annotation and model evaluation solutions for computer vision, generative AI and other major applications of artificial intelligence. Our solutions minimize the risk of model failure and lower the total cost of ownership through an enterprise ready ML-powered platform, actionable data insights uncovered by proprietary algorithms, and a highly skilled on-staff team of over 5,000 data experts. 25% of Fortune 50 companies, including GM, Ford, Microsoft and Google, trust Sama to help deliver industry-leading ML models. Ethical AI is responsible AI, and as a Certified B-Corp, we’ve pioneered an impact model that harnesses the power of markets for social good, and has been proven to meaningfully improve employment and income outcomes for those with the greatest barriers to formal work. So far, helping more than 65,000 people lift themselves out of poverty.
Here’s the full interview.
AI Technology Insights (AIT): Hi, welcome to the AI Technology Top Voice Interview Series. Please tell us about your current role at Sama and your involvement with the latest announcement.
Duncan: I am currently the Senior Vice President of Product at Sama, where I focus on helping companies build comprehensive training data strategies through consulting services, data labelling and model evaluation solutions . My background includes roles as Head of Product at Zoox (now part of Amazon) and VP of Product at Aptiv, along with experience at Google managing products for over one billion daily active users. With a degree in Computer Software Engineering from Queensland University of Technology, I am passionate about merging technology with impactful outcomes at Sama.
We recently launched Agentic Capture, a data capture framework for multi-modal agentic AI. Built on our configurable capture platform, this framework addresses complex use cases across various industries, ensuring accurate annotation through collaborative quality rubrics. Agentic Capture enhances our understanding of AI agent behaviors and facilitates effective model performance iteration.
At Sama, we collaborate with leading AI model builders, positioning ourselves at the forefront of innovation. Our proprietary human-in-the-loop (HITL) approach integrates expert feedback throughout the model development process, ensuring high performance and quality, backed by our SamaAssure™ program, which boasts a 98% first-batch acceptance rate.
AIT: What are the core capabilities of Agentic AI? How do customers benefit from using Sama’s Agentic AI?
Duncan: Agentic AI represents a significant shift beyond what capabilities we’ve expected from traditional LLMs. While built upon traditional LLMs as a foundation, agentic AI is designed to mimic human decision-making and actions through autonomous learning and interaction.
Unlike standard LLMs, agentic AI can process multiple types of data simultaneously including text, images, audio, video and most importantly, procedural data. This means they can make plans and execute them with a variety of tools along the way. However, AI agents require explicit instruction for each step that humans might combine subconsciously – much like teaching a five-year-old to make a sandwich requires breaking down “get the bread” into “go to the fridge, open the door, get the bread.” It doesn’t help to teach them how to spread the peanut butter if they never got the bread out of the fridge in the first place.
A key challenge with agentic AI is maintaining visibility into these processes, as hidden steps can cause AI agents to fail without clear paths to recovery. Sama’s Agentic Capture addresses this by providing real-time feedback and model validation through its comprehensive framework that helps map out and validate each necessary step. Customers benefit from this by gaining access to a robust data strategy that optimizes the training of AI agents, enabling them to effectively solve specific industry problems while ensuring high-quality performance through expert feedback.
Recommended: Sama Launches Agentic Capture: Data Framework for Multi-Modal AI
AIT: Sama’s Agentic Capture framework aims to significantly impact industries by integrating multi-modal data for AI agents. How do you see the evolution of Agentic AI affecting various sectors over the next five years, and how does Sama plan to stay ahead of the curve as this market expands to USD 47.1 billion by 2030?
Duncan: The launch of Sama’s Agentic Capture framework represents a pivotal shift in our industry’s approach to AI integration. Integrating multi-modal data into autonomous systems will completely transform operational paradigms across sectors within the next five years. As the agentic AI market expands toward that projected $47.1B valuation by 2030, Sama is positioning itself at the intersection of innovation and practical deployment—where theoretical capability meets real-world application.
The reality facing most organizations today is stark: without the resources of tech giants, building and implementing effective AI agents remains daunting. Many leadership teams approach me with the same fundamental questions: Where do we start? Is an agent-based approach even appropriate for our use case? This uncertainty has created a troubling pattern where companies adopt generative and agentic AI technologies without fully comprehending their strategic implications or operational requirements.
Despite the democratization of agent-building tools, we’re still operating in an environment where feedback mechanisms and audit capabilities lag significantly behind development frameworks. The technical barrier to effectively evaluate an agent’s decision-making process remains prohibitively high for most implementation teams. This creates not just operational blind spots but potentially significant compliance and governance risks.
Our framework directly confronts these challenges by implementing transparent operation logging and intuitive audit mechanisms. We’ve engineered the system to surface agent actions in human-readable formats that don’t require specialized AI expertise to interpret. This transparency doesn’t just improve operational confidence—it fundamentally changes the risk profile of agent deployment and accelerates the integration timeline for organizations at any technical maturity level.
Recommended: AITech Top Voice: Interview with Heather Pringle, CEO at Space Foundation
AIT: The HITL approach is central to Sama’s platform, ensuring that AI models are validated through expert feedback. Can you elaborate on how this framework allows for real-time human judgment to influence AI agent behavior, and what specific challenges have you faced in maintaining the integrity of human feedback in AI model development?
Duncan: The Human-in-the-Loop (HITL) approach is central to our platform at Sama. It enables expert feedback to validate and refine AI models. This framework allows domain experts to monitor AI behavior closely, providing immediate insights that guide adjustments. Increasingly, top AI models are starting to show more of their decision-making processes to end users for this exact reason – to build trust and transparency in how determinations are made. Such visibility into the AI’s reasoning combined with real-time human judgment is crucial for adapting AI decision-making to complex, real-world scenarios that may not be fully captured in initial training data.
However, maintaining the integrity of human feedback presents challenges. One significant issue is ensuring the quality and relevance of the feedback provided. Experts may not always grasp the technical nuances, leading to vague or inaccurate input that can impair the AI’s learning process. The volume of data generated by AI agents can overwhelm reviewers, making it difficult to extract actionable insights. This volume can result in oversight and delayed responses to critical issues. Human behavior is often complex and context-dependent, complicating the AI’s ability to learn effectively from feedback.
Sama’s Agentic Capture framework addresses these challenges by offering strategic guidance and a user-friendly technology interface. It helps businesses establish a solid data strategy, identifying suitable AI models and developing quality rubrics for performance assessment. The interface translates complex AI processes into understandable steps for domain experts, facilitating better monitoring and issue flagging.
The operational process begins with consultations to define the goals for Agentic AI models. We work with clients to establish frameworks that log AI behavior in detail, allowing experts to supervise actions effectively. By translating low-level agent traces into human-readable steps, we empower experts to identify and address issues promptly. For example, if an AI agent makes an inappropriate recommendation, expert feedback can guide it toward better decision-making in future scenarios. This iterative feedback loop is essential for enhancing AI performance and ensuring alignment with business objectives.
AIT: With AI agents mimicking human-like decision-making across industries, transparency and accountability become critical. How does the Agentic Capture framework ensure that users can trust the behavior and decisions of these AI agents, particularly in high-stakes environments like fraud detection and customer service?
Duncan: The Agentic Capture framework addresses trust concerns in AI decision-making through a comprehensive oversight system. Beginning with strategic consultation to identify automation goals, the system captures detailed logs of AI agent behavior for review by domain experts who may lack technical expertise but possess crucial subject knowledge. What sets this framework apart is its focus on process steps rather than just outputs, accessibility for non-technical users, consultative approach to quality measurement, and integrated execution capabilities. In retail scenarios, for example, the AI agent fails to move from the customer chat interface where they’ve said they’re processing the return into the order management system that actually processes the return and ships out a new product.
AIT: Given that Agentic AI requires data across multiple modalities (text, images, audio, video), how does Sama address the complexities of integrating diverse data types seamlessly, and what unique challenges arise in ensuring high-quality annotations for each modality?
Duncan: Though still in early deployment stages, internal testing has revealed that optimizing AI agents requires multiple learning iterations based on real user interactions, emphasizing the need for proper partner support during fine-tuning. Implementation is straightforward: teams save agent logs to cloud storage, grant Sama access, and use the platform interface to provide insights that are then saved for continuous improvement. The system has already demonstrated value through proactive error detection and addressing understanding gaps in task management scenarios, while keeping additional cost overhead minimal—primarily requiring cloud storage and initial integration setup. This practical approach allows companies to avoid critical errors that might otherwise erode employee or customer trust when deploying AI agents at scale.
AIT: How does Sama measure the success of its Agentic Capture framework in improving model performance, and can you provide specific examples of how this framework has successfully facilitated the iteration and optimization of AI agents for your clients?
Duncan: Sama measures the success of its Agentic Capture framework through a multifaceted approach that tracks both immediate improvements in agent behavior and long-term performance gains. The framework employs quantitative metrics to evaluate error reduction rates, decision accuracy improvements, and the speed at which agents learn from feedback. Additionally, it incorporates qualitative assessments from domain experts who evaluate the appropriateness of agent responses in complex scenarios. This balanced measurement methodology ensures that improvements are substantive rather than superficial, with success defined not just by reduced error rates but by meaningful enhancements in decision quality.
While still in early implementation stages, Sama has already observed promising results from internal deployments and initial client applications. In one notable case involving a task management system, the framework identified a fundamental gap in the AI agent’s understanding of priority assignment protocols. By capturing the agent’s decision-making process and enabling domain experts to provide targeted feedback, the team successfully recalibrated the agent’s behavior, resulting in significantly improved task prioritization accuracy. Another key success has been in proactive error detection, where the framework has enabled teams to identify and address potential issues before deployment in production environments. These early wins highlight the framework’s potential while reinforcing Sama’s observation that optimal performance requires multiple feedback iterations and a dedicated fine-tuning partner to mitigate the substantial risks associated with deploying undertrained AI agents at scale.
Recommended: HIMITSU8 – Open-Source Agentic AI Framework Intelligent, Adaptive and Autonomous
AIT: At AI Tech Insights, we focus on the companies that have embraced Ethical AI Development frameworks. As AI agents become more autonomous, ethical considerations around biases and decision-making become crucial. How does Sama’s platform ensure that AI models are not only efficient but also fair and ethical in their decision-making processes, especially when deployed in sensitive industries such as finance and retail?
Duncan: At Sama, we’ve built our platform with ethical AI as a foundational principle, not an afterthought. Our approach ensures AI systems remain fair and unbiased, particularly when deployed in sensitive sectors like finance and retail. Three core pillars of our ethical framework include:
- Diverse data foundation: we maintain gender-balanced annotation teams and provide living wages to our global workforce. This diversity directly translates to more representative training data, helping eliminate inherent biases that emerge when AI is developed from homogeneous perspectives.
- Systemic bias detection: our platform includes advanced analytics specifically designed to identify potentially problematic patterns in AI decision making. These tools can detect when models consistently make different choices for specific demographic groups – essential for applications in lending, pricing or customer service automation.
- Ethical feedback loops: at Sama we have implemented structured feedback loops that evaluate AI processes against ethical benchmarks. When issues are identified, our consulting team works directly with clients to address those concerns through model adjustments, data rebalancing or governance improvements
The agentic capture framework integrates these ethical considerations throughout the entire AI lifecycle ensuring responsible development from initial data collection through deployment and ongoing monitoring and evaluation.
AIT: Given the wide variety of industries Sama serves, how do you work with your clients to tailor the Agentic Capture framework to their specific use cases and ensure that their AI agents are capable of solving unique industry challenges effectively?
Duncan: Sama follows a structured approach that ensures AI agents are equipped to address unique challenges. The process begins with a strategy consultation, where the goals of the Agentic AI models are defined. This initial step is crucial for identifying specific workflows that require automation, allowing clients to articulate their needs clearly.
Once the workflows are identified, the framework incorporates detailed logging of the AI agent’s actions. This logging provides transparency and accountability, capturing not just the outcomes but also the processes that led to them. This data is essential for users who may not have extensive technical expertise, as it enables domain experts to supervise and monitor the agent’s actions effectively.
Feedback is a critical component of this framework. Unlike traditional models that focus solely on outputs, Sama emphasizes the importance of process steps. This approach allows for a deeper understanding of the agent’s performance and facilitates improvements that outcome-only supervision cannot achieve. By gathering feedback from non-technical users, the framework empowers them to contribute meaningfully, breaking down barriers that often limit involvement to developers.
Sama’s consultative methodology further enhances the process by measuring quality and defining the necessary data for effective feedback. This ensures that clients have a clear roadmap for success. Additionally, by combining strategic guidance with an in-house execution team, Sama provides clients with both the insight and practical support required to implement their AI solutions successfully.
In summary, Sama’s Agentic Capture framework stands out by focusing on process-oriented feedback, ensuring accessibility for non-technical users, and offering a consultative approach that integrates strategy with execution. This comprehensive methodology equips AI agents to effectively tackle industry-specific challenges, fostering ongoing improvement and responsiveness.
Recommended: Botify Pioneers Agentic AI: Transforming Search in the Age of AI Bots
To share your insights, please write to us at news@intentamplify.com
