Human in the Loop

Agents such as Support Agents, Coding or Research Agents might require human oversight.

By combining AgentKit with Inngest, you can create Tools that can wait for human input.

Creating a “Human in the Loop” tool

“Human in the Loop” tools are implemented using Inngest’s waitForEvent() step method:

import { createTool } from "@inngest/agent-kit";

createTool({
  name: "ask_developer",
  description: "Ask a developer for input on a technical issue",
  parameters: z.object({
    question: z.string().describe("The technical question for the developer"),
    context: z.string().describe("Additional context about the issue"),
  }),
  handler: async ({ question, context }, { step }) => {
    if (!step) {
      return { error: "This tool requires step context" };
    }

    // Example: Send a Slack message to the developer

    // Wait for developer response event
    const developerResponse = await step.waitForEvent("developer.response", {
      event: "app/support.ticket.developer-response",
      timeout: "4h",
      match: "data.ticketId",
    });

    if (!developerResponse) {
      return { error: "No developer response provided" };
    }

    return {
      developerResponse: developerResponse.data.answer,
      responseTime: developerResponse.data.timestamp,
    };
  },
});

The ask_developer tool will wait up to 4 hours for a "developer.response" event to be received, pausing the execution of the AgentKit network. The incoming "developer.response" event will be matched against the data.ticketId field of the event that trigger the AgentKit network. For this reason, the AgentKit network will need to be wrapped in an Inngest function as demonstrated in the next section.

Example: Support Agent with Human in the Loop

Let’s consider a Support Agent Network automously triaging and solving tickets:

const customerSupportAgent = createAgent({
  name: "Customer Support",
  description:
    "I am a customer support agent that helps customers with their inquiries.",
  system: `You are a helpful customer support agent.
Your goal is to assist customers with their questions and concerns.
Be professional, courteous, and thorough in your responses.`,
  model: anthropic({
    model: "claude-3-5-haiku-latest",
    max_tokens: 1000,
  }),
  tools: [
    searchKnowledgeBase,
    // ...
  ],
});

const technicalSupportAgent = createAgent({
  name: "Technical Support",
  description: "I am a technical support agent that helps critical tickets.",
  system: `You are a technical support specialist.
Your goal is to help resolve critical tickets.
Use your expertise to diagnose problems and suggest solutions.
If you need developer input, use the ask_developer tool.`,
  model: anthropic({
    model: "claude-3-5-haiku-latest",
    max_tokens: 1000,
  }),
  tools: [
    searchLatestReleaseNotes,
    // ...
  ],
});

const supervisorRoutingAgent = createRoutingAgent({
  // ...
});

// Create a network with the agents and default router
const supportNetwork = createNetwork({
  name: "Support Network",
  agents: [customerSupportAgent, technicalSupportAgent],
  defaultModel: anthropic({
    model: "claude-3-5-haiku-latest",
    max_tokens: 1000,
  }),
  router: supervisorRoutingAgent,
});

Note

You can find the complete example code in the examples/support-agent-human-in-the-loop directory.

To avoid the Support Agent to be stuck or classifying tickets incorrectly, we’ll implement a “Human in the Loop” tool to enable a human to add some context.

To implement a “Human in the Loop” tool, we’ll need to embed our AgentKit network into an Inngest function.

Transforming your AgentKit network into an Inngest function

First, you’ll need to create an Inngest Client:

import { Inngest } from "inngest";

const inngest = new Inngest({
  id: "my-agentkit-network",
});

Then, transform your AgentKit network into an Inngest function as follows:

import { createAgent, createNetwork, openai } from "@inngest/agent-kit";
import { createServer } from "@inngest/agent-kit/server";

const customerSupportAgent = createAgent({
  name: "Customer Support",
  // ..
});

const technicalSupportAgent = createAgent({
  name: "Technical Support",
  // ..
});

// Create a network with the agents and default router
const supportNetwork = createNetwork({
  name: "Support Network",
  agents: [customerSupportAgent, technicalSupportAgent],
  // ..
});

const supportAgentWorkflow = inngest.createFunction(
  {
    id: "support-agent-workflow",
  },
  {
    event: "app/support.ticket.created",
  },
  async ({ step, event }) => {
    const ticket = await step.run("get_ticket_details", async () => {
      const ticket = await getTicketDetails(event.data.ticketId);
      return ticket;
    });

    if (!ticket || "error" in ticket) {
      throw new NonRetriableError(`Ticket not found: ${ticket.error}`);
    }

    const response = await supportNetwork.run(ticket.title);

    return {
      response,
      ticket,
    };
  }
);

// Create and start the server
const server = createServer({
  functions: [supportAgentWorkflow as any],
});

server.listen(3010, () =>
  console.log("Support Agent demo server is running on port 3010")
);

The network.run() is now performed by the Inngest function.

Don’t forget to register the function with createServer’s functions property.

Add a ask_developer tool to the network

Our AgentKit network is now ran inside an Inngest function triggered by the "app/support.ticket.created" event which carries the data.ticketId field.

The Technical Support Agent will now use the ask_developer tool to ask a developer for input on a technical issue:

import { createTool } from "@inngest/agent-kit";

createTool({
  name: "ask_developer",
  description: "Ask a developer for input on a technical issue",
  parameters: z.object({
    question: z.string().describe("The technical question for the developer"),
    context: z.string().describe("Additional context about the issue"),
  }),
  handler: async ({ question, context }, { step }) => {
    if (!step) {
      return { error: "This tool requires step context" };
    }

    // Example: Send a Slack message to the developer

    // Wait for developer response event
    const developerResponse = await step.waitForEvent("developer.response", {
      event: "app/support.ticket.developer-response",
      timeout: "4h",
      match: "data.ticketId",
    });

    if (!developerResponse) {
      return { error: "No developer response provided" };
    }

    return {
      developerResponse: developerResponse.data.answer,
      responseTime: developerResponse.data.timestamp,
    };
  },
});

Our ask_developer tool will now wait for a "developer.response" event to be received (ex: from a Slack message), and match it against the data.ticketId field.

The result of the ask_developer tool will be returned to the Technical Support Agent.

Look at the Inngest step.waitForEvent() documentation for more details and examples.

Try it out

Support Agent with "Human in the loop" This Support AgentKit Network is composed of two Agents (Customer Support and Technical Support) and a Supervisor Agent that routes the ticket to the correct Agent. The Technical Support Agent can wait for a developer response when facing complex technical issues.