How can I test an AI workflow before I let it handle real work?
Test an AI workflow by giving it past requests in a place where it cannot contact customers or change live records. Write down what should happen before each run. Check the draft, the records it touched, and the next step it chose. Include ordinary work, missing information, and requests outside its authority. When it can handle those cases, let it prepare work alongside your team. Expand what it can do only after someone checks the results.
What does a passing result look like?
Suppose your team wants help handling requests for copies of documents. An employee reads the email, identifies the customer, finds the requested file, and prepares a reply. Choosing the right file can save time. Choosing a file for the wrong customer can expose information. Both possibilities belong in your test. Begin with one sentence that defines success in terms the employee can verify. For a routine request, the workflow should prepare a draft for the correct customer with the correct document ready for review.
Make the expected result specific enough for two reviewers to agree. Use a test customer, a document number, a version, and an intended recipient. Decide whether an attachment belongs on the draft or whether a link to an approved location is required. Record whether the workflow may send anything during this stage. If the reviewers disagree about which version the customer should receive, settle that question before counting either answer as correct.
Inspect the place where the work ends. A message that says “I attached the document” is only a claim. Open the saved draft and see which attachment is actually there. Check that its contents match the right customer and version. Then check the test mailbox to confirm whether anything was delivered. These checks let you catch a workflow that writes a convincing account of a job it has only partly done. Anthropic makes this distinction between what an agent reports and the actual outcome of a trial. Read Anthropic’s guide to evaluating agents.
How do you build a small set of useful cases?
Ask the person who handles the queue to choose several recent jobs. Include a quiet day, a busy day, and one request they remember having to correct. Replace identifying details with test values and create matching sample documents. Keep the important relationships intact. If two customers shared a surname in the original problem, give the sample customers the same invented surname. Removing that detail would remove the very problem you meant to test.
Start with three cases you can explain to a colleague. In the routine case, the sender and document reference match the records. Expect a draft with one specific attachment. In the incomplete case, the sender asks for “last month’s copy” and two files fit. Expect the workflow to ask which document is needed or send the question to the coordinator. In the exception case, the sender asks for someone else’s file. Expect the request to stop for an authorized review, with no attachment released.
Add a fourth case that looks easy but has a different answer. A customer asks for a copy, then sends a second message saying they have found it. Give the workflow both messages in order. A correct outcome may be a closed request with no draft. This checks whether it can use the latest information in the conversation. A pile of tests in which every correct answer includes an attachment encourages a system that always tries to attach something.
Keep a few examples aside while you adjust the workflow. You want to know whether a change that fixes one missing reference also works on a request worded differently. Once you use a held-back case to design a fix, treat it as a known case and choose another for the next independent check. A small set can expose clear faults. It cannot tell you the frequency of every error you will see in live work.
Where can you run a realistic test?
Use a separate mailbox, sample records, and a folder containing only the test documents. Point every delivery action at a destination your team controls. Check the actual connection settings with the person who manages the software. Calling a folder “test” does little if the final send step still uses a live account. If you cannot isolate delivery, disable it and inspect the prepared draft. Record that delivery itself still needs a separate check.
Make the test resemble the work you plan to permit. If the real workflow must search by document number, give it a test folder with similar documents and revisions. Giving it just one perfectly named file makes the search meaningless. Include an outdated copy and a record with an empty field. Keep enough detail to exercise the choice while limiting access to the material needed for this run.
Set up a way to see changes before and after the run. For a document request, note the initial queue status, existing drafts, and attachment count. Afterward, inspect the same items again. This takes a few minutes in a small test and lets you answer a practical question if the workflow runs twice. Did it update an existing draft, produce two drafts, or quietly mark the request complete without creating either?
How do you record failures so someone can fix them?
Use one row per run. Record the case, expected result, observed result, and evidence. “Wrong file” is a start. “The draft for account A contains revision two from account B” gives the person making the repair a place to look. Note whether the workflow retrieved the wrong record, chose the wrong attachment from the right record, or saved a different file from the one it selected. Each of those faults needs a different fix.
Separate wording problems from action problems. An awkward greeting can go back for an edit. An attachment from another account should block a release decision even if every other sample reads well. Decide which failures block which actions before you review results. A team might allow staff to use draft text while continuing to require manual file selection. That gives the team a useful improvement with a limit they can explain.
Repeat important cases more than once, especially after a change to the instructions or the source documents. If the same request sometimes gets a draft and sometimes gets a question, record both outcomes. Ask whether both are acceptable under the procedure. If one is unsafe, keep the relevant action restricted while you investigate. An average across easy requests can hide a failure in the one case that matters most.
What happens if the job stops halfway through?
Test one ordinary interruption. Make a sample document unavailable after the workflow finds the record. Check whether the item stays open and whether the next person can see what is missing. Restore the file and try again. The second run should finish the existing job without losing its history. Check that an old draft with no attachment does not remain available for someone to send by mistake.
Then submit the same sample request twice. Use the same identifier for a repeat delivery of the original event. Use a different identifier for a genuinely new request from the same customer. Your team needs a clear rule for distinguishing the two. If an employee would see two identical drafts and wonder which one to use, the retry behavior needs work. Put the expected outcome in the test record so future changes preserve it.
When should real staff try it?
After the sample cases pass, run the workflow alongside the current process. Let the employee handle requests as usual while the workflow prepares separate suggestions. Have a reviewer compare the two after the employee finishes. Record how many suggestions they would use unchanged, how many need edits, and how long checking them takes. Also record cases where the employee missed something and the workflow found it. The original human response can contain errors too.
Choose a narrow first use based on what you learned. For example, allow the workflow to prepare drafts for requests with one confirmed customer, an exact document reference, and one current matching file. Keep final sending with an employee. Send ambiguous requests to the existing queue and assign someone to check that queue. Otherwise a system can appear successful simply because the difficult work is waiting somewhere nobody looks.
What should you keep after the first live run?
Keep the cases that caught a meaningful mistake, the written expected outcomes, and the name of the person who can pause the workflow. Agree on how to handle unfinished requests if the workflow stops. Staff need a way to return to the ordinary queue and see which items still need attention. Try that handoff once with a test request while everyone who manages the process is available.
When a new kind of request fails in live use, turn it into a de-identified sample. Add the expected result before changing the instructions, then rerun both the new case and the cases that already passed. Over time, this gives the business a practical record of what the workflow can do under its current permissions. The decision to expand its role can then depend on the work you have inspected and the cost of the checks you still need. See how to include review time in the payback calculation.