Call Evaluations

An evaluation is a question you want asked of every recorded call — "Did the CSR offer a membership?", "Was a date and time confirmed?", "How many times did the caller have to repeat themselves?" — answered the same way every time, on every call, without anyone listening to them.

You'll find them under Call Analysis → Evaluations. The page is for Admins, and we switch it on account by account. You can also create, test and change evaluations by asking Jenny — see Asking Jenny instead below for what she can and can't do.

The three sub-tabs

• All evaluations — your questions, their results, and where you create new ones.

• AI Voice Agent — the same idea pointed at what your voice agent says on a call, rather than at your people.

• Correction rules — using an evaluation's answer to fix how a call got labelled in ServiceTitan.

A banner at the top of All evaluations keeps count: "Up to 10 call evaluations per organization and 10 voice checks per agent. Pause an evaluation when its limit is reached." Ten is the ceiling, so spend them on questions you'll actually act on.

Asking a question that answers the same way every time

An evaluation is only worth having if two people reading the same call would agree with its answer. Four habits get you there:

• One question per evaluation. "Did the CSR offer a membership and mention financing?" is two questions. When it answers No, you won't know which half failed. Split it.

• Ask about what was said, not how it felt. "Was the CSR friendly and agreeable?" gets different answers on the same call. "Did the CSR use the customer's name?" doesn't.

• Put your policy in the instructions. The evaluation only has the transcript. If the right answer depends on something the call doesn't say — your dispatch fee, which plans you sell, your service area — write it in.

• Say what doesn't count. UpSmith's own example is worth copying: "Return Yes only when a date and time are confirmed" is clearer than "Did the call book?"

Writing one

Click Create evaluation. The form walks you through it, and you can't save until it's been checked.

1. Under What should this evaluation determine?, give it a Name your team will recognize, then write the Evaluation Instructions. This is the part that decides whether the whole thing works "Explain exactly what qualifies, what does not qualify, and how to handle ambiguous calls."

2. Under How should the evaluation answer?, pick one of Yes / No, Multiple choice, Free text or Number. Multiple choice asks you for the possible answers; Number asks for a Minimum and Maximum.

3. Under Which answers count as success?, tick the answers you want counted as the good outcome. It's optional, and it's what makes the result show up as a rate rather than a tally.

4. Under Which calls should this evaluation run on?, choose All calls or Only matching calls. Filters available are Direction (Inbound / Outbound), Audio type (Recording / Voicemail) and Call handler (UpSmith Voice Agent / Someone else). "Calls must match every filter below. Select multiple values within a filter to match any of them."

5. Click Check evaluation. You'll get either "This looks good!" or "I suggest changing this:" with a proposed rewrite you can Accept suggestion (it's applied and checked again) or Dismiss.

6. Click Create evaluation.

Things worth knowing before you start:

• The answer format is locked once you save. "Changes apply to future evaluations. The answer format is fixed after creation." Get it wrong and you're creating a new evaluation, not editing this one.

• New evaluations start paused. "New evaluations start paused, so you can review and test them before they run." Nothing happens until you activate it.

• You can't skip the check. Until you've run Check evaluation on the current wording, the save button stays disabled and reads "Check evaluation to enable saving." Any edit after a check means checking again.

• Saving pops the backfill dialog open. Every time you save a call evaluation, Review recent-call backfill opens with that evaluation already ticked. Nothing has run and no credits have been spent — it's an offer, not a result. Close it if you don't want it.

Testing it before it runs

At the bottom of the form, Test on a sample call runs your draft against one real transcript — "Run this unsaved draft against a recent transcript. The result is not saved and does not change the call's saved evaluation results."

Pick a call under Sample call and click Test draft. You get back the Returned value, the Reasoning, and the Transcript evidence it leaned on. If the reasoning is sound but the answer is wrong, your instructions are ambiguous — that's the signal to go back and tighten them.

Test a handful: one call you'd expect to pass, one you'd expect to fail, and one awkward one. It costs nothing and it's the difference between a number you trust and a number you argue with.

Turning it on and off

Open an evaluation and you'll see a single button that flips between Pause and Activate, next to Edit. A badge beside the title reads Active or Paused.

• Paused means it stops scoring new calls. The banner says so: "Paused — this evaluation will not run until activated." Past results stay.

• Pause rather than delete. The delete dialog tells you the same: "This removes the evaluation from this page. Pause it instead if you want to preserve it for later use."

• There's no bulk control. Pausing, activating and deleting are one evaluation at a time.

• Hitting the limit gives you a plain error. "At most 10 customer call evaluations can be active. Pause an existing evaluation or contact support."

Reading the results

An evaluation opens on two tabs: Details — "Instructions, answers, and call filters" — and Results — "See answers, trends, and evaluated calls". You can also get there with View results.

Results gives you Evaluated calls and, where you set success answers, a Success rate. A numeric evaluation shows a Numeric summary instead — Minimum, Median, Average and Maximum.

Below that are two ways into the calls themselves:

• Overall breakdown — "Results across the full selected date range. Select a result to see all matching calls." Click an answer to list every call that got it.

• Results by day — "Daily call counts by result, in your local timezone. Select a segment to see its calls." A stacked bar per day. Click one segment and you get just that answer on just that day, which is how you find the Tuesday something changed rather than averaging it away across a month.

A call whose result was later corrected carries a Corrected badge in the list.

Running it on calls you already have

A new evaluation only scores calls from the moment you activate it, which is no help when you want to know whether the question is worth asking. Run on past calls fixes that — "Apply this evaluation to previously recorded calls." Nothing ever runs on past calls on its own; a run only starts when you start it.

The dialog is headed Review recent-call backfill:

1. Pick How far back? — 1 day, 3 days, 7 days or 14 days.

2. Optionally narrow it under Temporary call filters — Direction, Media and Conversation type (Abandoned, Unbooked, Excused, Booked, Not lead). "These filters apply only to this backfill." Conversation type is worth knowing about — it's the one filter you can't set on the evaluation itself.

3. Tick the evaluations to include. "Select up to 10 evaluations. Previewing does not spend credits, and overlapping evaluations share the same call credit."

4. Click Preview backfill. You'll see "This run will use {n} credits." with the real figure in place of {n}, plus four tiles: Allowance, Used / reserved, Remaining before and Remaining after.

5. Click Start backfill with the credit count on it.

The honest limits here:

• Fourteen days is as far back as it goes.

• Credits are per call, not per evaluation. "Each distinct call costs one credit even when several evaluations run on it." So it's cheaper to backfill five evaluations in one run than five runs of one.

• Your allowance is set per account. There's no standard number. If the tile reads Not configured you'll see "Historical evaluation is not available for this organization yet."

• Previewing is free. Preview as often as you like; nothing is spent until you start the run.

• Paused evaluations can be backfilled. That's the best way to judge one before you activate it.

• You can stop it. Cancel backfill on a running job — "This stops calls that have not been prepared yet. Completed and submitted evaluations remain and may still finish."

Past runs are listed under Recent-call backfills, each showing what it cost and where it got to — Queued, Finding calls, Evaluating, Completed, Partially completed, Failed or Canceled.

Correction rules

If an evaluation can tell a real lead from a wrong number more reliably than the label on the call, you can have it fix the label. That's ServiceTitan correction rules — "Choose a result and the ServiceTitan call classification and reason to set. Each rule can suggest a correction for approval or apply it automatically. Calls linked to jobs are excluded."

A rule reads across the row: When evaluation … Returns … Set ServiceTitan to … Permission, where permission is one of Suggest for approval, Auto-apply or Analyze only — no correction. Add rules with Add correction rule and click Save correction rules.

• Start on Suggest for approval. Suggestions land in Needs attention → ServiceTitan corrections for a person to wave through. Move a rule to Auto-apply once you've watched it be right for a few weeks.

• Rules don't go back over old calls. "If matching rules disagree, no correction is made. Saved rules apply when evaluations finish; saving does not scan past calls."

• One action per result. "Each evaluation result can have only one action."

• Calls already on a job are left alone. A correction only ever touches a call ServiceTitan hasn't tied to work.

• Saving replaces your old lead/non-lead setup. "Your existing lead/non-lead setup is still active. Saving these rules replaces that setup." You'll be asked to confirm, and told that "Existing suggestions will not be automatically approved."

Checks on your AI voice agent

The AI Voice Agent sub-tab is a lighter version of the same thing, pointed at your voice agent's own speech. Click Add check, give it a Name and describe the Spoken behavior to check — "Describe what a passing spoken exchange sounds like. Keep this limited to the supplied AI conversation."

• Voice checks are always Yes/No. There's no answer-format choice.

• They're about words, not outcomes. "Check what the agent says in each AI conversation. Booking and escalation actions need separate checks based on records." A check can tell you the agent offered the slot; it can't tell you the booking landed.

• They're shared across your agents' versions, and sometimes across agents. "Shared with other agents. Changes affect everyone using this check."

• Same ten-check ceiling, per agent. The tab keeps count: "{n} of 10 active checks for this agent."

• Two things have to be on. "These checks apply across agent versions. New checks start paused. Automatic evaluation must also be enabled for this organization." If your checks never return anything, that second one is usually why — ask us.

Asking Jenny instead

Jenny can do most of this in conversation: list your evaluations, write a new one, test it on recent calls, change the wording, and pause, activate or delete one. She asks you to confirm before anything is written, and evaluations she creates start paused like any other. The ten-active limit applies to her too.

Two things she can't do, which you'll need the page for:

• Running an evaluation over past calls. Backfills are started from the page.

• Correction rules. Those are set up on the Correction rules tab.

Troubleshooting

The save button won't enable. You haven't run Check evaluation since your last edit. Run it again.

Clicking save brings up "You have an unresolved suggestion". The check suggested a change you haven't accepted or dismissed. Go back and resolve it, or Save anyway to keep your wording.

An evaluation is Active but nothing is being scored. Look for the amber banner "This evaluation has saved filters that aren't available for live evaluation." Some filters only work on a backfill, not on new calls. Click Remove unavailable filters and save. The list also flags these rows with a Needs filter update badge.

You can't activate another evaluation. You're at ten active. Pause one you've stopped reading.

"Run on past calls" isn't there. Backfill hasn't been enabled for your account. Email support@upsmith.com.

The preview changed between previewing and starting. You'll see a banner ending "Confirm the updated preview to continue." New calls landed while you were looking. Click Confirm updated preview.

A backfill finished as "Partially completed". Some calls were evaluated and some weren't. Open the run to see where it stopped — the breakdown counts calls in the window, calls with a usable transcript, and calls that matched your filters.

You can't add a correction rule. Either you've hit the rule limit, or you'll see "Create or activate a multiple-choice, yes/no or text evaluation first." — correction rules need an evaluation with discrete answers to key off.

Common questions

How many evaluations should I actually run? Fewer than ten. Every one you add is a column somebody has to read. Two or three questions you'll act on beat ten you'll skim.

Can I change an evaluation's wording later? Yes — Edit, then Save changes. But "Changes apply to future evaluations", so past results were scored on the old wording. If a change is significant, treat the before and after as two different numbers, or run the new wording over past calls to compare.

Can two evaluations ask about the same call? Yes, and on a backfill they share the credit for it.

Will deleting an evaluation delete its results? It removes the evaluation from this page. If you might want it or its history again, pause it instead.

Can Jenny write one for me? Yes. Describe what you want to know, and ask her to test it on a few calls before you turn it on. See Asking Jenny instead above for the two things she can't do.

Can my CSRs see their own scores? Call Analysis is an Admin page. If you want a CSR looking at their own calls, that's a conversation to have with us — email support@upsmith.com.

Where to go next

• Call Analysis — reading the results these evaluations produce

• Giving Your Agents Documents to Work From — the other way to tell your agents what good looks like