How we score sentiment across four models (without making it up)

When a Promptive sentiment score drops from +0.48 to +0.12 overnight, customers ask one question: is that real? This is the full answer, the rubric, the grader, the calibration loop, the failure modes we've caught, and why we publish all of it so customers can audit us.

LLM-graded metrics are useful and genuinely novel, but they're easy to get wrong in ways that are hard to detect. A poorly calibrated scorer can produce numbers that feel precise and trend in consistent directions while measuring something subtly different from what the label says. We've seen this in competitor products. We've caught early versions of it in our own pipeline. The only honest response is full transparency about the methodology.

What we're actually measuring

Sentiment, in the context of AI engine monitoring, means: how does this engine describe this brand when answering this prompt? That's narrower than general brand sentiment. We're not scoring social listening data, press coverage, or customer reviews. We're scoring the language a specific model uses in a specific answer to a specific question, and tracking how that language changes over time.

A brand can have strong positive social sentiment and poor AI sentiment simultaneously. If a model was trained on a period of negative press, or if a Reddit thread cementing a negative narrative earned enough upvotes to get cited frequently, the model's language will reflect that regardless of the current social conversation. AI sentiment is its own signal, and it requires its own measurement system.

The three-axis rubric

Every brand mention inside an AI answer is graded on three axes, each scored from −1.0 to +1.0 before being blended into a final score:

• Affect, how positive or negative is the emotional valence of the language? "Leading," "trusted," and "reliable" push positive. "Limited," "expensive," or "controversial" push negative. The axis measures tone independently of what's being claimed.
• Stance, what is the engine's position toward recommending this brand? A direct, unqualified recommendation scores high. "Some teams prefer X" is neutral. "X may work if Y and Z" is qualified and scores lower.
• Framing, how does the mention position the brand within the category? Category leader, viable option, budget alternative, niche choice, or afterthought are meaningfully different frames, each with a corresponding score range. Framing is weighted highest because it correlates most strongly with downstream brand perception.

Why a continuous scale instead of three buckets

We built the three-bucket version first. We hated it within six weeks. The problem is the gray zone, which is where most real brand mentions live. "Solid choice for mid-market teams" and "a reasonable option if you're working within a tight budget" are both technically positive, but they describe meaningfully different brand positions. Bucketing them together destroys the signal.

The continuous −1 to +1 scale preserves those distinctions and makes trends detectable earlier. A −0.04 shift per week over four weeks is a real trend worth investigating. In a three-bucket system, that same drift is invisible until the brand crosses the neutral/negative line, by which time the underlying cause is usually well established and harder to address.

The grader architecture

We grade with Claude Haiku. Not because it's the most capable model, but because structured output tasks at high volume require consistency and cost efficiency. Haiku produces the most reliable adherence to scoring instructions in our testing, less token-level creativity, more rule-following. For a task where the goal is calibrated consistency, that's the right trade-off.

Every mention is graded twice: once using the full three-axis rubric prompt, once using a condensed single-pass version. When the two scores diverge by more than 0.2, the mention is flagged for human review rather than being auto-published to the customer dashboard. That review queue runs daily. The flag rate is about 2.3% of all mentions. Roughly 0.7% are genuinely ambiguous in ways where both scores are defensible. The remaining 1.6% resolve clearly on second review.

The published score is the grader output plus a monotonic correction coefficient. We'd rather be consistently accurate than purely raw.

The calibration loop

The single worst failure mode for LLM-graded metrics is unacknowledged scorer drift. A model update changes the grader's internal prior. Every score shifts subtly, not enough to look wrong in any individual answer, but enough to create a systematic bias across the whole dataset. A customer sees a "trend" that is actually measurement error. That permanently erodes trust in the metric.

Our calibration procedure: every Monday, we run the current grader against a frozen set of 400 manually-rated mentions spanning the full sentiment range. Mean absolute error is computed for each axis separately and for the blended score. If any axis moves more than 0.03 from baseline, we ship a correction coefficient and log the change in the customer-facing changelog. The correction is monotonic, it rescales but does not reorder the scores, so trend lines remain intact.

We've run 14 calibration cycles since launch. Five produced corrections smaller than 0.01, logged but not applied. Four were between 0.01 and 0.025. Two exceeded 0.03, both triggered by Haiku version updates, both caught within a week of the model change. Our full calibration history is published publicly so customers can correlate any anomalies they observe with documented corrections.

What the numbers mean in practice

Per-engine variation and what it signals

The same brand often scores differently across the four engines, and that variation is real information. Perplexity, which is citation-heavy by design, tends to score brands closely in line with the content of the citations it pulls. ChatGPT's sentiment correlates more strongly with training data recency and brand entity clarity. Claude tends to be the most carefully qualified, phrases like "solid choice for teams with X requirements" produce mid-range sentiment scores that can look deceptively flat but actually represent stable positive positioning.

Per-engine sentiment divergence is one of the most actionable signals in the Promptive dashboard. If your ChatGPT sentiment is +0.55 and your Claude sentiment is +0.08, you have an engine-specific problem worth diagnosing. It usually means Claude's training data or inference-time citations contain a different signal about your brand than ChatGPT's does. Both are fixable with targeted interventions. But you can only see the divergence if you're tracking all four engines, not just the one with the most users.