SQL Query
SQL Query
SQL Query
SQL Query
| Time | Question | DBX Gold | Fabric Gold | DBX Raw | Fabric Raw | Summary |
|---|
| Engine / Architecture | Original Script | Claude Opus 5.5 Adj. | 2nd Blind Audit | Unambiguous Acc.* | Latency (Mean / Med / P90 / P99) | Calls >30s |
|---|---|---|---|---|---|---|
| Databricks Genie โ Gold | 90.2% | 90.8% (+0.6) | 90.8% (+0.5) | 96.5% | 14.2s / 13.1s / 18.4s / 25.0s | 1 / 200 (0.5%) |
| Fabric Data Agent โ Gold | 88.2% | 89.5% (+1.3) | 89.0% (+0.8) | 95.2% | 15.0s / 12.4s / 24.9s / 42.4s | 12 / 200 (6.0%) |
| Databricks Genie โ Raw | 85.0% | 83.2% (-1.8) | 82.8% (-2.2) | 87.0% | 19.5s / 17.6s / 25.0s / 49.8s | 15 / 196 (+1 timeout) |
| Fabric Data Agent โ Raw | 73.8% | 70.2% (-3.6) | 68.3% (-5.5) | 74.7% | 20.6s / 15.9s / 36.0s / 75.7s | 34 / 200 (17.0%) |
COM-064..075 prompts wording practice_area / firm_practice_group values as "external template folder" (2nd Blind Audit de-ambiguated: 96.5% / 94.7% / 86.4% / 72.6%).
Medallion Gold Lift: +7.6โ8.0 pts (DBX) & +19.3โ20.8 pts (Fabric) ยท P99 Latency Cut ~50%
On VER-056..075, 3 of 4 expected tokens were matter IDs in the prompt (75% threshold), awarding Accurate to explicit Fabric Raw "no matching data" refusals (VER-064/065/075). Single-digit tokens ('1','2','3','5') also matched inside IDs or "112,500".
StandardRate vs Rate 121% Realization BugWithout Gold semantic definitions, Fabric Raw multiplied hours by jt.StandardRate ($1,140) instead of te.Rate ($1,500), reporting impossible 121.14% partner realization & -$561.6M leakage (VER-001/007/009), while DBX Raw used IsBillable=1 (100%) on 3 Medium runs.
is_calderwood_dms=0 Inversions & Enum GuessingFabric Raw inverted is_calderwood_dms = 0 (external templates) on 5 Firm DMS queries, confused MatterType with PracticeArea ($0 revenue on 4 CRM queries), and hit 4 content-filter blocks. Fabric Gold guessed string literals without value sampling.
'folder' vs firm_practice_group Ambiguity Misses12 of Gold's ~20 non-perfect calls (COM-064..075) asked for "folder '<X>'" when <X> lives in firm_practice_group. Excluding those lifts Gold to 96.5% (DBX) & 95.2% (Fabric) (DBX Raw caught 3/12 via automatic column value sampling).
This PoC ran single-turn paced queries on 12 pre-warmed tables. Real enterprise estates (Elite 3E / Aderant + iManage: 300โ1,000+ tables, 10โธ+ time entries, millions of .docx files, cold SQL starts, dynamic ethical-wall RLS, and 3โ6 agent retry loops ร 20โ40s on key/rate mismatches) push uncurated Raw queries to 3โ4+ minutes. Medallion Gold collapses multi-table joins and full-text LIKE scans into governed tables, halving P99 tail latency (25.0s DBX Gold vs. 49.8s Raw; 42.4s Fabric Gold vs. 75.7s Raw).
Average Response Time by Complexity (Seconds โ Lower is Better)
Accuracy Score by Complexity (% โ Higher is Better)
| Category | DBX Gold | Fabric Gold | DBX Raw | Fabric Raw |
|---|
| Dataset | DBX Gold | Fabric Gold | DBX Raw | Fabric Raw |
|---|
| ID | Tier | Dataset | Question & Ground Truth | DBX Gold | Fabric Gold | DBX Raw | Fabric Raw |
|---|
Two independent AI auditors manually inspected every non-Accurate run (141 calls) and scanned all 659 Accurate runs against SQL & Parquet/CRM ground truth.
| Engine / Architecture | Original Script (%) | Claude Opus 5.5 Adjusted (%) | 2nd Blind Audit (%) | Unambiguous Accuracy* | Mean / Median / P90 / P99 | CRM Gold/Raw ยท Harvey |
|---|---|---|---|---|---|---|
| Databricks Genie โ Gold | 90.2% | 90.8% (+0.6) | 90.8% (90.75%) | 96.5% | 14.2s / 13.1s / 18.4s / 25.0s | 100.0% CRM ยท 81.5% (92.6%*) |
| Fabric Data Agent โ Gold | 88.2% | 89.5% (+1.3) | 89.0% (89.00%) | 95.2% | 15.0s / 12.4s / 24.9s / 42.4s | 97.5% CRM ยท 81.5% (92.6%*) |
| Databricks Genie โ Raw | 85.0% | 83.2% (-1.8) | 82.8% (82.75%) | 87.0% | 19.5s / 17.6s / 25.0s / 49.8s | 98.5% CRM ยท 68.0% (73.9%*) |
| Fabric Data Agent โ Raw | 73.8% | 70.2% (-3.6) | 68.3% (68.25%) | 74.7% | 20.6s / 15.9s / 36.0s / 75.7s | 84.0% CRM ยท 56.5% (64.2%*) |
20 Very Complex prompts (VER-056..075) echoed 3/4 expected matter IDs in the question text, letting explicit Fabric Raw refusals (VER-064/065/075) score 1.0 Accurate. Conversely, Gold answers on MED-055..057 & VER-051 were under-scored for omitting unrequested matter IDs.
StandardRate vs Rate 121% Realization BugOn Raw CRM billing queries (VER-001/007/009/012/013), Fabric Raw joined jt.StandardRate ($1,140) instead of te.Rate ($1,500), producing impossible 121.14% Equity Partner realization and negative leakage while still passing automated title-echo checks.
is_calderwood_dms=0 Inversions & Schema DriftFabric Raw inverted WHERE is_calderwood_dms = 0 on 5 Firm DMS queries (SIM-073, MED-064/065/068, COM-051), mapped MatterType as PracticeArea ($0 revenue ร4), and hit 4 content-filter blocks. Genie's automatic value sampling avoided enum guessing.
'folder' vs firm_practice_group Ambiguity MissesCOM-064..075 asked for "external template folder '<X>'" where <X> (corporate-ma, diligence) is stored in practice_area / firm_practice_group. Excluding those 12 prompts lifts Gold accuracy to 96.5% (DBX) and 95.2% (Fabric).
Single-turn PoC queries ran against 12 pre-warmed tables with 2s pacing. Unmodeled law-firm production environments expose hundreds of cryptic Elite 3E / Aderant / iManage tables, 10โธ+ time entries, millions of .docx files, cold SQL warehouse spin-up, multi-turn chat context, ethical-wall RLS on every join, and 3โ6 agent retry loops (ร 20โ40s) fighting key/rate mismatches โ pushing Raw queries to 3โ4+ minutes. Medallion Gold pre-joins keys and materializes legal flags, cutting P99 latency in half and widening the true Gold advantage to +7.6โ8.0 pts (DBX) and +19.3โ20.8 pts (Fabric).