Specialist teams across the post-training stack.
We field small, senior teams that plug directly into your workflows and tooling. Every project is scoped up front and managed end to end.
Model evaluations
Human evals designed and run by calibrated raters, built around rubrics your researchers can trust.
- Rubric architecture: multi-dimension scoring for accuracy, sourcing, and treatment of uncertainty
- Calibration rounds and gold-set anchoring to keep grader agreement high
- Blind side-by-side comparisons across model versions
- Error taxonomies and capability reports that feed the next training cycle
RLHF & preference data
Rankings from experts trained in comparative judgment, measured against gold sets for the life of the project.
- Preference rankings tuned to what a finance professional would actually sign off on
- Reward-model datasets with documented rater rationale
- Agreement tracking and drift checks across the life of a project
- Fast escalation paths when tasks fall outside rubric coverage
SFT & instruction data
Gold-standard demonstrations and worked examples, written and reviewed by credentialed professionals.
- Expert-written completions with step-by-step reasoning traces
- Domain corpora built to spec: accounting treatments, modeling tasks, market analysis
- Two-layer review with senior sign-off before anything reaches your pipeline
- Format-flexible delivery matched to your ingestion tooling
Red-teaming & adversarial testing
Structured probing for the failure modes that automated testing misses.
- Finance-specific threat models: hallucinated figures, unsupported claims, misapplied standards
- Compliance-sensitive scenario testing designed with practitioners
- Jailbreak and edge-case discovery with reproducible prompts
- Fixed-scope pre-launch packages that run against your ship date
- Prioritized failure-mode reports your engineers can act on directly
Task suites for finance agents
Verifiable task suites that measure whether an agent can do real finance work, end to end.
- Realistic task design: three-statement model builds, reconciliations, audit tie-outs, messy-export cleanup
- Programmatic end-state checks so results are verifiable, not vibes
- Expert-written references for partial credit and error analysis
- Harnesses designed to run as regression suites across model releases
Recent engagements in finance and accounting.
A sample of projects, with details anonymized to protect client confidentiality.
Rubric architecture for financial reasoning evals
A scoring system that turned subjective grading into repeatable measurement.
Side-by-side capability evals across finance domains
Blind comparative grading across valuation, accounting, and market analysis.
Stress-testing an AI finance assistant
Hunting hallucinated figures and misapplied standards in a production system.
Gold-standard accounting datasets
Expert-written walkthroughs of the treatments models get wrong most.
Preference data for analyst-grade output
Rankings tuned to the standard a professional would put their name on.
Task suites for spreadsheet agents
Realistic modeling and reconciliation tasks with verifiable end states.
Low-risk to start. Built to scale.
Scope call
We define the tasks, rubric needs, volume, and tooling. You talk to the people who will run the work.
Calibration pilot
A short, paid, bounded task set. You judge the quality before committing to volume.
Production
A trained team ships against the rubric, with QA layers and throughput reporting as standard.
Scale or wind down
Expand the team, extend the rubric, or close the project cleanly. No lock-in.
Quality is a process, not a promise.
Every engagement runs on the same QA spine, and the artifacts it produces ship with the handoff: rubrics, gold sets, error taxonomies, and QA logs are yours.
Gold sets before production
Raters work a gold set built with your team before touching live tasks, and only those who clear it move on. Calibration is per-project, against your rubric, never a generic badge.
Blind QA through every batch
Senior reviewers audit blind duplicates through live batches. Disagreements go to adjudication, and adjudication outcomes feed back into rater guidance the same week.
Rubric change control
When live tasks fall outside rubric coverage, the ambiguity is logged, the rubric is amended by a single owner, and the change propagates with re-anchoring. Drift gets managed, not discovered.
Frontier models are built on expert judgment.
Marketplaces optimize for volume. We optimize for the quality of the fiftieth hour of work, when calibration and consistency are what actually move your metrics.
Trained, not just credentialed
Every expert on the bench completes hands-on training before touching a client project: annotation tooling, rubric calibration, model comparison, and quality review.
Curated and quality-dense
We know every person we place. The bench stays deliberately small, so agreement rates stay high and you never wonder who is behind the labels.
Managed end to end
S46 handles staffing, calibration, QA, and throughput reporting. You get a single point of contact and data you can trust, not a dashboard of freelancers to manage.
Fast to deploy
Because the bench is trained in advance, we can scope a project and field a team in days, from a single senior reviewer to a full evaluation team, then flex as your roadmap shifts.
Deep expertise across finance.
Finance and accounting is not one vertical among twenty for us. It is the entire practice. Our specialists come from public accounting, banking, buy-side research, and corporate finance teams: selected for judgment, then trained for data work.
Common questions.
How do engagements start?
With a scope call and, usually, a short paid calibration pilot: a bounded task set that shows you the quality bar before anything scales.
How do you handle confidentiality?
NDAs as standard, with flow-down agreements for every expert staffed. Wherever possible, work happens inside your tooling and under your agreements, and we don't retain client data beyond the engagement. Engagements are conflict-screened so experts are not staffed across competing projects.
How is pricing structured?
Scoped up front, per task or per hour depending on the work. No platform fees and no volume minimums. Rate card available on request.
How fast can a team start?
The bench trains before it's staffed, so a team can typically start within days of the scope call, pilot included. Pre-launch red-team packages run against your ship date.
Why not hire a CPA contractor directly?
Domain knowledge is necessary, but it isn't the product. Training-grade data takes rubric design, calibration, agreement measurement, and delivery into your harness. We field finance professionals who have been trained to do exactly that, inside a QA structure built for it.
Who owns what you build?
You do. Rubrics, gold sets, task suites, error taxonomies, and the data itself transfer to you at handoff. Your team can keep running them after we're gone.
Experts: join the bench.
Paid project work on frontier AI systems for finance and accounting professionals. We provide the training, match you to projects in your domain, and handle the logistics. You bring the judgment.
- Remote and project-based, designed to fit around practice
- For credentialed professionals and practitioners with real deal or close experience
- Apply with your CV and a note on your domain
Tell us what you're building.
Whether you need a standing eval team or a one-off data sprint, we'll scope it with you and field the right people.
Include the model or product, the task domain, and rough volume. You'll hear back with a proposed scope within one business day.
Get in touchhello@s46holdings.com