Methodology
The Algodai Methodology
Algodai combines AI application security testing with behavioral accountability review, then measures whether a model or product holds up on fresh, practical tasks — not just public benchmarks. Four of the six modules below are tested directly against the AI surface; the other two are scored from the company's own published disclosures, cited by source.
1. Security and Misuse Resistance Tested
Prompt injection, jailbreaks, sensitive-data disclosure, tool abuse, excessive agency, cost amplification, and unsafe output handling.
2. Truth Integrity Tested
Hallucination behavior, source substitution, correction consistency, uncertainty disclosure, and false authority signals.
3. Political Fairness Tested
Mirrored left/right prompts, selective refusal, loaded language, selective skepticism, persuasion asymmetry, and framing-anchor questions — naturally-phrased questions where the "factual" framing itself splits along ideological lines, scored against defined left and right anchor answers to catch a default lean that explicit steelman requests can't.
4. Data, Privacy, and Memory Tested
Retention claims, user-data boundaries, cross-user leakage, vendor exposure, deletion/correction paths, and hidden memory behavior.
5. Accountability and Appeal Documentation
Human escalation, reviewability, recovery after mistakes, proportionality, explanation quality, and user correction paths — scored from the company's own published policies, cited by source.
6. Governance and Release Controls Documentation
Model inventory, release scope, change control, monitoring, incident response, rollback readiness, and evidence retention — scored from the company's own published disclosures, cited by source.
The Reality Gap Signal
Public benchmarks are gameable and often don't predict practical behavior — a model can top a leaderboard and still hallucinate an API or break working code the moment the task isn't a clean, self-contained problem. Algodai runs a separate, rotating set of fresh, practical tasks: a real bug fix against a real test suite, a browser Snake game, an advanced seeded roguelike, consistency under rephrasing, confident fabrication under pressure, long-context multi-amendment synthesis, calibration/hedging, operational-friction probes, and agentic secrets-handling. Published artifact tasks expose inspectable outputs; secrets-handling evidence stays private because the fixture structurally resembles a real credentials file. Findings are reported with methodology, limitations, and hashed or redacted evidence rather than a raw prompt pool that can be memorized.
Critical findings cap the grade.
Algodai does not average away severe failures. A system with cross-user data leakage, unrestricted agent access, falsified audit evidence, or no human escalation in a high-risk decision workflow cannot receive a high overall grade because other categories scored well.
Who does the judging
Algodai's own tooling runs partly on the same models it evaluates — a model grading its own maker's product would be a real conflict of interest. The mitigation is cross-company independent judging, not pretending the conflict doesn't exist: a transcript is never scored by a model made by the same company as the surface being tested. Where more than one independent judge scores the same transcript, disagreement is flagged in the record rather than averaged away silently. Each record's judge(s) and any disagreement are disclosed on the record itself, not just here.
Where the original research fits.
Algodai's early work on probation, appeal, restoration, and never-forgetting systems remains the foundation for the accountability layer above: correction, escalation, auditability, and safe release controls, expressed here in operational terms.