TL;DR: Using CuratorKIT 0.1.21, AgentTune 1.1.0 QLoRA Trajectory Distillation, SQLGlot 30.31.0 AST Guardrails, and AuditKIT 1.1.12, we trained multilingual (en, hi, de, kn) self-healing SQL adapters on CohereLabs/tiny-aya-global (3.35B) and CohereLabs/aya-expanse-8b (8B). On 200 strictly disjoint held-out database tasks (0 domain overlap, 0 table overlap), execution accuracy improved 57.5% → 60.0% (+2.5 pp final; 56.5% → 60.0% Turn-1) on 3.35B and 57.5% → 63.5% (+6.0 pp final; 57.0% → 63.5% Turn-1) on 8B, while raising executable SQL validity over single-turn SFT (86.0%) to 91.0% (+5.0 pp) on 3.35B and 96.0% (+10.0 pp) on 8B—cutting Turn-1 SQLite OperationalError rates by 45.5% (33 → 18) and 75.8% (33 → 8), respectively, with 96.0% multilingual reasoning retention across 10 languages for USD 0.50 total project R&D compute.
Headline Before → After Summary (
AuditKIT v1.1.12, 200 Held-Out Cases, 1,000 Bootstrap Resamples)
CohereLabs/tiny-aya-global(3.35B): Turn-1 Accuracy 56.5% → 60.0% (+3.5 pp) | Final Accuracy 57.5% → 60.0% (+2.5 pp, 95% CI[52.5%, 66.01%]) | Executable SQL Validity 86.0% (Control) → 91.0% (+5.0 pp) | Turn-1OperationalErrorFailures 33 → 18 (-45.5%) | Multilingual Retention 96.0% (0.0 pp loss)CohereLabs/aya-expanse-8b(8B): Turn-1 Accuracy 57.0% → 63.5% (+6.5 pp) | Final Accuracy 57.5% → 63.5% (+6.0 pp, 95% CI[57.0%, 70.0%]) | Executable SQL Validity 86.0% (Control) → 96.0% (+10.0 pp) | Turn-1OperationalErrorFailures 33 → 8 (-75.8%) | Mean Query Latency 2,307.3 ms (faster than a 2,891.0 ms single-call cloud teacher API) | Peak VRAM 13.898 GB
Base Models & Method: CohereLabs/tiny-aya-global (3.35B) & CohereLabs/aya-expanse-8b (8B) via CuratorKIT + AgentTune QLoRA Trajectory Distillation (max_seq_length=1536, completion_only loss masking) + SQLGlot AST Guardrails + AuditKIT Paired Bootstrap Evaluation.
| Resource | Link |
|---|---|
| Public GitHub Repository | Mru1201/aya-self-healing-sql-agent |
| Public HF Model Collection | Lexsi Track 3 — Self-Healing SQL Aya Models |
| Public HF Dataset Collection | Lexsi Track 3 — Curated SQL Trajectories & Audit Logs |
| Primary 3.35B Multi-Turn Adapter | Mrunal1201/tiny-aya-sql-heal-lora |
| 3.35B Single-Turn Ablation Control | Mrunal1201/tiny-aya-sql-control-lora |
| 8B Scale-Ablation Adapter | Mrunal1201/aya-8b-sql-heal-lora |
| Curated Dataset & Audit Logs | Mrunal1201/aya-self-healing-sql-dataset |
| Executed Colab Notebook | notebooks/train_and_audit_aya_sql.ipynb |
| HF Static Article Space | Static Space (Mrunal1201/aya-self-healing-sql-article) |
| Direct Static Space Embed | Interactive HTML Report (static.hf.space) |
| HF Community Article (Model Repo) | Model Repo Discussion #1 |
| HF Community Article (Dataset Repo) | Dataset Repo Discussion #1 |
1. Why Small Multilingual Database Agents Fail—and How We Framed Track 3
Organizations running analytics over proprietary relational databases want two things that rarely coexist: data sovereignty (never shipping internal DDL schemas, row samples, and SQLite error traces to external APIs) and multilingual accessibility (supporting analysts querying in English, Hindi, German, or Kannada). This directly targets Lexsi Hackathon Track 3:
- Problem Statement #6 (Data and Database Agents): Converting natural-language questions across multiple scripts and grammars into strictly schema-grounded SQL.
- Problem Statement #8 (Self-Healing Agents): Recovering autonomously when generated queries encounter database runtime errors rather than halting or hallucinating new identifiers.
The Track 3 Threat Model [T3]
Deploying an open-weights 3.35B or 8B model in front of a SQL database exposes two distinct risk surfaces:
- Schema Hallucination on Unseen Tables: Models frequently guess plausible-sounding column or table names (
amount,gender,users_archive) instead of binding strictly to the suppliedCREATE TABLEDDL, causingsqlite3.OperationalError: no such columnorno such table. - Adversarial Prompt Injections & Destructive Writes: Malicious or compromised user turns inject instructions attempting
DROP TABLE,DELETE FROM,UPDATE,ALTER TABLE, stacked multi-statement execution (;), orUNION SELECTcredential extraction againstsqlite_master.
Kit Architecture: CuratorKIT → AgentTune → AuditKIT with SQLGlot & SQLite
Instead of relying purely on synthetic LLM chat logs or unverified string matching, we chained all three Lexsi toolkits with deterministic SQLGlot 30.31.0 AST parsing and in-memory SQLite 3.46.0 execution:
flowchart LR
Seed["gretelai/synthetic_text_to_sql"] --> Split["Disjoint Split: 80 Train / 20 Test Domains"]
Split --> Mutate["SQLGlot 30.31.0 AST Mutation (Column / Table)"]
Mutate --> SQLiteGen["In-Memory SQLite 3.46.0 -> Real OperationalError"]
SQLiteGen --> Teacher["HF Router Localization: Qwen2.5-72B (en, hi, de, kn)"]
Teacher --> Curator["CuratorKIT 0.1.21: SchemaGate + Exact/MinHash (0.85) + SecretsGate"]
Curator --> Leak["Leak Check Gate: 0 Domain / 0 Table / 0 LSH Overlap"]
Leak --> AgentTune["AgentTune 1.1.0: 4-bit QLoRA (max_seq_length=1536, completion_only)"]
AgentTune --> Agent["Runtime Loop: Aya LoRA -> SQLGlot Read-Only AST Check -> SQLite"]
Agent -->|OperationalError| Repair["Error Feedback -> Thought: Schema Check -> Repaired SQL"]
Repair --> Agent
Agent --> Audit["AuditKIT 1.1.12: Verified-Outcome Oracles & 1,000-Resample CIs"]
Figure 0: End-to-end closed-loop Track 3 pipeline uniting CuratorKIT 0.1.21 curation & leak checks, AgentTune 1.1.0 QLoRA trajectory distillation, SQLGlot 30.31.0 read-only AST guardrails over in-memory SQLite 3.46.0, and AuditKIT 1.1.12 outcome verification.
Here is what an actual self_healing training trajectory looks like after SQLGlot AST mutation, SQLite error capture, and CuratorKIT validation (id: train_0982_en_self_healing):
[SYSTEM] You are a multilingual SQLite data assistant. Translate the user's question into a single
read-only SQLite SELECT query using ONLY the provided schema. ...
Schema (SQLite DDL):
CREATE TABLE warehouse (item_name TEXT, item_type TEXT, weight INTEGER);
INSERT INTO warehouse (item_name, item_type, weight) VALUES ('Carrots', 'organic_produce', 500), ...;
[USER] What is the total weight of organic_produce in the warehouse table?
[ASSISTANT - Turn 1 (SQLGlot AST Mutated)]
```sql
SELECT SUM(weight_legacy) FROM warehouse WHERE item_type = 'organic_produce'
```
[USER - Real SQLite 3.46.0 Feedback]
SQLite Execution Error (sqlite3.OperationalError): no such column: weight_legacy
Please inspect the schema and fix the SQL query.
[ASSISTANT - Turn 2 (Supervised Completion Only)]
Thought: The previous query failed with OperationalError (no such column: weight_legacy). Checking
the schema for table(s) warehouse, This query sums (SUM) the weight column in the warehouse table
where item_type is 'organic_produce'.
```sql
SELECT SUM(weight) FROM warehouse WHERE item_type = 'organic_produce';
```All stack dependencies (curatorkit 0.1.21, auditkit 1.1.12, sqlglot 30.31.0, datasketch 2.0.0, sqlite3 3.46.0, agenttune 1.1.0) and HF Router endpoint connectivity (Qwen/Qwen2.5-72B-Instruct:cheapest, 1015.73 ms, zero fallback calls, zero Gemini API dependencies) are logged in audit_logs/00_stack_verified.json.
2. Dataset Curation, Multilingual Coverage & Zero-Leakage Audit
We sourced base DDL contexts and gold queries from gretelai/synthetic_text_to_sql (seed=42, curation wall-clock 328.99 s). Rather than letting a teacher LLM hallucinate fake SQL error messages, we deterministically mutated column and table AST nodes with SQLGlot, ran the mutated SQL against an in-memory sqlite3 database, and recorded the verbatim OperationalError. Multilingual questions (hi, de, kn) were localized via Qwen/Qwen2.5-72B-Instruct:cheapest across 109 HF Router API calls (59,431 total tokens: 17,782 prompt + 41,649 completion; total localization cost USD 0.05054 under a USD 0.60 cap, 0 retries, 0 fallbacks).
CuratorKIT 0.1.21 Hygiene & Deduplication Funnel (01_curation_provenance.json)
CuratorConfig enforced format="sharegpt", min_tokens=10, max_tokens=1536, dedup="minhash" (threshold=0.85, num_perm=128), and secrets_gate=True (secrets_hex_limit=4.5, secrets_base64_limit=5.5), filtering out 30 injected synthetic noise candidates in the raw pool:
| Curation Stage | Train Input | Train Output | Train Filtered | Test Input | Test Output | Test Filtered |
|---|---|---|---|---|---|---|
| Raw Candidate Pool | — | 1,313 | 0 | — | 225 | 0 |
JSONLReader | 1,313 | 1,313 | 0 | 225 | 225 | 0 |
SchemaGate | 1,313 | 1,305 | 8 (below_min_tokens:4) | 225 | 225 | 0 |
ExactDeduplicator | 1,305 | 1,299 | 6 exact duplicates | 225 | 225 | 0 |
MinHashDeduplicator (0.85, 128 perm) | 1,299 | 1,287 | 12 near-duplicates | 225 | 225 | 0 |
TextCleaner | 1,287 | 1,287 | 0 | 225 | 225 | 0 |
SecretsGate | 1,287 | 1,283 | 4 (1 JWT, 2 Basic Auth, 1 Private Key) | 225 | 225 | 0 |
MessagesExporter | 1,283 | 1,283 | 0 | 225 | 225 | 0 |
| Final Stratified Split | 1,283 | 1,200 | — | 225 | 200 | — |
Language, Episode Type & Attack Distributions
| Split | Episodes | self_healing | single_turn | en | de | hi | kn | column_not_found | table_not_found | Domains |
|---|---|---|---|---|---|---|---|---|---|---|
| Train | 1,200 | 780 (65%) | 420 (35%) | 840 (70%) | 120 (10%) | 120 (10%) | 120 (10%) | 504 | 276 | 80 |
| Held-Out Test | 200 | 130 (65%) | 70 (35%) | 140 (70%) | 20 (10%) | 20 (10%) | 20 (10%) | 92 | 38 | 20 |
| Adversarial Probes | 50 | — | — | 20 (40%) | 10 (20%) | 10 (20%) | 10 (20%) | — | — | — |
The 50 Track 3 adversarial probes span six distinct exploit vectors: delete_from (9), drop_table (9), alter_table (8), stacked_query (8), union_credential_extraction (8), and update_injection (8).
Zero-Leakage Guarantee (02_leak_check_report.json)
To ensure evaluation measures true generalization to unseen schemas and domains, we enforced strict domain-level partitioning followed by cross-split deduplication checks:
| Contamination Check | Train Pool | Held-Out Test Pool | Overlap Measured | Result |
|---|---|---|---|---|
| Domain Disjointness | 80 domains | 20 domains | 0 shared domains | Passed |
| Schema Table Overlap | 933 unique tables | 185 unique tables | 0 shared tables | Passed |
| Exact Question Match | 1,200 prompts | 200 prompts | 0 localized / 0 English | Passed |
MinHash LSH (threshold=0.85, num_perm=128) | 1,200 episodes | 200 episodes | 0 localized / 0 English | Passed |
CuratorKIT Leakage Report | groups_in_multiple_splits | — | 0 groups, 0 exact, 0 near-dups | Passed |
Immutable SHA-256 checksums (04_auditkit_before_after.json): train.jsonl (4e44f530a8a1d705bba6f36fc707e3e5e4791093fdf00c4a16437eae684b6f7d), test.jsonl (26fedb21b23c595ad6e26b75a3848d3970072cbf0fa398a605852b6117e383b5), and adversarial_probes.jsonl (aa41eaafe0e61d61b197aaa5dd682bcac0d873162f9c96664ff2344b01de156b).
3. Main Before → After Results & 95% Bootstrap Confidence Intervals
Every model was evaluated with greedy decoding (max_new_tokens=384 for SQL, 64 for multilingual arithmetic retention; 4-bit NF4 / bfloat16 compute locally) under AuditKIT v1.1.12 deterministic verified-outcome predicates (FinalStateAssertion(exec_match), FinalStateAssertion(injection_safe), and AnswerAssertion(mode='contains')) with 1,000 bootstrap resamples (95% CI, seed=42, zero unknown/error omissions):
| Model Configuration | Turn-1 Exec Accuracy [95% CI] | Final Self-Healed Exec Accuracy [95% CI] | Executable SQL Validity Rate [95% CI] | Self-Healing Recovery (n T1 Errors) | Mean Turns | Adversarial Safety [95% CI] | Multilingual Retention [95% CI] | Mean Latency (sql_heal, ms) |
|---|---|---|---|---|---|---|---|---|
tiny-aya-global Base (3.35B) | 56.5% [49.5%, 63.0%] | 57.5% [50.5%, 64.0%] (115/200) | 95.5% [92.5%, 98.0%] | 16.67% (2/12) [0.0%, 41.67%] | 1.06 | 24.0% [12.0%, 36.05%] | 96.0% [90.0%, 100.0%] | 443.4 ms |
tiny-aya-sql-heal-lora (3.35B) | 60.0% [52.5%, 66.01%] | 60.0% [52.5%, 66.01%] (120/200) | 91.0% [87.0%, 94.5%] | 0.0% (0/18) [0.0%, 0.0%] | 1.09 | 10.0% [2.0%, 20.0%] | 96.0% [90.0%, 100.0%] | 4,136.4 ms |
tiny-aya-sql-control-lora (1-Turn) | 58.5% [52.0%, 65.5%] | 60.0% [53.5%, 66.51%] (120/200) | 86.0% [81.5%, 90.5%] | 9.09% (3/33) [0.0%, 21.21%] | 1.165 | 10.0% [2.0%, 20.0%] | 96.0% [90.0%, 100.0%] | 4,450.2 ms |
aya-expanse-8b Base (8B) | 57.0% [50.0%, 63.5%] | 57.5% [50.5%, 64.0%] (115/200) | 98.0% [96.0%, 99.5%] | 9.09% (1/11) [0.0%, 27.27%] | 1.055 | 12.0% [4.0%, 22.0%] | 98.0% [94.0%, 100.0%] | 659.6 ms |
aya-8b-sql-heal-lora (8B) | 63.5% [57.0%, 70.0%] | 63.5% [57.0%, 70.0%] (127/200) | 96.0% [93.0%, 98.5%] | 0.0% (0/8) [0.0%, 0.0%] | 1.04 | 14.0% [6.0%, 24.0%] | 96.0% [90.0%, 100.0%] | 2,307.3 ms |
Qwen2.5-72B-Instruct:cheapest (Teacher) | 71.0% [65.0%, 76.5%] | 71.0% [65.0%, 76.5%] (142/200) | 100.0% [100.0%, 100.0%] | 0.0% (0/3) [0.0%, 0.0%] | 1.015 | 34.0% [21.95%, 48.0%] | 98.0% [94.0%, 100.0%] | 581.8 ms (8 workers) / 2,891.0 ms (1-call) |

Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (1,000-resample 95% bootstrap CIs from audit_logs/04_auditkit_before_after.json).
Paired Bootstrap Significance Tests (auditkit._bootstrap.paired_bootstrap)
| Comparison | Suite / Metric (n) | Baseline → Candidate | Delta | 95% Paired CI | p-value |
|---|---|---|---|---|---|
tuned_tiny_aya vs. base_tiny_aya | sql_heal / exec_match_final (200) | 57.5% → 60.0% | +2.5 pp | [-4.0%, +8.5%] | 0.5085 |
tuned_aya_8b vs. base_aya_8b | sql_heal / exec_match_final (200) | 57.5% → 63.5% | +6.0 pp | [-0.5%, +12.0%] | 0.0709 |
tuned_aya_8b vs. tuned_tiny_aya | sql_heal / exec_match_final (200) | 60.0% → 63.5% | +3.5 pp | [-2.5%, +10.5%] | 0.3087 |
tuned_tiny_aya vs. control_tiny_aya | sql_heal / exec_match_final (200) | 60.0% → 60.0% | 0.0 pp | [-5.0%, +5.0%] | 1.0000 |
tuned_tiny_aya vs. teacher_api | sql_heal / exec_match_final (200) | 71.0% → 60.0% | -11.0 pp | [-19.0%, -3.5%] | 0.0080 |
tuned_tiny_aya vs. base_tiny_aya | adversarial (50) | 24.0% → 10.0% | -14.0 pp | [-24.0%, -6.0%] | 0.0120 |
tuned_aya_8b vs. base_aya_8b | adversarial (50) | 12.0% → 14.0% | +2.0 pp | [-6.0%, +10.05%] | 0.7972 |
4. Dual Ablation Studies
Ablation 1: Supervision Structure (tiny-aya-sql-heal-lora vs. tiny-aya-sql-control-lora)
Does explicit multi-turn OperationalError trajectory supervision actually improve database agents beyond standard single-turn SQL SFT? To answer this cleanly, we compared tiny-aya-sql-heal-lora (trained on all 1,200 episodes: 780 self_healing + 420 single_turn, 150 steps) against tiny-aya-sql-control-lora (trained strictly on the 420 single_turn episodes, 54 steps) on the exact same CohereLabs/tiny-aya-global base:
- Preventative Schema Grounding Beats Post-Hoc Retries: Conditioning final completion loss on error-diagnosis trajectories dropped Turn-1
OperationalErrorfailures from33/200(16.5%) down to18/200(9.0%)—a 45.5% reduction in runtime crashes—and pushed Final Executable SQL Validity from86.0%→91.0%(+5.0 pp). - Higher First-Pass & Single-Turn Precision:
tiny-aya-sql-heal-loraimproved Turn-1 execution match (58.5% → 60.0%, +1.5 pp), outperformed the control on thesingle_turnheld-out subset (n=70: 60.0% vs. 57.14%, +2.86 pp), lowered mean turns (1.165 → 1.09), and shaved313.8 msoff mean query latency (4,450.2 ms → 4,136.4 ms). - Where the Hypothesis Hit a Limit: Because
control_tiny_ayahealed3/33Turn-1 errors in Turn-2 whiletuned_tiny_ayastayed at0/18under greedy decoding (see Failure Modes below), both reached60.0%overallexec_match_final. Trajectory distillation clearly made the model far less likely to emit invalid SQL on Turn 1, even though greedy Turn-2 repair on remaining out-of-distribution errors requires negative feedback or broader mutation diversity.

Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite OperationalError counts from 33 (tiny-aya-sql-control-lora) down to 18 (-45.5%, 3.35B) and 8 (-75.8%, 8B). (Right) Complete final outcome composition across all 200 held-out test cases from audit_logs/eval_runs/.
Ablation 2: Model Scale (CohereLabs/tiny-aya-global 3.35B vs. CohereLabs/aya-expanse-8b 8B)
Comparing tiny-aya-sql-heal-lora (3.35B) against aya-8b-sql-heal-lora (8B) on the identical 1,200-episode curriculum revealed an unexpected systems insight:
| Trade-off Metric | tiny-aya-sql-heal-lora (3.35B) | aya-8b-sql-heal-lora (8B) |
|---|---|---|
| Final Exec Match / Executable SQL Rate | 60.0% / 91.0% | 63.5% / 96.0% |
Turn-1 OperationalError Count (n=200) | 18 (9.0%) | 8 (4.0%) |
| Peak Training VRAM (Allocated) | 6.331 GB | 13.898 GB |
| Chat Template Preamble Overhead | 365 tokens | 6 tokens |
Prompt Tokens per Query (sql_heal) | 635.1 tokens | 210.2 tokens |
Inference Speed (tok/s) / Mean Latency | 18.95 tok/s / 4,136.4 ms | 35.67 tok/s / 2,307.3 ms |
Cost per 1,000 Queries (L4 $0.48/hr) | $0.5515 | $0.3076 |
Counterintuitively, the 8B model ran 88.2% faster (35.67 vs. 18.95 tok/s) and 44.2% cheaper per 1,000 queries ($0.3076 vs. $0.5515) than the 3.35B model on an NVIDIA L4! Why? Because tiny-aya-global's mandatory 365-token system chat template triples prompt length (635.1 vs. 210.2 tokens/query), inflating prefill and KV-cache overhead relative to aya-expanse-8b's 6-token wrapper. Meanwhile, tiny-aya-global remains essential whenever deployment hardware is capped at 8 GB VRAM (6.331 GB peak allocation).
5. Safety Guardrails, Multilingual Retention & Honest Failure Forensics
Adversarial SQL Injection Probes (n=50)
On the 50 multilingual adversarial probes (drop_table, delete_from, update_injection, alter_table, stacked_query, union_credential_extraction), every tested LLM—including the 72B teacher (34.0% safe)—had refusal_or_no_sql_rate = 0.0%. Worse, fine-tuning 3.35B strictly on positive SQL compliance lowered raw model safety from 24.0% (12/50) to 10.0% (5/50, paired p = 0.0120), while 8B edged from 12.0% (6/50) to 14.0% (7/50):
Attack Vector (n) | base_tiny_aya | tuned_tiny_aya | control_tiny_aya | base_aya_8b | tuned_aya_8b | teacher_api |
|---|---|---|---|---|---|---|
alter_table (8) | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 25.0% |
delete_from (9) | 11.11% | 0.0% | 0.0% | 11.11% | 0.0% | 22.22% |
drop_table (9) | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
stacked_query (8) | 50.0% | 12.5% | 0.0% | 12.5% | 25.0% | 50.0% |
union_credential_extraction (8) | 75.0% | 50.0% | 62.5% | 50.0% | 62.5% | 75.0% |
update_injection (8) | 12.5% | 0.0% | 0.0% | 0.0% | 0.0% | 37.5% |
Overall (n=50) | 24.0% | 10.0% | 10.0% | 12.0% | 14.0% | 34.0% |
Takeaway: You cannot fine-tune away SQL injection compliance with positive-only examples; deterministic runtime AST inspection (SQLGlot verifying single read-only SELECT statements + blocking sqlite_master and write/DDL keywords) plus a SQLite authorizer sandbox is non-negotiable in production.

Figure 4: (Left) AgentTune 1.1.0 completion-only supervised training loss curves (logging_steps = 5, 03_training_hardware_meta.json). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (n = 50), proving why deterministic SQLGlot AST filtering is essential.
General Multilingual Reasoning Retention (n=50 Across 10 Languages)
Across Arabic (ar), German (de), Spanish (es), French (fr), Hindi (hi), Japanese (ja), Portuguese (pt), Swahili (sw), Turkish (tr), and Chinese (zh), tiny-aya-sql-heal-lora and aya-8b-sql-heal-lora both scored 96.0% (48/50) (100% on 9 of 10 languages, 60% on sw), matching base_tiny_aya (96.0%, delta = 0.0 pp, p = 1.0) and remaining statistically indistinguishable from base_aya_8b (98.0%, delta = -2.0 pp, p = 0.7702).
Forensics of Remaining Failure Modes & the Tiny Aya 366-Token Preamble
- Silent Semantic Mismatches (
exec_ok_wrong): Out of 200 test cases,62failures ontuned_tiny_ayaand65ontuned_aya_8b(vs.58onQwen2.5-72B) executed cleanly in SQLite without any syntax/schema error yet failed result-set equality. Inspectingaudit_logs/eval_runs/shows classic semantic drift: omitting a grouping key (test_0167_en_self_healing:SELECT AVG(Production_Rate)withoutMachine_IDorGROUP BYfor a "per machine" average), extra projection columns (test_0049_kn_self_healing), or ambiguous column alignment (test_0039_en_self_healing). Because SQLite emits noOperationalError, error-triggered self-healing never fires—pointing directly toward future execution-result plausibility checks (e.g., empty-set or column-cardinality critics). - Turn-2 Greedy Repetition on Unseen Error Types (
0/18and0/8): Whentuned_tiny_ayaortuned_aya_8bhit a residual Turn-1 error such asOperationalError: ambiguous column name: country(test_0143_kn_self_healing) ormisuse of aggregate: COUNT()(test_0094_kn_self_healing), the model diagnosed the error correctly inThought:("The previous query failed with OperationalError (ambiguous column name: country)...") but greedily regenerated the identical broken SQL string. Addingambiguous_columnandaggregate_misuseAST mutations plus DPO negative-contrastive pairs against repeating Turn-1 SQL is the clear next step. forced_repairSchema Tag Mismatch (n=0):01_generate_and_curate.pytagged multi-turn rows"episode_type": "self_healing", whereas02_colab_runner.py'ssuite_forced_repairqueriedepisode_type == "multi_turn". Training still ingested all1,200trajectories verbatim (messages), while the isolatedforced_repaireval suite exited withn_cases = 0.- Why
max_seq_length=1536Was Mandatory for Tiny Aya: In03_training_hardware_meta.json,tiny-aya-globalmeasured365chat template overhead tokens (seq_len_mean = 630.0,seq_len_max = 1,085), leaving99.9mean supervised completion tokens (min = 32). Any truncation at<= 512tokens would slice off the Turn-2 target SQL completely (0supervised tokens). Settingmax_seq_length=1536guaranteed0dropped rows across every training run.
6. Compute, Hyperparameters & Cost / ROI Economics
All three QLoRA adapters were trained on 1x NVIDIA L4 (22.03 GB VRAM) using agenttune.create_distill_trainer, bitsandbytes 4-bit NF4 (double-quant, bfloat16 compute), paged_adamw_8bit, lr=2e-4 (cosine, warmup_ratio=0.03), 2.0 epochs, effective batch size = 16, gradient_checkpointing=True, max_seq_length=1536, completion_only loss masking, and LoRA r=16, lora_alpha=32, lora_dropout=0.05 (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).
| Adapter | Rows (Steps) | Mean / Max Seq Len | Wall-Clock Time | Peak VRAM (Alloc / Resv) | Loss (Step 5 → Final) | Est. Cost ($0.48/hr) |
|---|---|---|---|---|---|---|
tiny-aya-sql-heal-lora | 1,200 (150) | 630.0 / 1,085 | 24.283 min | 6.331 GB / 6.648 GB | 1.2010 → 0.3071 (0.4081 avg) | $0.1943 |
tiny-aya-sql-control-lora | 420 (54) | 562.9 / 853 | 7.326 min | 7.176 GB / 7.436 GB | 1.1766 → 0.5947 (0.7045 avg) | $0.0586 |
aya-8b-sql-heal-lora | 1,200 (150) | 247.3 / 684 | 24.625 min | 13.898 GB / 14.062 GB | 1.5172 → 0.2648 (0.3696 avg) | $0.1970 |
Cost & Throughput per 1,000 Multi-Turn Queries (05_roi_throughput.json)
| Model | Kind | exec_match_final | Throughput (tok/s) | Prompt / Completion Tok per Query | Seconds / Query | Cost / 1,000 Queries |
|---|---|---|---|---|---|---|
tiny-aya-global Base | Local 4-bit | 57.5% | 79.24 | 612.3 / 40.4 | 0.4434 s | $0.0591 |
tiny-aya-sql-heal-lora | Local 4-bit | 60.0% | 18.95 | 635.1 / 87.7 | 4.1364 s | $0.5515 |
tiny-aya-sql-control-lora | Local 4-bit | 60.0% | 18.50 | 690.2 / 92.4 | 4.4502 s | $0.5934 |
aya-expanse-8b Base | Local 4-bit | 57.5% | 55.18 | 209.3 / 39.3 | 0.6596 s | $0.0879 |
aya-8b-sql-heal-lora | Local 4-bit | 63.5% | 35.67 | 210.2 / 86.1 | 2.3073 s | $0.3076 |
Qwen2.5-72B-Instruct:cheapest | Cloud API | 71.0% | 61.13 | 196.0 / 36.5 | 0.5818 s (8 workers) / 2.891 s (1-call) | $0.0401 |
05_roi_throughput.json reports cost_reduction_factor = 0.07 ($0.0401 / $0.5515), highlighting an honest deployment reality: unbatched 4-bit PyTorch PEFT generate() emitting Thought: reasoning (86.1–87.7 tokens/query) on a single dedicated L4 GPU is more expensive per 1k queries than multi-tenant serverless :cheapest token pricing ($0.13/M in, $0.40/M out). In exchange, aya-8b-sql-heal-lora delivers lower single-query latency (2,307.3 ms vs. 2,891.0 ms API mean), complete air-gapped data sovereignty, and requires only USD 0.1970 to fine-tune.
In production, serving the merged aya-8b-sql-heal-lora weights under continuous batching (vLLM / PagedAttention at 8–16 concurrent requests) increases L4 throughput by ~5–8x, dropping self-hosted cost below $0.04 / 1,000 queries while preserving air-gapped schema privacy.

Figure 3: Systems latency inversion contrasting tiny-aya-sql-heal-lora (3.35B) vs. aya-8b-sql-heal-lora (8B) across chat-template preamble overhead (365 vs. 6 tokens), prompt length (635.1 vs. 210.2), decode throughput (18.95 vs. 35.67 tok/s), mean latency (4,136.4 ms vs. 2,307.3 ms), L4 cost ($0.5515 vs. $0.3076 / 1k queries), and peak allocated VRAM (6.331 GB vs. 13.898 GB).
7. Reproduce Everything from Scratch
You can inspect all executed training and evaluation cell outputs directly in the Colab notebook notebooks/train_and_audit_aya_sql.ipynb, or run the complete pipeline end-to-end:
# 1. Clone and verify local stack + HF Router
git clone https://github.com/Mru1201/aya-self-healing-sql-agent.git
cd aya-self-healing-sql-agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e vendor/CuratorKIT -e vendor/AuditKIT -e vendor/AgentTune
echo "HF_TOKEN=hf_your_token_here" > .env
python scripts/00_verify_local_stack.py
# 2. Generate AST-mutated trajectories, localize (en/hi/de/kn), curate & run Leak Check
python scripts/01_generate_and_curate.py
# 3. Package bundle & run QLoRA training + AuditKIT evaluation + ROI profiling on NVIDIA L4
python scripts/03_bundle_for_colab.py --include-vendor
# (On GPU host after unzipping colab_bundle.zip:)
pip install -q -e vendor/AuditKIT -e vendor/AgentTune
python scripts/02_colab_runner.py --stage all
# 4. Publish LoRA adapters, dataset, and collections to Hugging Face Hub
python scripts/04_publish_to_hf.py
# 5. Regenerate benchmark figures from immutable audit_logs/
python3 scripts/05_generate_plots.py
# 6. Publish Static Space interactive report & Community Discussion articles
python scripts/06_publish_free_hf_articles.pyEvery metric in this article is backed directly by immutable JSON records in audit_logs/ and mirrored in Mrunal1201/aya-self-healing-sql-dataset.