@misc{wong2026-malay-llm-safety-score,
  author = {Wong, Sam},
  title = {Malay AI Safety Evaluation: A Case Study with Oaica’s Fine-Tuned 35B-A3B Model, and an Indonesian Cross-Language Probe},
  year = {2026},
  howpublished = {Oaica Research},
  url = {https://research.oaica.com/2026/10/malay-llm-safety-score/}
}

Malay AI Safety Evaluation: A Case Study with Oaica’s Fine-Tuned 35B-A3B Model, and an Indonesian Cross-Language Probe

What the SEA-HELM Malay safety benchmark measures, an Indonesian cross-language probe, and what they cannot resolve

Sam Wong · [email protected]

Oaica (oaica.com)

1 October 2026

Keywords: Malay LLM safety benchmark; LLM safety evaluation; Malay (Bahasa Melayu); Indonesian (Bahasa Indonesia); cross-language transfer; multilingual and culturally grounded safety; Southeast Asian languages; SEA-HELM; SEA-SafeguardBench; confidence intervals; annotation disagreement; benchmark reliability; leaderboard ranking uncertainty; reasoning models; open-weight models.

Evidence notice. This is an observational case study, not an independent model certification. Historical scores are distinguished from recomputed runs. The evidence document supplies text-free counts, exact score-recomputation code and an explicit list of missing provenance. Historical interval reconstruction and the mapping of saved results to checkpoint hashes remain open; no claim of complete reproducibility or verified absence of contamination is made.

1  Introduction

In our experience with one regional language, safety work failed more often at measurement than at training. The harms that matter are culturally specific, and they concern religion, ethnicity, institutions and local politics. The public benchmarks that measure them in Malay are small [2], and their labels necessarily encode one annotation convention, which other competent annotators may not share [7]. SEA-HELM and SEA-SafeguardBench are valuable public instruments [1, 7]; the scorer comparison in Section 2 and the interval analysis in Section 5 are possible because the leaderboard’s maintainers document the scorer and publish intervals over repeated runs [2], and the assessment in Section 5 draws on the annotator agreement that the source benchmark reports [7].

We chose Malay as the worked example because a public, culturally grounded safety benchmark exists for it at a size small enough for the measurement problems to be visible and quantifiable [2, 7], and because national guidance on AI governance is in place in Malaysia [19]. We expect the same problems wherever culturally grounded safety subtasks are small, which we expect to be the usual case for regional languages (Section 5.3).

In this regime a subtask may hold only tens of items, so one changed prediction can move it by several points. Differences between training seeds of the same recipe are as large as the effects practitioners hope to find. Without discipline, iterative development ends up optimising noise: the best of many runs is reported, and the gain does not replicate.

This paper reports what the Malay safety track of SEA-HELM [1, 2] can support at its current size and published precision. It gives the size of the effects that evaluation choices alone produce, the spread of a single training lineage, and an assessment of what the benchmark’s headline number can and cannot resolve. Training methods, data sources and model internals are out of scope. It is a contribution to evaluation methodology, not a competitive claim about any model’s safety.

Our contributions are:

1. Examples showing that evaluation choices, including aggregation, reasoning mode and request concurrency, can move observed safety scores by amounts comparable to historical variation among closely matched runs.
2. Measurements of the spread of one training lineage: 12 variants spanning 3.3 points on the composite, and 20 variants drawn from that lineage’s final runs and from earlier lineages, whose accuracy on the toxicity subtask spans 0.577 to 0.614 (standard deviation (SD) 0.008).
3. Worked examples of measurement artefacts that would otherwise have been reported as model effects.
4. An assessment of what the public benchmark can and cannot resolve, including indirect evidence that agreement with its annotation convention, rather than harm detection alone, may limit scores on the toxicity subtask.
5. A practical placement of the two checkpoints on the wider Malay and regional benchmark set, and candidate uses for supervised pilots, subject to validation in the deployment context.
6. Guidance for readers of the leaderboard.

2  The Malay safety track and its scorer

SEA-HELM [1] is a public evaluation suite for Southeast Asian languages. The Malay safety competency of its leaderboard [2], one of the task groups the leaderboard reports, is what this paper calls the safety track. It averages two parts. The first is toxicity detection: 1,000 texts, 500 toxic and 500 non-toxic. The second is a safeguard group of three culturally grounded subtasks drawn from SEA-SafeguardBench [7] with binary labels. Each asks whether a prompt or a response is harmful. Two subtasks share one prompt set: cultural prompts (71 items, 32 harmful) and cultural responses (the responses to the same 71 prompts, judged with the prompt as context; 11 harmful), which together use 71 of the 215 Malay prompt–response pairs in [7]. The third, cultural content in the wild, uses a separate set of 430 Malay items of [7], half harmful. The repository also defines two general safeguard subtasks of 600 items each, which are not run by any of the repository’s task sets, including the Malay set [2]. The official scorer computes each subtask’s balanced accuracy, the mean of the per-class recalls, and rescales it against chance as (balanced accuracy − 1/k)/(1 − 1/k), floored at zero, where k is the number of classes. It averages the three safeguard scores, then averages that mean with the toxicity score, so toxicity carries half of the 0 to 100 composite [2]. Plain accuracy, used below for comparison, is the fraction of items answered correctly, rescaled against chance in the same way.

The composite, its aggregation and the subtasks it includes must be declared with every number, because they move the result. One checkpoint scored anywhere from 45.17 to 50.47 depending on the aggregation (Table 1).

Table 1: One checkpoint under four aggregations (composite, points).

The 5.3-point spread is larger than the 3.3-point range of the 12 training variants in Section 3.3. Choosing an aggregation after seeing results is a form of selection. Every number in this paper therefore carries its definition, and the definition that matches the official scorer is the one of record.

3  How far evaluation choices move the score

The evaluation choices in Sections 3.1 and 3.2 change nothing about the weights. On at least one of the two checkpoints, each of them moves the official composite by more than the average gap between two variants of the same training recipe.

3.1 Reasoning mode

Switching reasoning on lowered the official composite of both models by more than the average gap between any two of the 12 variants of the lineage (1.2 points, the mean absolute difference over all 66 pairs), so reasoning mode belongs to the evaluation protocol and must be declared with every number.

With greedy decoding on both sides and a reasoning budget of 1,536 tokens when reasoning was on, enabling reasoning lowered the composite of the two checkpoints, Oaica 35B-A3B Malay Safety v1.0 260923 and Oaica 35B-A3B Malay v1.0 260923, by 1.5 and 2.3 points (paired 95% intervals for the change, reasoning on minus off: −5.5 to +3.4 and −7.4 to +3.4; Figure 1). An earlier check of two other variants under the benchmark’s own reasoning flag, which allows 20,000 additional reasoning tokens and leaves decoding at each model’s packaged sampling defaults [2], found scores 6 to 8 points lower; those runs could not be re-scored under the official formula, so we treat them as indicative only.

Individually neither drop is statistically significant at α = 0.05, since both paired intervals include zero, and the two checkpoints share a base model, so they are not independent evidence. Both point the same way, and each exceeds the 1.2-point average gap between variants given above.

3.2 Request concurrency

The number of concurrent requests moves the three cultural safeguard subtasks without changing any weights. Re-running those subtasks with one concurrent request instead of four changed only four of the 568 predictions that could be matched between the two runs (0.7 per cent), yet moved their mean score by 3.56 points (69.47 to 73.03), worth about 1.8 points on the composite. For Oaica 35B-A3B Malay v1.0 260923, the same three subtasks moved the other way, about −0.5 on the composite, while seven more correct toxicity predictions on balance added about +0.7, a net change of about 0.2 points. The size and sign of the effect are therefore those of a few flipped predictions on high-leverage items rather than a systematic bias.

3.3 Spread of one training lineage

The 12 training variants of one lineage spanned 3.3 points on the official composite (mean 45.36, SD 1.04): all inside the 95% interval of any single run, comparable to the paired resolution of the reasoning-mode comparison above, and comparable in size to the effects of the evaluation choices in Sections 3.1 and 3.2. Across all 20 variants retained from earlier and final lineages, the composite spanned 40.8 to 47.2 points.

The historical analysis reported a bootstrap 95% confidence interval (CI) of about ±6 points for a single run. These are marginal item-sampling intervals, not uncertainty on a paired difference or an estimate of training and selection uncertainty. This follows recent calls to put error bars on language-model evaluations [3] and to avoid relying on normal approximations for small samples [4].

The safety-tuned checkpoint, Oaica 35B-A3B Malay Safety v1.0 260923, scored 47.19 on the official safety composite (95% CI 40.59 to 52.98, from one greedy run), and the general-use checkpoint, Oaica 35B-A3B Malay v1.0 260923, scored 45.64 (CI 39.22 to 52.20). Re-run through a production serving stack with single requests, they scored 47.11 and 44.94. Across the seven competencies (task groups) of the Malay leaderboard, the two checkpoints’ means differed by about a third of a point, well inside their intervals of about ±2 points. Our reproduction is not the leaderboard’s runner and does not reproduce every published entry, so neither the safety composites above nor these means can be compared like-for-like with the published scores in Section 5, and comparing their marginal intervals cannot determine pairwise significance. We report the lineage and make no confirmatory claim of a gain over the siblings. The historical scores and intervals are retained as archived results; their full interval reconstruction and immutable checkpoint mapping remain incomplete, as documented in the evidence document.

4  Findings

Measured this way, two findings stand out: one about the toxicity subtask, and one about how many apparent effects are artefacts of the measurement.

4.1 The toxicity subtask and its annotation convention

Toxicity detection carries half the composite, and no training variant raised it above a narrow band (Figure 2). Across the 20 variants we retained (the 12 of the final lineage in Section 3.3 and eight from earlier lineages), accuracy on the 1,000-item toxicity subtask averaged 0.600 (SD 0.008, range 0.577 to 0.614). The spread across variants is about half the item-sampling standard error of a single score (0.015). Similar aggregate accuracies do not establish shared errors; that would require an item-level disagreement analysis. Further variants from lineages abandoned during development scored between 0.50 and 0.58 and are not shown: training changes could lower the subtask, but none raised it above the band. Because the subtask is balanced and carries half the composite, each 0.01 of toxicity accuracy is worth one composite point.

Several results bear on why the scores stop there. For one model, a decision threshold tuned on the test labels themselves reached only 0.617 accuracy, against 0.612 for its ordinary predictions (Figure 2), and its scores separated the classes with an area under the receiver operating characteristic (ROC) curve (AUC) of just 0.632. This exploratory threshold sweep offered little improvement for the tested score on these test items. It does not bound other calibration methods or models, and it cannot identify whether model limitations, policy mismatch or labels explain the low performance. The leaderboard’s own toxicity column, produced under the maintainers’ protocol rather than ours and so not comparable entry by entry with our figures, also shows low scores in that dated snapshot: across its 58 open-weight entries the highest published score is 17.51 normalised points, about 0.59 balanced accuracy, and the median entry scores about 0.55 [2]. Automated annotators, language models prompted to assign labels in the benchmark’s convention, agreed with human labels in that convention at only 0.51 to 0.64 balanced accuracy; each of these figures is uncertain by several points.

Scores on this subtask were low across the historical runs and the dated leaderboard snapshot. These observations do not distinguish model limitations, annotation-policy mismatch, missing context or label error, and do not quantify their relative contributions. Automated annotators are not an independent human reference. A blinded audit with native Malay speakers, a specified policy and disagreement reporting is needed; a new held-out sample would also help assess generalisation. Work on annotator disagreement [11, 12] and imperfect alignment between automated and human annotations [13] motivates that investigation, but does not establish the explanation in this dataset.

4.2 Measurement artefacts explain many apparent effects

Two cases show the size of such effects. In one, a composite computed with a different formula from the one an archived result used read as a 1.8-point gain for one model until both were recomputed with the same formula. In another, a result file with no record of its configuration read 47.27 against 45.75 for the configuration-matched run, both under the same earlier formula. Several effects in this paper happen to be about 1.8 points; they are distinct measurements, and only the formula cases share a cause.

5  Assessment of the SEA-HELM Malay safety track

SEA-HELM’s Malay safety track [2], built on the SEA-HELM suite [1] and SEA-SafeguardBench [7], is a valuable public instrument for regional-language safety. Its small classes make some scores sensitive to individual outcomes. Published marginal intervals alone do not identify which differences between models are distinguishable, and low toxicity scores alone do not identify their cause.

It is one of very few public instruments that test culturally grounded harms in the region, and its cultural subtasks target failures that general-purpose safety sets do not cover [7]. The weaknesses listed in Table 2 concern what its headline number can support, not whether it is worth running. Its maintainers average eight runs per model and publish 95% bootstrap intervals from 2,000 resamples of the items [2], which is what makes the interval analysis in Section 5.1 possible.

Table 2: Properties of the safety track that limit what its headline number can support.

5.1 Item leverage and interval overlap

Under the official scorer, away from the zero floor, one corrected prediction moves the composite by 50/(3n) points on a safeguard subtask and by 50/n points on toxicity, where n is the size of the item’s class within its subtask (Figure 3). One of the 11 harmful items in the cultural-response subtask therefore moves the composite by 1.5 points, 15 times as much as one toxicity item (0.10) and about 20 times as much as one in-the-wild item (0.078). The cultural-prompt and cultural-response subtasks score only 71 distinct prompts (142 of the 1,572 scored items, 9.0 per cent), yet this is where single predictions move the composite most. These figures are for a single run, which is how we evaluate; published entries average eight runs per item, so an answer that changes in one of the eight runs moves an entry by an eighth of these amounts.

The data behind the leaderboard’s Malay page hold 58 open-weight entries of up to 200 billion parameters (dated 18 September 2026), of which the page’s default view displays 18; we use the full set. Its ten highest safety scores, from 46.22 down to 42.47, have 95% intervals that all share the range 40.8 to 47.4 points (Figure 4). Non-overlap of individual intervals is a conservative test for a difference, so we also approximated a 95% interval for each pairwise difference from the published half‑widths, treating entries as independent: every approximate interval includes zero, and the largest standardised difference is 0.93. This is an independence sensitivity calculation, not a paired test: the models share evaluation items, and the necessary covariance is unavailable in the published marginal intervals. The five highest entries in the default view likewise share a range (40.8 to 46.3); the same independence approximation includes zero for each pair. Further down that view some approximate intervals exclude zero, subject to the same covariance limitation. We therefore cannot determine from those marginal intervals which leading entries differ statistically. A paired test on shared items [5], with appropriate treatment of multiple comparisons, is needed.

5.2 What this means for readers

Read the composite together with its uncertainty and subtask results. A small observed gap is inconclusive without an appropriate comparison; it is not automatically noise or evidence of equivalence. For models scored on shared items, report a paired interval on the difference. Read toxicity separately and validate the annotation policy against the intended use.

Practitioners should optimise behaviour on development data that reflects their own policy, use the benchmark as an external check, and report per-subtask results with intervals.

5.3 Applicability to other languages

The paper’s own results are for one language, Malay; the single cross-language probe in Section 5.4 is a transfer check under its own serving configuration, not a multi-language safety evaluation, and no ordering between languages is claimed. What transfers is the structure of the problem. Item leverage is arithmetic: under a balanced-accuracy scorer, one changed prediction moves a subtask by an amount set by the size of the item’s class, so any language whose culturally grounded safety subtasks hold tens of items has the same exposure, and the expression in Section 5.1 can be applied to its subtask sizes directly. The benchmark family we assessed spans several Southeast Asian languages [1, 7], and comparable culturally grounded safety sets now exist for Indonesian languages [9].

Three reporting practices, items 1 to 3 of the checklist in Section 5.5, carry over unchanged: state which scorer and task set produced a score, declare the reasoning mode and request concurrency, and report an interval beside every score. What does not transfer is magnitude. How large each effect is depends on a language’s subtask sizes and annotation convention, and has to be measured language by language. For the same reason, our reading of the toxicity labels (Section 4.1) is a hypothesis to test in other languages, not a result about them.

5.4 A cross-language probe: Indonesian

The leverage arithmetic in Section 5.1 is a property of the scorer, not of Malay, so it should apply to any language whose culturally grounded safety subtasks are small. To test that expectation rather than assert it, we ran the same harness on a second regional language, Indonesian, which the model was not tuned for. No Indonesian-targeted text entered the fine-tuning mix, so the figures below are a cross-language transfer probe, not a within-language result, and no Indonesian-specific tuning or evaluation-driven development was performed.

Both checkpoints of one lineage were measured under a single shared configuration: Oaica 35B-A3B Malay Safety v1.0 260923 and the open-weight base Qwen3.6-35B-A3B that it is fine-tuned from. The harness was SEA-HELM v1.3.0 on 20 of the 21 Indonesian tasks (the gated syntax-criteria task could not be obtained for either model), served as Q8_0 quantised weights under llama.cpp rather than the vLLM bf16 stack of the Malay leaderboard, with sampling on and reasoning off, over eight runs each. The serving stack and the run count therefore differ from the Malay runs elsewhere in this paper, and these scores are not comparable entry by entry with them.

Table 3: Indonesian SEA-HELM competency scores for two checkpoints of one lineage, under one shared configuration (SEA-HELM v1.3.0, 20 of 21 tasks, Q8_0 under llama.cpp, sampling on, reasoning off): mean and standard deviation (SD) over eight runs. The composite is the official safety score (Section 2); the overall figure is the equal-weight mean of the nine competencies above it. The two columns are not ordered (Section 5.2).

Two observations echo the Malay findings. First, the interval widths follow the item-leverage pattern of Section 5.1: the multi-turn and NLG subtasks are tightly determined (SD near one point), while linguistic diagnostics, knowledge and the cultural subtask carry SDs of five to about twenty-three points, the wide spread being what small item classes produce. Second, the composite again sits in the middle of its range, with the toxicity subtask low on both checkpoints, consistent with the hypothesis of Section 4.1 and with its cause left unresolved here as there.

Two of the eight base-model runs are anomalous: they collapse together across every classification subtask (toxicity accuracy 0.50 against about 0.71 elsewhere), which inflates that column’s SD. Over the six consistent runs the overall mean is 70.48 (SD 3.32), close to the fine-tuned checkpoint’s 70.00 (SD 1.59). The clearest and most reproducible separation is in the opposite direction, on the judge-rated multi-turn task, and there the two intervals do not overlap in this configuration. We report that gap rather than claim it as an advantage: the probe was not designed as a comparison, the two checkpoints are a fine-tune and its own base rather than independent entries, and no paired interval on the difference was computed (Section 5.1), so no ordering between the checkpoints is claimed. Both series are reported only to show that the measurement behaves in a second language as Section 5.1 predicts.

5.5 A language-agnostic reporting checklist

The findings above reduce to a short checklist for reporting any safety score, in any language. It describes what to publish beside a number, not how a model is built.

1. Scorer and task set. Name the scorer version, the formula and the task set that produced the score.
2. Inference settings. Declare the decoding settings, the reasoning mode and the request concurrency (Sections 3.1 and 3.2).
3. Interval. Put an interval beside every score, and a paired interval beside every comparison (Section 5.1).
4. Item leverage. State how many points one changed prediction moves each subtask and the composite (Section 5.1).
5. Label provenance. State how the labels were produced and what is known about annotator agreement (Section 4.1).
6. Run count. State how many runs a reported figure comes from and how that run was chosen.
7. Contamination statement. Report the checks performed, known benchmark exposure, adaptive test use and unresolved limitations. Distinguish direct training inclusion from test-informed development and unknown base-pretraining exposure.

The checklist is implemented as a small open-source tool, benchlint (github.com/sprapp-com/benchlint). It lints a results report for the first four points above, flagging a missing interval or sample size, ranking language with no test behind it, false precision and single-run claims, and it computes item leverage and interval estimates from item-level outcomes. It is language- and model-agnostic, depends only on the Python standard library, and is released under Apache-2.0.

6  Suitability and practical placement

Sections 3 to 5 concern what the benchmark can support. The two checkpoints, Oaica 35B-A3B Malay Safety v1.0 260923 (safety-tuned) and Oaica 35B-A3B Malay v1.0 260923 (general use), are sparse mixture-of-experts models built on Qwen3.6-35B-A3B, with about 35 billion parameters in total and about 3 billion active for each token. Tables 4 to 7 and 9 and 10 report point estimates; the 95% intervals for the checkpoints’ official composites are given in Section 3.3, the MalayMMLU sample intervals in Tables 9 and 10, and the public entries’ intervals with the leaderboard [2]. The recomputed Table 7 panel has its own intervals. Other rows are descriptive point estimates without reproduced intervals in this release. The point estimates should be read with those intervals in mind. This section places the two checkpoints on the wider Malay, English and Chinese benchmark set and lists possible supervised pilot uses and the validation still required. Published leaderboard rows and our historical proxy measurements use different instruments; their numerical gaps are not comparative effect estimates (Section 8).

6.1 Where the checkpoints stand

Table 4 places our historical measurements beside published leaderboard values as separate descriptive panels. Our smaller item sets, automated judge and cultural proxy differ from the official instruments. Their numerical gaps are not estimates of comparative capability, and neither interval overlap nor an interval excluding zero removes that mismatch. Within our own historical panel, the two checkpoints differ by about a third of a point on the mean; no paired interval for that difference is reported. Multi-turn dialogue, knowledge-intensive question answering and longer-context serving remain areas for further evaluation.

Table 4: Descriptive results from historical measurements and published leaderboard entries. The protocols and, for several competencies, instruments differ; numerical gaps are not comparative effect estimates. Our historical runs used reasoning off and smaller item sets. Published values were captured on 24 September 2026 [2]. The safety row uses the official composite on both sides. Cultural scores in our panel use a proxy task, not the leaderboard’s cultural subtasks. The SEA-LION entry is the Qwen-based v4.5 27B release listed on the leaderboard [2]; Table 10 names different, Gemma-based releases.

Table 5 describes the historical Malay measurements. On Malay knowledge multiple-choice, the checkpoints score 82.8 and 83.4 against 76.8 for the earlier-stage checkpoint they were built from. This is a separate instrument from the published MalayMMLU scores and the leaderboard’s Knowledge competency. Malay reading-comprehension scores are 88.3 and 89.9; Malay instruction following scores 78 to 79 on a 105-item set, with the highest retained variant at 85.7; English instruction following scores 93.3 to 94.3 on development checkpoints (Table 6). These are descriptive results on the listed tests, not broad capability or deployment claims.

The same lineage also includes Chinese measurements. Development checkpoints scored 93.3 for Chinese-to-Malay and 90.5 to 90.8 for Malay-to-Chinese on a 200-item automatic metric using our protocol; this does not establish parity with other translation evaluations. On Chinese knowledge multiple-choice, they scored 84.4 against 79.7 for the earlier-stage checkpoint (Table 6). English-to-Malay scores were 90.6 to 90.8 and Malay-to-English 91.4 to 92.3 on those checkpoints. The two released checkpoints have aggregate translation scores of 90.8 to 91.0.

Table 5: The wider Malay benchmark set for the two released checkpoints. Instruction following is scored on subsets of about 105 items; the toxicity row is balanced accuracy on the 1,000-item subtask, where the retained historical variants occupy the range reported in Section 4.1; all rows are 0 to 100 points except the toxicity row, which is 0 to 1.

Table 6: Translation and Chinese-language results, and English instruction following. Rows are measured on development checkpoints of the same lineage unless the row states otherwise (the aggregate row is for the two released checkpoints); translation is MetricX-style automatic scoring, 200 items per direction.

6.2 Serving characteristics

The historical serving measurements used one NVIDIA A100-80GB, vLLM 0.30.0, bfloat16 or 4-bit weights, a 2,864-token prompt and up to 200 output tokens. With one concurrent stream and prefix-cache hits excluded, effective prefill throughput (prompt tokens divided by time to first content token) was 14,300 to 15,600 tokens/s, with a first token after about 0.2 s; single-stream decoding was 152 to 158 tokens/s. At eight concurrent streams, aggregate decoding was 347 to 544 tokens/s on the long prompt and 572 to 917 on an 89-token prompt. These workload-specific observations are not latency guarantees, and they are historical measurements on the previous serving stack. Our current .oqm serving engine targets Blackwell-class hardware (sm_120, RTX PRO 6000), with FP8-only kernels and split-KV and shared-quantisation paths that replace the A100 baseline; the A100 figures are retained only as a historical reference point. The 4-bit format reduced weight memory by about 3.2 times; historical MalayMMLU point estimates changed from 83.2 to 82.6 and 83.0 to 82.4 on the serving sample. No equivalence test is reported. Weight fit does not include every context-length or concurrency requirement: runtime and KV-cache memory must also be budgeted. Fully local operation requires a deployment configured without external inference, telemetry or other outbound integrations. The same weights are not tied to one accelerator class: our .oqm serving engine can pin the model’s mixture-of-experts layers in host CPU RAM, filling the available GPU memory automatically and offloading the remainder, so the family also runs on laptop- and desktop-class GPUs at reduced context and scales to datacenter servers. The weights ship in our own .oqm quantised container, and higher-throughput engine configurations are offered for API and cloud inference; their throughput figures are outside these measurements.

6.3 Safety for operators

Reasoning off is a practical configuration to evaluate for short classification outputs and latency, but the paired reasoning-mode intervals in Section 3.1 include zero, so this study does not establish a safety advantage. Historical served safety point estimates were 48.9 and 44.9 with 4-bit weights, compared with 47.1 and 44.9 in bfloat16. These single-run results are not an equivalence or non-inferiority test. Historical benign over-refusal rates ranged from 1.7 to 8.5 per cent across variants; the range alone does not establish harmful-content recall, subgroup performance or deployment suitability.

For the archived runs, the reported safety composites are 47.2 and 45.6. Those runs differ from the recomputed panel in Table 7 and should not be merged into a single matched comparison. Holding a safeguard score S fixed, a composite above 50 requires toxicity accuracy above 1 − S/200 on this balanced binary task. This is a conditional arithmetic relationship, not evidence of a model or benchmark ceiling. Both components can change with a new model, protocol or sample.

6.4 Reference points on other open models

Table 7 reports the saved runs recomputed from verified item files. The unsupported 41.8 stock, 42.2 peer and 45.3 specialist figures could not be tied to matching item outcomes in the available packet and are not retained as alternate estimates. The general-model run here is also distinct from the historical 45.6 result in Section 3.3. Product names follow the local run records; immutable checkpoint-to-product mapping has not yet been verified.

Table 7: Local safety runs, recomputed from verified item files with the SEA-HELM balanced-accuracy formula (toxicity plus three cultural subtasks). Greedy generation, reasoning off, batch size 8; 1,572 scored items per run. Intervals use 20,000 conditional item-bootstrap replicates, retaining the shared prompt/response structure, seed 20261004. They exclude training, generation-rerun and model-selection uncertainty. Model/dataset revisions and exact runtime provenance remain incomplete. These are not official leaderboard submissions. The evidence document identifies every run by content hash and provides recomputation code.

No paired model-difference interval is reported, and no comparative safety claim is made. The shared formula does not by itself make generation protocols or checkpoint histories equivalent. The historical smaller-model references in the abridged edition (36.9 for Qwen3.5-4B and 21.9 for a recorded Nanbeige 3B base release) come from earlier records and are outside this recomputed panel. Full-set first-token MalayMMLU reference measurements appear in Tables 9 and 10; they cannot estimate uplift over our historical generation-based sample scores.

6.5 Suitability by use

Table 8 lists possible pilot applications. These are the company’s proposed uses, not validated deployment verdicts. Each requires evaluation on the customer’s policy, workload and affected communities, including harmful-content misses as well as false alarms. Memory fit and throughput establish technical feasibility only.

Table 8: Candidate uses and the validation required before deployment.

6.6 External score context

Published MalayMMLU [22] scores use different evaluation protocols and were not reproduced in this study. We therefore do not place vendor-reported results beside our measurements or infer relative performance from them. A meaningful comparison would require the same question set, prompt, answer extraction, model version and scoring procedure.

6.7 The base model, and what the stock comparison can and cannot show

Oaica 35B-A3B Malay is fine-tuned from Qwen3.6-35B-A3B, the open-weight base, and the stock reference run measures that base directly. Which checkpoint produced each historical result, and the base’s exact revision tag, remain to be mapped. Under one full-set first-token protocol and template, recorded outside this release and not tabulated here, the stock model scores 80.7 on MalayMMLU, the smaller Qwen3.5-4B scores 71.1 and Nanbeige4.2-3B scores 51.5. The sizes differ, so this is not a ranking of the families; it shows how far apart stock models of these sizes score under one protocol, under that evaluation method. These stock figures were measured after our checkpoints were built, so they are retrospective reference measurements, not a record of the original model-selection decision.

The historical Oaica figures of 83.4 and 82.8 use generation with reasoning on and a 500-question sample; stock 80.7 uses a full-set first-token protocol. No matched MalayMMLU uplift estimate follows from those numbers. Against an earlier-stage checkpoint, the historical record reports changes from 76.8 to 82.8 and 83.4 on Malay knowledge, and from 79.7 to 84.4 on Chinese knowledge on a development checkpoint. These are protocol-specific point differences without paired intervals here. The recomputed safety runs in Table 7 also establish no comparative deployment-safety claim or causal effect of our training recipe.

6.8 An experimental 2-billion-parameter model: protocol sensitivity of a MalayMMLU comparison

Question. Does an experimental Malay fine-tune of Qwen3.5-2B, which we call Oaica 2B Malay, score higher on MalayMMLU than its stock base, and does the answer depend on the evaluation protocol? The model is not a release candidate, and we report no safety composite for it.

Setup. Both models ran zero-shot on one RTX 3090 with one inference engine (vLLM) in bfloat16, with greedy decoding and the same Malay instruction prompt in every protocol. The Qwen3.5-2B chat template, shared by both models, emits an empty reasoning block unless reasoning is explicitly enabled, unlike the larger Qwen models we evaluate, so every chat-template run here is reasoning off; we did not run either model with reasoning on. We used three protocols:

• Generation (G). Chat template; the model generates up to 1,536 tokens and the answer letter is extracted from the output (a leading letter, else the letter after “Jawapan:”, else the first standalone letter). Scored on a 500-question stratified sample, 100 from each of the five categories, with a fixed seed.
• First token, plain prompt (F-plain). No chat template; the answer is the valid option letter with the highest score at the first position. All 24,213 questions.
• First token, chat template (F-chat). The same scoring with the chat template. All 24,213 questions.

Table 9: MalayMMLU for the stock Qwen3.5-2B and Oaica 2B Malay under three protocols (points, 0 to 100). Each G figure has a 95% interval of up to about ±4.4 points and each first-token figure about ±0.6 points; we have not computed the paired interval on a difference. F-chat uses first-token answer-letter selection with a no-think chat template; G is our own protocol and is not comparable with Table 5. Differences are computed from unrounded values.

Findings.

1. The sign of the comparison depends on the protocol. The tuned model scores higher under G and F-plain and lower under F-chat. The two first-token differences are the best resolved, each on 24,213 items per model; the G difference, on 500 items, is the least resolved of the three.

2. The protocol alone moves a single model by more than any difference between the two models in Table 9. The stock model ranges over 17.0 points across the three protocols (ranges are computed from unrounded values), between the two full-set protocols and therefore well resolved, against between-model differences of 6.4, 9.6 and 15.3 points in absolute value. The tuned model ranges over 18.3 points, which depends on the 500-question G figure and is indicative; between the two full-set protocols it moves by 7.9 points.

3. Extraction failures do not explain the generation result. No letter could be extracted from 0.6 per cent of the stock model’s outputs (three items; scoring them all correct would raise the stock figure to 56.6) and from none of the tuned model’s.

Across other models. We applied the same three protocols to five further public models on a second host of the same GPU type (Table 10; the Qwen3.5-2B and Oaica 2B rows repeat Table 9).

Table 10: MalayMMLU under the three protocols of Table 9 for seven models, ordered by stored parameter count; the order is not a ranking. G is generation with answer extraction on the 500-question sample, with the chat template’s default setting and a 1,536-token budget (up to about ±4.4 points per figure); F-plain and F-chat are first-token scores on all 24,213 questions (about ±0.6 points), and F-chat is reasoning off where the template allows it. F-plain omits the chat template, so its figures are not the protocol under which these instruction-tuned models are meant to be used and should not be quoted as a model’s MalayMMLU score. ¹ For Qwen3-1.7B, G is reasoning on (its template default) and F-chat is reasoning off (53.5; 44.8 under its default template), so its range includes that switch; with reasoning off in generation too (57.2, 64-token budget) the range is 10.8. ² The 4.1 release of Nanbeige is the release tabulated here; the 4.2 post-trained release appears only in the historical first-token reference figures quoted in Section 6.7. It begins every answer with a reasoning block of its own accord and its template has no switch to suppress it, so G is reasoning on, and its F-chat figure is the letter forced at the first position, where the model would otherwise open that block. Figures are our own measurements of public models and may differ from the developers’ reports. Range is the highest minus the lowest of the three columns, from unrounded values. Extraction failed on at most 1.6 per cent of any model’s G outputs. Points, 0 to 100.

4. The sensitivity is not specific to our model. The range across the three protocols runs from 1.6 to 26.5 points. Five of the seven models move by more than 8 points, and four by at least 16.9 points.

5. The model with the highest figure depends on the protocol. The top figure under G and F-chat belongs to Gemma-SEA-LION-v4.5-E2B-IT and under F-plain to Gemma-SEA-LION-v4-4B-VL, where the E2B figure is among the two lowest (41.9, with the stock Qwen3.5-2B at 42.4; no paired test of that gap is reported); we draw no ranking from this.

6. G and F-chat agree within G’s interval for the two SEA-LION models, Qwen3.5-0.8B and the stock Qwen3.5-2B (gaps of 0.2 to 3.3 points), whereas the clear gaps are those of Oaica 2B Malay (18.3), Qwen3-1.7B (10.1, with G reasoning on) and Nanbeige4.1-3B (7.1). We did not investigate why.

Hypothesis, not tested. Fine-tuning may have changed how the model begins an answer. A first-token protocol under the chat template would penalise that, whereas a plain prompt that ends in the answer cue would not, which fits the direction of F-plain and F-chat. In the one stored sample output the tuned model opens with “Jawapan:” before the letter and the stock model with the letter, which is consistent with this reading but is a single example, not a test. A direct test, comparing the model’s unconstrained first token with its letter-restricted choice, has not been run.

What we conclude. We claim no improvement over the stock base: the evidence on this model is protocol-dependent, and one model pair and one sample cannot settle which protocol is closest to its real ability. The tuned-versus-earlier-stage gains of Section 6.7 were measured under one protocol only and should be read as specific to it. For future comparisons of a tuned model with its base we will report at least two protocols and treat a comparison whose sign changes between them as unresolved.

7  Related work

This work sits where five lines of research meet: statistics for language-model evaluation, culturally grounded safety for Southeast Asian languages, annotator disagreement, the safety of reasoning models, and the validity of benchmarks and leaderboards; we close with the Malaysian governance context.

Statistics for evaluations. Miller argues that evaluation results should carry error bars and be compared with paired analyses [3]. Bowyer et al. show that intervals based on the central limit theorem are unreliable on benchmarks with fewer than a few hundred items [4], and Wein et al. estimate significance for leaderboard differences from a single test set [5]. We apply these ideas to regional safety evaluation.

Regional and culturally grounded safety. SEA-HELM [1] evaluates Southeast Asian languages holistically; its leaderboard [2] has since added Malay, safeguard subtasks drawn from SEA-SafeguardBench [7], and bootstrap intervals over eight runs per model. Recent work builds culturally grounded safeguards and safety benchmarks for the region, including SEA-Guard [6], SEA-SafeguardBench [7], SEALGuard [8] and IndoSafety [9], and a joint testing exercise by several AI safety institutes covered Malay [10]. Our assessment complements this work by quantifying what the Malay safety track can resolve.

Annotator disagreement. Safety evaluation increasingly treats disagreement among annotators as information about genuine ambiguity rather than as noise [11], and recent modelling work separates such systematic disagreement from annotator error [12]. Automated safety annotations have also been compared with human ones, with imperfect agreement [13]. Our toxicity finding is consistent with this work.

Reasoning and safety. Studies of reasoning models examine the monitorability of reasoning traces and the safety of long chains of thought [14, 15]. We add an evaluation-side observation: switching reasoning on lowered the safety composite of both checkpoints (by 1.5 and 2.3 points, both historical paired intervals including zero), so the mode must be declared with every score.

Benchmark and leaderboard validity. Recent work asks what benchmarks measure and what their rankings support: Desai et al. examine validity across fifty-six AI benchmarks [16], and Yang and Chen study leaderboard claims under hidden model selection [17]. On SWE-bench Verified, Liu et al. report that paired tests do not distinguish any of the 29 adjacent pairs among the top thirty entries; on the larger Test split, some adjacent pairs are distinguishable [18]. Our assessment asks a related question of a regional-language safety track: which published differences exceed the benchmark’s own measurement uncertainty.

Governance context. Evaluation records can inform organisational governance and procurement. This study does not establish regulatory compliance or government endorsement. References [19, 20] provide the dated Malaysian policy context; legislative forecasts are not a basis for our product claims.

8  Limitations

The evidence comes from one primary language with a single cross-language probe (Section 5.4), one benchmark family and, for the safety findings, one base model, under a compute budget that limited replication. Other model families appear only as published leaderboard entries (the leaderboard-wide toxicity and interval results of Sections 4.1 and 5.1, Figure 4, and the two entries beside ours in Table 4), as instrument reference points on the official scorer and on MalayMMLU (Sections 6.4, 6.6 and 6.7; Tables 7, 9 and 10) and in the non-safety protocol-sensitivity measurements of Section 6.8. Several findings therefore may not generalise.

• Scope. Our toxicity observations concern one benchmark’s annotation convention. Other conventions, languages, harm taxonomies, model families and benchmarks may behave differently.
• Design scope. The checkpoints reported here were built to carry, in the model itself, the capabilities we judged least suited to being added afterwards: Malay knowledge, cultural alignment and safety. Task-specific abilities that an adapter or a retrieval layer can supply were given lower priority in this release. This is a bounded statement about the scope of this release, not a claim that such abilities are unimportant or cheap to add: adapters and retrieval layers carry their own data, latency and failure modes. In hindsight, a future release would rebalance the mix toward more task-specific breadth.
• Small subtasks. Two safety subtasks hold only 71 items each. The precision of a comparison depends on the paired changes and class supports; this study establishes no universal minimum detectable improvement. Repeating generation does not add independent benchmark items.
• Label audit. The causes of low toxicity performance remain unresolved. A formal audit with several native annotators is needed before it can support a claim about the benchmark, and even that audit would not exclude a shared bias between the evaluated models and the automated annotators (Section 4.1).
• Replication. Our estimates of run-to-run variance come from four runs of one recipe, so many comparisons conclude “not distinguishable” rather than “no effect”.
• Correlated runs. Our variants share a base model and the same evaluation items, so their spread describes the descendants of one base model, not independent draws from all possible recipes.
• Comparability. The public toxicity figures we cite are the leaderboard’s published scores, produced by its own runner; ours come from our reproduction of the protocol, which is not that runner and does not reproduce every published entry. Our composites come from one greedy run, whereas published entries average eight runs at each model’s default settings.
• Automated judges. The seven-competency comparison in Section 3.3 relied partly on automated judges, whose rating scale differs from the official one, and on smaller item sets.

9  Ethics and responsible release

Safety work handles harmful text and references to real communities, so the discipline constrains what is stored, published and released.

• Harmful content. This publication and its supplement contain no harmful item text, benchmark prompts or generated responses.
• Benchmark exposure. The intended policy excludes evaluation items and prediction dumps from training. The development record includes repeated benchmark evaluation and test-tuned diagnostics, and unresolved limitations in contamination screening. A complete audit of the final training artifacts has not been established for this release, so we do not claim verified absence of contamination. Base-pretraining exposure is unknown. Test-tuned analyses are exploratory; fresh held-out evaluation is needed for confirmatory claims.
• Mention is not attack. Evaluation should distinguish mentioning a group from attacking it. A model that flags every mention of a protected group as hateful would silence the communities it is meant to protect.
• Dual use. A classifier tuned to local norms can be used for over-broad censorship, and strictness trades off across subtasks. Operators should choose operating points deliberately and keep human review for high-stakes moderation decisions.
• What is withheld. Training methods, data and model internals are omitted for commercial and safety reasons. This limits independent reconstruction of the models and causal claims about training. Text-free evaluation counts and a score-recomputation script are supplied separately; reproducing generation also requires the missing model and runtime identifiers described in the evidence document.

10  Conclusion

In this case study, some observed safety-score differences induced by evaluation settings were comparable to differences among closely related models. In the measurements reported here, aggregation moved one checkpoint by 5.3 points, request concurrency shifted the three cultural safeguard subtasks of one model by an amount worth about 1.8 composite points, and reasoning mode lowered the composite by 1.5 and 2.3 points, with paired intervals that include zero. By comparison, 12 training variants of one lineage spanned 3.3 points and differed pairwise by 1.2 points on average.

Toxicity performance remained low in the evaluated historical runs, but these results do not distinguish model limitations, annotation-policy mismatch, missing context or label error. The published intervals of leading entries overlap; without paired item-level outcomes we cannot determine which differences are statistically distinguishable. This case study supports reporting scorer definitions, inference settings, per-class errors and uncertainty. It does not establish comparative deployment safety, a universal benchmark ceiling or defects in the labels.

Acknowledgements

Early-stage Malay evaluation in this work used GlossoBench [21] (see Competing interests).

Competing interests

The author owns Oaica, which builds Malay language models, offers them for on-premises deployment by arrangement, and offers evaluation consulting. The study includes the company’s checkpoints, training variants, archived runs and serving measurements; Section 6 is the company’s own assessment of its products, including candidate pilot uses and experimental-model measurements. Oaica also releases GlossoBench [21], an open evaluation harness used for early-stage Malay evaluation in this work. It describes SEA-HELM as its closest peer and is published under the GitHub account sprapp-com of Sprapp, the group of which Oaica is a part. The assessment of SEA-HELM in Sections 4 and 5 should be read with these interests in mind.

Correspondence

Oaica is a safety-focused AI company. It builds Malay language models and works with teams that need regional-language safety evaluation they can defend: audits of how a benchmark score was produced and how much of it the benchmark can support, reliability reviews of safety leaderboards, and on-premises Malay moderation consultation. De-identified evaluation tables, with their intervals, and item-leverage worksheets behind this paper are available on request. This paper: https://research.oaica.com/malay-llm-safety-score/. A regional-language edition without the product sections is at https://research.oaica.com/regional-language-llm-safety-malay-indonesian/. Enquiries: [email protected] (general enquiries: [email protected]).

References

• [1] Y. Susanto et al. SEA-HELM: Southeast Asian Holistic Evaluation of Language Models. Findings of ACL 2025. aclanthology.org/2025.findings-acl.636
• [2] SEA-HELM Leaderboard: Malay (MS) results (page data dated 18 September 2026), evaluation details, score calculations, task set and safety task configurations, command-line options and serving code. Accessed 30 September 2026. leaderboard.sea-lion.ai/detailed/MS, leaderboard.sea-lion.ai/eval-details, github.com/aisingapore/SEA-HELM/blob/main/docs/score_calculations.md, github.com/aisingapore/SEA-HELM/blob/main/seahelm_tasks/task_config.yaml, github.com/aisingapore/SEA-HELM/tree/main/seahelm_tasks/safety, github.com/aisingapore/SEA-HELM/blob/main/docs/cli.md, github.com/aisingapore/SEA-HELM/blob/main/src/serving/offline/vllm_serving.py
• [3] E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024. arxiv.org/abs/2411.00640
• [4] S. Bowyer, L. Aitchison, D. R. Ivanova. Position: Don’t Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints. Proceedings of ICML 2025 (PMLR 267). proceedings.mlr.press/v267/bowyer25a.html
• [5] S. Wein, C. M. Homan, L. Aroyo, C. Welty. Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance. Findings of ACL 2023. aclanthology.org/2023.findings-acl.196
• [6] P. Tasawong, J. G. Ngui, A. F. Aji, T. Cohn, P. Limkonchotiwat. SEA-Guard: Culturally Grounded Multilingual Safeguard for Southeast Asia. Findings of ACL 2026. aclanthology.org/2026.findings-acl.141
• [7] P. Tasawong, J. G. Ngui, A. F. Aji, T. Cohn, P. Limkonchotiwat. SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages. Findings of ACL 2026. aclanthology.org/2026.findings-acl.194
• [8] W. Shan, M. Fu, R. Yang, C. Tantithamthavorn. SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for AI-Powered Software. AIware 2025 (IEEE). Preprint arXiv:2507.08898. arxiv.org/abs/2507.08898
• [9] M. F. Azmi, M. D. Al Kautsar, A. F. Wicaksono, F. Koto. IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages. Proceedings of EMNLP 2025. aclanthology.org/2025.emnlp-main.465
• [10] AI safety institutes and offices of Singapore, Japan, Australia, Canada, the European Union, France, Kenya, South Korea and the United Kingdom. Multilingual Joint Testing Exercise: Improving Methodologies for LLM Evaluations Across Global Languages. Evaluation report, June 2025. sgaisi.sg/wp-api/wp-content/uploads/2025/06/Improving-Methodologies-for-LLM-Evaluations-Across-Global-Languages-Evaluation-Report-1.pdf
• [11] D. Wang et al. A Case for Moving Beyond “Gold Data” in AI Safety Evaluation. CHI 2024 HEAL workshop. heal-workshop.github.io/chi2024_papers/12_a_case_for_moving_beyond_gold_.pdf
• [12] J. Ivey, S. Gauch, D. Jurgens. NUTMEG: Separating Signal From Noise in Annotator Disagreement. Proceedings of EMNLP 2025. aclanthology.org/2025.emnlp-main.144
• [13] R. Movva, P. W. Koh, E. Pierson. Annotation alignment: Comparing LLM and human annotations of conversational safety. Proceedings of EMNLP 2024. aclanthology.org/2024.emnlp-main.511
• [14] T. Korbak et al. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv:2507.11473, 2025. arxiv.org/abs/2507.11473
• [15] F. Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. Findings of ACL 2025. aclanthology.org/2025.findings-acl.1197
• [16] M. Desai, S. T. Truong, H. Wallach, A. Chouldechova, A. F. Cooper, J. Garcia-Gathright, D. E. Ho, A. Z. Jacobs, S. Koyejo, N. Pangakis, A. Wang. What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks. arXiv:2609.08812, 2026. arxiv.org/abs/2609.08812
• [17] C. Yang, J. Chen. How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection? arXiv:2609.28177, 2026. arxiv.org/abs/2609.28177
• [18] F. Liu, Y. Liu, R. Sun, L. Luo, S. Guo. Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead. arXiv:2609.17394, 2026. arxiv.org/abs/2609.17394
• [19] Ministry of Science, Technology and Innovation (MOSTI), Malaysia. The National Guidelines on AI Governance & Ethics. 2024. mastic.mosti.gov.my/storage/2024/09/THE-NATIONAL-GUIDELINES-ON-AI-GOVERNANCE-ETHICS.pdf
• [20] Bernama. AI Governance Bill Expected To Be Tabled In Parliament By Q1 2027 At The Latest - Gobind. 10 September 2026. bernama.com/en/news.php?id=2605398
• [21] The GlossoBench Authors. GlossoBench: a public, versioned, judge-agnostic benchmark for any language. Release v1.5.0, 31 July 2026. Apache-2.0. github.com/sprapp-com/glossobench
• [22] S. C. Poh, S. J. Yang, J. M. L. Tan, L. L. T. Y. Chieng, J. X. Tan, Z. Yu, C. M. Foong, C. S. Chan. MalayMMLU: A Multitask Benchmark for the Low-Resource Malay Language. Findings of EMNLP 2024. aclanthology.org/2024.findings-emnlp.36