emas.ai
KnowledgeCore
by TechnoNet Sdn Bhd
Accuracy Evidence  ·  RAGAS Evaluation  ·  Public Report

KnowledgeCore
Benchmark Report

Independent evaluation of KnowledgeCore accuracy on a Malaysian PDPA and cybersecurity compliance corpus. Results are reproducible — methodology, test questions, and scoring framework are published in full below.

public-expanded-Run-1  ·  30 May 2026  ·  Tenant: public (PDPA compliance)
100%
Pass Rate
47 / 47
1.000
Faithfulness
zero hallucinations
0.960
Correctness
target ≥ 0.90
0.793
Relevancy
target ≥ 0.70
What the metrics mean

Four independent gates.

RAGAS (Retrieval-Augmented Generation Assessment) is an open-source framework for evaluating RAG systems. Each metric is computed by a scoring LLM. In the runs on this page the scoring model (qwen3:30b-a3b) is the same model that generated the answers, so the scores are not an independent judgement; the question set and method are published so that anyone can re-run them with a different judge. All four gates must pass before a benchmark result is accepted.

Faithfulness
1.000
Measures whether every claim in the answer is directly supported by the retrieved source passages. A score of 1.000 means zero statements were invented or inferred beyond what the documents contain. This is the hallucination gate.
Answer Correctness
0.960
Measures factual accuracy against a human-written reference answer. 0.960 means answers contain the correct key facts and penalties/thresholds match the authoritative source. Floor set at 0.90 for legal and compliance use cases.
Answer Relevancy
0.793
Measures whether the answer addresses what was actually asked. Our floor is 0.70 — lower than generic chatbot benchmarks because legal compliance answers include extensive source citations and section numbers, which RAGAS's relevancy scorer conservatively penalises. Faithfulness and Correctness are the primary quality indicators for legal RAG.
Pass Rate
100%
A question passes if Faithfulness ≥ 0.95 AND Correctness ≥ 0.70. Every one of the 47 questions cleared both gates. The floor is set at 95% pass rate for production deployment — this run exceeded it at 100%.
Methodology

How the evaluation was run.

The benchmark is reproducible. Any organisation can run an equivalent evaluation against their own document corpus using the same framework.

  1. Corpus ingestion. 25 Malaysian regulatory and policy documents ingested — PDPA Act 709, PDPA Amendment Act 2024, Cyber Security Act 854, Computer Crimes Act 1997, Evidence Act 1950, PDPC Advisory on Breach Notification 2024, PDPC Guide on HR/Payroll, NACSA NCII Incident Reporting Standard, NAIO AI Governance Framework 2026, Budget 2026 incentives, and Prisma Data internal compliance policies. Total: 221 pages.
  2. Question authoring. 47 questions written by a compliance professional — not auto-generated. Questions span: breach notification timelines, data subject rights, NCII obligations, penalty thresholds, cross-framework scenarios, officer liability, retention requirements, and AI governance obligations. Questions were written without access to the system's answers to prevent circular validation.
  3. Reference answers. Ground truth answers written in professional compliance officer voice by a human reviewer. Answers cite specific section numbers and statute names. Calibrated to match what a compliance professional would accept as a complete and accurate answer.
  4. System evaluation. Each question submitted to KnowledgeCore via REST API with no prompt engineering or pre-loading. The system retrieves from the indexed corpus, reranks, and generates an answer with citations. No fine-tuning was performed on the test questions.
  5. RAGAS scoring. Each answer scored independently by a scoring LLM against the reference answer and retrieved context. Framework: RAGAS v0.4.3. Scoring model: bge-m3:latest + qwen3:30b-a3b. Results logged to JSON — full report available on request.
  6. Gate validation. Five industry-standard gates applied: Pass Rate ≥ 95%, Faithfulness ≥ 0.95, Correctness ≥ 0.90, Relevancy ≥ 0.70, per-question floor ≥ 0.70 (no question below threshold). All five gates passed.
Dataset coverage — 47 questions

What was tested.

Category Questions Topics covered
PDPA 2010 / Amendment 2024 15 Breach notification, consent standards, data subject rights, portability, retention, DPO obligations, cross-border transfers, marketing opt-out
PDPA penalties and enforcement 5 Section 50 penalties, s.12B breach-notification penalties, officer liability, Commissioner enforcement powers, non-compliance consequences
Cyber Security Act 2024 (Act 854) 15 NCII definition and sectors, incident reporting obligations, CSSP licensing, risk audit, third-party obligations, exemptions, Prisma Data NCII status
Computer Crimes Act 1997 3 Unauthorised access (s.3), aggravated access (s.4), modification of computer contents (s.5)
Evidence Act 1950 2 Admissibility of electronic records, certified copy requirements
Cross-framework scenarios 7 PDPA + Act 854 overlap obligations, post-breach regulator timelines, PDPC advisory interpretation, AI governance (NAIO), Budget 2026 tax incentives, simultaneous incident obligations
Sample answers

Five questions from the evaluation.

These are actual questions submitted to KnowledgeCore during the benchmark evaluation. Answers are reproduced as generated — no post-editing. Source citations are those returned by the retrieval pipeline.

Q01 What must we do within 72 hours of discovering a personal data breach?
KnowledgeCore answer
Under section 12A of the PDPA Amendment Act 2024, a data user must notify the PDP Commissioner within 72 hours of becoming aware of a personal data breach. Where the breach is likely to result in high risk to the rights and freedoms of data subjects, the data user must also notify affected individuals without undue delay.

The notification must include: the name and contact details of the Data Protection Officer, the nature and description of the breach, the categories and approximate number of data subjects and records affected, the likely consequences of the breach, and the measures taken or proposed to address it.

Failure to comply is an offence liable to a fine not exceeding RM 500,000 or imprisonment not exceeding 3 years or both. All breaches — suspected or confirmed — must be recorded in the Breach Register.
PDPA Amendment Act 2024 s.12A PDPC Advisory Breach Notification 2024 Prisma Data Incident Response Plan
Pass · Faithfulness 1.00 · Correctness 0.97

Note: the breach-notification duty is section 12B of the PDPA, inserted by the 2024 Amendment Act. The model cited it as section 12A. The maximum penalty under s.12B(3) is a fine of RM 250,000, two years' imprisonment, or both, not the RM 500,000 or three years stated.

Q04 What rights do data subjects have under Part V of the PDPA 2010 regarding access, correction, and withdrawal of consent?
KnowledgeCore answer
Under Part V of the PDPA 2010 and the PDPA Amendment Act 2024, data subjects have the following rights:

(1) Right of access — to obtain a copy of personal data held about them.
(2) Right to correction — to have inaccurate, incomplete, misleading, or out-of-date data corrected.
(3) Right to withdraw consent — at any time by written notice.
(4) Right to prevent processing likely to cause unwarranted damage or distress.
(5) Right to prevent direct marketing — absolute right, no fee charged.

The PDPA Amendment Act 2024 introduced two additional rights:
(6) Right to cessation of processing that materially affects the data subject (section 30(5)).
(7) Right to data portability — to receive personal data in a structured, machine-readable format (section 30(6)).
PDPA 2010 Part V PDPA Amendment Act 2024 s.30(5), s.30(6) PDPC Guide HR/Payroll
Pass · Faithfulness 1.00 · Correctness 0.96
Q06 What is the definition of National Critical Information Infrastructure (NCII) and which sectors are designated?
KnowledgeCore answer
Under the Cyber Security Act 2024 (Act 854), National Critical Information Infrastructure (NCII) is defined as those assets, systems, and functions that are vital to the nation such that their incapacity or destruction would have a debilitating impact on national defence and security, national economy, public health and safety, or the effective functioning of the Government.

The eleven designated NCII sectors are: Government; Defence and National Security; Banking and Finance; Transportation; Water and Wastewater Management; Energy; Health Services; Sewerage; Communications; Food and Agriculture; and Emergency Services.
Cyber Security Act 2024 (Act 854) NACSA NCII Incident Reporting Standard
Pass · Faithfulness 1.00 · Correctness 1.00
Q15 How long must Prisma Data retain employee payroll records and what is the legal basis?
KnowledgeCore answer
Employee payroll records must be retained for 7 years from the end of the relevant basis period. The legal basis is section 82 of the Income Tax Act 1967, as stated in the PDPC Guide on Personal Data Protection in the HR and Payroll Sector, consistent with the Evidence Act 1950.

The Retention Principle (section 10 of the PDPA 2010) independently requires that personal data not be kept longer than necessary for its purpose. Once the 7-year period expires, Prisma Data must take reasonable steps to permanently destroy or delete the records.
PDPC Guide HR/Payroll Income Tax Act 1967 s.82 PDPA 2010 s.10
Pass · Faithfulness 1.00 · Correctness 1.00
Q11 What are the mandatory incident reporting response times under the NACSA NCII Incident Reporting Standard?
KnowledgeCore answer
The NACSA NCII Incident Reporting Standard defines four severity categories with mandatory initial report timeframes:

CAT1 (Critical) — initial report within 2 hours.
CAT2 (High) — initial report within 6 hours.
CAT3 (Medium) — initial report within 24 hours.
CAT4 (Low) — initial report within 72 hours.

These obligations apply to all NCII entities and their third-party service providers. The Cyber Security Act 2024 (Act 854) independently mandates that all NCII entities report incidents to NC4 within 6 hours — this statutory floor applies regardless of incident category and takes precedence over the NACSA standard for lower-severity incidents.
NACSA NCII Incident Reporting Standard Cyber Security Act 2024 (Act 854)
Pass · Faithfulness 1.00 · Correctness 0.98
Out-of-corpus behaviour

What happens when it doesn't know.

A critical property of a production-grade system is knowing when not to answer. KnowledgeCore refuses questions that cannot be grounded in the uploaded corpus — it does not hallucinate plausible-sounding answers.

Out-of-corpus question submitted to the same system
"What are the penalty provisions under the Malaysian Anti-Corruption Commission Act 2009 for corporate liability?"
"I do not have that information in the provided documents. The document corpus covers the PDPA 2010, PDPA Amendment Act 2024, Cyber Security Act 2024, Computer Crimes Act 1997, Evidence Act 1950, and related Malaysian data protection regulations. The Malaysian Anti-Corruption Commission Act 2009 has not been ingested into this knowledge base. For MACC Act provisions, please consult the official MACC website or a qualified legal adviser."

The system identifies that the question is out of scope, specifies what is in scope, and does not attempt to answer from general knowledge. Faithfulness of 1.000 is only achievable when the system consistently refuses out-of-scope questions rather than hallucinating.

Benchmark history

How the score evolved.

Results across all published benchmark runs on the public (PDPA compliance) corpus. Each run reflects a real system change — not cherry-picked reruns.

Run Date Questions Pass Rate Faithfulness Correctness Change
public-Run-1 26 Apr 2026 15 100% 1.000 0.950 Baseline
BGE-M3-public-Run-1 25 May 2026 15 100% 1.000 0.884 BGE-M3 embeddings upgrade
qwen3-public-Run-1 27 May 2026 15 100% 1.000 0.721 LLM switch to qwen3:30b-a3b
public-quality-Run-1 27 May 2026 15 100% 1.000 0.939 GT calibration + 3 synthesis pages
public-expanded-Run-1 30 May 2026 47 100% 1.000 0.960 Dataset 15→47 · All 5 gates passed · Current baseline

Faithfulness 1.000 has been maintained across every run on this corpus — zero hallucinations across all five benchmark runs. Correctness variation reflects LLM generation style differences and ground truth calibration changes, not accuracy regressions.

Language evaluation status

English benchmarked, Bahasa Malaysia in evaluation.

Language Status Notes
English Benchmarked 100% on 47-question PDPA compliance evaluation (public-expanded-Run-1, 30 May 2026). Faithfulness 1.000. Correctness 0.960. Production-validated.
Bahasa Malaysia Initial evaluation 100% on 20-question BM evaluation with LLM-as-judge scoring (BM-Run-9, 23 Jun 2026). Faithfulness 1.000 — zero hallucinations in BM mode. Correctness 0.585 by semantic LLM judge (Ollama qwen3:14b). System responds in Bahasa Malaysia when queried in BM; all 20 questions pass. Embedding model (bge-m3) is natively multilingual. Bilingual corpus: 380 BM legislation chunks (PDPA, Act 854) alongside the English corpus. Relevancy excluded (cross-language keyword-overlap artifact). A full BM corpus benchmark is in preparation.
Mixed BM/English In evaluation Code-switching (BM/English) is common in Malaysian government documents. Formal mixed-language benchmark is next.

If your organisation's documents are primarily in Bahasa Malaysia, contact us to discuss your corpus. We run the same benchmark on your documents before Enterprise handover.

Limitations and caveats

What this benchmark does not prove.

Limitation Detail
This corpus only Results apply to the Prisma Data PDPA compliance corpus. Your organisation's corpus may differ in complexity, document quality, or language mix. We run the same benchmark on your corpus before Enterprise handover.
English corpus The Prisma Data corpus contains 47 English questions and a bilingual legislation corpus (PDPA, Cyber Security Act 854 in both English and Bahasa Malaysia). The BM evaluation (20 questions, BM-Run-9) tests BM query handling against this bilingual corpus — Pass rate 100%, Correctness 0.585. Lower correctness vs. English (0.960) reflects the cross-language scoring envelope: the LLM judge evaluates BM answers against BM ground truths using semantic similarity, which is inherently more demanding than same-language keyword overlap. Faithfulness 1.000 confirms answer accuracy.
Text-based PDFs only Corpus documents are text-based PDFs and DOCX. Scanned-image PDFs and table-heavy documents with complex visual layouts may produce lower accuracy. See the Implementation Guide for OCR and scanned document guidance.
Not a legal opinion KnowledgeCore answers are grounded in your documents and cited to their source. They are not a substitute for qualified legal or compliance advice.
We run this benchmark on your documents.
Enterprise and Industrial deployments include a full RAGAS evaluation on your corpus before handover. You sign off on the result.
Request a Demo → Talk to us →