muis-assistant

Why it exists

The public guidance people need is spread across muis.gov.sg and AskGov's MUIS Q&A, and people ask about it in different languages. On a religious topic a confident wrong answer costs more than no answer, so the design goal was narrow: never answer without a source, and never give a personal ruling.

The index covers 1,080 pages in 3,784 chunks: 534 AskGov Q&A pages and about 546 muis.gov.sg pages across Haj, halal, zakat, wakaf, faraid, fatwa and irsyad, madrasahs, SCIS, asatizah, mosques and speeches.

How it works

question -> small talk? (templates) -> guard (personal ruling / override) -> tool router (prayer times)
         -> embed (bge-m3) -> pgvector cosine search, k=6 -> similarity gate
         -> model (JSON: answer + citations) -> validate citations
         -> answered | abstained | referred | refused
         -> handoff draft (on-topic abstentions) -> redacted query log

• Cite it or abstainEvery claim must cite a retrieved source. An answer without a valid citation becomes an abstention, and figures of 10 or more are checked against the cited text.
• Rules before the modelPersonal-ruling requests and instruction-override attempts are caught by deterministic guards and never reach retrieval or the model. Greetings, thanks and distress messages get templated replies in all three languages with no model call.
• Tools for factsPrayer times come straight from MUIS's official timetable on data.gov.sg. No model is involved and nothing is guessed.
• A way outWhen it cannot answer an on-topic question, it drafts a message to MUIS with NRIC, phone, email and account numbers redacted. The query log holds no user id, IP address or session.
• A content auditIts own logs feed a report of unanswered-topic clusters, pages that changed between crawls, possible conflicts between sources, and dated COVID-era pages. It flags these for people and never edits official content.
• Cost and abuse controlsPer-client rate limits, a daily cap on paid model calls, a password gate on the question endpoints, and caches for repeated questions.

Evaluation

Averaged over repeated runs of the current build (3,784 chunks, Claude Haiku 5.5, top 6 chunks retrieved).

The main set is 30 curated questions plus 120 derived from AskGov, in English, Malay and Arabic. The second set covers mosques, madrasahs, SCIS, speeches and about-us pages that were added later.

What these numbers are not. The questions were written or generated by me and are not independently reviewed. Generated cases take their gold page by construction and are easier than real questions. Gold is one page per case, so citation precision (about 0.5 to 0.6) understates answers that cite a sibling page that also answers. Model output varies, so every figure is an average, and there are no real users yet. A second person should review the gold labels before anyone treats these as benchmarks.

What the evals changed

The eval harness earned its keep by overturning things I expected to be improvements.

• Plain vector search beat hybrid searchOn the 150-case set, vector retrieval scored MRR 0.85 against 0.58 for hybrid and 0.17 for keyword-only. The keyword arm added noise, so vector is the default and hybrid stays behind a flag.
• The reranker made things worseIt lowered MRR (0.578 against 0.642) and added about 35 seconds per query, so it is off.
• Query rewriting lost recallOn the 30 curated cases, recall@5 fell from 0.82 to 0.65, and the looser version let off-topic questions through the relevance gate. It is built but off.
• A bigger model was not betterClaude Sonnet 5.5 abstained far more often than Haiku 5.5 (false-abstain 0.18 against 0.04 in that test), so the answer model stayed on Haiku. Turning hidden thinking off cut median latency from 4.1 s to 2.3 s with no meaningful quality change.
• Two bugs were inflating the early numbersOutput was capped at 900 tokens shared with hidden reasoning, so JSON got truncated and counted as a fake abstention. Separately, the model often put unescaped quotes inside its answer string, which made invalid JSON and fake abstentions on about 40% of some questions. A prompt rule and a repair step fixed both, and any remaining failure is now flagged and excluded from content-gap reports.
• 13% of the index was junkBreadcrumb fragments were outranking real content. Removing them cut false-abstains on the new sections from 0.085 to 0.059 at no extra cost.
• A harmless-looking prompt change raised false-abstains from 0.036 to 0.107Only the eval caught it.
• Long speeches crowded out official pagesA ranking prior of 0.90 on speeches raised recall@5 on two coverage sets (0.875 to 0.938 and 0.85 to 0.95). It was tuned on 36 unreviewed cases, so I treat it as provisional.
• Concurrent requests could crash the embedderFound as a segfault, fixed with locks, and covered by a 16-thread regression test.

Limits and next steps

• The guards are heuristic. About 2.5% of generated answerable questions are wrongly referred as personal rulings. I left that on purpose, because wrongly referring someone to MUIS is the safe error.
• Tone was judged by an LLM (the newer voice was preferred in 42 of 44 answers). A blind human read and a native Malay and Arabic review are still to do.
• The gold labels need a second reviewer, and a permanent public demo needs written permission from the content owners first.

Stack

Want to see it run? A live demo from my laptop can be arranged. Get in touch.