Supported Resources

Note: This page is auto-generated during the CI/CD pipeline. Last updated: 2026-06-19 10:04:07 UTC

The following resources are currently supported by KARMA:

Datasets

Currently supported datasets (40 total):

Dataset	Task Type	Metrics	Required Args	Processors	Split
ChuGyouk/MedXpertQA	mcqa	exact_match	—	—	test
GBaker/MedQA-USMLE-4-options	mcqa	exact_match	—	—	test
HFXM/CARES-18K	safety_classification	accuracy	—	—	train
Showing-KCL/AMQA	mcqa_bias	exact_match, bias_consistency, bias_disparity	—	—	train
Showing-KCL/AMQA:gender	mcqa_bias	exact_match, bias_consistency, bias_disparity	—	—	train
Showing-KCL/AMQA:race	mcqa_bias	exact_match, bias_consistency, bias_disparity	—	—	train
Showing-KCL/AMQA:ses	mcqa_bias	exact_match, bias_consistency, bias_disparity	—	—	train
Tonic/Health-Bench-Eval-OSS-2025-07	rubric_evaluation	rubric_evaluation	—	—	oss_eval
TsinghuaC3I/MedXpertQA	mcqa	exact_match	—	—	test
ai4bharat/IN22-Conv	translation	bleu	source_language, target_language	devnagari_transliterator	test
ai4bharat/IndicVoices	transcription	wer, cer, asr_semantic_metric	language	multilingual_text_processor	valid
ekacare/Eka-IndicMTEB	retrieval	ndcg@k, recall@k, map@k, precision@k	—	—
ekacare/Eka_NFI_MCQA	mcqa	exact_match	—	—	test
ekacare/MedMCQA-Indic	mcqa	exact_match	subset	—	test
ekacare/clinical_note_generation_dataset	text_to_json	json_rubric_evaluation	—	—	test
ekacare/denoising-impact-evaluation-dataset	transcription	wer, cer, semantic_wer_metric	—	—	test
ekacare/eka-medical-asr-evaluation-dataset	transcription	wer, cer	language	—	test
ekacare/ekacare_medical_history_summarisation	rubric_evaluation	rubric_evaluation	—	—	test
ekacare/indian_drug_mcqa	mcqa	exact_match	—	—	test
ekacare/indian_protocols_based_clinical_QnA	conversation_rubric_evaluation	answer_rubric_evaluation, query_rubric_evaluation	—	—	test
ekacare/medical_calculator_eval	calculator	numeric_tolerance	—	—	test
ekacare/medical_records_parsing_validation_set	image_to_json	json_rubric_evaluation	—	—	test
ekacare/mmlu-medical-mcqs-evaluation-dataset	mcqa	exact_match	subset	—	test
ekacare/vistaar_small_asr_eval	transcription	wer, cer, semantic_wer_metric	language	general_text_processor	test
flaviagiammarino/vqa-rad	vqa	exact_match, tokenised_f1	—	—	test
intronhealth/afrimedqa_v2	open_ended_qa	tokenised_f1, bleu, rouge	—	—	train
intronhealth/afrimedqa_v2/mcq	mcqa	exact_match, geographic_accuracy, demographic_accuracy	—	—	train
intronhealth/afrimedqa_v2:harm	harm_detection	harm_detection_accuracy	—	—	train
mdwiratathya/SLAKE-vqa-english	vqa	exact_match, tokenised_f1	—	—	test
microsoft/PatientSafetyBench	safety_classification	accuracy	—	—	train
nfi_mcqa	mcqa	exact_match	—	—	test
openlifescienceai/medmcqa	mcqa	exact_match	—	—	validation
openlifescienceai/medqa	mcqa	exact_match	—	—	test
openlifescienceai/mmlu_anatomy	mcqa	exact_match	—	—	test
openlifescienceai/mmlu_clinical_knowledge	mcqa	exact_match	—	—	test
openlifescienceai/mmlu_college_biology	mcqa	exact_match	—	—	test
openlifescienceai/mmlu_college_medicine	mcqa	exact_match	—	—	test
openlifescienceai/mmlu_professional_medicine	mcqa	exact_match	—	—	test
openlifescienceai/pubmedqa	mcqa	exact_match	—	—	test
zou-lab/MedCaseReasoning	open_ended_qa	medcasereasoning_diagnostic_accuracy, medcasereasoning_reasoning_recall	—	—	test

Recreate this through

karma list datasets

Models

Currently supported models (44 total):

Model Name
Qwen/Qwen3-0.6B
Qwen/Qwen3-1.7B
aws-bedrock-knowledge-retriever
aws-transcribe
docassistchat/default
ekacare/EmbeddingGemma-finetuned
ekacare/IndicBertV2-finetuned
ekacare/SapBERT-finetuned
ekacare/parrotlet-e
ekacare/parrotlet-v-lite-4b
gemini-2.0-flash
gemini-2.5-flash
gemini-2.5-flash-lite
gemini-2.5-flash-text
gemini-2.5-pro
gemini-2.5-pro-text
gemini-3-flash-preview
gemini-3.1-flash-lite-preview
gemini-3.1-pro-preview
google/gemma-3-27b-it
google/gemma-3-4b-it
google/medgemma-1.5-4b-it
google/medgemma-27b-text-it
google/medgemma-4b-it
gpt-3.5-turbo
gpt-4.1
gpt-4.1-mini
gpt-4o
gpt-4o-mini
gpt-4o-transcribe
gpt-5
gpt-5-mini
gpt-5-nano
gpt-5.2
o3
saaras:v2
saaras:v3
sarvam-105b
sarvam-30b
us.anthropic.claude-3-5-sonnet-20240620-v1:0
us.anthropic.claude-3-5-sonnet-20241022-v2:0
us.anthropic.claude-sonnet-4-20250514-v1:0
us.anthropic.claude-sonnet-4-5-20250929-v1:0
us.anthropic.claude-sonnet-4-6

Recreate this through

karma list models

Metrics

Currently supported metrics (26 total):

Metric Name
accuracy
answer_rubric_evaluation
bias_consistency
bias_disparity
bleu
cer
demographic_accuracy
exact_match
f1
geographic_accuracy
harm_detection_accuracy
json_rubric_evaluation
map@k
medcasereasoning_diagnostic_accuracy
medcasereasoning_reasoning_recall
ndcg@k
numeric_tolerance
precision@k
query_rubric_evaluation
recall@k
rouge
rubric_evaluation
silence_accuracy
stratified_accuracy
tokenised_f1
wer

Recreate this through

karma list metrics

Quick Reference

Use the following commands to explore available resources:

# List all models
karma list models

# List all datasets
karma list datasets

# List all metrics
karma list metrics

# List all processors
karma list processors

# Get detailed information about a specific resource
karma info model "Qwen/Qwen3-0.6B"
karma info dataset "openlifescienceai/pubmedqa"

Adding New Resources

To add new models, datasets, or metrics to KARMA:

• See Adding Models
• See Adding Datasets
• See Metrics Overview