Caching

KARMA saves the model’s predictions locally to avoid redundant computations. This ensures that running multiple metrics or extending datasets is trivial.

How are items cached?

KARMA caches at a sample level for each evaluated model + configuration and dataset combinations. For example, if we run evalution on pubmedqa with the Qwen3-0.6B model, we will cache for each of the configurations. So if temperature is changed and evalution is run once again, then model will be invoked again. However, if only a new metric has been added along with exact_match on the dataset, then the cached model outputs are reused.

Caching is hugely beneficial for ASR related models as well since the metric computation also evolves over time. For example, if we run evaluation on a dataset with a new metric, the cached model outputs are reused.

DuckDB Caching

DuckDB is a lightweight, in-memory, columnar database that is used by KARMA to cache the model’s predictions. This the default way of caching.

DynamoDB Caching

DynamoDB is a NoSQL database service provided by Amazon Web Services (AWS). KARMA can also use DynamoDB to cache model predictions. This is useful for large-scale deployments where the model predictions need to be stored in a highly scalable and durable manner.

To use DynamoDB caching, you need to configure the following environment variables:

• AWS_ACCESS_KEY_ID: Your AWS access key ID.
• AWS_SECRET_ACCESS_KEY: Your AWS secret access key.
• AWS_REGION: The AWS region where your DynamoDB table is located.

Once you have configured these environment variables, you can enable DynamoDB caching by setting the KARMA_CACHE_TYPE environment variable to dynamodb.

DynamoDB table layout

KARMA writes to three DynamoDB tables that share the same prefix (default: parrotlet_omni_benchmark_cache). Besides the primary cache table and the _run_metadata collection, metric persistence expects a parrotlet_omni_benchmark_cache_metric_results table with:

• config_hash as the partition key.
• target_scope#target_id#metric_name as the sort key (stored in the target_sort_key attribute).
• A config_hash-index global secondary index that uses config_hash (HASH) + metric_name (RANGE) and projects all attributes.
• On‑demand (PAY_PER_REQUEST) billing unless you have a specific capacity plan.

This layout mirrors the uniqueness constraints DuckDB enforces across config_hash, metric_name, target_scope, and target_id, ensuring run-level and sample-level metrics can coexist without clobbering each other.