Fine-Tuning GPT-OSS-20B for DAX: From 15/30 to 20/30 on DAXBench

I run DAXBench, a project where I test LLMs on their ability to write correct DAX.

The benchmark currently contains 30 tasks covering different areas of DAX: filter context, iterators, time intelligence, ranking, metadata adherence, syntax and other common sources of errors.

So far, no model has solved all 30 tasks, although the strongest frontier models are getting close.

Local models are still noticeably behind.

I wanted to see how much of that gap could be reduced by fine-tuning a local model specifically for DAX.

The result was a LoRA fine-tune of OpenAI's GPT-OSS-20B that improved its DAXBench score from 15/30 to 20/30, produced 20% more correct results on an independent evaluation set, and dramatically reduced response length and latency.

I am now open-sourcing the LoRA on Hugging Face:

https://huggingface.co/maxanatsko/gpt-oss-20b-dax-lora

Why fine-tune a local model for DAX?

Frontier cloud models are generally better at DAX than smaller local models.

They reason more reliably about filter context, follow complex instructions better and make fewer syntax mistakes. But they also come with several disadvantages:

• API costs
• network latency
• data privacy concerns
• rate limits
• dependence on an external provider
• unnecessarily large responses for relatively narrow tasks

DAX generation is also a fairly specialized problem.

A model does not need to know everything about the world to write a measure. It needs to understand the supplied semantic model, follow the requested output format and reason correctly about DAX evaluation context.

That makes it a potentially good use case for a specialized local model.

My broader view is that the future is probably not purely local or purely cloud-based.

A practical architecture may combine:

• specialized local models for narrow and repeatable tasks
• frontier cloud models for harder, ambiguous or highly complex tasks

The local model can handle the majority of routine requests quickly and cheaply, while the frontier model remains available as an escalation path.

Goals of the experiment

I had four main goals.

1. Improve DAXBench performance

The aspirational goal was to push the model as close as possible to solving all 30 DAXBench tasks.

I did not expect a relatively small LoRA to suddenly turn GPT-OSS-20B into a frontier model, but I wanted to test how much domain-specific improvement was possible.

2. Improve reasoning about DAX

Producing syntactically valid DAX is not enough.

Many incorrect measures look reasonable at first glance. They may use valid functions and compile successfully while still returning the wrong result.

The model needs to reason about:

• current filter context
• which filters should be removed
• which filters should be preserved
• whether a new filter should replace or intersect an existing one
• the correct iterator and calculation grain
• partial-period versus full-period time intelligence
• relationships and metadata from the semantic model

3. Improve metadata adherence

The model should use the tables, columns and measures that actually exist in the supplied model.

It should not invent plausible-looking fields or ignore the metadata included in the prompt.

4. Improve syntax and output adherence

When a user asks for only a DAX expression, the model should return only a DAX expression.

This sounds simple, but general-purpose reasoning models often produce a long explanation before the final answer, even when explicitly instructed not to.

For an interactive tool or coding assistant, this matters.

Verbose output increases latency, consumes more tokens and can make it harder to extract and execute the generated expression.

Why GPT-OSS-20B?

For the experiment, I chose OpenAI's GPT-OSS-20B.

It is no longer a new model, but it remains a capable local model that can run on many modern laptops.

The model is large enough to have useful reasoning capabilities, while still being practical for local inference and LoRA fine-tuning on Apple Silicon.

My training environment was based on MLX running on a Mac.

This is important because the released adapter is an MLX LoRA intended for Apple Silicon. It is not a standard cross-platform adapter that I have validated on NVIDIA or Windows hardware.

I currently do not have a sufficiently powerful NVIDIA system to train and test a broadly compatible release.

Results

The fine-tuned model improved across several dimensions.

DAXBench

The base model solved: 15 out of 30 tasks

The fine-tuned model solved: 20 out of 30 tasks

That is a five-task absolute improvement, or roughly a one-third increase in the number of solved benchmark tasks.

Independent evaluation

I also evaluated the model on a separate set of examples that was not used to select the final checkpoint.

On that independent evaluation, the fine-tuned model produced:

20% more correct results than the base model

This was important because it reduced the likelihood that the improvement was limited only to the original benchmark tasks.

Metadata and syntax adherence

Metadata and syntax adherence also improved, although less dramatically than the DAXBench score.

The fine-tuned model was more likely to:

• use the supplied table and column names
• avoid inventing unsupported metadata
• follow the required output structure
• avoid unnecessary explanatory text

Output length and latency

One of the largest practical improvements was output discipline.

The base model often produced verbose reasoning even when instructed to return "just DAX."

The fine-tuned model followed this requirement much more reliably.

Across the DAXBench evaluations, the average response changed from approximately:

This is a major difference.

The fine-tuned model was not only more accurate. It also returned answers much faster and with far less unnecessary text.

For a user-facing DAX assistant, that can be as important as the benchmark improvement itself.

Examples of the improvements

The benchmark numbers show that the fine-tune improved, but individual examples make the difference easier to understand.

The strongest gains were concentrated in filter semantics, time-intelligence alignment, iterator usage and selecting the correct business grain.

1. Respecting the existing filter context

The task required calculating the median funded amount within the current filter context.

Base model

MEDIANX(
    ALL('LoanLedger'),
    'LoanLedger'[Funded Amount]
)

Fine-tuned model

MEDIANX(
    'LoanLedger',
    'LoanLedger'[Funded Amount]
)

The base model removes all filters from LoanLedger.

That means slicers and filters such as product, year or borrower segment no longer affect the result. The measure returns a global median instead of the median for the currently selected context.

The fine-tuned model iterates over the currently filtered rows.

This is a semantic correction, not merely a formatting difference.

2. Preserving the requested date grain

The task required returning a yearly total while removing the filter from a particular item.

Base model

CALCULATE(
    [Room Revenue Total],
    ALL('RatePlan')
)

Fine-tuned model

CALCULATE(
    [Room Revenue Total],
    ALLEXCEPT(
        'StayCalendar',
        'StayCalendar'[Stay Year]
    ),
    REMOVEFILTERS('RatePlan')
)

The base model removes the item filter, but it leaves all existing calendar filters intact.

If the current context is a month or quarter, the result still changes within the year.

The fine-tuned model removes the lower date-grain filters while preserving the year. It therefore returns the requested year total while ignoring the selected item.

3. Intersecting a filter instead of replacing it

The task required applying a "Major" market-size filter while preserving the existing user selections.

Fine-tuned model

CALCULATE(
    [Ticket Revenue Total],
    KEEPFILTERS(
        'VenueMarket'[Market Size] = "Major"
    )
)

The important part here is KEEPFILTERS.

Without it, a filter argument inside CALCULATE can replace the existing filter on the same column.

With KEEPFILTERS, the new condition is intersected with the current filter context.

This preserves the user's selections rather than silently broadening or changing them.

4. Correct partial-period year-over-year logic

The task required comparing the current visible period with the equivalent period in the previous year.

Base model

[Contract Value Total]
    - CALCULATE(
        [Contract Value Total],
        PREVIOUSYEAR('ContractCalendar'[Date])
    )

Fine-tuned model

VAR CurrentAmount =
    [Contract Value Total]

VAR PriorAmount =
    CALCULATE(
        [Contract Value Total],
        SAMEPERIODLASTYEAR('ContractCalendar'[Date])
    )

RETURN
    CurrentAmount - PriorAmount

PREVIOUSYEAR returns the previous calendar year.

That can be wrong for a partial-period comparison. For example, if the current selection is January through March, the base expression may compare those three months with the entire previous year.

SAMEPERIODLASTYEAR shifts the visible date period back by one year.

The fine-tuned model therefore performs a like-for-like comparison.

5. Implementing a genuine weighted calculation

The task required calculating a quantity-weighted parts cost.

Base model

DIVIDE(
    SUM('RepairLedger'[Parts Cost]),
    SUM('RepairLedger'[Repair Units]),
    0
)

Fine-tuned model

DIVIDE(
    SUMX(
        'RepairLedger',
        'RepairLedger'[Parts Cost]
            * 'RepairLedger'[Repair Units]
    ),
    SUM('RepairLedger'[Repair Units])
)

The base model calculates a ratio of totals.

That is not a weighted average.

The fine-tuned model multiplies each row's cost by its weight, sums those products and divides the result by the total weight.

This is a textbook use of an iterator, but it is also a consequential semantic improvement.

6. Ranking at the requested business grain

The task required identifying the top five ad-format labels by revenue.

The base model ranked all rows of the product table.

That can produce incorrect results when multiple underlying keys map to the same displayed label.

The fine-tuned model explicitly builds the ranking set from the requested label column:

VAR TopLabels =
    TOPN(
        5,
        ADDCOLUMNS(
            ALL('AdProduct'[Ad Format]),
            "Metric", [Booked Revenue Total]
        ),
        [Metric],
        DESC
    )

It then returns the measure only when the current label belongs to the top five.

The improvement is not simply that the model used TOPN.

It ranked at the user-facing business grain requested by the task.

Where the fine-tune improved most

The gains were not evenly distributed across every type of task.

The strongest improvements were concentrated in:

• preserving and modifying filter context correctly
• distinguishing filter replacement from filter intersection
• aligning partial-period time-intelligence calculations
• selecting the correct iterator
• calculating at the requested grain
• following metadata more closely

These are also areas where general-purpose models often make subtle mistakes.

The expression may look reasonable and use familiar DAX functions, but the result can still be incorrect because the model misunderstood the evaluation context or business grain.

What the results do not mean

The fine-tuned model is still not perfect.

It solved 20 out of 30 DAXBench tasks, which means it still failed one-third of the benchmark.

It also does not consistently match the reasoning ability of the best frontier models.

This experiment does not show that a 20-billion-parameter local model can replace a frontier model for every DAX problem.

It shows something narrower and, in my view, still useful:

A targeted LoRA can materially improve a local model's accuracy, instruction adherence, latency and output efficiency for a specialized domain.

The improvement is especially valuable when the model is part of a repeatable workflow where the prompt format and expected output are relatively stable.

Limitations

There are several important limitations.

The adapter is MLX-specific

The released LoRA was trained using MLX on Apple Silicon.

I have not trained or validated an equivalent adapter for CUDA, Windows or standard Transformers inference.

Evaluation remains difficult

DAX evaluation is not fully captured by string comparison or syntax validation.

Two expressions can look different while being semantically equivalent. Conversely, an expression can parse successfully while still producing the wrong result.

This is why benchmark results need a combination of:

• automated validation
• parser checks
• execution where possible
• manual adjudication of semantic correctness

The benchmark is still limited

DAXBench currently contains 30 primary tasks.

It covers a range of important DAX concepts, but it cannot represent the complete complexity of real-world semantic models.

Additional evaluation sets are necessary to test whether improvements generalize.

Fine-tuning can make behavior narrower

A specialized model can become better at the target task while becoming less useful outside that domain.

For this use case, that is largely acceptable. The goal is not to create a general assistant. The goal is to create a better DAX assistant.

Local specialists and frontier generalists

The result reinforces my view that local and cloud models do not need to be treated as mutually exclusive.

A local fine-tuned model can handle:

• routine measure generation
• metadata-constrained tasks
• syntax repair
• common filter-context patterns
• low-latency interactive suggestions
• privacy-sensitive semantic models

A frontier model can remain available for:

• ambiguous requirements
• unfamiliar business logic
• complex debugging
• large multi-step reasoning tasks
• cases where the local model has low confidence
• review and correction of local-model output

This kind of routing can reduce cost and latency without requiring users to accept the limitations of a local model for every request.

Open-source release

The LoRA is available on Hugging Face:

https://huggingface.co/maxanatsko/gpt-oss-20b-dax-lora

It is released for experimentation and further evaluation.

The current version is intended for MLX and Apple Silicon. I would be interested to see equivalent fine-tunes trained and tested on NVIDIA hardware, as well as comparisons with newer local base models.

Conclusion

Fine-tuning GPT-OSS-20B for DAX produced meaningful improvements.

Compared with the base model:

• DAXBench improved from 15/30 to 20/30
• the independent evaluation produced 20% more correct results
• metadata and syntax adherence improved
• output became significantly more concise
• average response time decreased from approximately 18 seconds to 2.4 seconds
• average output length decreased from approximately 2,200 characters to 170 characters

The model is still not a replacement for the strongest frontier systems.

But it is a much better DAX model than the base checkpoint, and it is fast enough to run locally on accessible hardware.

For me, the most interesting result is not only the five additional benchmark tasks.

It is that a relatively focused fine-tune improved accuracy, latency and usability at the same time.

That makes specialized local models a credible part of a practical LLM architecture for tools such as DAX assistants, semantic-model copilots and automated analytics workflows.