Comparative Evaluation of Open-Source LLMs on Professional Domain Knowledge Tasks: A Multi-Domain Benchmark with Multi-Judge and Expert Assessment
About the Publication
This research was conducted by Eduard Pankratov and Aleksandr Yakimov and presents a comparative evaluation of modern open-source large language models across 16 professional domains.
The study, “Comparative Evaluation of Open-Source LLMs on Professional Domain Tasks: A Multi-Domain Benchmark with Multi-Judge and Expert Evaluation,” was published in the peer-reviewed scientific journal Software Systems and Computational Methods, 2026, No. 3, pp. 40–60.
Authors: Eduard R. Pankratov, Aleksandr B. Yakimov
Publication date: July 7, 2026
DOI: 10.7256/2454-0714.2026.3.80458
EDN: NAGOPN
The original journal publication, including the complete methodology, experimental results, tables, figures, references, and peer-review materials, is available on the publisher’s website.
The study, “Comparative Evaluation of Open-Source LLMs on Professional Domain Tasks: A Multi-Domain Benchmark with Multi-Judge and Expert Evaluation,” was published in the peer-reviewed scientific journal Software Systems and Computational Methods, 2026, No. 3, pp. 40–60.
Authors: Eduard R. Pankratov, Aleksandr B. Yakimov
Publication date: July 7, 2026
DOI: 10.7256/2454-0714.2026.3.80458
EDN: NAGOPN
The original journal publication, including the complete methodology, experimental results, tables, figures, references, and peer-review materials, is available on the publisher’s website.
Keywords: large language models; LLM; benchmark; LLM-as-a-Judge; open-source; professional domains; professional knowledge; multi-judge evaluation; expert validation; comparative evaluation.
Abstract. This paper presents a comparative evaluation of 15 open-source large language models (LLMs) ranging from 8B to 120B parameters on professional-domain knowledge tasks. An original Russian-language benchmark was developed, covering 16 professional domains. Each domain comprises 90 questions stratified across three difficulty levels, totaling 1,440 questions. Response evaluation was performed in two stages: automated multi-judge assessment by three independent LLM judges following the LLM-as-a-Judge methodology (semantic correspondence to reference answers), followed by an independent expert validation by domain practitioners. Results indicate that recent-generation models in the 27−31B-parameter range (including Mixture-of-Experts architectures and FP8-optimized variants) perform on par with 120B-parameter models on professional tasks with several-fold lower computational resource requirements; differences within the leading group lie within the estimation margin and are not claimed as statistically significant. A recommendation framework for model selection in domain-specific applications is proposed.
Introduction
Large language models (LLMs) are rapidly transforming professional workflows, with applications spanning legal analysis, clinical analytics, DevOps automation, and supply chain management. The enterprise AI market is already valued at tens of billions of dollars and continues to expand at an annual rate exceeding 30%. At the same time, open-source solutions are steadily gaining ground because they offer greater transparency, controllability, and the flexibility of on-premises deployment. These characteristics are particularly important in industries with stringent information security and personal data protection requirements.
The relevance of this study stems partly from the needs of specific industry projects. We are currently developing a medical record analytics platform built on multi-agent systems that use open-source neural network models deployed on local servers in the Russian Federation. The platform architecture includes RAG-based databases for storing and contextualizing medical information, as well as API integration with a 1C-based medical information system (MIS). The platform is being developed in collaboration with practicing physicians. Selecting suitable language models for such a system requires a systematic evaluation of their performance on professional tasks across multiple domains, ranging from medicine and pharmaceuticals to law and finance.
Existing LLM benchmarks, including MMLU, HellaSwag, ARC, and TruthfulQA, primarily target English-language general-knowledge tasks [1]. Russian-language benchmarks such as Russian SuperGLUE and MERA assess general language capabilities but do not cover specialized professional domains that underpin real-world enterprise applications of LLMs [2, 3]. Widely used leaderboards, including Chatbot Arena and the Open LLM Leaderboard, also do not account for the effect of generation parameters, such as temperature and repeat penalty, on response quality and rely on a single evaluator [4].
The aim of this study is to develop and empirically evaluate a reproducible methodology for the comparative evaluation of open-source LLMs on Russian-language professional tasks across multiple domains. The methodology combines a two-stage evaluation framework-multi-judge LLM-as-a-Judge assessment followed by independent expert evaluation-with an analysis of the effects of generation parameters on response quality.
The novel contributions of this study are threefold. First, we introduce a Russian-language benchmark of professional knowledge comprising 1,440 questions across 16 professional domains and three difficulty levels. Second, we propose a two-stage methodology for response evaluation: automated multi-judge LLM-as-a-Judge assessment by three independent judge models, followed by independent expert validation by domain practitioners and an analysis of agreement between automated and expert assessments [5, 6]. Third, we systematically analyze the effects of generation parameters on response quality in professional tasks. We explicitly distinguish these contributions from findings and approaches already established in the literature. Neither the use of language models as judges nor the observation that smaller models can remain competitive with larger ones is new; both have already been widely documented [4−6] and are not claimed here as original contributions. The originality of this study lies specifically in the combination of Russian-language coverage across 16 professional domains, practitioner-verified tasks, and expert validation of automated assessments across the entire benchmark corpus. To our knowledge, this combination has not been reported in either Russian-language studies or the broader international literature.
The practical value of this study lies in providing an evidence base for LLM selection in the development of industry-specific AI systems, including multi-agent systems, RAG platforms, document workflow automation modules, and medical record analytics systems. The methodology can also be adapted to other sets of domains and models.
The relevance of this study stems partly from the needs of specific industry projects. We are currently developing a medical record analytics platform built on multi-agent systems that use open-source neural network models deployed on local servers in the Russian Federation. The platform architecture includes RAG-based databases for storing and contextualizing medical information, as well as API integration with a 1C-based medical information system (MIS). The platform is being developed in collaboration with practicing physicians. Selecting suitable language models for such a system requires a systematic evaluation of their performance on professional tasks across multiple domains, ranging from medicine and pharmaceuticals to law and finance.
Existing LLM benchmarks, including MMLU, HellaSwag, ARC, and TruthfulQA, primarily target English-language general-knowledge tasks [1]. Russian-language benchmarks such as Russian SuperGLUE and MERA assess general language capabilities but do not cover specialized professional domains that underpin real-world enterprise applications of LLMs [2, 3]. Widely used leaderboards, including Chatbot Arena and the Open LLM Leaderboard, also do not account for the effect of generation parameters, such as temperature and repeat penalty, on response quality and rely on a single evaluator [4].
The aim of this study is to develop and empirically evaluate a reproducible methodology for the comparative evaluation of open-source LLMs on Russian-language professional tasks across multiple domains. The methodology combines a two-stage evaluation framework-multi-judge LLM-as-a-Judge assessment followed by independent expert evaluation-with an analysis of the effects of generation parameters on response quality.
The novel contributions of this study are threefold. First, we introduce a Russian-language benchmark of professional knowledge comprising 1,440 questions across 16 professional domains and three difficulty levels. Second, we propose a two-stage methodology for response evaluation: automated multi-judge LLM-as-a-Judge assessment by three independent judge models, followed by independent expert validation by domain practitioners and an analysis of agreement between automated and expert assessments [5, 6]. Third, we systematically analyze the effects of generation parameters on response quality in professional tasks. We explicitly distinguish these contributions from findings and approaches already established in the literature. Neither the use of language models as judges nor the observation that smaller models can remain competitive with larger ones is new; both have already been widely documented [4−6] and are not claimed here as original contributions. The originality of this study lies specifically in the combination of Russian-language coverage across 16 professional domains, practitioner-verified tasks, and expert validation of automated assessments across the entire benchmark corpus. To our knowledge, this combination has not been reported in either Russian-language studies or the broader international literature.
The practical value of this study lies in providing an evidence base for LLM selection in the development of industry-specific AI systems, including multi-agent systems, RAG platforms, document workflow automation modules, and medical record analytics systems. The methodology can also be adapted to other sets of domains and models.
1. Methodology
Table 3. Experimental series and generation parameters
Table 2. Characteristics of the evaluated models
Table 1. Professional domains in the benchmark
1.6. Expert Validation of the Results
To strengthen the reliability of the automated evaluation results, we included an independent expert validation stage conducted by practitioners from the respective domains. The validation was performed on the complete full_details dataset, which contains row-level results for responses generated by all evaluated models, including the scores and rationales provided by each of the three judge models and the reference answer for each question.
The expert validation procedure consisted of three sequential steps:
Expert validation was qualitative and verification-oriented. Its purpose was to confirm that the automated evaluations were meaningful and professionally sound rather than to derive a formal statistical measure of agreement. Based on the qualitative judgment of the participating domain practitioners, expert assessments were generally consistent with those of the LLM judges, while the observed discrepancies were considered predominantly nonsystematic. The experts also considered the benchmark questions to be appropriate to the objectives of the study. Formal quantitative measures of agreement (correlation coefficients, Cohen’s kappa, etc.) were not calculated in this study. Nor did the study include formal, documented verification of expert qualifications for every domain. Computing such metrics using a larger and systematically documented sample of experts is therefore left for future work. Accordingly, the qualitative expert verification should be regarded as supporting evidence for the applicability of the automated multi-judge evaluation procedure rather than as a comprehensive statistical validation of that procedure.
To strengthen the reliability of the automated evaluation results, we included an independent expert validation stage conducted by practitioners from the respective domains. The validation was performed on the complete full_details dataset, which contains row-level results for responses generated by all evaluated models, including the scores and rationales provided by each of the three judge models and the reference answer for each question.
The expert validation procedure consisted of three sequential steps:
- Question verification. Practitioners from the respective domains reviewed the questions for correctness, completeness, and relevance to current professional practice, confirming that they reflect real-world professional tasks and comply with current regulatory requirements.
- Independent expert assessment of model responses. Experts independently evaluated the responses generated by the evaluated LLMs, relying on their own professional expertise and using the same 0−10 scale as the LLM judges. During the assessment, the experts had no access to the scores or rationales provided by the LLM judges, thereby precluding anchoring bias arising from the automated evaluations.
- Assessment of agreement between expert and automated evaluations. For each model-question pair, the experts compared their own scores with the mean score assigned by the three LLM judges and subsequently reviewed the judges' rationales, recording discrepancies and potential systematic biases.
Expert validation was qualitative and verification-oriented. Its purpose was to confirm that the automated evaluations were meaningful and professionally sound rather than to derive a formal statistical measure of agreement. Based on the qualitative judgment of the participating domain practitioners, expert assessments were generally consistent with those of the LLM judges, while the observed discrepancies were considered predominantly nonsystematic. The experts also considered the benchmark questions to be appropriate to the objectives of the study. Formal quantitative measures of agreement (correlation coefficients, Cohen’s kappa, etc.) were not calculated in this study. Nor did the study include formal, documented verification of expert qualifications for every domain. Computing such metrics using a larger and systematically documented sample of experts is therefore left for future work. Accordingly, the qualitative expert verification should be regarded as supporting evidence for the applicability of the automated multi-judge evaluation procedure rather than as a comprehensive statistical validation of that procedure.
1.5. Generation Parameters and Experimental Series
The evaluation was conducted in four experimental series using different generation settings (Table 3). We varied temperature, which controls generation stochasticity, and repeat penalty, which penalizes token repetition. The primary series used temperature = 0 to minimize generation stochasticity and improve reproducibility. This setting also yielded the highest response quality on the professional tasks in our experiments.
The evaluation was conducted in four experimental series using different generation settings (Table 3). We varied temperature, which controls generation stochasticity, and repeat penalty, which penalizes token repetition. The primary series used temperature = 0 to minimize generation stochasticity and improve reproducibility. This setting also yielded the highest response quality on the professional tasks in our experiments.
1.4. Metrics and Scoring Formulas
The per-question score for each model is computed as the arithmetic mean of the scores assigned by the three judge models:
S = (S₁ + S₂ + S₃) / 3, (1)
where S₁, S₂, and S₃ are the scores assigned by GLM-4.7-Flash, DeepSeek-V4-Flash, and GPT-OSS-120B, respectively.
The mean score for model m at difficulty level d is computed by averaging across all domains:
Score (m, d) = (1/N) · ∑ Sᵢ(m, d), (2)
where N = 16 is the number of domains and Sᵢ is the mean score for domain i based on Equation (1).
Performance degradation is defined as the difference between the mean scores at Level 1 and Level 3:
D (m) = Score (m, 1) − Score (m, 3). (3)
Inter-judge variability is quantified using the mean standard deviation:
σ(m) = (1/K) · ∑ √[ ((S₁ − S)² + (S₂ − S)² + (S₃ − S)²) / 3 ], (4)
where the sum is taken over all K observations (question x domain x level), and S is the mean of the three judge scores defined in Equation (1).
For the overall ranking of models, we use a weighted scoring scheme that assigns greater weight to higher-difficulty questions, reflecting their greater professional significance:
W (m) = [0,6 · Score (m, 1) + 0,8 · Score (m, 2) + 1,0 · Score (m, 3)] / 3. (5)
The same weighting scheme is used to rank models within individual domains.
The per-question score for each model is computed as the arithmetic mean of the scores assigned by the three judge models:
S = (S₁ + S₂ + S₃) / 3, (1)
where S₁, S₂, and S₃ are the scores assigned by GLM-4.7-Flash, DeepSeek-V4-Flash, and GPT-OSS-120B, respectively.
The mean score for model m at difficulty level d is computed by averaging across all domains:
Score (m, d) = (1/N) · ∑ Sᵢ(m, d), (2)
where N = 16 is the number of domains and Sᵢ is the mean score for domain i based on Equation (1).
Performance degradation is defined as the difference between the mean scores at Level 1 and Level 3:
D (m) = Score (m, 1) − Score (m, 3). (3)
Inter-judge variability is quantified using the mean standard deviation:
σ(m) = (1/K) · ∑ √[ ((S₁ − S)² + (S₂ − S)² + (S₃ − S)²) / 3 ], (4)
where the sum is taken over all K observations (question x domain x level), and S is the mean of the three judge scores defined in Equation (1).
For the overall ranking of models, we use a weighted scoring scheme that assigns greater weight to higher-difficulty questions, reflecting their greater professional significance:
W (m) = [0,6 · Score (m, 1) + 0,8 · Score (m, 2) + 1,0 · Score (m, 3)] / 3. (5)
The same weighting scheme is used to rank models within individual domains.
1.3. Evaluation Methodology: LLM-as-a-Judge
Responses were evaluated in two independent stages. The first stage used automated multi-judge evaluation following the LLM-as-a-Judge methodology with reference answers (reference-based grading) [5, 6]. The second stage consisted of independent expert validation (see Section 1.6). Each response generated by an evaluated model was assessed by three independent judge models for semantic correspondence to the reference answer, using a 0−10 scale:
— GLM-4.7-Flash (NVFP4);
— DeepSeek-V4-Flash (API);
— GPT-OSS-120B (MXFP4).
GLM-4.7-Flash and GPT-OSS-120B served both as evaluated models and as judge models, whereas DeepSeek-V4-Flash was used exclusively as a judge. The overall evaluation pipeline followed established practices in automated LLM evaluation [15].
To reduce potential bias in the automated evaluation, model responses were presented to the LLM judges in anonymized form, without disclosing which model generated each response. All responses were standardized to a uniform text format and stripped of metadata that could reveal the identity of the generating model. The judges were not informed which models were included in the evaluation.
Despite these precautions, the possibility of systematic bias arising from the architectural characteristics of the judge models cannot be entirely excluded, particularly because some models serve both as evaluated models and as judge models. We mitigate the potential impact of such bias by using multiple independent judges, averaging their scores, and conducting additional independent expert validation (see Section 1.6). The implications of these potential biases for the evaluation results are discussed in detail in Section 3.
Responses were evaluated in two independent stages. The first stage used automated multi-judge evaluation following the LLM-as-a-Judge methodology with reference answers (reference-based grading) [5, 6]. The second stage consisted of independent expert validation (see Section 1.6). Each response generated by an evaluated model was assessed by three independent judge models for semantic correspondence to the reference answer, using a 0−10 scale:
— GLM-4.7-Flash (NVFP4);
— DeepSeek-V4-Flash (API);
— GPT-OSS-120B (MXFP4).
GLM-4.7-Flash and GPT-OSS-120B served both as evaluated models and as judge models, whereas DeepSeek-V4-Flash was used exclusively as a judge. The overall evaluation pipeline followed established practices in automated LLM evaluation [15].
To reduce potential bias in the automated evaluation, model responses were presented to the LLM judges in anonymized form, without disclosing which model generated each response. All responses were standardized to a uniform text format and stripped of metadata that could reveal the identity of the generating model. The judges were not informed which models were included in the evaluation.
Despite these precautions, the possibility of systematic bias arising from the architectural characteristics of the judge models cannot be entirely excluded, particularly because some models serve both as evaluated models and as judge models. We mitigate the potential impact of such bias by using multiple independent judges, averaging their scores, and conducting additional independent expert validation (see Section 1.6). The implications of these potential biases for the evaluation results are discussed in detail in Section 3.
1.2. Evaluated Models
The evaluation included 15 open-source LLMs spanning different architectures and parameter scales (Table 2), including two models from the Gemma 4 family developed by Google DeepMind: the dense Gemma-4−31B model and the Gemma-4−26B-A4B Mixture-of-Experts (MoE) variant. The models were run locally on servers using FP8 weights, with NVFP4 quantization used for GLM-4.7-Flash and native MXFP4 weights for the GPT-OSS family. We also evaluated GGUF builds quantized as Q4_K_M, Q4_K_S, or Q6_K using Ollama. All models were evaluated in instruct/chat mode. QwQ-32B was tested with two quantization settings (Q6_K and Q4_K_M), resulting in 16 configurations in total. Model architectures and key characteristics are described in the corresponding technical reports and official publications by the model developers [7−14, 17−21]. For model versions released without dedicated research papers at the time of writing (Qwen3.5, Gemma 4, GLM-4.7-Flash, and DeepSeek-V4), we cite official model cards and release pages published by the developers. For earlier generations of the same model families, we retain references to the corresponding technical reports describing their core architectural principles.
The evaluation included 15 open-source LLMs spanning different architectures and parameter scales (Table 2), including two models from the Gemma 4 family developed by Google DeepMind: the dense Gemma-4−31B model and the Gemma-4−26B-A4B Mixture-of-Experts (MoE) variant. The models were run locally on servers using FP8 weights, with NVFP4 quantization used for GLM-4.7-Flash and native MXFP4 weights for the GPT-OSS family. We also evaluated GGUF builds quantized as Q4_K_M, Q4_K_S, or Q6_K using Ollama. All models were evaluated in instruct/chat mode. QwQ-32B was tested with two quantization settings (Q6_K and Q4_K_M), resulting in 16 configurations in total. Model architectures and key characteristics are described in the corresponding technical reports and official publications by the model developers [7−14, 17−21]. For model versions released without dedicated research papers at the time of writing (Qwen3.5, Gemma 4, GLM-4.7-Flash, and DeepSeek-V4), we cite official model cards and release pages published by the developers. For earlier generations of the same model families, we retain references to the corresponding technical reports describing their core architectural principles.
1.1. Benchmark Design
The benchmark covers 16 professional domains (Table 1). Each domain comprises 90 questions evenly distributed across three difficulty levels: Level 1 (basic professional questions, 30 questions), Level 2 (advanced questions, 30 questions), and Level 3 (expert-level questions, 30 questions). The complete benchmark therefore contains 1,440 questions. We developed the questions and reference answers in collaboration with practitioners from the respective domains to reflect real-world professional practice more closely. The medical component of the benchmark is divided into two domains: Clinical Care Processes, covering clinical documentation workflows, diagnostics, and organization of healthcare delivery; and General Internal Medicine, covering initial patient visits, diagnosis, and pharmacotherapy. This distinction reflects the requirements of the intended application—a medical record analytics platform.
The benchmark covers 16 professional domains (Table 1). Each domain comprises 90 questions evenly distributed across three difficulty levels: Level 1 (basic professional questions, 30 questions), Level 2 (advanced questions, 30 questions), and Level 3 (expert-level questions, 30 questions). The complete benchmark therefore contains 1,440 questions. We developed the questions and reference answers in collaboration with practitioners from the respective domains to reflect real-world professional practice more closely. The medical component of the benchmark is divided into two domains: Clinical Care Processes, covering clinical documentation workflows, diagnostics, and organization of healthcare delivery; and General Internal Medicine, covering initial patient visits, diagnosis, and pharmacotherapy. This distinction reflects the requirements of the intended application—a medical record analytics platform.
2. Results
Figure 3. Top-ranked models across 16 professional domains (weighted average)
Table 6. Leading models by professional domain (weighted average)
Table 5. Inter-judge agreement (σ: mean standard deviation of judge scores)
Figure 2. Model robustness to increasing task difficulty (mean scores by difficulty level)
Figure 1. Weighted average score by model size
Table 4. Overall ranking of model configurations by difficulty level
Table 8. Final ranking of models for professional tasks
2.6. Final Ranking and Recommendations
The overall final ranking of model configurations, grouped into S/A/B/C tiers, is presented in Table 8. Tier assignments take into account the weighted average score, degradation D (m), and inter-judge variability σ. The recommendations reflect the domain-specific strengths observed for each model.
The overall final ranking of model configurations, grouped into S/A/B/C tiers, is presented in Table 8. Tier assignments take into account the weighted average score, degradation D (m), and inter-judge variability σ. The recommendations reflect the domain-specific strengths observed for each model.
Table 7. Effect of generation parameters on mean score
2.5. Effect of Generation Parameters
We compared results obtained under different generation settings during preliminary tuning of the evaluation methodology on a representative subset of models (Table 7). Increasing temperature from 0 to 0.8 was associated with a decrease in mean score of approximately 0.2−0.3 points on the 0−10 scale for most models in the subset. By contrast, reducing repeat penalty from 1.2 to 1.05 at temperature = 0 had only a minor effect (approximately ±0.1 points), suggesting that temperature had a greater influence on response quality within the tested parameter ranges. Based on these results, the primary experimental series used deterministic generation with temperature = 0 and repeat penalty = 1.2.
We compared results obtained under different generation settings during preliminary tuning of the evaluation methodology on a representative subset of models (Table 7). Increasing temperature from 0 to 0.8 was associated with a decrease in mean score of approximately 0.2−0.3 points on the 0−10 scale for most models in the subset. By contrast, reducing repeat penalty from 1.2 to 1.05 at temperature = 0 had only a minor effect (approximately ±0.1 points), suggesting that temperature had a greater influence on response quality within the tested parameter ranges. Based on these results, the primary experimental series used deterministic generation with temperature = 0 and repeat penalty = 1.2.
2.4. Domain Specialization
Domain-level analysis based on Equation (5) shows clear model specialization across professional domains (Table 6 and Figure 3). The top rank across the 16 domains is shared by four models: Qwen3.5−27B ranks first in 8 domains, GPT-OSS-120B in 6, and Gemma-4−31B and Gemma-4−26B-A4B in one domain each. No single model ranks first across all domains, which supports the potential value of combining multiple models in multi-agent systems.
Domain-level analysis based on Equation (5) shows clear model specialization across professional domains (Table 6 and Figure 3). The top rank across the 16 domains is shared by four models: Qwen3.5−27B ranks first in 8 domains, GPT-OSS-120B in 6, and Gemma-4−31B and Gemma-4−26B-A4B in one domain each. No single model ranks first across all domains, which supports the potential value of combining multiple models in multi-agent systems.
2.3. Inter-Judge Agreement
The inter-judge variability quantified by Equation (4) reflects the degree of agreement among the three judge models when evaluating the same responses (Table 5, which shows representative configurations from the full set). On the 0−10 scoring scale, the lowest mean standard deviations were observed for GPT-OSS-120B (σ = 0.35), Nemotron-70B (σ = 0.40), Gemma-4−31B (σ = 0.41), and Qwen3.5−27B (σ = 0.42). Higher values for GLM-4.7-Flash (0.64) and GPT-OSS-20B (0.59) indicate greater disagreement among the judge models when evaluating their responses. The relatively low σ values observed for several of the highest-ranked models indicate that the three independent judges assigned similar scores to the same responses.
The inter-judge variability quantified by Equation (4) reflects the degree of agreement among the three judge models when evaluating the same responses (Table 5, which shows representative configurations from the full set). On the 0−10 scoring scale, the lowest mean standard deviations were observed for GPT-OSS-120B (σ = 0.35), Nemotron-70B (σ = 0.40), Gemma-4−31B (σ = 0.41), and Qwen3.5−27B (σ = 0.42). Higher values for GLM-4.7-Flash (0.64) and GPT-OSS-20B (0.59) indicate greater disagreement among the judge models when evaluating their responses. The relatively low σ values observed for several of the highest-ranked models indicate that the three independent judges assigned similar scores to the same responses.
2.2. Analysis of Robustness to Increasing Task Difficulty
The degradation metric D (m), calculated using Equation (3) and visualized in Figure 2, characterizes model robustness to increasing task difficulty. Lower values of D indicate greater robustness. The lowest degradation values (D ≤ 0.30) were observed for Gemma-4−31B (0.21), Qwen3.5−27B (0.22), Qwen3.5−9B (0.23), and Gemma-4−26B-A4B (0.26). By contrast, Calme-2.4-RYS-78B (1.17) and Mistral-Nemo-12B (1.14) had degradation exceeding 1.0. A pronounced decline in performance on expert-level questions may increase the risk of generating confident but inaccurate responses (hallucinated confidence) [16]. Within the evaluated model set, the Qwen3.5 and Gemma 4 families exhibited lower degradation as task difficulty increased than the earlier-generation models included in the comparison.
The degradation metric D (m), calculated using Equation (3) and visualized in Figure 2, characterizes model robustness to increasing task difficulty. Lower values of D indicate greater robustness. The lowest degradation values (D ≤ 0.30) were observed for Gemma-4−31B (0.21), Qwen3.5−27B (0.22), Qwen3.5−9B (0.23), and Gemma-4−26B-A4B (0.26). By contrast, Calme-2.4-RYS-78B (1.17) and Mistral-Nemo-12B (1.14) had degradation exceeding 1.0. A pronounced decline in performance on expert-level questions may increase the risk of generating confident but inaccurate responses (hallucinated confidence) [16]. Within the evaluated model set, the Qwen3.5 and Gemma 4 families exhibited lower degradation as task difficulty increased than the earlier-generation models included in the comparison.
As shown in Figure 1 and Table 4, the results do not show a simple monotonic relationship between parameter count and response quality. Qwen3.5−27B (27B, FP8) achieved the highest weighted average score of 9.33, followed closely by GPT-OSS-120B (120B parameters) at 9.28. The Gemma 4 family also ranked among the leading models: Gemma-4−31B (31B parameters) scored 9.28, while the MoE variant Gemma-4−26B-A4B (26B parameters) scored 9.27. Thus, the four highest-ranked models span a wider range of parameter counts, from 26B to 120B, while achieving very similar weighted average scores. The maximum difference within this group is only 0.06 points. However, because confidence intervals and formal statistical significance tests were not calculated in this study, these small numerical differences should not be interpreted as evidence that any one of the four models outperforms the others. By contrast, some substantially larger models performed markedly worse than the smaller leading models; for example, Calme-2.4-RYS-78B (78B parameters) achieved a weighted average score of 8.17, more than one point below the highest-ranked model.
2.1. Overall Model Ranking
Table 4 presents the overall ranking of the evaluated models. The results were calculated using Equations (1)-(5) by aggregating data across all 16 domains. All scores are reported on a 0−10 scale. The Weighted Average column reports the overall score calculated using Equation (5), with difficulty-level weights of 0.6, 0.8 and 1.0, reflecting the increasing professional significance of higher-difficulty questions.
Table 4 presents the overall ranking of the evaluated models. The results were calculated using Equations (1)-(5) by aggregating data across all 16 domains. All scores are reported on a 0−10 scale. The Weighted Average column reports the overall score calculated using Equation (5), with difficulty-level weights of 0.6, 0.8 and 1.0, reflecting the increasing professional significance of higher-difficulty questions.
3. Discussion
Competitiveness of Compact Models. A central result of this study is the empirical confirmation that recent-generation models in the 27−31B parameter range can achieve results comparable to those of substantially larger 120B-parameter models and rank among the leading models in several professional domains. In the overall ranking, Qwen3.5−27B (27B parameters, FP8) achieved the highest weighted average score of 9.33, compared with 9.28 for GPT-OSS-120B (120B parameters). The Gemma 4 models also ranked among the leaders: the dense Gemma-4−31B model (FP8) and the MoE variant Gemma-4−26B-A4B achieved weighted average scores of 9.28 and 9.27, respectively. These results demonstrate that performance comparable to that of substantially larger models can be achieved with considerably fewer parameters, particularly in architectures with sparse parameter activation. Similar trends have been reported in the broader LLM literature: advances such as Mixture-of-Experts architectures, reasoning-oriented optimization, and improvements in training data quality can enable compact models to compete with substantially larger counterparts [7, 8]. Quantization schemes such as FP8 can further reduce the computational and memory requirements of model deployment while largely preserving model quality.
Robustness as a Key Differentiator. On basic questions, most models achieve scores between 8.5 and 9.5 on the 0−10 scale. On expert-level questions, however, the degradation metric D varies substantially, ranging from 0.2 for Gemma-4−31B to 1.17 for Calme-2.4-RYS-78B. Models from the Qwen3.5 and Gemma 4 families lose only 0.21−0.26 points when moving from basic to expert-level tasks, indicating high robustness on complex professional tasks. For domain-specific applications, model selection should therefore be based on robustness to increasing task difficulty rather than on peak performance on simple questions. The difference of more than one point between the leading and lowest-ranked models on expert-level questions has direct implications for the quality of professional responses in real-world use cases.
Systematic Judge Bias and Expert Validation. The dual role of GLM-4.7-Flash and GPT-OSS-120B as both evaluated models and judge models creates a potential source of evaluation bias. This concern is partially mitigated by the use of three judges, score averaging, and DeepSeek-V4-Flash as an external judge not included among the evaluated models. However, this study did not conduct a formal quantitative analysis of the score distributions produced by individual judges, including an assessment of whether judges systematically assign higher scores to closely related models. Such an analysis is therefore left for future work, and the mitigation measures described above should be regarded as procedural rather than statistically validated measures. Qualitative validation conducted by domain practitioners (see Section 1.6) was generally consistent with the assessments of the LLM judges. Differences between expert scores and scores assigned by the LLM judges were within the range of natural variability and, according to the experts' qualitative assessment, showed no pronounced systematic pattern. Future work should further diversify the judge pool by including models from different architectures and providers and should formalize quantitative measures of agreement between expert and automated evaluations, including correlation coefficients and other agreement metrics [5, 6].
Practical Recommendations for Domain-Specific Applications. For multi-agent systems, including medical record analytics platforms integrated via API with medical information systems (MISs) and RAG-based databases, an effective configuration is to combine multilingual models from the Qwen3.5 family (Qwen3.5−27B as a general-purpose professional agent) with models from the Gemma 4 family (Gemma-4−31B for robustness on complex tasks and Gemma-4−26B-A4B as a resource-efficient MoE alternative). For more structured domains, including Law, Clinical care processes, and Sales, GPT-OSS-120B emerges as the leading model, while also exhibiting the highest inter-judge agreement (σ = 0.35) among the evaluated models despite its larger parameter count. For resource-constrained systems, Qwen3.5−9B (9B parameters) offers a particularly strong quality-to-size trade-off, with a weighted average score of 9.09. It is also suitable for edge deployments [7, 13].
Study Limitations. The benchmark focuses on Russian-language tasks, and extrapolation to other languages requires additional validation. The evaluation was conducted in a single-question setting and did not account for multi-turn dialogue or integration with RAG systems. The benchmark questions were verified by the participating domain practitioners, and the assessments provided by the LLM judges were compared with independent expert evaluations, as described in Section 1.6. However, no formal, documented verification of expert qualifications was conducted for each of the 16 domains. The experimental series using different generation parameters (Section 2.5) were conducted using a representative subset of models during preliminary tuning of the evaluation methodology. Full validation across all 15 models is left for future work.
Data and Code Availability. To support reproducibility, the set of evaluation tasks with reference answers, the system prompts used in the experiments, instructions provided to the LLM judges, and the source code of the evaluation pipeline can be made available upon reasonable request to the authors. The elements described in this study—including the benchmark design, score aggregation formulas (1)-(5), generation parameters, and the composition of the judge-model panel—are reported in sufficient detail to enable independent reproduction of the methodology on any set of domains and models.
Robustness as a Key Differentiator. On basic questions, most models achieve scores between 8.5 and 9.5 on the 0−10 scale. On expert-level questions, however, the degradation metric D varies substantially, ranging from 0.2 for Gemma-4−31B to 1.17 for Calme-2.4-RYS-78B. Models from the Qwen3.5 and Gemma 4 families lose only 0.21−0.26 points when moving from basic to expert-level tasks, indicating high robustness on complex professional tasks. For domain-specific applications, model selection should therefore be based on robustness to increasing task difficulty rather than on peak performance on simple questions. The difference of more than one point between the leading and lowest-ranked models on expert-level questions has direct implications for the quality of professional responses in real-world use cases.
Systematic Judge Bias and Expert Validation. The dual role of GLM-4.7-Flash and GPT-OSS-120B as both evaluated models and judge models creates a potential source of evaluation bias. This concern is partially mitigated by the use of three judges, score averaging, and DeepSeek-V4-Flash as an external judge not included among the evaluated models. However, this study did not conduct a formal quantitative analysis of the score distributions produced by individual judges, including an assessment of whether judges systematically assign higher scores to closely related models. Such an analysis is therefore left for future work, and the mitigation measures described above should be regarded as procedural rather than statistically validated measures. Qualitative validation conducted by domain practitioners (see Section 1.6) was generally consistent with the assessments of the LLM judges. Differences between expert scores and scores assigned by the LLM judges were within the range of natural variability and, according to the experts' qualitative assessment, showed no pronounced systematic pattern. Future work should further diversify the judge pool by including models from different architectures and providers and should formalize quantitative measures of agreement between expert and automated evaluations, including correlation coefficients and other agreement metrics [5, 6].
Practical Recommendations for Domain-Specific Applications. For multi-agent systems, including medical record analytics platforms integrated via API with medical information systems (MISs) and RAG-based databases, an effective configuration is to combine multilingual models from the Qwen3.5 family (Qwen3.5−27B as a general-purpose professional agent) with models from the Gemma 4 family (Gemma-4−31B for robustness on complex tasks and Gemma-4−26B-A4B as a resource-efficient MoE alternative). For more structured domains, including Law, Clinical care processes, and Sales, GPT-OSS-120B emerges as the leading model, while also exhibiting the highest inter-judge agreement (σ = 0.35) among the evaluated models despite its larger parameter count. For resource-constrained systems, Qwen3.5−9B (9B parameters) offers a particularly strong quality-to-size trade-off, with a weighted average score of 9.09. It is also suitable for edge deployments [7, 13].
Study Limitations. The benchmark focuses on Russian-language tasks, and extrapolation to other languages requires additional validation. The evaluation was conducted in a single-question setting and did not account for multi-turn dialogue or integration with RAG systems. The benchmark questions were verified by the participating domain practitioners, and the assessments provided by the LLM judges were compared with independent expert evaluations, as described in Section 1.6. However, no formal, documented verification of expert qualifications was conducted for each of the 16 domains. The experimental series using different generation parameters (Section 2.5) were conducted using a representative subset of models during preliminary tuning of the evaluation methodology. Full validation across all 15 models is left for future work.
Data and Code Availability. To support reproducibility, the set of evaluation tasks with reference answers, the system prompts used in the experiments, instructions provided to the LLM judges, and the source code of the evaluation pipeline can be made available upon reasonable request to the authors. The elements described in this study—including the benchmark design, score aggregation formulas (1)-(5), generation parameters, and the composition of the judge-model panel—are reported in sufficient detail to enable independent reproduction of the methodology on any set of domains and models.
Conclusion
The study presents a reproducible methodology for multi-domain comparative evaluation of open-source large language models on Russian-language professional-domain tasks. The benchmark covers 16 domains and 1,440 questions and employs a two-stage evaluation framework combining multi-judge LLM-as-a-Judge assessment with independent expert validation.
The results demonstrate that recent-generation models in the 27−31B parameter range (Qwen3.5−27B, Gemma-4−31B, and Gemma-4−26B-A4B) can achieve performance comparable to that of a 120B-parameter model (GPT-OSS-120B), with several-fold lower computational resource requirements. Performance degradation across difficulty levels ranges from 0.21 points for Gemma-4−31B to 1.17 points for Calme-2.4-RYS-78B. A pronounced pattern of domain specialization was also observed: no single model ranks first across all 16 domains, supporting the strategy of combining multiple models in multi-agent systems. On the representative subset of models used for preliminary methodology tuning, deterministic generation (temperature = 0) yielded scores approximately 0.2−0.3 points higher than stochastic generation for most of the evaluated models. Qualitative expert validation was generally consistent with the automated assessments provided by the LLM judges.
These findings are of practical value for organizations deploying AI systems based on open-source LLMs, including multi-agent analytics systems, RAG platforms, document workflow automation modules, and medical record analytics systems integrated with medical information systems (MISs). The proposed methodology can be adapted to broader sets of domains, languages, and models.
The results demonstrate that recent-generation models in the 27−31B parameter range (Qwen3.5−27B, Gemma-4−31B, and Gemma-4−26B-A4B) can achieve performance comparable to that of a 120B-parameter model (GPT-OSS-120B), with several-fold lower computational resource requirements. Performance degradation across difficulty levels ranges from 0.21 points for Gemma-4−31B to 1.17 points for Calme-2.4-RYS-78B. A pronounced pattern of domain specialization was also observed: no single model ranks first across all 16 domains, supporting the strategy of combining multiple models in multi-agent systems. On the representative subset of models used for preliminary methodology tuning, deterministic generation (temperature = 0) yielded scores approximately 0.2−0.3 points higher than stochastic generation for most of the evaluated models. Qualitative expert validation was generally consistent with the automated assessments provided by the LLM judges.
These findings are of practical value for organizations deploying AI systems based on open-source LLMs, including multi-agent analytics systems, RAG platforms, document workflow automation modules, and medical record analytics systems integrated with medical information systems (MISs). The proposed methodology can be adapted to broader sets of domains, languages, and models.
References
1. Hendrycks D., Burns C., Basart S. et al. Measuring Massive Multitask Language Understanding // arXiv. 2021. arXiv: 2009.3 300. DOI: 10.48 550/arXiv.2009.3 300. URL: arxiv.org/abs/2009.3 300 (accessed 16 April 2026).
2. Fenogenova A., Chervyakov A., Martynov N. et al. MERA: A Comprehensive LLM Evaluation in Russian // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok: Association for Computational Linguistics, 2024. pp. 9920−9948. DOI: 10.18 653/v1/2024.acl-long.534.
3. Shavrina T., Fenogenova A., Emelyanov A. et al. RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg: Association for Computational Linguistics, 2020. pp. 4717−4726. DOI: 10.18 653/v1/2020.emnlp-main.381.
4. Chiang W.-L., Zheng L., Sheng Y. et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference // Proceedings of the 41st International Conference on Machine Learning (ICML). Vienna: PMLR, 2024. pp. 8359−8388.
5. Zheng L., Chiang W.-L., Sheng Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena // Advances in Neural Information Processing Systems (NeurIPS). 2023. Vol. 36, pp. 46 595−46 623. DOI: 10.48 550/arXiv.2306.5 685.
6. Zhu L., Wang X., Wang X. JudgeLM: Fine-tuned Large Language Models are Scalable Judges // Proceedings of the 13th International Conference on Learning Representations (ICLR). 2025. DOI: 10.48 550/arXiv.2310.17 631.
7. Qwen Team. Qwen3 Technical Report // arXiv. 2025. arXiv: 2505.9 388. DOI: 10.48 550/arXiv.2505.9 388. URL: arxiv.org/abs/2505.9 388 (accessed 22 April 2026).
8. Yang A., Yang B., Hui B. et al. Qwen2 Technical Report // arXiv. 2024. arXiv: 2407.10 671. DOI: 10.48 550/arXiv.2407.10 671. URL: arxiv.org/abs/2407.10 671 (accessed 25 April 2026).
9. Google DeepMind. Gemma 3 Technical Report // arXiv. 2025. arXiv: 2503.19 786. DOI: 10.48 550/arXiv.2503.19 786. URL: arxiv.org/abs/2503.19 786 (accessed 28 April 2026).
10. Abdin M., Aneja J., Behl H. et al. Phi-4 Technical Report // arXiv. 2024. arXiv: 2412.8 905. DOI: 10.48 550/arXiv.2412.8 905. URL: arxiv.org/abs/2412.8 905 (accessed 30 April 2026).
11. Jiang A. Q., Sablayrolles A., Roux A. et al. Mixtral of Experts // arXiv. 2024. arXiv: 2401.4 088. DOI: 10.48 550/arXiv.2401.4 088. URL: arxiv.org/abs/2401.4 088 (accessed 4 May 2026).
12. Adler B., Agarwal N., Aithal A. et al. Nemotron-4 340B Technical Report // arXiv. 2024. arXiv: 2406.11 704. DOI: 10.48 550/arXiv.2406.11 704. URL: arxiv.org/abs/2406.11 704 (accessed 6 May 2026).
13. GLM Team. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools // arXiv. 2024. arXiv: 2406.12 793. DOI: 10.48 550/arXiv.2406.12 793. URL: arxiv.org/abs/2406.12 793 (accessed 8 May 2026).
14. Groeneveld D., Beltagy I., Walsh P. et al. OLMo: Accelerating the Science of Language Models // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok: Association for Computational Linguistics, 2024. pp. 15 789−15 809. DOI: 10.18 653/v1/2024.acl-long.841.
15. Gao L., Tow J., Abbasi B. et al. A Framework for Few-Shot Language Model Evaluation. Zenodo, 2023. DOI: 10.5281/zenodo.10 256 836. URL: doi.org/10.5281/zenodo.10 256 836 (accessed 12 May 2026).
16. Kadavath S., Conerly T., Askell A. et al. Language Models (Mostly) Know What They Know // arXiv. 2022. arXiv: 2207.5 221. DOI: 10.48 550/arXiv.2207.5 221. URL: arxiv.org/abs/2207.5 221 (accessed 14 May 2026).
17. Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. URL: qwen.ai/blog?id=qwen3.5 (accessed 2 June 2026).
18. Google DeepMind. Gemma 4: The most capable open models. 2026. URL: blog.google/innovation-and-ai/technology/developers-tools/gemma-4/ (accessed 3 June 2026).
19. Zhipu AI (Z.ai). GLM-4.7-Flash: An efficient MoE model for local coding and agents. 2026. URL: www.zhipuai.cn/en/news/148 (accessed 4 June 2026).
20. OpenAI. gpt-oss-120b & gpt-oss-20b Model Card // arXiv. 2025. arXiv: 2508.10 925. DOI: 10.48 550/arXiv.2508.10 925. URL: arxiv.org/abs/2508.10 925 (accessed 5 June 2026).
21. DeepSeek-AI. DeepSeek-V4 Technical Report. 2026. URL: huggingface.co/deepseek-ai/DeepSeek-V4-Pro (accessed 6 June 2026).
2. Fenogenova A., Chervyakov A., Martynov N. et al. MERA: A Comprehensive LLM Evaluation in Russian // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok: Association for Computational Linguistics, 2024. pp. 9920−9948. DOI: 10.18 653/v1/2024.acl-long.534.
3. Shavrina T., Fenogenova A., Emelyanov A. et al. RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg: Association for Computational Linguistics, 2020. pp. 4717−4726. DOI: 10.18 653/v1/2020.emnlp-main.381.
4. Chiang W.-L., Zheng L., Sheng Y. et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference // Proceedings of the 41st International Conference on Machine Learning (ICML). Vienna: PMLR, 2024. pp. 8359−8388.
5. Zheng L., Chiang W.-L., Sheng Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena // Advances in Neural Information Processing Systems (NeurIPS). 2023. Vol. 36, pp. 46 595−46 623. DOI: 10.48 550/arXiv.2306.5 685.
6. Zhu L., Wang X., Wang X. JudgeLM: Fine-tuned Large Language Models are Scalable Judges // Proceedings of the 13th International Conference on Learning Representations (ICLR). 2025. DOI: 10.48 550/arXiv.2310.17 631.
7. Qwen Team. Qwen3 Technical Report // arXiv. 2025. arXiv: 2505.9 388. DOI: 10.48 550/arXiv.2505.9 388. URL: arxiv.org/abs/2505.9 388 (accessed 22 April 2026).
8. Yang A., Yang B., Hui B. et al. Qwen2 Technical Report // arXiv. 2024. arXiv: 2407.10 671. DOI: 10.48 550/arXiv.2407.10 671. URL: arxiv.org/abs/2407.10 671 (accessed 25 April 2026).
9. Google DeepMind. Gemma 3 Technical Report // arXiv. 2025. arXiv: 2503.19 786. DOI: 10.48 550/arXiv.2503.19 786. URL: arxiv.org/abs/2503.19 786 (accessed 28 April 2026).
10. Abdin M., Aneja J., Behl H. et al. Phi-4 Technical Report // arXiv. 2024. arXiv: 2412.8 905. DOI: 10.48 550/arXiv.2412.8 905. URL: arxiv.org/abs/2412.8 905 (accessed 30 April 2026).
11. Jiang A. Q., Sablayrolles A., Roux A. et al. Mixtral of Experts // arXiv. 2024. arXiv: 2401.4 088. DOI: 10.48 550/arXiv.2401.4 088. URL: arxiv.org/abs/2401.4 088 (accessed 4 May 2026).
12. Adler B., Agarwal N., Aithal A. et al. Nemotron-4 340B Technical Report // arXiv. 2024. arXiv: 2406.11 704. DOI: 10.48 550/arXiv.2406.11 704. URL: arxiv.org/abs/2406.11 704 (accessed 6 May 2026).
13. GLM Team. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools // arXiv. 2024. arXiv: 2406.12 793. DOI: 10.48 550/arXiv.2406.12 793. URL: arxiv.org/abs/2406.12 793 (accessed 8 May 2026).
14. Groeneveld D., Beltagy I., Walsh P. et al. OLMo: Accelerating the Science of Language Models // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok: Association for Computational Linguistics, 2024. pp. 15 789−15 809. DOI: 10.18 653/v1/2024.acl-long.841.
15. Gao L., Tow J., Abbasi B. et al. A Framework for Few-Shot Language Model Evaluation. Zenodo, 2023. DOI: 10.5281/zenodo.10 256 836. URL: doi.org/10.5281/zenodo.10 256 836 (accessed 12 May 2026).
16. Kadavath S., Conerly T., Askell A. et al. Language Models (Mostly) Know What They Know // arXiv. 2022. arXiv: 2207.5 221. DOI: 10.48 550/arXiv.2207.5 221. URL: arxiv.org/abs/2207.5 221 (accessed 14 May 2026).
17. Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026. URL: qwen.ai/blog?id=qwen3.5 (accessed 2 June 2026).
18. Google DeepMind. Gemma 4: The most capable open models. 2026. URL: blog.google/innovation-and-ai/technology/developers-tools/gemma-4/ (accessed 3 June 2026).
19. Zhipu AI (Z.ai). GLM-4.7-Flash: An efficient MoE model for local coding and agents. 2026. URL: www.zhipuai.cn/en/news/148 (accessed 4 June 2026).
20. OpenAI. gpt-oss-120b & gpt-oss-20b Model Card // arXiv. 2025. arXiv: 2508.10 925. DOI: 10.48 550/arXiv.2508.10 925. URL: arxiv.org/abs/2508.10 925 (accessed 5 June 2026).
21. DeepSeek-AI. DeepSeek-V4 Technical Report. 2026. URL: huggingface.co/deepseek-ai/DeepSeek-V4-Pro (accessed 6 June 2026).