Introduction

博学之 ，审问之 ，慎思之 ，明辨之 ，笃行之。

—— 《礼记 · 中庸》

The Scientific Knowledge Evaluation (SciKnowEval) benchmark for Large Language Models (LLMs) is inspired by the profound principles outlined in the “Doctrine of the Mean” from ancient Chinese philosophy. This benchmark is designed to assess LLMs based on their proficiency in Studying Extensively, Enquiring Earnestly, Thinking Profoundly, Discerning Clearly, and Practicing Assiduously. Each of these dimensions offers a unique perspective on evaluating the capabilities of LLMs in handling scientific knowledge.

L1: Studying Extensively
(Knowledge Coverage)

This dimension evaluates the breadth of an LLM’s knowledge across various scientific domains. It measures the model’s ability to remember and understand a wide range of scientific concepts.

L2: Enquiring Earnestly
(Knowledge Enquiry and Exploration)

This aspect focuses on the LLM’s capacity for deep enquiry and exploration within scientific contexts, such as analyzing scientific texts, identifying key concepts, and questioning relevant information.

L3: Thinking Profoundly
(Knowledge Reflection and Reasoning)

This criterion examines the model’s capacity for critical thinking, logical deduction, numerical calculation, function prediction, and the ability to engage in reflective reasoning to solve problems.

L4: Discerning Clearly
(Knowledge Discernment and Safety Assessment)

This aspect evaluates the LLM’s ability to make correct, secure, and ethical decisions based on scientific knowledge, including assessing the harmfulness and toxicity of information, and understanding the ethical implications and safety concerns related to scientific endeavors.

L5: Practicing Assiduously
(Knowledge Practice and Application)

The final dimension assesses the LLM’s capability to apply scientific knowledge effectively in real-world scenarios, such as analyzing complex scientific problems and creating innovative solutions.

SciKnowEval represents a comprehensive benchmark for assessing the capability of LLMs in processing and utilizing scientific knowledge. It aims to promote the development of scientific LLMs that not only possess extensive knowledge but also demonstrate ethical discernment and practical applicability, ultimately contributing to the advancement of scientific research.

Leaderboards

Last updated: 22 July, 2024

Overall Biology Chemistry Physics Materials
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models Overall Biology Chemistry Material Physics
331 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Claude-3.5-Sonnet-20240620 1 5.36 2.83 3.06 3.42
332 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM GPT-4o-2024-05-13 2 5.20 5.50 4.18 2.83
333 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Qwen2-72B-Inst 3 8.12 7.17 4.88 3.17
334 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM GPT-4-Turbo-2024-04-09 4 8.48 7.17 5.53 5.17
335 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Gemini1.5-Pro-latest 5 7.88 6.58 6.18 7.42
336 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Llama3-70B-Inst 6 8.12 7.13 5.71 7.50
337 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Qwen-Max 7 10.44 8.04 7.65 6.33
338 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Claude3-Sonnet-20240229 8 8.24 9.63 8.24 10.58
339 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM SciKnowMind-7b-v0.1 9 8.56 8.96 11.53 10.00
340 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Qwen2-7B-Inst 10 12.48 13.63 12.06 9.17
341 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Qwen1.5-14B-Chat 11 13.16 12.21 12.35 11.92
342 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM GPT-3.5-Turbo-0125 12 12.52 12.42 12.82 12.92
343 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Llama3-8B-Inst 13 13.12 12.21 14.88 15.75
344 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM ChemDFM-13B 14 15.72 12.67 15.00 16.33
345 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM ChemLLM-20B-Chat 15 14.32 14.83 16.06 17.25
346 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM MolInst-Llama3-8B 16 16.00 15.00 16.06 16.58
347 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Qwen1.5-7B-Chat 17 15.36 15.29 18.12 18.17
348 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Gemma1.1-7B-Inst 18 19.44 18.42 15.94 16.33
349 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Mistral-7B-Inst-v0.2 19 19.60 20.25 14.12 15.83
350 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM ChatGLM3-6B 20 17.04 20.04 20.35 21.17
351 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Galactica-30B 21 17.84 19.63 21.24 19.58
352 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Llama2-13B-Chat 22 18.24 19.92 20.00 21.33
353 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM SciGLM-6B 23 20.40 20.17 20.29 22.83
354 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM ChemLLM-7B-Chat 24 19.60 20.79 22.53 20.25
355 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM Galactica-6.7B 25 21.28 21.04 22.47 21.33
356 user 25/07/2024 02:24 AM user 25/07/2024 02:24 AM LlaSMol-Mistral-7B 26 22.16 22.17 23.47 24.17
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models L1 L2 L3 L4 L5 Avg
103 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Claude-3.5-Sonnet-20240620 3.00 6.13 8.86 3.00 1.60 5.36
104 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM GPT-4o-2024-05-13 4.50 4.25 8.43 5.33 2.40 5.20
105 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Qwen2-72B-Inst 3.50 5.88 13.86 5.33 7.20 8.12
106 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM GPT-4-Turbo-2024-04-09 8.50 8.63 10.71 6.67 6.20 8.48
107 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Gemini1.5-Pro-latest 8.00 7.88 8.14 4.67 9.40 7.88
108 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Llama3-70B-Inst 13.00 5.00 12.00 7.00 6.40 8.12
109 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Qwen-Max 4.50 11.50 11.29 7.33 11.80 10.44
110 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Claude3-Sonnet-20240229 11.00 6.88 12.43 8.00 3.60 8.24
111 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM SciKnowMind-7b-v0.1 2.00 10.38 5.14 11.67 11.20 8.56
112 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Qwen2-7B-Inst 9.50 11.50 14.57 9.67 14.00 12.48
113 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Qwen1.5-14B-Chat 10.50 15.38 12.71 14.67 10.40 13.16
114 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM GPT-3.5-Turbo-0125 7.00 12.00 16.57 9.00 12.00 12.52
115 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Llama3-8B-Inst 14.00 9.38 16.57 12.00 14.60 13.12
116 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM ChemDFM-13B 11.50 15.88 16.43 15.00 16.60 15.72
117 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM ChemLLM-20B-Chat 18.00 10.50 14.14 20.33 15.60 14.32
118 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM MolInst-Llama3-8B 19.50 14.13 11.71 17.67 22.60 16.00
119 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Qwen1.5-7B-Chat 14.50 15.25 19.14 13.67 11.60 15.36
120 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Gemma1.1-7B-Inst 21.50 22.75 16.00 20.00 17.80 19.44
121 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Mistral-7B-Inst-v0.2 16.50 18.38 20.14 18.33 22.80 19.60
122 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM ChatGLM3-6B 17.50 19.63 15.43 15.33 16.00 17.04
123 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Galactica-30B 17.00 19.13 12.00 22.67 21.40 17.84
124 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Llama2-13B-Chat 25.00 16.63 22.71 15.00 13.80 18.24
125 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM SciGLM-6B 21.00 21.63 18.57 20.00 21.00 20.40
126 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM ChemLLM-7B-Chat 22.00 19.63 15.86 22.00 22.40 19.60
127 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM Galactica-6.7B 23.50 22.13 16.57 24.00 24.00 21.28
128 user 25/07/2024 02:20 AM user 25/07/2024 02:20 AM LlaSMol-Mistral-7B 25.50 22.25 19.00 25.67 23.00 22.16
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models L1 L2 L3 L4 L5 Avg
103 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude-3.5-Sonnet-20240620 1.67 3.71 2.14 2.33 3.75 2.83
104 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4o-2024-05-13 2.33 4.86 6.71 10.67 3.00 5.50
105 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-72B-Inst 5.67 4.29 10.86 6.00 7.75 7.17
106 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4-Turbo-2024-04-09 6.67 2.71 11.43 9.67 6.00 7.17
107 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemini1.5-Pro-latest 5.33 6.86 6.29 4.00 9.50 6.58
108 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-70B-Inst 6.00 6.57 8.57 3.67 9.00 7.13
109 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen-Max 7.67 5.43 10.00 7.67 9.75 8.04
110 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude3-Sonnet-20240229 10.67 7.86 12.00 12.00 6.00 9.63
111 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciKnowMind-7b-v0.1 2.33 12.29 2.43 10.67 18.25 8.96
112 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-7B-Inst 15.33 11.71 16.14 8.67 15.00 13.63
113 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-14B-Chat 14.67 11.86 10.14 13.67 13.50 12.21
114 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-3.5-Turbo-0125 14.33 13.00 12.43 12.33 10.00 12.42
115 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-8B-Inst 11.33 11.00 12.86 12.00 14.00 12.21
116 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemDFM-13B 12.00 14.86 12.29 14.00 9.00 12.67
117 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-20B-Chat 15.67 12.57 15.57 22.33 11.25 14.83
118 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM MolInst-Llama3-8B 14.67 14.86 11.43 15.00 21.75 15.00
119 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-7B-Chat 16.33 15.43 13.57 17.33 15.75 15.29
120 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemma1.1-7B-Inst 23.00 21.57 18.14 16.00 11.75 18.42
121 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Mistral-7B-Inst-v0.2 20.33 20.00 21.14 12.67 24.75 20.25
122 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChatGLM3-6B 21.33 21.00 20.43 17.67 18.50 20.04
123 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-30B 12.33 22.43 18.29 22.33 20.50 19.63
124 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama2-13B-Chat 24.67 19.86 21.14 15.00 18.00 19.92
125 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciGLM-6B 21.00 20.86 20.71 21.67 16.25 20.17
126 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-7B-Chat 22.00 21.29 18.43 23.67 21.00 20.79
127 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-6.7B 18.67 23.86 18.00 18.67 25.00 21.04
128 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM LlaSMol-Mistral-7B 25.33 24.71 21.86 26.00 13.00 22.17
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models L1 L2 L3 L4 L5 Avg
103 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude-3.5-Sonnet-20240620 3.50 3.50 2.67 4.50 3.00 3.42
104 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4o-2024-05-13 1.00 2.50 4.33 4.00 1.00 2.83
105 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-72B-Inst 4.50 2.75 2.33 3.00 5.00 3.17
106 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4-Turbo-2024-04-09 5.00 6.75 5.00 4.00 2.00 5.17
107 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemini1.5-Pro-latest 3.50 12.50 4.33 7.50 4.00 7.42
108 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-70B-Inst 6.50 9.75 6.00 6.50 7.00 7.50
109 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen-Max 7.00 5.25 7.67 6.00 6.00 6.33
110 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude3-Sonnet-20240229 10.50 11.00 11.00 10.50 8.00 10.58
111 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciKnowMind-7b-v0.1 5.50 14.50 11.00 2.50 13.00 10.00
112 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-7B-Inst 9.50 9.25 9.67 7.50 10.00 9.17
113 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-14B-Chat 13.00 11.75 11.67 12.00 11.00 11.92
114 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-3.5-Turbo-0125 11.00 14.50 12.33 14.50 9.00 12.92
115 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-8B-Inst 13.00 13.50 19.00 13.00 26.00 15.75
116 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemDFM-13B 17.00 15.50 16.33 16.50 18.00 16.33
117 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-20B-Chat 14.50 12.50 24.00 15.50 25.00 17.25
118 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM MolInst-Llama3-8B 12.50 18.00 17.33 17.50 15.00 16.58
119 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-7B-Chat 17.00 15.50 20.33 18.50 24.00 18.17
120 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemma1.1-7B-Inst 19.00 19.50 12.33 15.50 12.00 16.33
121 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Mistral-7B-Inst-v0.2 19.50 14.25 17.00 14.50 14.00 15.83
122 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChatGLM3-6B 23.00 21.00 20.33 20.00 23.00 21.17
123 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-30B 19.00 22.75 14.33 23.50 16.00 19.58
124 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama2-13B-Chat 23.00 19.75 21.33 23.50 20.00 21.33
125 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciGLM-6B 24.00 22.25 24.33 23.50 17.00 22.83
126 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-7B-Chat 21.00 18.75 21.67 21.00 19.00 20.25
127 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-6.7B 23.00 25.00 13.67 23.50 22.00 21.33
128 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM LlaSMol-Mistral-7B 26.00 24.75 22.00 26.00 21.00 24.17
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models L1 L2 L3 L4 L5 Avg
103 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude-3.5-Sonnet-20240620 1.00 4.50 2.00 3.00 2.67 3.06
104 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4o-2024-05-13 2.00 5.00 4.00 3.50 4.00 4.18
105 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-72B-Inst 4.00 6.67 2.80 1.50 7.33 4.88
106 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-4-Turbo-2024-04-09 5.00 5.67 4.60 8.50 5.00 5.53
107 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemini1.5-Pro-latest 6.00 7.17 5.00 6.00 6.33 6.18
108 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-70B-Inst 7.00 4.50 7.00 4.50 6.33 5.71
109 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen-Max 8.00 6.33 7.40 10.00 9.00 7.65
110 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Claude3-Sonnet-20240229 9.00 9.83 8.20 7.50 5.33 8.24
111 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciKnowMind-7b-v0.1 3.00 13.50 11.00 7.00 14.33 11.53
112 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen2-7B-Inst 10.00 10.83 14.20 13.50 10.67 12.06
113 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-14B-Chat 11.00 12.00 15.00 12.50 9.00 12.35
114 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM GPT-3.5-Turbo-0125 12.00 12.67 18.40 7.00 8.00 12.82
115 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama3-8B-Inst 13.00 15.50 13.20 9.50 20.67 14.88
116 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemDFM-13B 16.00 14.00 15.00 14.50 17.00 15.00
117 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-20B-Chat 18.00 15.00 16.60 18.00 15.33 16.06
118 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM MolInst-Llama3-8B 14.00 17.67 13.20 16.50 18.00 16.06
119 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Qwen1.5-7B-Chat 15.00 15.67 20.00 17.00 21.67 18.12
120 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Gemma1.1-7B-Inst 20.00 16.17 14.00 18.00 16.00 15.94
121 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Mistral-7B-Inst-v0.2 19.00 12.17 14.00 14.50 16.33 14.12
122 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChatGLM3-6B 22.00 20.83 20.40 19.50 19.33 20.35
123 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-30B 17.00 22.67 21.20 22.50 19.00 21.24
124 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Llama2-13B-Chat 25.00 18.50 21.80 21.50 17.33 20.00
125 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM SciGLM-6B 24.00 20.00 20.60 22.50 17.67 20.29
126 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM ChemLLM-7B-Chat 23.00 21.00 25.20 22.50 21.00 22.53
127 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM Galactica-6.7B 21.00 24.33 20.20 24.50 21.67 22.47
128 user 25/07/2024 02:21 AM user 25/07/2024 02:21 AM LlaSMol-Mistral-7B 26.00 24.33 20.80 26.00 23.67 23.47

Datasets

Last updated: 22 July, 2024

Biology Chemistry Physics Materials
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Ability Level Task Name Task Type Data Source #Questions
1 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L1 Biological Literature QA MCQ Literature Corpus 14,869
2 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L1 Protein Property Identification MCQ UniProtKB 1,500
3 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Drug-Drug Relation Extraction RE Bohrium 464
4 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Biomedical Judgment and Interpretation T/F PubMedQA 904
5 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Compound-Disease Relation Extraction RE Bohrium 867
6 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Gene-Disease Relation Extraction RE Bohrium 203
7 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Detailed Understanding MCQ LibreTexts 828
8 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Text Summary GEN LibreTexts 1,291
9 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Hypothesis Verification T/F LibreTexts 619
10 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L2 Reasoning and Interpretation MCQ LibreTexts 647
11 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 Solubility Prediction MCQ PEER, DeepSol 201
12 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 $\beta$-lactamase Activity Prediction MCQ PEER, Envision 209
13 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 Fluorescence Prediction MCQ PEER, Sarkisyan's 205
14 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 GB1 Fitness Prediction MCQ PEER, FLIP 201
15 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 Stability Prediction MCQ PEER, Rocklin's 203
16 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 Protein-Protein Interaction MCQ STRING, SHS27K, SHS148K 205
17 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L3 Biological Calculation MCQ MedMCQA, SciEval, MMLU 60
18 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L4 Biological Harmful QA GEN Self-generated 297
19 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L4 Proteotoxicity Prediction MCQ, T/F UniProtKB 510
20 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L4 Biological Laboratory Safety Test MCQ, T/F LabExam (ZJU) 194
21 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L5 Biological Protocol Procedure Design GEN Protocol Journal 591
22 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L5 Biological Protocol Reagent Design GEN Protocol Journal 565
23 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L5 Protein Captioning GEN UniProtKB 937
24 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L5 Protein Design GEN UniProtKB 860
25 user 22/07/2024 01:23 PM user 22/07/2024 01:23 PM L5 Single Cell Analysis GEN SHARE-seq 300
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Ability Level Task Name Task Type Data Source #Questions
1 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L1 Molecular Name Conversion MCQ PubChem 1,008
2 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L1 Molecular Property Identification MCQ, T/F MoleculeNet 1,625
3 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L1 Chemical Literature QA MCQ Literature Corpus 6,316
4 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Reaction Mechanism Inference MCQ LibreTexts 269
5 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Compound Identification and Properties MCQ LibreTexts 497
6 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Doping Extraction RE NERRE 821
7 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Detailed Understanding MCQ LibreTexts 626
8 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Text Summary GEN LibreTexts 692
9 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Hypothesis Verification T/F LibreTexts 544
10 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L2 Reasoning and Interpretation MCQ LibreTexts 516
11 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Molar Weight Calculation MCQ PubChem 1,042
12 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Molecular Property Calculation MCQ MoleculeNet 740
13 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Molecular Structure Prediction MCQ PubChem 608
14 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Reaction Prediction MCQ USPTO-Mixed 1,122
15 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Retrosynthesis MCQ USPTO-50k 1,122
16 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Balancing Chemical Equation GEN WebQC 535
17 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L3 Chemical Calculation MCQ XieZhi, SciEval, MMLU 269
18 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L4 Chemical Harmful QA GEN Proposition-65, ILO 454
19 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L4 Molecular Toxicity Prediction MCQ, T/F Toxric 870
20 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L4 Chemical Laboratory Safety Test MCQ, T/F LabExam (ZJU) 531
21 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L5 Molecular Captioning GEN ChEBI-20 943
22 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L5 Molecular Generation GEN ChEBI-20 897
23 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L5 Chemical Protocol Procedure Design GEN Protocol Journal 74
24 user 22/07/2024 01:24 PM user 22/07/2024 01:24 PM L5 Chemical Protocol Reagent Design GEN Protocol Journal 129
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Ability Level Task Name Task Type Data Source #Questions
1 user 23/07/2024 04:40 AM user 24/07/2024 04:47 AM L1 Physics Literature QA MCQ Literature Corpus 4,403
2 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L1 Fundamental Physics Exam MCQ SciQ 2,375
3 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L2 Detailed Understanding MCQ Literature Corpus 400
4 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L2 Text Summary GEN Literature Corpus 400
5 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L2 Hypothesis Verification T/F Literature Corpus 400
6 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L2 Reasoning and Interpretation MCQ Literature Corpus 400
7 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L3 High School Physics Calculation MCQ tiku.cn 698
8 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L3 General Physics Calculation MCQ SciEval, SciBench 800
9 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L3 Physics Formula Derivation MCQ Physics Inference Dataset 218
10 user 23/07/2024 04:40 AM user 24/07/2024 04:47 AM L4 Physics Safety QA GEN Nature Portfolio 341
11 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L4 Laboratory Safety Test MCQ LabExam (ZJU) 606
12 user 23/07/2024 04:40 AM user 23/07/2024 04:40 AM L5 Physics Problem Solving GEN Qualifying Exam 302
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Ability Level Task Name Task Type Data Source #Questions
19 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L1 Material Literature QA MCQ Literature Corpus 5534
20 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Chemical Composition Extraction GEN Literature Corpus 203
21 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Digital Data Extraction MCQ Literature Corpus 170
22 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Detailed Understanding MCQ Literature Corpus 400
23 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Text Summary GEN Literature Corpus 400
24 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Hypothesis Verification T/F Literature Corpus 300
25 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L2 Reasoning and Interpretation MCQ Literature Corpus 359
26 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L3 Valence Electron Difference Calculation MCQ Metallic Glass Forming Database 146
27 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L3 Material Calculation MCQ MaScQA 348
28 user 24/07/2024 07:52 AM user 24/07/2024 07:52 AM L3 Lattice Volume Calculation MCQ Materials Project 160

Task Scores

Last updated: 22 July, 2024

Biology Chemistry Physics Materials
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models biology_literature_QA protein_property_identification drug_drug_relation_extraction biomedical_judgment_and_interpretation compound_disease_relation_extraction gene_disease_relation_extraction biological_detailed_understanding biological_text_summary biological_hypothesis_verification biological_reasoning_and_interpretation solubility_prediction beta_lactamase_activity_prediction fluorescence_prediction GB1_ftness_prediction stability_prediction Protein_Protein_Interaction biological_calculation biological_harmful_QA proteotoxicity_prediction biological_laboratory_safety_test biological_procedure_generation biological_reagent_generation protein_description_generation protein_design single_cell_analysis molecule_name_conversion molecular_property_prediction
52 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Claude-3.5-Sonnet-20240620 0.8415 0.3700 0.1307 0.9852 0.1691 0.3929 0.9952 4.8104 0.9547 0.9815 0.4686 0.5369 0.4975 0.2823 0.3088 0.3140 0.5833 0.9933 0.8235 0.8454 3.0711 2.5556 0.1298 0.0143 0.0197 0.8996 0.4363
53 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-4o-2024-05-13 0.8371 0.3420 0.1741 0.9546 0.3770 0.3620 0.9940 4.7701 0.9482 0.9769 0.4831 0.5025 0.5172 0.3254 0.2402 0.3527 0.5833 0.5556 0.8588 0.8667 3.1334 2.5333 0.1219 0.0098 0.0165 0.8665 0.4403
54 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen2-72B-Inst 0.8151 0.4587 0.1555 0.9388 0.3477 0.3197 0.9940 4.7709 0.9385 0.9799 0.5459 0.4975 0.4975 0.2440 0.1520 0.2415 0.5000 0.9327 0.7529 0.8557 2.7782 2.2427 0.1103 0.0027 0.0074 0.6487 0.4055
55 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-4-Turbo-2024-04-09 0.8006 0.3173 0.0973 0.9134 0.2955 0.2359 0.9952 4.8104 0.9676 0.9784 0.5121 0.5123 0.4384 0.2919 0.1471 0.3333 0.5500 0.8721 0.7941 0.7667 2.6638 2.3418 0.1103 0.0074 0.0156 0.7787 0.3821
56 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Gemini1.5-Pro-latest 0.8160 0.2740 0.1687 0.9483 0.3117 0.2382 0.9928 4.3197 0.9450 0.9753 0.5266 0.5123 0.5074 0.2919 0.2059 0.2609 0.5667 1.0000 0.7765 0.7444 2.7851 2.3929 0.0477 0.0039 0.0038 0.8360 0.3770
57 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama3-70B-Inst 0.8047 0.2500 0.1720 0.9176 0.3669 0.4859 0.9928 4.7941 0.9417 0.9784 0.5121 0.5025 0.5025 0.2440 0.2745 0.1981 0.5000 0.9596 0.7686 0.7423 2.5078 2.4000 0.1140 0.0005 0.0138 0.7249 0.3883
58 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen-Max 0.8050 0.4093 0.1227 0.9324 0.1372 0.0230 0.9915 4.8118 0.9385 0.9815 0.5024 0.5025 0.5025 0.1675 0.2892 0.3043 0.4167 0.9091 0.6314 0.8299 2.6944 2.2222 0.0745 0.0032 0.0007 0.6909 0.3698
59 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Claude3-Sonnet-20240229 0.7644 0.2793 0.1602 0.9620 0.3585 0.2918 0.9867 4.7670 0.9498 0.9660 0.4686 0.4384 0.5172 0.2440 0.3039 0.2754 0.3833 1.0000 0.4088 0.7111 2.8614 2.3707 0.1180 0.0082 0.0183 0.6846 0.3450
60 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM SciKnowMind-7b-v0.1 0.8309 0.7480 0.1529 0.9704 0.2672 0.2289 0.9819 4.6471 0.8883 0.9722 0.6860 0.4729 0.5911 0.8325 0.8039 0.6135 0.4167 0.0168 0.5157 0.8454 1.4506 1.0319 0.2009 0.0003 0.0918 0.8262 0.4843
61 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen2-7B-Inst 0.7716 0.2827 0.1455 0.9155 0.2888 0.0779 0.9831 4.8599 0.9094 0.9630 0.5169 0.4877 0.5025 0.1962 0.2696 0.1498 0.4167 0.5488 0.5569 0.8402 2.2010 1.9573 0.1002 0.0003 0.0014 0.4113 0.3218
62 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen1.5-14B-Chat 0.7466 0.3407 0.1256 0.8701 0.2161 0.0535 0.9879 4.5967 0.8932 0.9645 0.4879 0.4926 0.5222 0.3445 0.2598 0.1498 0.3333 0.4916 0.3971 0.6667 2.3102 2.2171 0.1081 0.0002 0.0073 0.4489 0.3406
63 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-3.5-Turbo-0125 0.7667 0.4013 0.1536 0.9187 0.2370 0.2757 0.9771 4.7291 0.8900 0.9552 0.4686 0.4778 0.5025 0.2057 0.2647 0.2560 0.3167 0.9764 0.4706 0.7111 2.1698 2.1060 0.0693 0.0027 0.0054 0.4767 0.3278
64 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama3-8B-Inst 0.7482 0.2560 0.1675 0.8912 0.2890 0.3916 0.9879 4.5944 0.9045 0.9676 0.4928 0.5025 0.4975 0.1914 0.2451 0.2126 0.3333 0.9832 0.3824 0.6444 2.1906 2.2393 0.1120 0.0001 0.0006 0.5538 0.3636
65 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemDFM-13B 0.7187 0.3693 0.1488 0.8448 0.2293 0.1234 0.9831 4.3676 0.8835 0.9460 0.4686 0.5025 0.4975 0.3062 0.2500 0.2222 0.2667 0.7475 0.3794 0.6444 1.8856 1.6564 0.0906 0.0001 0.0032 0.6353 0.3527
66 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemLLM-20B-Chat 0.6746 0.2540 0.1606 0.9641 0.2292 0.2740 0.9879 4.5875 0.9061 0.9552 0.5217 0.5025 0.4975 0.2679 0.2549 0.2367 0.2500 0.0337 0.3353 0.5667 2.0312 1.6003 0.0737 0.0002 0.0021 0.5717 0.3214
67 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM MolInst-Llama3-8B 0.7282 0.2053 0.1486 0.9271 0.5026 0.1038 0.9650 2.9574 0.8948 0.9321 0.5362 0.4778 0.4483 0.2823 0.2892 0.2560 0.4167 0.1515 0.3706 0.6556 1.1317 1.0272 0.0091 0.0001 0.0014 0.5690 0.3495
68 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen1.5-7B-Chat 0.7206 0.2640 0.1693 0.8733 0.2127 0.0874 0.9758 4.6200 0.8560 0.9522 0.4928 0.2759 0.4631 0.1340 0.2255 0.2222 0.3667 0.5488 0.3971 0.6778 2.2340 2.0884 0.1004 0.0004 0.0033 0.4005 0.3495
69 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Gemma1.1-7B-Inst 0.4386 0.2520 0.0110 0.9060 0.0452 0.0078 0.1775 4.0495 0.8689 0.3858 0.5604 0.5025 0.4975 0.2249 0.2206 0.1546 0.3167 0.8687 0.0618 0.4889 2.2582 1.9609 0.0967 0.0000 0.0000 0.3737 0.2952
70 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Mistral-7B-Inst-v0.2 0.7136 0.2627 0.1024 0.0116 0.2613 0.2514 0.9710 3.6176 0.1424 0.9599 0.5459 0.4483 0.1872 0.1435 0.1176 0.1159 0.3500 0.3266 0.3735 0.5889 1.0289 1.0359 0.0156 0.0000 0.0015 0.3871 0.3003
71 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChatGLM3-6B 0.6400 0.2627 0.1304 0.7054 0.1906 0.1355 0.9396 4.2353 0.8074 0.8966 0.4734 0.5025 0.5123 0.2440 0.2304 0.2222 0.3167 0.7542 0.3147 0.6667 1.5303 1.4813 0.1096 0.0001 0.0038 0.2778 0.3195
72 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Galactica-30B 0.7294 0.2480 0.1104 0.8817 0.2107 0.3066 0.9408 1.9133 0.6084 0.9090 0.5700 0.4975 0.5025 0.2488 0.2549 0.2609 0.2667 0.0000 0.2588 0.5778 1.0260 1.1077 0.0171 0.0001 0.0018 0.4364 0.3936
73 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama2-13B-Chat 0.4985 0.1687 0.0923 0.8110 0.2662 0.3685 0.9312 4.3963 0.8722 0.9336 0.4734 0.0000 0.4286 0.0909 0.2500 0.1884 0.2167 0.9865 0.3735 0.4667 1.8839 1.9026 0.1160 0.0001 0.0043 0.2437 0.2594
74 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM SciGLM-6B 0.6165 0.2220 0.0647 0.2608 0.1334 0.1265 0.9601 2.8947 0.6197 0.9167 0.4686 0.6158 0.2069 0.1627 0.2647 0.2077 0.2167 0.4343 0.2206 0.6111 1.0855 1.0940 0.0652 0.0001 0.0008 0.4471 0.2038
75 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemLLM-7B-Chat 0.6096 0.2093 0.1141 0.9261 0.1080 0.1151 0.9287 3.6246 0.8204 0.9090 0.4879 0.4975 0.5025 0.2201 0.2353 0.2560 0.2833 0.0135 0.3088 0.5222 1.1478 1.1721 0.0567 0.0000 0.0000 0.3405 0.3176
76 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Galactica-6.7B 0.6058 0.2053 0.1017 0.8501 0.2416 0.0743 0.8237 1.2765 0.5777 0.7855 0.4831 0.4975 0.4828 0.2201 0.2745 0.2415 0.2333 0.0034 0.2382 0.3556 1.0017 1.0120 0.0333 0.0001 0.0004 0.2706 0.3610
77 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM LlaSMol-Mistral-7B 0.3980 0.1753 0.1318 0.5871 0.0902 0.1255 0.7186 2.5834 0.4223 0.7978 0.1836 0.0197 0.0049 0.1914 0.3725 0.2415 0.3167 0.0000 0.0853 0.2889 1.1361 1.0000 0.0616 0.0000 0.0007 0.2742 0.1891
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models chemical_literature_QA reaction_mechanism_inference compound_identification_and_properties extract_doping chemical_detailed_understanding chemical_text_summary chemical_hypothesis_verification chemical_reasoning_and_interpretation molar_weight_calculation molecular_property_calculation molecule_structure_prediction reaction_prediction retrosynthesis balancing_chemical_equation chemical_calculation chemical_harmful_QA mol_toxicity_prediction chemical_laboratory_safety_test molecule_captioning molecule_generation chemical_procedure_generation chemical_reagent_generation
52 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Claude-3.5-Sonnet-20240620 0.8735 0.9888 0.9879 0.5676 0.9968 4.8032 0.9505 0.9767 0.4568 0.3973 0.4730 0.5535 0.4563 0.4430 0.6766 0.7225 0.6057 0.8249 0.1199 0.7105 2.9459 2.4320
53 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-4o-2024-05-13 0.8575 1.0000 0.9859 0.4994 0.9952 4.7598 0.9486 0.9689 0.2591 0.3473 0.4608 0.4804 0.4118 0.1047 0.5242 0.0154 0.3740 0.8523 0.1390 0.6086 2.9595 2.4240
54 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen2-72B-Inst 0.8420 0.9963 0.9879 0.5743 0.9920 4.7945 0.9376 0.9767 0.2006 0.2743 0.4301 0.2843 0.3387 0.1776 0.5428 0.2974 0.5391 0.7985 0.1198 0.3796 2.7027 2.1680
55 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-4-Turbo-2024-04-09 0.8233 0.9963 0.9859 0.6063 0.9936 4.8017 0.9541 0.9825 0.2774 0.2608 0.3885 0.3610 0.2834 0.0991 0.4647 0.2247 0.3415 0.7764 0.1061 0.5762 2.6486 2.4419
56 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Gemini1.5-Pro-latest 0.8328 0.9926 0.9819 0.5929 0.9936 4.3271 0.9156 0.9612 0.2706 0.3865 0.3897 0.2941 0.3182 0.2673 0.6989 0.8194 0.3902 0.8017 0.0720 0.5729 2.3784 2.3178
57 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama3-70B-Inst 0.8313 0.9814 0.9759 0.5789 0.9856 4.8292 0.9229 0.9650 0.2841 0.3392 0.3713 0.2807 0.2201 0.3290 0.4535 0.5793 0.6172 0.7966 0.0919 0.5287 2.4189 2.1840
58 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen-Max 0.8309 0.9926 0.9759 0.5502 0.9920 4.8090 0.9376 0.9748 0.1891 0.3189 0.3137 0.3993 0.3984 0.2542 0.4758 0.1872 0.5563 0.7797 0.1031 0.3627 2.6892 2.1360
59 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Claude3-Sonnet-20240229 0.8033 0.9926 0.9638 0.5847 0.9856 4.7048 0.8991 0.9592 0.2466 0.2730 0.2439 0.4349 0.3217 0.2804 0.3606 0.7731 0.2520 0.6582 0.1245 0.5321 2.8243 2.1318
60 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM SciKnowMind-7b-v0.1 0.8687 0.9740 0.9618 0.4644 0.9760 4.5890 0.8422 0.9709 0.5931 0.8932 0.7512 0.7513 0.5169 0.2935 0.4015 0.0000 0.5839 0.8079 0.1073 0.2131 1.3014 1.0588
61 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen2-7B-Inst 0.7977 0.9480 0.9598 0.5253 0.9617 4.8509 0.8752 0.9553 0.1871 0.2716 0.3051 0.2077 0.3182 0.1514 0.3866 0.0396 0.5586 0.7834 0.0831 0.2215 2.2162 1.9200
62 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen1.5-14B-Chat 0.7811 0.9777 0.9618 0.4750 0.9696 4.6151 0.8844 0.9592 0.2188 0.3595 0.2917 0.4269 0.3913 0.1551 0.3978 0.0264 0.3089 0.7173 0.1031 0.2446 2.1622 2.1040
63 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM GPT-3.5-Turbo-0125 0.7855 0.9703 0.9396 0.5070 0.9649 4.7453 0.8826 0.9417 0.1900 0.4108 0.3456 0.2647 0.2531 0.2355 0.3606 0.0837 0.2846 0.6878 0.1201 0.5012 2.1622 2.1200
64 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama3-8B-Inst 0.7873 0.9851 0.9577 0.5265 0.9744 4.5398 0.8771 0.9650 0.1996 0.3351 0.2745 0.3957 0.3556 0.1720 0.3457 0.4361 0.2683 0.6751 0.0452 0.3780 2.2568 2.0320
65 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemDFM-13B 0.7724 0.9703 0.9598 0.4729 0.9744 4.3198 0.8569 0.9495 0.1823 0.3108 0.3811 0.6399 0.4332 0.1327 0.3086 0.0374 0.3740 0.5992 0.2530 0.8493 1.7568 1.6320
66 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemLLM-20B-Chat 0.7523 0.9888 0.9638 0.4083 0.9728 4.5760 0.9009 0.9320 0.2015 0.3446 0.2598 0.3084 0.1551 0.1458 0.2974 0.0000 0.2602 0.4684 0.1755 0.5155 2.0676 1.4880
67 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM MolInst-Llama3-8B 0.7457 0.9814 0.9618 0.4976 0.9601 2.9768 0.8624 0.9495 0.2006 0.3189 0.3493 0.2709 0.4055 0.1813 0.4015 0.0154 0.3333 0.6793 0.0236 0.2785 1.0405 1.0775
68 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Qwen1.5-7B-Chat 0.7598 0.9628 0.9316 0.4799 0.9473 4.6006 0.8422 0.9495 0.2131 0.3338 0.3272 0.2799 0.3824 0.1047 0.3383 0.0308 0.2520 0.6245 0.0752 0.2091 2.0946 2.1085
69 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Gemma1.1-7B-Inst 0.4490 0.1673 0.1972 0.4769 0.1709 3.9783 0.8661 0.4272 0.1603 0.2459 0.2770 0.1248 0.2709 0.2505 0.3234 0.2379 0.2358 0.5907 0.1306 0.2226 2.3514 1.7752
70 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Mistral-7B-Inst-v0.2 0.7327 0.9591 0.9155 0.1136 0.9409 3.7757 0.1119 0.9495 0.1881 0.2527 0.2941 0.0588 0.0125 0.0000 0.3309 0.1079 0.3821 0.5570 0.0333 0.1328 1.0638 1.0000
71 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChatGLM3-6B 0.6937 0.9108 0.8431 0.3758 0.9105 4.1447 0.7945 0.8796 0.2399 0.2297 0.2586 0.2094 0.0588 0.0916 0.2862 0.0441 0.2439 0.5865 0.0921 0.1522 1.4459 1.6279
72 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Galactica-30B 0.7663 0.9219 0.8189 0.3919 0.8946 2.0275 0.5596 0.8835 0.2514 0.2595 0.2586 0.0098 0.1622 0.2187 0.2862 0.0110 0.2114 0.5148 0.0375 0.2401 1.0811 1.1680
73 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Llama2-13B-Chat 0.5058 0.9219 0.8410 0.4558 0.8850 4.2967 0.8073 0.9165 0.2524 0.2608 0.1164 0.0357 0.0499 0.0822 0.2119 0.4537 0.2764 0.3586 0.0747 0.1662 1.8378 1.8720
74 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM SciGLM-6B 0.6657 0.9257 0.8974 0.2774 0.9249 2.8567 0.5908 0.8913 0.2063 0.3000 0.2365 0.0187 0.0062 0.1121 0.2788 0.0154 0.2439 0.5021 0.1228 0.2984 1.0135 1.2320
75 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM ChemLLM-7B-Chat 0.6617 0.9071 0.8491 0.2095 0.9042 3.5326 0.8055 0.8971 0.2361 0.2851 0.2549 0.0428 0.2201 0.1364 0.2862 0.0000 0.2276 0.5063 0.0480 0.1983 1.1081 1.1680
76 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM Galactica-6.7B 0.7038 0.7546 0.7002 0.4199 0.7732 1.2173 0.0000 0.7553 0.2495 0.2514 0.3064 0.2888 0.1649 0.0523 0.2379 0.0396 0.2683 0.4177 0.0351 0.1314 1.0000 1.0240
77 user 25/07/2024 02:22 AM user 25/07/2024 02:22 AM LlaSMol-Mistral-7B 0.3911 0.6431 0.6258 0.1857 0.6214 2.4916 0.4532 0.7437 0.2495 0.1959 0.2059 0.0134 0.0027 0.0037 0.3383 0.0000 0.2033 0.2743 0.2600 0.6840 1.0769 1.0000
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models physics_literature_QA fundamental_physics_exam physics_detailed_understanding physics_text_summary physics_hypothesis_verification physics_reasoning_and_interpretation high_school_physics_calculation general_physics_calculation physics_formula_derivation physics_safety_QA physics_laboratory_safety_test physics_problem_solving
77 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Claude-3.5-Sonnet-20240620 0.8617 0.9453 0.9950 4.8225 0.9850 0.9975 0.5716 0.4963 4.8257 0.8830 0.7805 3.7616
78 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-4o-2024-05-13 0.8756 0.9659 0.9950 4.8775 0.9875 0.9975 0.5745 0.3962 4.8624 0.8801 0.7871 3.8576
79 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen2-72B-Inst 0.8429 0.9524 0.9975 4.8900 0.9850 0.9950 0.6705 0.4500 4.8119 0.8743 0.8152 3.0563
80 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-4-Turbo-2024-04-09 0.8322 0.9558 0.9925 4.8800 0.9750 0.9925 0.5444 0.4188 4.7982 0.8509 0.7987 3.7682
81 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Gemini1.5-Pro-latest 0.8474 0.9558 0.9925 4.4150 0.9725 0.9825 0.6046 0.4662 4.4495 0.8480 0.7789 3.5728
82 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama3-70B-Inst 0.8404 0.9398 0.9900 4.8300 0.9750 0.9900 0.4599 0.4313 4.6101 0.8772 0.7558 2.7781
83 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen-Max 0.8236 0.9406 0.9925 4.9200 0.9775 0.9950 0.6117 0.3175 4.7936 0.8304 0.7970 2.9603
84 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Claude3-Sonnet-20240229 0.7891 0.9048 0.9925 4.7925 0.9600 0.9875 0.3711 0.3337 4.5688 0.8129 0.7541 2.6391
85 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM SciKnowMind-7b-v0.1 0.8590 0.9234 0.9925 4.4125 0.9375 0.9900 0.4441 0.3488 3.3303 0.9006 0.7954 1.4238
86 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen2-7B-Inst 0.7832 0.9124 0.9925 4.7650 0.9600 0.9925 0.4341 0.3550 4.2339 0.8304 0.7822 1.8477
87 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen1.5-14B-Chat 0.7579 0.9023 0.9900 4.7725 0.9575 0.9900 0.4771 0.3075 3.7752 0.8099 0.7327 1.7682
88 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-3.5-Turbo-0125 0.7600 0.9069 0.9850 4.6825 0.9400 0.9900 0.3367 0.3475 4.0505 0.7661 0.7178 1.9272
89 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama3-8B-Inst 0.7761 0.8926 0.9925 1.0200 0.9525 0.9950 0.3009 0.3475 1.2202 0.8129 0.7030 1.0099
90 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemDFM-13B 0.7228 0.8893 0.9925 4.3675 0.9275 0.9875 0.4226 0.2838 1.4725 0.7661 0.6799 1.0894
91 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemLLM-20B-Chat 0.7386 0.8931 0.9875 4.5000 0.9700 0.9900 0.2908 0.1850 1.1330 0.7632 0.7244 1.0199
92 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM MolInst-Llama3-8B 0.7550 0.9061 0.9900 2.5100 0.9400 0.9725 0.3066 0.3300 1.7661 0.7632 0.6815 1.2185
93 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen1.5-7B-Chat 0.7296 0.8644 0.9825 4.6925 0.9175 0.9900 0.3496 0.2387 1.0046 0.7632 0.6749 1.0232
94 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Gemma1.1-7B-Inst 0.6794 0.8531 0.9800 3.8975 0.9225 0.9650 0.3195 0.3613 3.3486 0.7339 0.7376 1.5497
95 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Mistral-7B-Inst-v0.2 0.7123 0.8008 0.9775 4.7550 0.9475 0.9900 0.3080 0.3038 2.4725 0.7719 0.6931 1.3775
96 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChatGLM3-6B 0.6358 0.7604 0.9700 4.2725 0.8675 0.9675 0.3109 0.2425 1.2798 0.7456 0.6271 1.0298
97 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Galactica-30B 0.7323 0.7693 0.9725 1.5000 0.5350 0.9525 0.3009 0.3987 2.2661 0.6754 0.3812 1.1490
98 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama2-13B-Chat 0.4332 0.7781 0.8975 4.4275 0.8825 0.9800 0.1862 0.2537 1.6560 0.4035 0.5990 1.0695
99 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM SciGLM-6B 0.6129 0.7200 0.9650 2.9425 0.8850 0.9375 0.2364 0.2213 1.1250 0.6228 0.4637 1.1074
100 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemLLM-7B-Chat 0.6358 0.8101 0.9700 2.9850 0.9500 0.9700 0.3352 0.2325 1.0000 0.6959 0.6271 1.0791
101 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Galactica-6.7B 0.6533 0.6623 0.9025 1.2550 0.4150 0.8300 0.3138 0.3962 2.0000 0.5936 0.5281 1.0430
102 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM LlaSMol-Mistral-7B 0.3597 0.2943 0.8000 2.6624 0.1600 0.9025 0.1232 0.2587 1.3807 0.3158 0.2096 1.0503
wdt_ID wdt_created_by wdt_created_at wdt_last_edited_by wdt_last_edited_at Models material_literature_QA material_component_extraction material_data_extraction material_detailed_understanding material_text_summary material_hypothesis_verification material_reasoning_and_interpretation valence_electron_difference_calculation material_calculation lattice_volume_calculation perovskite_stability_prediction diffusion_rate_analysis material_safety_QA material_toxicity_prediction property_and_usage_analysis crystal_structure_and_composition_analysis specified_band_gap_material_generation
77 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Claude-3.5-Sonnet-20240620 0.7765 0.5431 0.9588 0.9025 4.7800 0.8300 0.9916 0.6027 0.5489 0.7188 0.4896 0.7852 0.8870 0.6683 2.7966 1.2041 2.8267
78 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-4o-2024-05-13 0.7647 0.5616 0.8882 0.9025 4.8200 0.9000 0.9833 0.4658 0.3793 0.5312 0.6188 0.6913 0.8621 0.6715 2.6780 1.2908 2.5133
79 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen2-72B-Inst 0.7423 0.4791 0.9176 0.8975 4.7900 0.8133 0.9972 0.4521 0.4195 0.5625 0.5333 0.7987 0.8704 0.7008 2.6525 1.1633 2.5100
80 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-4-Turbo-2024-04-09 0.7407 0.4889 0.9176 0.9025 4.7875 0.8700 0.9889 0.3973 0.4770 0.5375 0.5167 0.6577 0.8383 0.6374 2.5932 1.2857 2.7067
81 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Gemini1.5-Pro-latest 0.7385 0.6084 0.9353 0.8975 4.3975 0.7900 0.9833 0.4795 0.5029 0.4188 0.4396 0.7785 0.8526 0.6520 2.8390 1.1837 2.2833
82 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama3-70B-Inst 0.7295 0.6786 0.9235 0.9025 4.7725 0.7133 0.9944 0.4315 0.3764 0.4500 0.4854 0.4497 0.8490 0.6764 2.6525 1.2551 2.5033
83 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen-Max 0.7161 0.5209 0.8941 0.8975 4.8250 0.8500 0.9833 0.3904 0.3391 0.5000 0.5021 0.4497 0.8407 0.6081 2.4831 1.1786 2.4567
84 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Claude3-Sonnet-20240229 0.7085 0.6207 0.8824 0.8925 4.6025 0.6867 0.9554 0.4658 0.3506 0.3937 0.3563 0.6980 0.8335 0.6650 2.7458 1.1276 2.6967
85 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM SciKnowMind-7b-v0.1 0.7454 0.4828 0.8471 0.9000 4.3125 0.5367 0.9805 0.3699 0.2931 0.4313 0.3875 0.3691 0.8609 0.6309 2.6949 1.0357 1.3867
86 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen2-7B-Inst 0.6877 0.5825 0.8118 0.8900 4.7425 0.6833 0.9805 0.3082 0.2500 0.3937 0.3542 0.4362 0.7895 0.5919 2.7119 1.0714 2.1267
87 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen1.5-14B-Chat 0.6789 0.5431 0.8412 0.8750 4.7175 0.6900 0.9638 0.2808 0.3218 0.4250 0.3187 0.3087 0.8014 0.5593 2.7542 1.0510 2.4367
88 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM GPT-3.5-Turbo-0125 0.6706 0.5185 0.8588 0.8725 4.5375 0.7600 0.9415 0.3151 0.2759 0.2313 0.3542 0.2148 0.7919 0.6846 2.6186 1.2551 2.3333
89 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama3-8B-Inst 0.6690 0.0000 0.9059 0.8875 1.1825 0.6500 0.9833 0.2877 0.2759 0.5000 0.3542 0.3893 0.8002 0.6390 1.0763 1.1173 1.0000
90 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemDFM-13B 0.6523 0.5185 0.8235 0.8750 4.3050 0.7200 0.9610 0.4110 0.2213 0.3312 0.3292 0.4161 0.7634 0.6081 2.3305 1.0204 2.0733
91 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemLLM-20B-Chat 0.6446 0.0468 0.8471 0.9025 4.4675 0.4500 0.9666 0.2945 0.2241 0.3187 0.3146 0.4564 0.7646 0.4878 1.9237 1.0918 2.0100
92 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM MolInst-Llama3-8B 0.6534 0.4113 0.8471 0.8850 2.4575 0.5133 0.9471 0.3699 0.2270 0.4437 0.3563 0.3624 0.7610 0.5480 1.4915 1.0816 1.2867
93 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Qwen1.5-7B-Chat 0.6531 0.3953 0.8235 0.8525 4.5475 0.6467 0.9694 0.1986 0.1954 0.4125 0.3417 0.2483 0.7872 0.5008 1.1525 1.0306 1.2767
94 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Gemma1.1-7B-Inst 0.6113 0.6527 0.8118 0.8700 3.7425 0.5667 0.9304 0.3219 0.2845 0.3625 0.3458 0.3691 0.7325 0.5268 2.4068 1.0000 2.4433
95 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Mistral-7B-Inst-v0.2 0.6297 0.5123 0.8706 0.8775 4.7000 0.7433 0.9387 0.3219 0.2787 0.3187 0.4292 0.3691 0.7515 0.6163 2.5424 1.0000 2.3167
96 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChatGLM3-6B 0.5593 0.1330 0.6118 0.8225 4.0975 0.6267 0.9164 0.2808 0.2155 0.3125 0.3125 0.2685 0.7229 0.5089 2.1102 1.0153 1.4867
97 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Galactica-30B 0.6456 0.4002 0.5882 0.8425 1.6598 0.2667 0.8914 0.2466 0.2816 0.2562 0.2687 0.2081 0.6778 0.3593 1.1795 1.1020 1.1933
98 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Llama2-13B-Chat 0.4387 0.5505 0.7059 0.7725 4.2475 0.5200 0.9387 0.2260 0.2241 0.3438 0.2771 0.1812 0.4185 0.5187 2.1864 1.0255 2.0067
99 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM SciGLM-6B 0.5365 0.4828 0.7647 0.8425 2.9223 0.5933 0.8830 0.2466 0.2471 0.3000 0.2729 0.2550 0.6409 0.4683 1.9068 1.0459 1.6633
100 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM ChemLLM-7B-Chat 0.5492 0.0012 0.6176 0.8375 3.3000 0.6633 0.9053 0.2329 0.2069 0.0563 0.2562 0.1611 0.7170 0.3041 1.9661 1.0204 1.0833
101 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM Galactica-6.7B 0.5799 0.2561 0.4471 0.7875 1.2323 0.2300 0.7521 0.2808 0.2816 0.2687 0.2646 0.2349 0.5672 0.2992 1.2143 1.0357 1.0333
102 user 25/07/2024 02:23 AM user 25/07/2024 02:23 AM LlaSMol-Mistral-7B 0.3150 0.0254 0.7059 0.6775 2.6624 0.1933 0.6964 0.2329 0.3851 0.0563 0.2354 0.2148 0.3341 0.2065 1.4576 1.0051 1.0100

Submission

Upload your results <json file>

    FAQ

    Where can I download the SciKnowEval dataset?

    The dataset can be found in our GitHub.

    What’s the format of uploaded results?

    The examples of results can be found in our GitHub.

    How long will it take for the results to appear in the leaderboard?

    The evaluation will take some time after submission. Please be patient, usually less than a week.

    How to contact us?

    Please contact Mr. Junjie Huang,   junjie6282@zju.edu.cn