What can and can't language models do? Lessons learned from BIGBench

So what exactly can and can’t language models do? What's the least impressive thing GPT-4 won't be able to do? What will GPT-4 be incapable of? BIGBench is kind of a way to figure this out. BigBench, aka “The Beyond the Imitation Game” Benchmark, is an attempt to explore the capabilities of large language models over a wide variety of tasks. All the tasks are enumerated here. I looked through every BIGBench task and took the ones that compared both GPT3 and PaLM against humans. * Spreadsheet

444 Authors From 132 Institutions Release BIG-bench: A 204-Task

Large language models encode clinical knowledge

📈 Chartpack: Measuring AI (3/3)

Train foundation model for domain-specific language model

DeWeese Lab (@DeWeeseLab) / X

PDF) Language Models Don't Always Say What They Think: Unfaithful

When training AI, we should escalate the frequency of capability

Dual Process Theory for Large Language Models: An overview of

Inverse scaling can become U-shaped — AI Alignment Forum