Surge AI Blog

Featured

30% of Google's Emotions Dataset is Mislabeled

Edwin Chen
Edwin Chen
AI

Last year, Google released their “GoEmotions” dataset: a human-labeled dataset of 58K Reddit comments categorized according to 27 emotions. The problem? A whopping 30% of the dataset is mislabeled! Check out some of the egregious errors, and learn how to build better datasets.

Read more

Recent 
articles

The latest in AI, language, and data labeling. Subscribe below!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Real-World ML Failures: The Violence, Racism, and Sexism Uncaught by Twitter's Content Moderation Systems

Real-World ML Failures: The Violence, Racism, and Sexism Uncaught by Twitter's Content Moderation Systems

Scott Heiner
Scott Heiner
Social Media
Nov 9, 2022
How TikTok is Evolving the Next Generation of Search

How TikTok is Evolving the Next Generation of Search

Edwin Chen
Edwin Chen
Social Media
Oct 25, 2022
Evaluating Generative AI: Did Astral Codex Ten Win His Bet on AI Progress?

Evaluating Generative AI: Did Astral Codex Ten Win His Bet on AI Progress?

Edwin Chen
Edwin Chen
AI
Sep 29, 2022
Hate Speech Datasets in English, Spanish, Japanese, Arabic, and More

Hate Speech Datasets in English, Spanish, Japanese, Arabic, and More

Bradley Webb
Bradley Webb
Datasets
Sep 20, 2022
Free Toxicity Dataset (September 2022)

Free Toxicity Dataset (September 2022)

Scott Heiner
Scott Heiner
Datasets
Sep 17, 2022
Why Instagram is Losing Gen Z: We Asked 100 Users to Compare TikTok vs. Reels

Why Instagram is Losing Gen Z: We Asked 100 Users to Compare TikTok vs. Reels

Andrew Mauboussin
Andrew Mauboussin
Social Media
Aug 31, 2022
The $250K Inverse Scaling Prize and Human-AI Alignment

The $250K Inverse Scaling Prize and Human-AI Alignment

Edwin Chen
Edwin Chen
AI
Aug 15, 2022
Labeled Dataset of Misinformation on Facebook

Labeled Dataset of Misinformation on Facebook

Jefferson Lee
Jefferson Lee
Datasets
Aug 1, 2022
Dataset of Financial Transactions, Labeled with Intent and Expense Category

Dataset of Financial Transactions, Labeled with Intent and Expense Category

Andrew Mauboussin
Andrew Mauboussin
Datasets
Jul 27, 2022
Human Evaluation of Large Language Models: How Good is Hugging Face's BLOOM?

Human Evaluation of Large Language Models: How Good is Hugging Face's BLOOM?

Edwin Chen
Edwin Chen
AI
Jul 19, 2022
30% of Google's Emotions Dataset is Mislabeled

30% of Google's Emotions Dataset is Mislabeled

Edwin Chen
Edwin Chen
AI
Jul 11, 2022
Search Behind-the-Scenes: How Neeva Uses Human Evaluation to Measure Search Quality

Search Behind-the-Scenes: How Neeva Uses Human Evaluation to Measure Search Quality

Edwin Chen
Edwin Chen
Case Studies
Jun 29, 2022
Generating Children's Stories Using GPT-3 and DALL·E

Generating Children's Stories Using GPT-3 and DALL·E

Edwin Chen
Edwin Chen
AI
Jun 29, 2022
AI Red Teams and Adversarial Data Labeling with Redwood Research

AI Red Teams and Adversarial Data Labeling with Redwood Research

Andrew Mauboussin
Andrew Mauboussin
Case Studies
Jun 28, 2022
Dataset of Roe v. Wade Tweets Labeled by Abortion Stance

Dataset of Roe v. Wade Tweets Labeled by Abortion Stance

Bradley Webb
Bradley Webb
Datasets
Jun 28, 2022
Case Study: Content Moderation for a Leading Social Media Company

Case Study: Content Moderation for a Leading Social Media Company

Jefferson Lee
Jefferson Lee
Case Studies
Jun 23, 2022
Humans vs. Gary Marcus vs. Slate Star Codex: When is an AI failure actually a failure?

Humans vs. Gary Marcus vs. Slate Star Codex: When is an AI failure actually a failure?

Edwin Chen
Edwin Chen
AI
Jun 22, 2022
Dataset: Reddit Crypto Sentiment

Dataset: Reddit Crypto Sentiment

Bradley Webb
Bradley Webb
Datasets
Jun 14, 2022
I wanted burritos. Facebook's search engine sent me to a dead restaurant 45 minutes away.

I wanted burritos. Facebook's search engine sent me to a dead restaurant 45 minutes away.

Jefferson Lee
Jefferson Lee
Human Evaluation
Jun 13, 2022
How We Built OpenAI's GSM8K Dataset of 8,500 Math Problems

How We Built OpenAI's GSM8K Dataset of 8,500 Math Problems

Edwin Chen
Edwin Chen
AI
Jun 13, 2022
New Data Labeling Tools for Training Chatbots and Conversational Assistants

New Data Labeling Tools for Training Chatbots and Conversational Assistants

Andrew Mauboussin
Andrew Mauboussin
Product
Jun 8, 2022
Dataset: Stock Market Sentiment on Social Media

Dataset: Stock Market Sentiment on Social Media

Bradley Webb
Bradley Webb
Datasets
Jun 8, 2022
10 Egregious Failures in Gmail Spam Detection

10 Egregious Failures in Gmail Spam Detection

Edwin Chen
Edwin Chen
Content Moderation
May 24, 2022
How to Pick the Best Data Labeling Platform

How to Pick the Best Data Labeling Platform

Scott Heiner
Scott Heiner
Product
May 21, 2022
Is Elon right? We labeled 500 Twitter users to measure the amount of Spam

Is Elon right? We labeled 500 Twitter users to measure the amount of Spam

Edwin Chen
Edwin Chen
Content Moderation
May 19, 2022
We asked 100 humans to draw the DALL·E prompts

We asked 100 humans to draw the DALL·E prompts

Edwin Chen
Edwin Chen
AI
May 12, 2022
The average number of ads on a Google Search recipe? 8.7

The average number of ads on a Google Search recipe? 8.7

Bradley Webb
Bradley Webb
Engineering
Apr 29, 2022
Google Search is Falling Behind

Google Search is Falling Behind

Edwin Chen
Edwin Chen
Engineering
Apr 12, 2022
Building a No-Code Machine Learning Model by Chatting with GitHub Copilot

Building a No-Code Machine Learning Model by Chatting with GitHub Copilot

Andrew Mauboussin
Andrew Mauboussin
Engineering
Mar 24, 2022
Surge AI Achieves SOC 2 Compliance

Surge AI Achieves SOC 2 Compliance

Andrew Mauboussin
Andrew Mauboussin
Engineering
Feb 25, 2022
Writing a Super Bowl Worthy Commercial with GPT-3

Writing a Super Bowl Worthy Commercial with GPT-3

Jefferson Lee
Jefferson Lee
Engineering
Feb 16, 2022
Moving Beyond Engagement: Optimizing Facebook's Algorithms for Human Values

Moving Beyond Engagement: Optimizing Facebook's Algorithms for Human Values

Andrew Mauboussin
Andrew Mauboussin
Social Media
Feb 10, 2022
Using Surge AI with a Hybrid Cloud Model

Using Surge AI with a Hybrid Cloud Model

Andrew Mauboussin
Andrew Mauboussin
Engineering
Feb 4, 2022
Holy $#!t: Are popular toxicity models simply profanity detectors?

Holy $#!t: Are popular toxicity models simply profanity detectors?

Edwin Chen
Edwin Chen
Content Moderation
Jan 22, 2022
Surge AI's Free Toxicity Dataset

Surge AI's Free Toxicity Dataset

Bradley Webb
Bradley Webb
Content Moderation
Jan 21, 2022
An Analysis of Omicron Tweets: 30% are Skeptical of the Medical Establishment

An Analysis of Omicron Tweets: 30% are Skeptical of the Medical Establishment

Jefferson Lee
Jefferson Lee
Engineering
Jan 21, 2022
Is Google Search Deteriorating? Measuring Google's Search Quality in 2022

Is Google Search Deteriorating? Measuring Google's Search Quality in 2022

Edwin Chen
Edwin Chen
Human Evaluation
Jan 10, 2022
The World's Largest Profanity List

The World's Largest Profanity List

Scott Heiner
Scott Heiner
Datasets
Jan 8, 2022
Inter-Annotator Agreement: An Introduction to Krippendorff’s Alpha

Inter-Annotator Agreement: An Introduction to Krippendorff’s Alpha

Andrew Mauboussin
Andrew Mauboussin
Engineering
Dec 29, 2021
Why I Joined Surge AI

Why I Joined Surge AI

Bradley Webb
Bradley Webb
Company
Dec 17, 2021
How Good is Your Chatbot? An Introduction to Perplexity in NLP

How Good is Your Chatbot? An Introduction to Perplexity in NLP

Engineering
Dec 9, 2021
ML School: Neural Networks

ML School: Neural Networks

Bradley Webb
Bradley Webb
Product
Dec 2, 2021
Inter-Annotator Agreement: An Introduction to Cohen's Kappa Statistic

Inter-Annotator Agreement: An Introduction to Cohen's Kappa Statistic

Bradley Webb
Bradley Webb
Engineering
Nov 30, 2021
5 Examples of the Importance of Context-Sensitivity in Data-Centric AI

5 Examples of the Importance of Context-Sensitivity in Data-Centric AI

Edwin Chen
Edwin Chen
Datasets
Nov 19, 2021
The Pitfalls of Inter-Rater Reliability in Data Labeling and Machine Learning

The Pitfalls of Inter-Rater Reliability in Data Labeling and Machine Learning

Scott Heiner
Scott Heiner
Engineering
Nov 13, 2021
A Visual Introduction to Language Models in NLP (Part 1: Intuition)

A Visual Introduction to Language Models in NLP (Part 1: Intuition)

Andrew Mauboussin
Andrew Mauboussin
Engineering
Nov 4, 2021
How Surge AI Helps NYU Study the Impact of Social Media

How Surge AI Helps NYU Study the Impact of Social Media

Bradley Webb
Bradley Webb
Case Studies
Oct 9, 2021
Manifold: Using Neural Embeddings to Explore the Shape of Your Data

Manifold: Using Neural Embeddings to Explore the Shape of Your Data

Jefferson Lee
Jefferson Lee
Product
Sep 18, 2021
Getting started with Surge AI's Python SDK

Getting started with Surge AI's Python SDK

Andrew Mauboussin
Andrew Mauboussin
Engineering
Aug 2, 2021
Supercharge your NLP with Surge AI’s NER labeling tool

Supercharge your NLP with Surge AI’s NER labeling tool

Jefferson Lee
Jefferson Lee
Product
Aug 2, 2021
The AI Bottleneck: High-Quality, Human-Powered Data

The AI Bottleneck: High-Quality, Human-Powered Data

Edwin Chen
Edwin Chen
Product
Aug 2, 2021