arxiv_cl 90% Match Research Paper NLP researchers,ML engineers,Data scientists,Developers of multilingual AI systems 2 weeks ago

DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection

large-language-models › training-methods

📄 Abstract

Abstract: The rapid development of multilingual large language models (LLMs) highlights the need for high-quality, diverse, and well-curated multilingual datasets. In this paper, we introduce DCAD-2000 (Data Cleaning as Anomaly Detection), a large-scale multilingual corpus constructed from newly extracted Common Crawl data and existing multilingual sources. DCAD-2000 covers 2,282 languages, 46.72TB of text, and 8.63 billion documents, spanning 155 high- and medium-resource languages and 159 writing scripts. To overcome the limitations of existing data cleaning approaches, which rely on manually designed heuristic thresholds, we reframe data cleaning as an anomaly detection problem. This dynamic filtering paradigm substantially improves data quality by automatically identifying and removing noisy or anomalous content. By fine-tuning LLMs on DCAD-2000, we demonstrate notable improvements in data quality, robustness of the cleaning pipeline, and downstream performance, particularly for low-resource languages across multiple multilingual benchmarks.

Authors (7)

Yingli Shen

Wen Lai

Shuo Wang

Xueren Zhang

Kangyang Luo

Alexander Fraser

+1 more

Submitted

February 17, 2025

arXiv Category

cs.CL

arXiv PDF

Key Contributions

Introduces DCAD-2000, a massive multilingual dataset (2,282 languages, 46.72TB) constructed using a novel 'Data Cleaning as Anomaly Detection' approach. This method dynamically filters noisy content, significantly improving data quality and demonstrating downstream performance gains for LLMs.

Business Value

Provides a foundational resource for developing more capable and equitable multilingual AI systems, accelerating research and application development across diverse linguistic communities.

Paper Metadata

Innovation Type

Dataset Creation and Methodology

Deployment Feasibility

High, as it provides a dataset resource.

Limitations Addressed

The limitations of existing data cleaning methods that rely on static, manually designed heuristics, and the need for large, high-quality multilingual datasets for LLMs.

Performance Gains

Demonstrates notable improvements in data quality, robustness, and downstream performance for LLMs fine-tuned on DCAD-2000.

Technical Tags

multilingual datasetslarge language models (LLMs)data cleaninganomaly detectionCommon Crawlcorpus constructiondata qualityrobustnessdownstream performancedynamic filteringheuristic thresholds

Research Topics

Data CurationMultilingual NLPLarge Language ModelsData MiningAnomaly Detection

Methods & Architectures

DCAD-2000 datasetData Cleaning as Anomaly Detection (DCAD)dynamic filtering paradigmanomaly detection techniquesCommon Crawl data extraction Large Language Models (LLMs)

Applications & Tasks

Natural Language Processing Machine Learning Need for high-quality, diverse multilingual datasetsLimitations of existing data cleaning approachesNoisy or anomalous content in web data Constructing a large-scale multilingual corpusImproving data quality through anomaly detectionEnhancing LLM performance on multilingual tasks

Datasets & Benchmarks

Datasets

DCAD-2000, Common Crawl

data qualityrobustness of cleaning pipelinedownstream performance

Related Fields

Data ScienceNatural Language ProcessingMachine LearningBig Data

Keywords

multilingual datasetLLMdata cleaninganomaly detectionCommon CrawlcorpusNLPdata qualityrobustnessDCAD-2000web data

Academic Context

#Data Curation#Multilingual NLP#Large Language Models#Data Mining#Anomaly Detection

Commercial Potential

Potential Products

Foundation models for multilingual NLPData curation servicesTools for building high-quality datasets

Target Industries

TechnologyAI ResearchGlobal CommunicationsContent Moderation

Use Case Examples

Training LLMs for translation across many languagesDeveloping chatbots for diverse linguistic marketsImproving content analysis tools for global platforms

Competitive Edge

Offers a significantly larger and more linguistically diverse dataset than many existing multilingual corpora, coupled with a novel, automated data cleaning methodology that surpasses traditional heuristic approaches.

Market Opportunity

Very large, essential for the development of global AI.

Revenue Models

Access fees for the datasetlicensing for commercial useservices based on the dataset.

Resource Requirements

Compute Needs

Significant compute resources required for constructing and processing the dataset.

Data Requirements

Leverages Common Crawl data and existing multilingual sources.

Deployment Constraints

Storage and management of large datasets,Ensuring ethical data sourcing and usage

Scalability

The methodology is designed for large-scale data processing.

Regulatory Considerations

Data privacy regulations (e.g., GDPR)Copyright considerations for web-scraped data

Production Readiness

Maturity Level

Dataset Release

Time to Market

Immediate availability as a resource.

Patent Potential

Low, focused on data curation methodology.

View Full Paper Back to Papers