Datasets
DATASETHugging Face
4/5

PanLex Language Vocabulary (lbourdois/panlex)

Beja(bej)TextMTNorth Africa
View Source

Dataset Info

Language
Beja (bej)
Modality
Text
Domain
General
Size
0
Institution
PanLex / lbourdois
Last Updated
19 May 2026

License & Access

SPDX identifierCC0-1.0

Public domain — no rights reserved; unrestricted use.

RedistributionYes

PII RISK

Unknown risk

Full Record

29 of 46 fields stated

Every column the registry tracks. Values marked derivedwere filled by a rule or read from the source's text rather than stated in its metadata; the rule or the quoted sentence is shown under the value.

Registry columns (22/27)

iso code
bej
Dataset name
PanLex Language Vocabulary (lbourdois/panlex)
Modality
Text
Type
dataset
Link
https://huggingface.co/datasets/lbourdois/panlex
NLP Task
MT
Domain
General
Data Format
csv
Size Metric
Tokens
Size Value
0
License
CC0-1.0
Open Access
Open
Source Repository
Hugging Face
Year Published
2024
Creator/Author(s)
PanLex / lbourdois
Funder/Partner
TWB
Dialect Coverage
Standard
Gender Balance
Not stated
Speaker Count
Not stated
Annotation Type
Human
Status
Active
Has Datasheet
Yes
PII Risk
Not stated
Consent Verified
Not stated
Verified By
Not stated
Date Indexed
2026-05-19
Notes / Flags
PanLex Language Vocabulary (January 2024), 24.65M rows across 6152 languages including Beja (bej confirmed via HuggingFace language tag). World's largest lexica…

Quality (DQI) (6/12)

Datasheet Fields Completed
Not stated
SPDX License ID
CC0-1.0
Redistribution Allowed?
Yes
Content Hash
feb3a878db0fdc93cadf739f3a2749be4d4698c8derivedHugging Face card: commit sha
Submitted By
hf_data pipeline (automated)
Fetch Date
2026-05-19
Last Updated Date
2024-02-03derivedHugging Face card: lastModified
Annotation Guidelines Available?
Not stated
IAA Score
Not stated
Cleaning Notes
Not stated
Benchmark Exists?
Not stated
Benchmark Citation
Not stated

Representativeness (DRI) (1/7)

Gender Distribution
Not stated
Region Coverage
Not stated
Region Distribution
Not stated
Age Range Coverage
Not stated
Register
Not stated
Marginalized Group Note
Not stated
Demographic Fields Reported Count
0

Readiness Score Breakdown

4/5
Data Coverage80%
Model Availability92%
Benchmarks33%
Documentation57%
Recency100%

Representativeness

Gender Balance
Regional Distribution
Dialect Coverage
Inclusivity

Provenance

Source URL
https://huggingface.co/datasets/lbourdois/panlex
Fetch Timestamp
19 May 2026
Content Hash
feb3a878db0fdc93cadf739f3a2749be4d4698c8

Audit Log

Record indexed from Hugging Face

19 May 2026

Source status: Active

19 May 2026

Other Copies

The same resource, also published at these locations. It is counted once.

dataset

PanLex Meanings (cointegrated/panlex-meanings)

Words across thousands of languages with semantic meaning links. 1012 languages including Beja (bej confirmed via HuggingFace language tag). Fork gtak1/panle…

113 languages5 copiesBeja(bej)TextMT
HUHugging Face
Open19 May 2026
dataset

panlex-meanings

Data format: csv

113 languages5 copiesBeja(bej)Tabulartranslation
HUHugging Face
Open27 Oct 2026
dataset

PanLex: World vocabulary listing

Parsed from Lanfrica name: panlex-world-vocabulary-listing. Raw license: https://panlex.org/license/.

Database entry113 languages5 copiesBeja(bej)TextLexicon
OTOther
Open26 May 2026

Related Datasets

dataset

MMS Ulab v2 (espnet/mms_ulab_v2)

Massively multilingual unlabeled speech, 4023 languages confirmed including Beja (bej) via HuggingFace language tag. 8900 total hours. 3 derived forks also t…

1,433 languages3 copiesBeja(bej)SpeechASR
HUHugging Face
Open19 May 2026
dataset

Universal Dependencies Treebank Collection (commul/universal_dependencies)

Universal Dependencies treebank collection on HuggingFace, 193 languages including Beja (UD_Beja-Autogramm: 763 sentences, 11,948 tokens). Bej confirmed via …

2 languages2 copiesBeja(bej)Texttoken-classification
HUHugging Face
Open19 May 2026
dataset

ChiKhaPo: Multilingual Lexical Benchmark (ec5ug/chikhapo)

Multilingual benchmark for lexical comprehension and generation, 2752 languages including Beja (bej confirmed via HuggingFace language tag). 8 subtasks. Draw…

857 languagesBeja(bej)TextClassification
HUHugging Face
Open19 May 2026