CORTEXA
← Browse
arxivcs.SEcs.AI2026-07-11

A Large-Scale Dataset of MCP Implementations on GitHub

Benny Toeppe, Amine Barrak, Emna Ksontini

The rapid emergence of the Model Context Protocol (MCP) has introduced a new standard for connecting large language models to external tools and services. Despite its rapid adoption in open-source development, systematic understanding of how MCP is implemented, structured, and maintained remains limited. This study presents the first large-scale, evidence-based dataset of real-world MCP implementation collected directly from GitHub. Using a hybrid pipeline that integrates the GitHub REST and GraphQL APIs with custom Python verification scripts, 3,238 candidate repositories were discovered, filtered, and validated through multi-stage evidence checks. Each verified project was classified by operational role (e.g., client, server, gateway) and exported in a reproducible JSONL schema. A manual review of a representative subset confirmed an overall precision of 83% at a 95% confidence level, and additionally revealed a set of repositories functioning primarily as educational samples, tutorials, or demonstration templates. A targeted exclusion rule was then applied to remove these non-operational repositories, resulting in a final dataset of 2,297 validated MCP projects. The analysis shows that Python and TypeScript dominate MCP development, with hybrid architectures emerging as the most common design pattern. By emphasizing transparent verification strategies, structured evidence tagging, and reproducible data organization, this work establishes a foundational benchmark for studying real-world MCP ecosystems and supports future research on integration, connectivity, and compatibility across the broader developer community.

View free PDFSource page

Related papers

arxivcs.SEcs.AI2026-06-27

Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

Balázs Szalontai, Ábel Szauter, Balázs Márton, Péter Verebics, Balázs Pintér, Tibor Gregorics

There are various benchmarks to evaluate bugfixing capabilities of Large Language Models. However, most widespread benchmarks do not fully reflect real-world bugfixing practices. They are small, weakening statistical reliability, and the buggy programs are often similar to one an…

View free PDFSource page
arxivcs.SEcs.AI2026-07-19

A Large-Scale Measurement of AI Bill of Materials Completeness in Hugging Face Models

Md Erfan, Ahmed Ryan, Md Rayhanur Rahman

Pretrained machine learning (ML) models help developers build ML-intensive software systems without training models from scratch. However, model repositories often provide incomplete machine-readable documentation about model provenance, licenses, datasets, limitations, and exter…

View free PDFSource page
arxivcs.MAcs.AIcs.SE2026-07-17

Automated Hardware Validation Test Plan Generation for Large Scale AI Datacenter Platforms Using a Generative AI Multi-Agents Architecture

Mohammed-Khalil Ghali, Saurabh Kulkarni, Prathamesh Kulkarni, Rohan Kulkarni, Sangwon Yoon, Daehan Won

Large-scale AI datacenter platforms comprise thousands of heterogeneous hardware components whose validation requires comprehensive fault injection test plans. Today these plans are authored manually: engineers review hardware self-healing validation documents and bills of materi…

View free PDFSource page
arxivcs.SEcs.AI2026-06-28

A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, et al.

LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data. However, existing benchmarks remain largely outcome-oriented: they score only the final answer and fail to assess the…

View free PDFSource page
arxivcs.IRcs.AIcs.SE2026-07-03

Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

Nishan Pantha, Pranath Reddy Kumbam, Sajil Awale, Pushwitha Krishnappa, Muthukumaran Ramasubramanian, Nidhi Jha, et al.

Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging. Existing code search benchmarks focus on general software engineering tasks and fail to capture…

View free PDFSource page
arxivcs.SEcs.AIcs.PL2026-07-01

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM co…

View free PDFSource page