Social Media Scraper API

What is SocialCrawl.dev?

SocialCrawl.dev is a unified social media scraping API designed for developers, researchers, and AI agents. It consolidates access to data from over 50 major social media, e-commerce, and web platforms—including TikTok, Instagram, YouTube, X (Twitter), Reddit, LinkedIn, Facebook, Amazon, and Google services—into a single, consistent API. Instead of managing dozens of different scrapers, authentication tokens, and data schemas, you can access nearly 400 endpoints with one API key and one unified response format. It is an agent-ready platform that simplifies the complex task of collecting public data at scale for use cases like brand monitoring, market research, competitive intelligence, and AI training.

What are the main features of SocialCrawl.dev?

1. Unified API for 50+ Platforms: A single API key and authentication token grants access to structured data from over 50 social media, search, e-commerce, and community platforms.

2. One Unified Schema: Every platform returns data in a consistent JSON structure (e.g., author, engagement, metadata), eliminating the need to write platform-specific parsing logic.

3. AI & Agent-First Design:

   • Build with AI: An integrated AI assistant can answer questions about endpoints, pricing, and integration, providing copy-paste ready code in Python, TypeScript, or cURL.

   • MCP (Model Context Protocol) Native: Offers a drop-in MCP server, making all API endpoints available as first-class tools for AI agents in Claude Desktop, Cursor, Windsurf, and other compatible clients.

   • Skill Installable: Developers can install a "skill" for their AI agents (like in Claude Code or Codex) to instantly equip them with scraping capabilities.

4. Comprehensive Platform Coverage: Supports deep data extraction from major platforms, including TikTok (profiles, videos, comments), Instagram, YouTube, Reddit (posts, subreddits), LinkedIn (profiles, posts), and many more, with specific endpoint counts for each.

5. Powerful Use Case Solutions: Provides built-in functionalities for real-world applications:

   • Brand Monitoring: Track mentions of brands, products, or executives across all platforms on a schedule.

   • Sentiment Analysis: Quantify public sentiment with built-in scoring for launches and campaigns.

   • Trend Detection: Identify emerging conversations and topics with velocity scores and spike alerts.

   • Competitive Intelligence: Monitor competitor activities, launches, and customer feedback.

How to use SocialCrawl.dev?

1. Get Started: Register on SocialCrawl.dev to receive 100 free credits without requiring a credit card. This provides you with an API key (sc_YOUR_API_KEY).

2. Explore and Build: Use the interactive documentation and the "Build with AI" chat feature. You can ask questions like "How do I scrape TikTok data?" and receive runnable code snippets tailored to your API key.

3. Make API Calls: Use your API key in the x-api-key header to make HTTP requests to any of the nearly 400 endpoints. The platform offers SDKs and examples in multiple programming languages.

4. Integrate with AI Agents: For autonomous AI workflows, you can:

   • Configure the MCP server by adding a JSON block to your agent's configuration.
   • Install the SocialCrawl skill via a command like npx skills add socialcrawl/skills.
   • Utilize the agent onboarding skill that allows AI agents to sign up and generate their own API keys.

What is the price of SocialCrawl.dev?

SocialCrawl.dev operates on a credit-based pricing model. New users start with ### 100 free credits upon registration (no credit card required). Beyond the free tier, you purchase credit packs based on your data volume needs. Pricing scales with usage, and specific plans and rates for credit packs can be found on their website. This model allows developers to pay for the data they actually scrape, making it cost-effective for both small projects and large-scale operations.

Helpful Tips for Using SocialCrawl.dev

• Start with Free Credits: Use the 100 free credits to test the API's reliability, data quality, and speed with your target platforms before committing to a paid plan.

• Leverage the Unified Schema: Design your data processing pipeline around the consistent author, engagement, metadata fields. This will make your codebase cleaner and more maintainable as you add more platforms.

• Use the AI Assistant for Development: When integrating, consistently use the "Build with AI" feature to generate boilerplate code and clarify parameter usage, which can significantly speed up development time.

• Plan for Rate Limits: Although the API handles retries, design your application to respect rate limits for sustained, high-volume scraping to ensure optimal performance.

• Explore MCP/Skill Integration Early: If your use case involves AI agents, integrate the MCP server or skill at the beginning of your project. This "agent-ready" design can automate complex data-fetching workflows from day one.

• Check Platform-Specific Docs: While the schema is unified, always check the specific endpoint documentation for each platform (e.g., TikTok API, Reddit API) to understand available parameters and data points.

Frequently Asked Questions (FAQs)

How many platforms does SocialCrawl.dev support?

SocialCrawl.dev supports over 50 platforms, including major social networks (TikTok, Instagram, YouTube, X, LinkedIn, Reddit, Facebook), e-commerce sites (Amazon, Walmart, eBay), search engines (Google Search, News, Trends), and community sites (GitHub, Hacker News).

Do I need a different API key for each platform?

No. One of the core features of SocialCrawl.dev is that you only need a single API key to access data from all supported platforms.

What is the "Unified Schema"?

The Unified Schema means that regardless of whether you scrape a TikTok profile, a YouTube video, or a Reddit post, the API response will structure the data in a consistent format (e.g., containing author, engagement, and metadata objects). This eliminates the need to write unique parsers for each platform.

Can I use SocialCrawl.dev for my AI agent or autonomous workflow?

Yes, it is explicitly designed for this. You can integrate it via MCP (Model Context Protocol) for compatible AI clients, install it as a "skill" in agent environments, or even use an onboarding skill that allows agents to self-signup.

What are credits, and how are they consumed?

Credits are the unit of consumption for the API. Each API call to an endpoint costs a certain number of credits, which varies based on the complexity and data volume of the request (e.g., scraping a user profile vs. searching through posts). You start with 100 free credits.

Is the data from SocialCrawl.dev real-time?

The API provides access to publicly available data. The freshness of the data depends on the specific platform and endpoint. Many endpoints are designed to fetch live data, and features like scheduled monitoring are built for tracking mentions as they happen.

What is Prism?

Prism is a feature of SocialCrawl.dev that consolidates multiple API calls (up to ten) into a single request. This is useful for batching operations or fetching related data points efficiently, reducing network overhead and simplifying your code.

How does SocialCrawl.dev handle platform changes or anti-scraping measures?

As a dedicated API service, SocialCrawl.dev maintains the infrastructure to handle updates to platform layouts and anti-scraping techniques (like rate limits or CAPTCHAs), which is a significant advantage over maintaining your own scrapers.