MediaCrawler Open-Source Tool for Public Content

Public content collection stops failing at the setup stage → verify the project license, platform rules, and intended data use before installing MediaCrawler. Need a fast decision? → use it for small, research-focused collection with a controlled browser session; do not treat it as an unrestricted commercial scraping service.

This guide is for developers learning Playwright and multi-platform collection, data engineers building research-oriented public data pipelines, and technical leads evaluating long-running browser automation environments.

Last updated August 13, 2026. Current project details were checked against the official repository README, English documentation, project license, dependency manifest, and data storage guide on August 13, 2026.

The decision before installation

MediaCrawler is a Python project that combines browser automation, login-state reuse, platform-specific collection logic, WebUI controls, and multiple storage backends. The official documentation lists support for Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, Zhihu, and other related workflows. Supported task types include keyword search, specific post collection, creator-page collection, comments, and cached login state. See the official MediaCrawler repository and English README for the current feature matrix.

The important distinction is not whether a page is visible without logging in. The important question is whether your planned collection is allowed, proportionate, and technically controlled.

The repository’s disclaimer describes the project as a learning and research tool. Its license limits use to non-commercial learning purposes, prohibits large-scale crawling and disruptive activity, and requires written consent for commercial use. That is a project-level restriction, not a legal opinion about your specific location or use case. Review the current project license before you create a pipeline.

A safe initial scope normally has these properties:

• You collect only the fields required for a stated research question.
• You use a small set of keywords, post IDs, or creator pages.
• You define a retention period before the first run.
• You avoid private content, restricted areas, access-control bypasses, and CAPTCHA workarounds.
• You can stop the process quickly when the platform changes behavior or your purpose changes.

Public visibility is not blanket permission for unlimited automated collection or commercial reuse. Platform terms, copyright rules, privacy obligations, database rights, and local law may all apply.

Timeline: scope, license, and platform fit

Stage 1: write the collection brief

Before opening a terminal, write down five items:

1. The platform or platforms.
2. The content type.
3. The exact fields required.
4. The expected retention period.
5. The human or automated decisions that will use the data.

For example, a research team studying public discussion around a product launch may need post text, publication time, public engagement counts, and a source URL. It may not need profile avatars, full comment threads, or historical creator metadata.

This distinction matters because collection risk expands with every unnecessary field. MediaCrawler can expose workflows for posts, comments, creator pages, and keyword searches. Selecting all available fields because the tool makes them accessible creates a data-governance problem that the repository cannot solve for you.

Can MediaCrawler collect public content legally?

There is no universal yes or no. The project disclaimer does not establish legality, and a page being publicly viewable does not remove platform terms or local legal requirements. Treat legality as a separate review involving your location, the data subjects, the collection method, the intended use, and the platform’s current rules.

If the purpose is commercial monitoring, customer profiling, resale, advertising enrichment, or large-scale archiving, stop before deployment and obtain a qualified legal review. The open-source license is especially important here because the current license describes non-commercial learning use rather than unrestricted commercial deployment.

Stage 2: confirm platform coverage

The current English README lists these named platforms:

• Xiaohongshu
• Douyin
• Kuaishou
• Bilibili
• Weibo
• Tieba
• Zhihu

The repository presents several common task patterns:

• Keyword-based search
• Specific post or content ID collection
• Creator homepage collection
• Comment and secondary-comment collection
• Login-state caching
• Data preview and export through WebUI

Support is not the same as stability. A platform may change HTML structure, request signatures, login flows, browser checks, or content availability without preserving compatibility with the current repository. A feature shown in the README can still require maintenance before it is suitable for a repeatable data pipeline.

Which platforms does MediaCrawler support?

Start with the platform matrix in the current README rather than an old tutorial or a copied command. The named open-source targets are Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Confirm the exact task type you need because keyword search, specific-content collection, creator pages, and comments may not have identical behavior across platforms.

Timeline: environment and first run

Stage 3: prepare Python, Node.js, and browser components

The repository’s current pyproject.toml requires Python 3.11 or newer and declares a Playwright dependency of at least 1.61.0. The README also lists Node.js 16 or newer as a project prerequisite. These are repository values checked on August 13, 2026, not permanent compatibility guarantees. Recheck the manifest after every major update. See the current dependency manifest before creating a reproducible environment.

The recommended setup path is based on uv:

git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler
uv sync

For the standard Playwright browser mode, install the required browser binaries:

uv run playwright install

Playwright maintains browser binaries separately from the Python package. Its official documentation notes that browser versions are tied to Playwright releases, so updating the package may require installing browsers again. The browser guide also explains how to install operating-system dependencies and how browser binaries are cached. Read the official Playwright browser installation guide before building a server image.

The current MediaCrawler documentation describes two browser connection approaches:

• Standard Playwright mode: MediaCrawler starts and controls the browser through Playwright.
• CDP mode: MediaCrawler connects to an existing local Chrome browser and can reuse its browser context, cookies, and login state.

The repository currently describes CDP support for selected platform workflows, including xhs and dy, and documents a local Chrome remote-debugging setup. CDP is Chromium-specific in Playwright, so it should not be treated as a universal cross-browser connection method. The Playwright Browser API documentation explains the relevant browser connection boundary.

Do not expose a remote-debugging port to the public internet. Bind management interfaces to localhost or a protected private network, use firewall rules, and keep one login state assigned to one controlled task owner.

Stage 4: establish the login state

Does MediaCrawler require QR-code login?

Often, yes, depending on the selected platform and login method. The official examples use a QR-code login parameter for sample tasks and instruct you to scan with the corresponding application. The project also supports cached login state, so you may not need to scan on every run if the session remains valid. Session expiration, platform changes, device verification, and account security checks can still force a new login.

A safer first-run sequence is:

1. Create a dedicated research account where the platform permits it.
2. Run the smallest possible task.
3. Complete the login interaction manually.
4. Confirm that the session is stored only in the intended local profile or cache location.
5. Test one keyword, one creator page, or one specific content ID.
6. Inspect the output before expanding the scope.

Do not paste cookies into tickets, shell history, shared chat, or source control. Cookies and browser profiles can be equivalent to account access. Treat them as secrets.

A useful separation is to maintain three environments:

• A local development environment for selectors and configuration.
• A controlled staging environment with a limited account and narrow scope.
• A production-like environment only after the task passes compliance and security review.

Do not move a personal browser profile directly from your workstation to a shared server. That action can copy session cookies, extensions, saved credentials, and unrelated browsing data.

Security reminder: A CDP port, browser profile, cookie file, or cached login state should be handled like a credential. If you cannot identify who can access it, do not use that environment for a logged-in collection task.

Timeline: task configuration and collection limits

Stage 5: choose the smallest task type

MediaCrawler supports several collection patterns, but they have different cost and risk profiles.

Keyword search is useful when you need to discover public content around a defined topic. Its risk is scope expansion: a broad keyword can produce far more material than your research design requires.

Specific content IDs are preferable when you already have a validated sample. They give you a clearer boundary and make reruns easier to audit.

Creator-page collection can support public profile or publishing-pattern research, but it may expose more personal or historical information than your question requires.

Comment collection adds another layer of personal data, moderation context, and retention risk. The repository states that comment crawling is not enabled by default in one of its documented paths. Treat that default as a useful control, not as a reason to activate comments automatically.

A practical configuration rule is:

• Start with specific IDs if you already have a sample.
• Use a narrow keyword set if discovery is necessary.
• Add creator pages only when the research question requires them.
• Add comments only after field-level review.
• Set a stop condition before increasing the sample.

Do not use instructions that bypass CAPTCHA, defeat access controls, impersonate users, or evade platform safety systems. When a task encounters a verification wall, the correct response is to stop, reduce scope, use an official interface, or request permission.

Scene example: a small monitoring study

Suppose you are comparing how a public product announcement is discussed across three platforms. A defensible pilot could use a fixed keyword list, a short collection window, selected public posts, and a defined set of text and timestamp fields.

An unsafe expansion would be to collect every creator page, download every media asset, retain all comments indefinitely, rotate accounts, and continue after repeated verification prompts. The second design is not merely “more engineering.” It changes the legal, privacy, security, and operational profile of the project.

Timeline: storage, preview, and data hygiene

MediaCrawler currently documents support for CSV, JSON, JSONL, Excel, SQLite, and MySQL storage. The repository also provides a data storage guide, while WebUI supports data preview and export.

Which data formats can MediaCrawler save?

You can choose among flat files, a local relational database, or a server database depending on the next pipeline stage:

• CSV: Easy to inspect and exchange, but weak for nested comments and evolving schemas.
• JSON: Flexible for nested objects, but less convenient for large analytical queries.
• JSONL: Useful for append-oriented pipelines because each record can be processed independently.
• Excel: Convenient for manual review, not a strong long-term system of record.
• SQLite: A practical local option for structured experiments and single-node workflows.
• MySQL: Better suited to a shared database service, provided you implement access control, backups, and retention rules.

A storage choice should follow the workflow, not the other way around. If a data engineer plans to transform records into a warehouse, JSONL may be useful during ingestion. If a researcher needs repeatable local queries, SQLite may be simpler. If a team needs concurrent access, a managed database may be more appropriate, but that adds credentials, network policy, backup, and deletion responsibilities.

Use a data contract before saving:

• Required fields
• Source URL or content identifier
• Collection timestamp
• Transformation history
• Retention deadline
• Deletion method
• Access owner

Redact tokens, cookies, account identifiers, and internal network details from logs. Store raw exports separately from cleaned analytical tables, and restrict access to raw data more tightly.

Timeline: remote deployment and long-running operations

Can MediaCrawler run on a remote server?

Technically, yes, but remote deployment is not automatically safe or reliable. A remote environment must provide Python, Node.js where required, browser dependencies, stable process supervision, protected login-state storage, and a secure way to complete interactive login. The repository’s WebUI documentation describes a backend API and a frontend development server, with a production build that can be served by the API process. The documented development ports are 8080 for the API and 5173 for the frontend, so treat those values as configuration details to verify rather than public endpoints.

For a remote setup, follow this sequence:

1. Provision a dedicated user account with no unnecessary administrative privileges.
2. Install the repository’s currently documented Python and Node.js dependencies.
3. Install Playwright browsers and operating-system dependencies if using standard Playwright mode.
4. Keep the WebUI and API bound to a private interface or protected tunnel.
5. Store login profiles outside the repository and outside backup sets that are accessible to unrelated users.
6. Run a limited test task and inspect memory, disk growth, browser processes, and log volume.
7. Add process supervision only after the stop and deletion procedures are tested.

Long-running browser jobs fail for reasons that a simple script does not show:

• Browser processes can remain after a task crashes.
• Login sessions can expire without a clear application-level error.
• Network interruptions can create partial files or duplicate records.
• Platform page changes can produce valid-looking but incomplete data.
• Logs can accidentally capture sensitive request details.
• A shared remote profile can mix accounts or expose one user’s session to another.

For an Apple-based remote environment, review the kvmboot company information page for general provider context, then confirm that the selected setup supports your browser, Python process, storage, and remote access requirements. Do not assume that renting a Mac solves the compliance or platform-access question. It only changes where the workload runs.

If you need to evaluate a temporary remote environment, write down the required browser mode, Python process, storage location, remote access method, and account-isolation controls before contacting a provider. Hosting can change the execution environment, but it cannot replace legal review, platform approval, or a defined data-retention policy. Keep provider communication focused on environment capabilities rather than treating hosting as a substitute for technical or legal review. A support channel can clarify operating-system, browser, and access constraints, but it cannot authorize your collection plan.

Stop conditions and fallback decisions

Pause the collection job immediately when any of these conditions appears:

• The account receives a security warning or unexpected verification request.
• The platform rules change in a way that affects your purpose.
• The repository license or disclaimer changes.
• The project begins collecting fields outside the approved data contract.
• The intended use expands from research to commercial profiling or resale.
• The output contains private, restricted, or unexpectedly sensitive information.
• A browser or network error causes repeated retries without clear progress.
• The team can no longer identify who has access to cookies, profiles, exports, or databases.

The correct fallback is not to increase evasion. Reduce the scope, stop the run, delete data that is no longer justified, and evaluate an official API or licensed data source where one exists.

For recurring jobs, add a monthly review of the repository README, license, dependency manifest, platform rules, account status, data fields, retention schedule, and incident logs. A crawler that ran successfully last month may still be unsuitable today because the project or target platform changed.

MediaCrawler strengths and limitations

Strengths

• It gives Python developers a concrete browser-automation architecture to study.
• It combines multiple platform adapters under one project structure.
• It demonstrates login-state reuse and browser-context handling.
• It supports several collection patterns instead of a single hard-coded workflow.
• It offers WebUI controls, status visibility, preview, and export.
• It supports flat files and database-oriented storage paths.
• It is useful for understanding where browser automation meets data engineering.

Limitations

• The open-source license is not an unrestricted commercial-use license.
• Platform compatibility can change without your pipeline being ready.
• Login states and browser profiles are sensitive assets.
• Remote deployment requires more than copying the repository to a server.
• A successful request does not prove that the resulting use is lawful.
• Browser automation introduces process, memory, storage, and recovery problems.
• The repository’s feature matrix should not be interpreted as a service-level guarantee.

MediaCrawler is a strong learning and prototyping choice when your scope is narrow, your account and browser state are isolated, and your data contract is explicit. It is a poor fit when you need guaranteed commercial rights, a stable high-volume feed, formal service-level commitments, or a platform-approved data interface.

Your current setup may be a personal laptop, an unmanaged Windows or Linux workstation, or a short-lived cloud host. Those options can be cheaper to start, but they often leave browser profiles mixed with personal data, make remote login-state control inconsistent, and provide weak recovery when long-running jobs fail. A dedicated Mac environment can be easier to isolate for temporary browser automation tests because you can separate the research session from your everyday workstation and retire the environment when the evaluation ends. Before choosing any remote environment, confirm the workload, permitted use, storage location, access controls, and expected rental period. The better choice is the one that keeps the collection narrow, auditable, and easy to stop—not the one that simply makes larger-scale collection possible.