Go modules · #287 by repository stars

github.com/crawlab-team/crawlab

v0.6.3crawlab-team/crawlab ↗New releases in your daily digest, changelog included.

Last release 10 months ago
03 Dec 2025
Ships unpredictably
gaps range from 8 days to 11 months
Most releases are documented
notes for 24 of 27 stable releases
Nothing withdrawn
no release was ever pulled
7 years old
143 releases · first in 2019

Release timeline

143 releases since 2019

20192020202120222023202420252026

Releases

v0.6.3-dev.0.20240710105935-13596436cd4710 Jul 2024pre-release
Nothing published for this version
v0.6.3-dev.0.20240705045709-059711a533f55 Jul 2024pre-release
Nothing published for this version
v0.6.3-dev.0.20240704050154-f4025a982d404 Jul 2024pre-release
Nothing published for this version
v0.6.3-dev.0.20240702095940-7eb8c08e0efc

v0.6.3-dev14 Jun 2024pre-release

chore: update version by @tikazyq in #1330

What's Changed

chore: update version by @tikazyq in #1330
feat: added export spider files by @tikazyq in #1332
fix: unable to terminate process when cancelling a task by @tikazyq in #1334
Develop by @tikazyq in #1341
chore: updated deps by @tikazyq in #1342
fix: unordered columns of export by @tikazyq in #1343
feat: added priority in spider and schedule by @tikazyq in #1344
Updated README by @tikazyq in #1345
feat: auto cleanup tasks over 30 days ago by @tikazyq in #1346
chore: updated deps by @tikazyq in #1351
chore: updated deps by @tikazyq in #1352
chore(deps): bump google.golang.org/grpc from 1.42.0 to 1.53.0 in /backend by @dependabot in #1355
Develop by @tikazyq in #1359
fix(sec): upgrade github.com/gin-gonic/gin to 1.9.1 by @chncaption in #1357

New Contributors

@chncaption made their first contribution in #1357

Full Changelog: v0.6.2...v0.6.3-dev

Open source →

Additional notes

v0.6.3-dev Pre-release

Pre-release

Compare

Choose a tag to compare

Open source →

v0.6.216 Jun 2023

Web Crawler Management Platform Crawlab v0.6.2 Official Release

Overview

Crawlab v0.6.2 is the latest iterative version of Crawlab v0.6.x, bringing a series of improvements, including bug fixes, feature enhancements, and enhanced functionality for environment variables.

Changelog

Bug Fixes

Feature Enhancements

Community

If you find Crawlab helpful for your daily development or your company, please consider starring it on GitHub. If you encounter any issues, feel free to raise them as issues on GitHub. Additionally, you're welcome to contribute to the development of Crawlab. You can also join the Crawlab technical discussion group by adding WeChat account tikazyq1, where you can communicate and discuss with other developers regarding technical development and deployment usage.

References

Official Website: https://www.crawlab.cn
Documentation: https://docs.crawlab.cn
GitHub: https://github.com/crawlab-team/crawlab
Demo: https://demo.crawlab.cn/

Open source →

Additional notes

v0.6.2

Compare

Choose a tag to compare

Open source →

v0.6.123 Mar 2023

Bump eventsource from 1.1.0 to 1.1.1 in /frontend by @dependabot in #1115

What's Changed

Bump eventsource from 1.1.0 to 1.1.1 in /frontend by @dependabot in #1115
Develop by @tikazyq in #1117
Develop by @tikazyq in #1119
Develop by @tikazyq in #1122
Develop by @tikazyq in #1124
Develop by @tikazyq in #1127
Develop by @tikazyq in #1145
Develop by @tikazyq in #1148
fix(mem): fixed memory leak issue resulted by log collector by @tikazyq in #1157
build(golang/dockerfile): rm unused tar by @ma-pony in #1165
Develop by @tikazyq in #1169
Develop by @tikazyq in #1170
Develop by @tikazyq in #1185
Develop by @tikazyq in #1191
Develop by @tikazyq in #1205
Develop by @tikazyq in #1208
fix(git): unable to pull code from remote by @tikazyq in #1211
Develop by @tikazyq in #1216
feat(filter): updated backend deps by @tikazyq in #1227
Develop by @tikazyq in #1229
fix(ui): code highlight issue by @tikazyq in #1230
Develop by @tikazyq in #1239
fix(performance): fixed memory leak issue by @tikazyq in #1240
fix(docker): api address incorrect issue by @tikazyq in #1241
Develop by @tikazyq in #1245
chore: updated deps by @tikazyq in #1249
Develop by @tikazyq in #1256
fix(doc): fix some broken links by @Codingendless in #1269
chore: updated deps by @tikazyq in #1276

New Contributors

@ma-pony made their first contribution in #1165
@Codingendless made their first contribution in #1269

Full Changelog: v0.6.0...v0.6.1

Open source →

Additional notes

v0.6.1

Compare

Choose a tag to compare

Open source →

v0.6.022 May 2022

As a major release, v0.6.0 is consisted of a number of large changes to enhance the performance, scalability, robustness and usability of Crawlab. Thi

Change Log (v0.6.0)

Overview

As a major release, v0.6.0 is consisted of a number of large changes to enhance the performance, scalability, robustness and usability of Crawlab. This beta version is theoretically more robust than older versions mainly in task execution, files synchronization and node management, yet we still recommend users to thoroughly run tests with various samples.

Enhancements

Backend

File Synchronization. Migrated file sync from MongoDB GridFS to SeaweedFS for better stability and robustness.
Node Communication. Migrated node communication from Redis-based RPC to gRPC. Worker nodes indirectly interact with MongoDB by making gRPC calls to the master node.
Task Queue. Migrated task queue from Redis list to MongoDB collection to allow more flexibility (e.g. priority queue).
Logging. Migrated logging storage system to SeaweedFS to resolve performance issue in MongoDB.
SDK Integration. Migrated results data ingestion from native SDK to task handler side.
Task Related. Abstracted task related logics into Task Scheduler, Task Handler and Task Runners to increase decoupling and improve scalability and maintainability.
Compotenization. Introduced DI (dependency injection) framework and componentized modules, services and sub-systems.
Plugin Framework. Crawlab Plugin Framework (CPF) has been released. See more info [here](https://docs.crawlab.cn/en/guide/plugin/).
Git Integration. Git integration is implemented as a built-in feature.
Scrapy Integration. Scrapy integration is implemented as a plugin [spider-assistant](https://docs.crawlab.cn/en/guide/plugin/plugin-spider-assistant).
Dependency Integration. Dependency integration is implemented as a plugin [dependency](https://docs.crawlab.cn/en/guide/plugin/plugin-dependency).
Notifications. Notifications feature is implemented as a plugin [notification](https://docs.crawlab.cn/en/guide/plugin/plugin-notification).

Frontend

Vue 3. Migrated to latest version of frontend framework Vue 3 to support more advanced features such as composition API and TypeScript.
UI Framework. Built with Vue 3-based UI framework Element-Plus from Vue-Element-Admin, more flexibility and functionality.
Advanced File Editor. Support more advanced file editor features including drag-and-drop copying/moving files, renaming, deleting, file editing, code highlight, nav tabs, etc.
Customizable Table. Support more advanced built-in operations such as columns adjustment, batch operation, searching, filtering, sorting, etc.
Nav Tabs. Support multiple nav tabs for viewing different pages.
Batch Creation. Support batch creating objects including spiders, projects, schedules, etc.
Detail Navigation. Sidebar navigation in detail pages.
Enhanced Dashboard. More stats charts in home page dashboard.

Miscellaneous

Documentation Site. Upgraded [documentation site](https://docs.crawlab.cn/en).
Official Plugins. Allow users to install [official plugins](https://docs.crawlab.cn/en/guide/plugin/) on Crawlab web UI.

Open source →

Additional notes2 sources agree

(TBC)

Open source →

Additional notes

v0.6.0

Compare

Choose a tag to compare

Open source →

v0.6.0-beta.2021122424 Dec 2021pre-release

Change Log (v0.6.0-beta.20211224)

Overview

This is the third beta release for the next major version v0.6.0. With more features and optimization coming in, the release of official version v0.6.0 is approaching soon.

Enhancement

Internationalization. Support Chinese.
CLI Upload Spider. #1020
Official Plugins. Allow users to install official plugins on Crawlab web UI.
More Documentation. Added documentation for plugins and CLI.

Bug Fixes

Upload File Issue #990 #1014 #1022
Unable to Rename File #1008

TODOs

Associated Tasks. There will be main tasks and their sub-tasks if task mode is "all nodes" or "selected nodes".
Crontab Editor. Frontend component that visualize the crontab editing.
Results Deduplication.
Environment Variables.
Frontend Utility Enhancement. Advanced features such as saved table customization.
Log Auto Cleanup.
More Documentation.
E2E Tests.
Frontend Output File Size Optimization.

What Next

The next version could the official release of v0.6.0, but not determined yet. There will be more tests running against the current beta version to ensure robustness and production-ready deployment.

Open source →

Additional notes

v0.6.0-beta.20211224

Compare

Choose a tag to compare

Open source →

v0.6.0-beta.2021112020 Nov 2021pre-release

Change Log (v0.6.0-beta.20211120)

Overview

This is the second beta release for the next major version v0.6.0 after the first beta release. With more features and optimization coming in, the release of official version v0.6.0 is approaching soon.

Enhancement

Backend

Plugin Framework. Crawlab Plugin Framework (CPF) has been released. See more info here.
Git Integration. Git integration is implemented as a built-in feature.
Scrapy Integration. Scrapy integration is implemented as a plugin spider-assistant.
Dependency Integration. Dependency integration is implemented as a plugin dependency.
Notifications. Notifications feature is implemented as a plugin notification.
Documentation Site. Set up documentation site.

Frontend

Bug Fixing.

TODOs

Associated Tasks. There will be main tasks and their sub-tasks if task mode is "all nodes" or "selected nodes".
Crontab Editor. Frontend component that visualize the crontab editing.
Results Deduplication.
Environment Variables.
Internationalization. Support Chinese.
Frontend Utility Enhancement. Advanced features such as saved table customization.
Log Auto Cleanup.
More Documentation.

What Next

The next version could the official release of v0.6.0, but not determined yet. There will be more tests running against the current beta version to ensure robustness and production-ready deployment.

Open source →

Additional notes

v0.6.0-beta.20211120

Compare

Choose a tag to compare

Open source →

v0.6.0-beta.202108033 Aug 2021pre-release

Change Log (v0.6.0-beta.20210803)

Overview

This is the beta release for the next major version v0.6.0. It recommended NOT to use it in production as it is not fully tested and thus not stable enough. Futhermore, more features including those not ready in the beta release (e.g. Git, Scrapy, Notification) are planned to be integrated into the live version, in the form of plugins.

Enhancement

As a major release, v0.6 (including beta versions) is consisted of a number of large changes to enhance the performance, scalability, robustness and usability of Crawlab. This beta version is theoretically more robust than older versions mainly in task execution, files synchronization and node management, yet we still recommend users to thoroughly run tests with various samples.

Backend

File Synchronization. Migrated file sync from MongoDB GridFS to SeaweedFS for better stability and robustness.
Node Communication. Migrated node communication from Redis-based RPC to gRPC. Worker nodes indirectly interact with MongoDB by making gRPC calls to the master node.
Task Queue. Migrated task queue from Redis list to MongoDB collection to allow more flexibility (e.g. priority queue).
Logging. Migrated logging storage system to SeaweedFS to resolve performance issue in MongoDB.
SDK Integration. Migrated results data ingestion from native SDK to task handler side.
Task Related. Abstracted task related logics into Task Scheduler, Task Handler and Task Runners to increase decoupling and improve scalability and maintainability.
Compotenization. Introduced DI (dependency injection) framework and componentized modules, services and sub-systems.

Frontend

Vue 3. Migrated to latest version of frontend framework Vue 3 to support more advanced features such as composition API and TypeScript.
UI Framework. Built with Vue 3-based UI framework Element-Plus from Vue-Element-Admin, more flexibility and functionality.
Advanced File Editor. Support more advanced file editor features including drag-and-drop copying/moving files, renaming, deleting, file editing, code highlight, nav tabs, etc.
Customizable Table. Support more advanced built-in operations such as columns adjustment, batch operation, searching, filtering, sorting, etc.
Nav Tabs. Support multiple nav tabs for viewing different pages.
Batch Creation. Support batch creating objects including spiders, projects, schedules, etc.
Detail Navigation. Sidebar navigation in detail pages.
Enhanced Dashboard. More stats charts in home page dashboard.

TODOs

As you may be aware that this is a beta release, some of the existing useful features such as Git and Scrapy integration may not be available. However, we are trying to include them in the official v0.6.0 release, as some of their core functionalities are already ready in the code base, and we will add to the stable version only if they are fully tested.

Plugin Framework. Advanced features will exist in the form of plugins, or pluggable modules.
Git Integration. To be included as a plugin.
Scrapy Integration. To be included as a plugin.
Notifications. To be included as a plugin.
Associated Tasks. There will be main tasks and their sub-tasks if task mode is "all nodes" or "selected nodes".
Crontab Editor. Frontend component that visualize the crontab editing.
Results Deduplication.
Environment Variables.
Internationalization. Support Chinese.
Frontend Utility Enhancement. Advanced features such as saved table customization.
Log Auto Cleanup.
Documentation.

What Next

This beta release is only a preview and a test ground for the core functionalies in Crawlab v0.6. Therefore, we will invite you guys to download and run more tests. The official release is expected to be ready after major issues from the beta version are sorted and Plugin Framework and other key features are developed and fully tested. With that beared in mind, a second beta version before the main release will also be possible.

Open source →

Additional notes

v0.6.0-beta.20210803

Compare

Choose a tag to compare

Open source →

v0.5.131 Jul 20203 sources agree

Added Golang programming language support .

Features / Enhancement

Added error message details.
Added Golang programming language support.
Added web driver installation scripts for Chrome Driver and Firefox.
Support system tasks. A "system task" is similar to normal spider task, it allows users to view logs of general tasks such as installing languages.
Changed methods of installing languages from RPC to system tasks.

Bug Fixes

Fixed first download repo 500 error in Spider Market page. #808
Fixed some translation issues.
Fixed 500 error in task detail page. #810
Fixed password reset issue. #811
Fixed unable to download CSV issue. #812
Fixed unable to install node.js issue. #813
Fixed disabled status for batch adding schedules. #814

Open source →

Additional notes

v0.5.1

Compare

Choose a tag to compare

Open source →

v0.5.019 Jul 20202 sources agree

Spider Market. Allow users to download open-source spiders into Crawlab.

Features / Enhancement

Spider Market. Allow users to download open-source spiders into Crawlab.
Batch actions. Allow users to interact with Crawlab in batch fashions, e.g. batch run tasks, batch delete spiders, ect.
Migrate MongoDB driver to MongoDriver.
Refactor and optmize node-related logics.
Change default task.workers to 16.
Change default nginx client_max_body_size to 200m.
Support writing logs to ElasticSearch.
Display error details in Scrapy page.
Removed Challenge page.
Moved Feedback and Dislaimer pages to navbar.

Bug Fixes

Fixed log not expiring issue because of failure to create TTL index.
Set default log expire duration to 1 day.
task_id index not created.
docker-compose.yml fix.
Fixed 404 page.
Fixed unable to create worker node before master node issue.

Open source →

v0.4.1023 Apr 20202 sources agree

Enhanced Log Management. Centralizing log storage in MongoDB, reduced the dependency of PubSub, allowing log error detection.

Features / Enhancement

Enhanced Log Management. Centralizing log storage in MongoDB, reduced the dependency of PubSub, allowing log error detection.
API Token. Allow users to generate API tokens and use them to integrate into their own systems.
Web Hook. Trigger a Web Hook http request to pre-defined URL when a task starts or finishes.
Auto Install Dependencies. Allow installing dependencies automatically from requirements.txt or package.json.
Auto Results Collection. Set results collection to results_<spider_name> if it is not set.
Optimized Project List. Not display "No Project" item in the project list.
Upgrade Node.js. Upgrade Node.js version from v8.12 to v10.19.
Add Run Button in Schedule Page. Allow users to manually run task in Schedule Page.

Bug Fixes

Cannot register. #670
Spider schedule tab cron expression shows second. #678
Missing daily stats in spider. #684
Results count not update in time. #689

Open source →

v0.4.931 Mar 20202 sources agree

Challenges. Users can achieve different challenges based on their actions.

Features / Enhancement

Challenges. Users can achieve different challenges based on their actions.
More Advanced Access Control. More granular access control, e.g. normal users can only view/manage their own spiders/projects and admin users can view/manage all spiders/projects.
Feedback. Allow users to send feedbacks and ratings to Crawlab team.
Better Home Page Metrics. Optimized metrics display on home page.
Configurable Spiders Converted to Customized Spiders. Allow users to convert their configurable spiders into customized spiders which are also Scrapy spiders.
View Tasks Triggered by Schedule. Allow users to view tasks triggered by a schedule. #648
Support Results De-Duplication. Allow users to configure de-duplication of results. #579
Support Task Restart. Allow users to re-run historical tasks.

Bug Fixes

CLI unable to use on Windows. #580
Re-upload error. #643 #640
Upload missing folders. #646
Unable to add schedules in Spider Page.

Open source →

v0.4.811 Mar 20202 sources agree

Support Installations of More Programming Languages. Now users can install or pre-install more programming languages including Java, .Net Core and PHP

Features / Enhancement

Support Installations of More Programming Languages. Now users can install or pre-install more programming languages including Java, .Net Core and PHP.
Installation UI Optimization. Users can better view and manage installations on Node List page.
More Git Support. Allow users to view Git Commits record, and allow checkout to corresponding commit.
Support Hostname Node Registration Type. Users can set hostname as the node key as the unique identifier.
RPC Support. Added RPC support to better manage node communication.
Run On Master Switch. Users can determine whether to run tasks on master. If not, all tasks will be run only on worker nodes.
Disabled Tutorial by Default.
Added Related Documentation Sidebar.
Loading Page Optimization.

Bug Fixes

Duplicated Nodes. #391
Duplicated Spider Upload. #603
Failure in dependencies installation results in unusable dependency installation functionalities.. #609
Create Tasks for Offline Nodes. #622

Open source →

v0.4.724 Feb 20202 sources agree

Better Support for Scrapy. Spiders identification, settings.py configuration, log level selection, spider selection. #435

Features / Enhancement

Better Support for Scrapy. Spiders identification, settings.py configuration, log level selection, spider selection. #435
Git Sync. Allow users to sync git projects to Crawlab.
Long Task Support. Users can add long-task spiders which is supposed to run without finishing. #425
Spider List Optimization. Tasks count by status, tasks detail popup, legend. #425
Upgrade Check. Check latest version and notifiy users to upgrade.
Spiders Batch Operation. Allow users to run/stop spider tasks and delete spiders in batches.
Copy Spiders. Allow users to copy an existing spider to create a new one.
Wechat Group QR Code.

Bug Fixes

Schedule Spider Selection Issue. Fields not responding to spider change.
Cron Jobs Conflict. Possible bug when two spiders set to the same time of their cron jobs. #515 #565
Task Log Issue. Different tasks write to the same log file if triggered at the same time. #577
Task List Filter Options Incomplete.

Open source →

v0.4.613 Feb 20202 sources agree

SDK for Node.js. Users can apply SDK in their Node.js spiders.

Features / Enhancement

SDK for Node.js. Users can apply SDK in their Node.js spiders.
Log Management Optimization. Log search, error highlight, auto-scrolling.
Task Execution Process Optimization. Allow users to be redirected to task detail page after triggering a task.
Task Display Optimization. Added "Param" in the Latest Tasks table in the spider detail page. #295
Spider List Optimization. Added "Update Time" and "Create Time" in spider list page.
Page Loading Placeholder.

Bug Fixes

Lost Focus in Schedule Configuration. #519
Unable to Upload Spider using CLI. #524

Open source →

v0.4.53 Feb 20202 sources agree

Interactive Tutorial. Guide users through the main functionalities of Crawlab.

Features / Enhancement

Interactive Tutorial. Guide users through the main functionalities of Crawlab.
Global Environment Variables. Allow users to set global environment variables, which will be passed into all spider programs. #177
Project. Allow users to link spiders to projects. #316
Demo Spiders. Added demo spiders when Crawlab is initialized. #379
User Admin Optimization. Restrict privilleges of admin users. #456
Setting Page Optimization.
Task Results Optimization.

Bug Fixes

Unable to find spider file error. #485
Click delete button results in redirect. #480
Unable to create files in an empty spider. #479
Download results error. #465
crawlab-sdk CLI error. #458
Page refresh issue. #441
Results not support JSON. #202
Getting all spider after deleting a spider.
i18n warning.

Open source →

v0.4.417 Jan 20202 sources agree

Email Notification. Allow users to send email notifications.

Features / Enhancement

Email Notification. Allow users to send email notifications.
DingTalk Robot Notification. Allow users to send DingTalk Robot notifications.
Wechat Robot Notification. Allow users to send Wechat Robot notifications.
API Address Optimization. Added relative URL path in frontend so that users don't have to specify CRAWLAB_API_ADDRESS explicitly.
SDK Compatiblity. Allow users to integrate Scrapy or general spiders with Crawlab SDK.
Enhanced File Management. Added tree-like file sidebar to allow users to edit files much more easier.
Advanced Schedule Cron. Allow users to edit schedule cron with visualized cron editor.

Bug Fixes

nil retuened error.
Error when using HTTPS.
Unable to run Configurable Spiders on Spider List.
Missing form validation before uploading spider files.

Open source →

v0.4.37 Jan 20202 sources agree

Dependency Installation. Allow users to install/uninstall dependencies and add programming languages (Node.js only for now) on the platform web interf

Features / Enhancement

Dependency Installation. Allow users to install/uninstall dependencies and add programming languages (Node.js only for now) on the platform web interface.
Pre-install Programming Languages in Docker. Allow Docker users to set CRAWLAB_SERVER_LANG_NODE as Y to pre-install Node.js environments.
Add Schedule List in Spider Detail Page. Allow users to view / add / edit schedule cron jobs in the spider detail page. #360
Align Cron Expression with Linux. Change the expression of 6 elements to 5 elements as aligned in Linux.
Enable/Disable Schedule Cron. Allow users to enable/disable the schedule jobs. #297
Better Task Management. Allow users to batch delete tasks. #341
Better Spider Management. Allow users to sort and filter spiders in the spider list page.
Added Chinese CHANGELOG.
Added Github Star Button at Nav Bar.

Bug Fixes

Schedule Cron Task Issue. #423
Upload Spider Zip File Issue. #403 #407
Exit due to Network Failure. #340
Cron Jobs not Running Correctly
Schedule List Columns Mis-positioned
Clicking Refresh Button Redirected to 404 Page

Open source →