github.com/compose/transporter

What's Changed

• Merge mongo doc improvements by @anthonyalberto in #511
• go mod download instructions by @anthonyalberto in #512
• Fix tail flag description for mongo by @anthonyalberto in #513
• Fix CI: Postgresql install was missing apt-get update by @atomicules in #516
• Adaptor: Add MySQL by @atomicules in #515

New Contributors

• @atomicules made their first contribution in #516

Full Changelog: v1.0.0...v1.1.0

Features

• Adds MySQL Adaptor via #515, thanks @atomicules!

Warning: only the mongodb and postgresql adaptors are tested on this new major.

• Bumps Go to 1.17
• Uses go mod
• Provides a standard way of running adaptor tests, either locally or on CI
• Switch from travis to Github actions

This release does not introduce any new functionalities, but catches up transporter with modern golang. It also introduces a new standard way of running adaptor tests. Only the mongodb and postgresql adaptors have been fully tested. We'll add more as we go.

• Bumped golang to 1.17
• Migrated to go mod
• Migrate from Travis CI to Github Actions
• Standardize how adaptor tests are run. Using docker-compose, we can run them the same way on CI and locally.
• Provide docker-compose and custom scripts to run mongodb and postgresql adaptor tests locally and on CI. See the respective adaptor README for more info.
• Fix for postgresql tailer when there are new lines in a record, via #456, thanks @erft-engineering!
• Reduce mongo bulk operation size to 1000, via #450, thanks @trinchan!

Order fixed to cd into transporter before building and get dependencies.

fixed growing memory issue via #436

See CHANGELOG for full details

Features

• new remap function for changing the namespace with a native function versus needing a custom JS function to do it, fixed via #400
• new opfilter function for whitelisting or blacklisting message operations versus needing a custom JS function to do it, fixed via #401
• elasticsearch adaptor now supports parent/child relationships, see fixed via #406, thanks @johnjjung!

Bugfixes

• when using an adaptor that was configured for flushing bulk messages based on an interval, it was possible for the bulk flush to error but not propagated back up to the pipeline, fixed via #399
• if using AWS Elasticsearch service, the process to determine the version of elasticsearch failed, fixed via #403

Changes

• the -log_dir flag is now -xlog_dir and the log_dir config property is now xlog_dir to match. This is to avoid confusion with the application logging's -log flags.
• elasticsearch adaptor now logs all errors returned from a bulk operation, #413, thanks @pmjhonwang!

See CHANGELOG for full details

Features

• updated to Go 1.8
• MongoDB Read Preferences #393, thanks @SamBartrum!
• ability to support Continuous Change Data Capture as a Beta feature, see BETA Feature section on README

Bugfixes

• when using MongoDB as a source with tailing enabled and namespace filtering, it was possible for documents from others collections to be sent down the pipeline, fixed via #386
• if transporter lost connection to MongoDB while tailing the oplog, the connection never successfully reconnected, fixed via #398

This is the first release candidate for v0.4.0 with support for resume and writer tracking. Additional information can be found in the BETA Feature section under about

See CHANGELOG for full details

Features

• added js alias for goja transform function #335
• init command will now prompt user when pipeline.js file already exists #336

Bugfixes

• fixed mongodb, rabbitmq, and rethinkdb adaptors from not trying to read from a file when provided in the ca_certs field #334

This release contained several breaking changes for configuring transporter, please take a look at the CHANGELOG for full details

Breaking changes

PLEASE READ

Transporter no longer requires a YAML file. All configuration is in the JS file where nodes are defined and a new DSL has been developed. Run the init command to see the new changes.

• if using transporter as a library, all packages have been moved out of pkg to the top-level
• eval command removed
• list command removed
• the namespace parameter now only expects a single part (the regexp filter), all adaptors have been updated to pull the "database name" from the provided URI

Features

• NEW RabbitMQ adaptor #298
• MongoDB adaptor supports per collection query filter when needing to copy only a subset of data #301
• goja added as an option for the JavaScript VM in transformers #294
• NEW native functions

Bugfixes

Features

Bugfixes

• attempted fix for mejson.S conversion to json column in postgres adaptor #314

Features

• added RethinkDB -> PostgreSQL integration test

Bugfixes

• fixed connection leak in PostgreSQL client

Breaking changes

• The etcd adaptor was removed

Features

• Integration tests are run weekly for the following pipelines:
  1. MongoDB -> MongoDB
  2. MongoDB -> Elasticsearch
  3. MongoDB -> RethinkDB
• RethinkDB SSL support added in #268
• RethinkDB performs bulk inserts now, #276
• transporter init [source] [sink] command added in #279

Bugfixes

• MongoDB adaptor could cause the pipeline to stop due to a concurrent flush operation, fixed via #271
• When being used as a library, several goroutines were leaking after the pipeline had stopped. Reported by @cognusion in #265 and addressed via #268 and #280

Breaking changes

• namespace processing no longer expects their the be a "db" portion (i.e. "database.collection") but an attempt to maintain backwards compatibility is still there for the time being. #258

Bugfixes

• #261: return a nil message to get skipped in the pipeline

This release is primarily aimed at getting the MongoDB and Elasticsearch adaptors into a stable/reliable state.

Breaking changes

• MongoDB adaptor SSL configuration is now defined as:

nodes:
  localmongo:
    type: mongodb
    ssl: true
    cacerts: ["/path/to/cert.pem"] # optional

Bugfixes

• #211: defer bulk channel init for mongo node reuse
• #213: track mongodb _id field so we can attempt to reissue queries
• #233: update elasticsearch adaptor with better support for multiple versions of elasticsearch as well as better performance with bulk indexing for most versions. Addresses #209, #222, #167 and #159.
• properly detect oplog access when attempting to use the tail option on the MongoDB adaptor.

This release contains the first step to getting savable state into adaptors for the ability to resume.

Features

• #116: Adaptor state phase 1, begins to address #33

Bugfixes

• #124: Update elasticsearch adaptor

This release contains several breaking changes but is a first step towards a stable API/DSL.

Features

• #95: add a Noop message type, and the ability to skip messages, fixes #93
• #100: replace environment variables, fixes #88
• #101: (Phase 1) Multi namespace support, addresses #78 and #23

Bugfixes

• #94: jsonlog, fixes [#92](Comment doesnt reflect jsonlog output)
• #98: Fixes influx_test.js, thanks @ExNexu!
• #97: throw transformer initialization errors early, fixes #96

This release contains several breaking changes:

• Transporter now uses godep for vendoring depedencies.
• transformer functions now receive a document in the following format:

{
  "op": "insert/update/delete",
  "data": "", // what use to be passed in the doc passed to the transformer function
  "ts": 1436794521, // the number of seconds elapsed since January 1, 1970 UTC
}

The same document must be returned from the function so any modifications must happen to doc["data"].

NOTE: Deletes are now passed to the transformer functions so this needs to be accounted for.

Features

• #46: add a JsonLog.
• #64: New Source Adapter: Implements RethinkDB as a source of documents. Thanks @alindeman!
• #68: Deletes from Mongo write adaptor.
• #72: Allows RethinkDB timeout to be configured.
• #83: RethinkDB: Changes for v1 RethinkDB driver.
• #86: Use godep to vendor required libraries.
• #87: add SSL support for MongoDB, bump mgo.
• #90: NOTE BREAKING CHANGE: send more info in doc to transformers.

Bugfixes

• #47: A few readme grammar fixes. Thanks @mm-!
• #49: fixes #48
• #51: fixes #50
• #54: import rethinkdb from gopkg.in. it's still v0, so this will change still.
• #55: remove influx until the state of the repository settles or we get proper vendoring in place.
• #63: use a smaller buffer for mongo adaptor.
• #65: Extracts only the relevant pieces of the version string. Thanks @alindeman!
• #85: ensure we're at version >=2.0 of rethink for this driver.

Features

• #36: Simplifies the config.yaml and node configuration.
• #38: add a quick benchmark to transformOne

Bugfixes

• #37: fixes #29
• #42: fixes #41
• #44: fix a logical error. check for transporter first.
• #45: add a bulk writer, and writeconcern options for mongo.

Features

• #22: adaptor constructor refactor, new transporter about command.
• #26: adds the name and description to about.
• #27: Instructions for building Transporter on Windows.
• #31: add some more information in errors that we get in the javascript.
• #32: Fix typo in registry comments for doc gen.

Bugfixes

• #7: Make list informative, stop arg panics.
• #9: beginners guide for os x. Thanks @sberryman!
• #10: Pretty print the list.
• #11: fix the config.yaml example to show the proper interval syntax.
• #13: makes the api in the config.yaml optional.
• #18: check that errors aren't nil.
• #20: log any adaptor.ERROR or adaptor.CRITICAL.