Go modules · #1343 by repository stars

github.com/compose/transporter

v1.1.0compose/transporter ↗New releases in your daily digest, changelog included.

Last release 3 years ago
no release in 18 months
Ships unpredictably
gaps range from 2 weeks to 2.2 years
Nearly every release is documented
notes for 18 of 19 stable releases
Nothing withdrawn
no release was ever pulled
12 years old
34 releases · first in 2014

Release timeline

34 releases since 2014

20142016201820202022

Releases

v1.1.1-0.20231017154441-11b03ff7354a17 Oct 2023pre-release
Nothing published for this version
v1.1.026 May 2022
Merge mongo doc improvements by @anthonyalberto in #511
What's Changed

Merge mongo doc improvements by @anthonyalberto in #511

go mod download instructions by @anthonyalberto in #512

Fix tail flag description for mongo by @anthonyalberto in #513

Fix CI: Postgresql install was missing apt-get update by @atomicules in #516

Adaptor: Add MySQL by @atomicules in #515

New Contributors

@atomicules made their first contribution in #516

Full Changelog: v1.0.0...v1.1.0

v1.0.018 Nov 2021

Warning: only the mongodb and postgresql adaptors are tested on this new major.

Bumps Go to 1.17
Uses go mod
Provides a standard way of running adaptor tests, either locally or on CI
Switch from travis to Github actions

Open source →

Additional notes2 sources agree

This release does not introduce any new functionalities, but catches up transporter with modern golang. It also introduces a new standard way of running adaptor tests. Only the mongodb and postgresql adaptors have been fully tested. We'll add more as we go.

Bumped golang to 1.17
Migrated to go mod
Migrate from Travis CI to Github Actions
Standardize how adaptor tests are run. Using docker-compose, we can run them the same way on CI and locally.
Provide docker-compose and custom scripts to run mongodb and postgresql adaptor tests locally and on CI. See the respective adaptor README for more info.
Fix for postgresql tailer when there are new lines in a record, via #456, thanks @erft-engineering!
Reduce mongo bulk operation size to 1000, via #450, thanks @trinchan!

Open source →

v0.5.011 Sept 2017

See CHANGELOG for full details

Open source →

Additional notes3 sources agree

Features

new remap function for changing the namespace with a native function versus needing a custom JS function to do it, fixed via #400
new opfilter function for whitelisting or blacklisting message operations versus needing a custom JS function to do it, fixed via #401
elasticsearch adaptor now supports parent/child relationships, see fixed via #406, thanks @johnjjung!

Bugfixes

when using an adaptor that was configured for flushing bulk messages based on an interval, it was possible for the bulk flush to error but not propagated back up to the pipeline, fixed via #399
if using AWS Elasticsearch service, the process to determine the version of elasticsearch failed, fixed via #403

Changes

the -log_dir flag is now -xlog_dir and the log_dir config property is now xlog_dir to match. This is to avoid confusion with the application logging's -log flags.
elasticsearch adaptor now logs all errors returned from a bulk operation, #413, thanks @pmjhonwang!

Open source →

v0.4.015 Aug 2017

See CHANGELOG for full details

Open source →

Additional notes3 sources agree

Features

updated to Go 1.8
MongoDB Read Preferences #393, thanks @SamBartrum!
ability to support Continuous Change Data Capture as a Beta feature, see BETA Feature section on README

Bugfixes

when using MongoDB as a source with tailing enabled and namespace filtering, it was possible for documents from others collections to be sent down the pipeline, fixed via #386
if transporter lost connection to MongoDB while tailing the oplog, the connection never successfully reconnected, fixed via #398

Open source →

v0.4.0-rc.120 May 2017pre-release

This is the first release candidate for v0.4.0 with support for resume and writer tracking. Additional information can be found in the BETA Feature se

This is the first release candidate for v0.4.0 with support for resume and writer tracking. Additional information can be found in the BETA Feature section under about

Open source →

v0.3.124 Mar 2017

See CHANGELOG for full details

Open source →

Additional notes3 sources agree

Features

added js alias for goja transform function #335
init command will now prompt user when pipeline.js file already exists #336

Bugfixes

fixed mongodb, rabbitmq, and rethinkdb adaptors from not trying to read from a file when provided in the ca_certs field #334

Open source →

v0.3.021 Mar 2017Breaking

This release contained several breaking changes for configuring transporter, please take a look at the CHANGELOG for full details

This release contained several breaking changes for configuring transporter, please take a look at the CHANGELOG for full details

Open source →

Additional notes3 sources agree

Breaking changes

PLEASE READ

Transporter no longer requires a YAML file. All configuration is in the JS file where nodes are defined and a new DSL has been developed. Run the init command to see the new changes.

if using transporter as a library, all packages have been moved out of pkg to the top-level
eval command removed
list command removed
the namespace parameter now only expects a single part (the regexp filter), all adaptors have been updated to pull the "database name" from the provided URI

Features

NEW RabbitMQ adaptor #298
MongoDB adaptor supports per collection query filter when needing to copy only a subset of data #301
goja added as an option for the JavaScript VM in transformers #294
NEW native functions

Bugfixes

Open source →

v0.2.028 Feb 20173 sources agree

Integration tests are run weekly for the following pipelines:

Breaking changes

The etcd adaptor was removed

Features

Integration tests are run weekly for the following pipelines:
1. MongoDB -> MongoDB
2. MongoDB -> Elasticsearch
3. MongoDB -> RethinkDB
RethinkDB SSL support added in #268
RethinkDB performs bulk inserts now, #276
transporter init [source] [sink] command added in #279

Bugfixes

MongoDB adaptor could cause the pipeline to stop due to a concurrent flush operation, fixed via #271
When being used as a library, several goroutines were leaking after the pipeline had stopped. Reported by @cognusion in #265 and addressed via #268 and #280

Open source →

v0.1.39 Feb 20173 sources agree

namespace processing no longer expects their the be a "db" portion (i.e. "database.collection") but an attempt to maintain backwards compatibility is

Breaking changes

namespace processing no longer expects their the be a "db" portion (i.e. "database.collection") but an attempt to maintain backwards compatibility is still there for the time being. #258

Bugfixes

#261: return a nil message to get skipped in the pipeline

Open source →

v0.1.227 Jan 20173 sources agree

This release is primarily aimed at getting the MongoDB and Elasticsearch adaptors into a stable/reliable state.

Breaking changes

MongoDB adaptor SSL configuration is now defined as:

nodes:
  localmongo:
    type: mongodb
    ssl: true
    cacerts: ["/path/to/cert.pem"] # optional

Bugfixes

#211: defer bulk channel init for mongo node reuse
#213: track mongodb _id field so we can attempt to reissue queries
#233: update elasticsearch adaptor with better support for multiple versions of elasticsearch as well as better performance with bulk indexing for most versions. Addresses #209, #222, #167 and #159.
properly detect oplog access when attempting to use the tail option on the MongoDB adaptor.

Open source →

v0.1.127 Aug 20153 sources agree

This release contains the first step to getting savable state into adaptors for the ability to resume.

Features

#116: Adaptor state phase 1, begins to address #33

Bugfixes

#124: Update elasticsearch adaptor

Open source →

v0.1.06 Aug 2015Breaking3 sources agree

This release contains several breaking changes but is a first step towards a stable API/DSL.

Features

#95: add a Noop message type, and the ability to skip messages, fixes #93
#100: replace environment variables, fixes #88
#101: (Phase 1) Multi namespace support, addresses #78 and #23

Bugfixes

#94: jsonlog, fixes [#92](Comment doesnt reflect jsonlog output)
#98: Fixes influx_test.js, thanks @ExNexu!
#97: throw transformer initialization errors early, fixes #96

Open source →

v0.0.413 Jul 2015Breaking3 sources agree

This release contains several breaking changes:

Transporter now uses godep for vendoring depedencies.
transformer functions now receive a document in the following format:

{
  "op": "insert/update/delete",
  "data": "", // what use to be passed in the doc passed to the transformer function
  "ts": 1436794521, // the number of seconds elapsed since January 1, 1970 UTC
}

The same document must be returned from the function so any modifications must happen to doc["data"].

NOTE: Deletes are now passed to the transformer functions so this needs to be accounted for.

Features

#46: add a JsonLog.
#64: New Source Adapter: Implements RethinkDB as a source of documents. Thanks @alindeman!
#68: Deletes from Mongo write adaptor.
#72: Allows RethinkDB timeout to be configured.
#83: RethinkDB: Changes for v1 RethinkDB driver.
#86: Use godep to vendor required libraries.
#87: add SSL support for MongoDB, bump mgo.
#90: NOTE BREAKING CHANGE: send more info in doc to transformers.

Bugfixes

#47: A few readme grammar fixes. Thanks @mm-!
#49: fixes #48
#51: fixes #50
#54: import rethinkdb from gopkg.in. it's still v0, so this will change still.
#55: remove influx until the state of the repository settles or we get proper vendoring in place.
#63: use a smaller buffer for mongo adaptor.
#65: Extracts only the relevant pieces of the version string. Thanks @alindeman!
#85: ensure we're at version >=2.0 of rethink for this driver.

Open source →

v0.0.314 Jan 20153 sources agree

### Features - #36: Simplifies the config.yaml and node configuration. - #38: add a quick benchmark to transformOne ### Bugfixes - #37: fixes #29 - #4

Features

#36: Simplifies the config.yaml and node configuration.
#38: add a quick benchmark to transformOne

Bugfixes

#37: fixes #29
#42: fixes #41
#44: fix a logical error. check for transporter first.
#45: add a bulk writer, and writeconcern options for mongo.

Open source →

v0.0.229 Dec 20143 sources agree

### Features - #22: adaptor constructor refactor, new transporter about command. - #26: adds the name and description to about. - #27: Instructions fo

Features

#22: adaptor constructor refactor, new transporter about command.
#26: adds the name and description to about.
#27: Instructions for building Transporter on Windows.
#31: add some more information in errors that we get in the javascript.
#32: Fix typo in registry comments for doc gen.

Bugfixes

#7: Make list informative, stop arg panics.
#9: beginners guide for os x. Thanks @sberryman!
#10: Pretty print the list.
#11: fix the config.yaml example to show the proper interval syntax.
#13: makes the api in the config.yaml optional.
#18: check that errors aren't nil.
#20: log any adaptor.ERROR or adaptor.CRITICAL.

Open source →

github.com/compose/transporter

Release timeline

Releases

What's Changed

New Contributors

Features

Features

Bugfixes

Changes

Features

Bugfixes

Features

Bugfixes

Breaking changes

Features

Bugfixes

Features

Bugfixes

Features

Bugfixes

Breaking changes

Features

Bugfixes

Breaking changes

Bugfixes

Breaking changes

Bugfixes

Features

Bugfixes

Features

Bugfixes

Features

Bugfixes

Features

Bugfixes

Features

Bugfixes