brickflows

What's Changed

• Task injection fix run time issue with tasktype support by @idurgakalyan in https://github.com/Nike-Inc/brickflow/pull/269

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.7.0...v1.7.1

What's Changed

• feature: add allow skip feature by @keerthipalanki in https://github.com/Nike-Inc/brickflow/pull/262
• Add description parameter to Workflow by @AmaniG77 in https://github.com/Nike-Inc/brickflow/pull/263
• Add workflow-specific task injection config files support by @idurgakalyan in https://github.com/Nike-Inc/brickflow/pull/265

New Contributors

• @keerthipalanki made their first contribution in https://github.com/Nike-Inc/brickflow/pull/262
• @AmaniG77 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/263

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.6.0...v1.7.0

What's Changed

• Add support for task injection based on template by @idurgakalyan in https://github.com/Nike-Inc/brickflow/pull/255
• [bugfix] Fix release workflow to trigger on published instead of crea… by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/258
• [bugfix] Fix CI workflow for Python 3.9 Poetry compatibility by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/260

New Contributors

• @idurgakalyan made their first contribution in https://github.com/Nike-Inc/brickflow/pull/255

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.5.0...v1.6.0

What's Changed

• [bugfix] Sync library versions for plugins with dev dependencies by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/249

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.4.1...v1.4.2

This release includes a fix for the regression introduced by #237, which causes the workflow deployment failure unless health rules are set.

What's Changed

• [BUG] 244 - Cannot deploy workflow without health rule by @riccamini in https://github.com/Nike-Inc/brickflow/pull/245

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.4.0...v1.4.1

What's Changed

• Add PythonWheelTask by @shinga in https://github.com/Nike-Inc/brickflow/pull/231
• [FIX] Add root parent project support for Tableau operators by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/230
• Add support for fail-on-active-runs flag by @nogitting in https://github.com/Nike-Inc/brickflow/pull/233
• Feature queueing workflows by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/226
• add comment why python version < 3.12 by @jacek6 in https://github.com/Nike-Inc/brickflow/pull/234
• Fixing a bug, including aarch64 by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/235
• Defaulting the jobs to PAUSED status for local, dev and test environments by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/236
• [FIX] 209 - Workflow jobs health type by @riccamini in https://github.com/Nike-Inc/brickflow/pull/237
• Run onpush workflow on multiple python versions by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/243
• Support intra-project workflow references for RunAsJob by @stikkireddy @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/242
• Add support for Python 3.12 and DBR 16.4.x by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/240

New Contributors

• @jacek6 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/234

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.3.3...v1.4.0

What's Changed

• [BUG] 222 - For each task executes nested brickflow task during deployment by @riccamini in https://github.com/Nike-Inc/brickflow/pull/228
• [BUG] 217 - Default task setting 'timeout' cannot be applied to if/else and for each task types by @riccamini in https://github.com/Nike-Inc/brickflow/pull/221
• [fix] Update Brickflow CLI version to 0.236.0 in bundle_synth by @mikita-sakalouski in https://github.com/Nike-Inc/brickflow/pull/220

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.3.2...v1.3.3

What's Changed

• [FIX] update the library versions by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/208
• [docs] update quick start guide for v1.3.1 by @meghana-0211 in https://github.com/Nike-Inc/brickflow/pull/210
• [FIX] - 46 Deployment failing instead of skipping when no files are synthesized by @riccamini in https://github.com/Nike-Inc/brickflow/pull/212
• [FIX] WorkflowDependencySensor doesn't fail the workflow run if invalid "dependency_job_id" is given by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/206
• [FIX] Added webhook notification into TaskSettings by @muhammadwaqas0505 in https://github.com/Nike-Inc/brickflow/pull/211
• [docs]: 35-Updated docs for Job Notification Settings by @muhammadwaqas0505 in https://github.com/Nike-Inc/brickflow/pull/214
• [FIX] - Provide option to override pickle+base64 serde for task values using BrickflowTaskComs by @riccamini in https://github.com/Nike-Inc/brickflow/pull/216

New Contributors

• @meghana-0211 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/210
• @muhammadwaqas0505 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/211

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.3.1...v1.3.2

What's Changed

• [FIX] Uncaught exception IndexError when calling get_project_parameter by @menathan in https://github.com/Nike-Inc/brickflow/pull/197
• [FIX] Add missing data security modes by @riccamini in https://github.com/Nike-Inc/brickflow/pull/198
• [FIX] Enhance support for debug mode by @nogitting in https://github.com/Nike-Inc/brickflow/pull/201
• [FIX] Replace CronUtils library by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/196
• [FIX] updating to fix python deployment by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/205

New Contributors

• @menathan made their first contribution in https://github.com/Nike-Inc/brickflow/pull/197
• @nogitting made their first contribution in https://github.com/Nike-Inc/brickflow/pull/201

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.3.0...v1.3.1

What's Changed

• [feature] Enable support for Serverless Compute by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/165
• [BUG] Missing common task parameters for Notebook Task by @riccamini in https://github.com/Nike-Inc/brickflow/pull/178
• [fix] UcToSnowflakeOperator multiple grantees and avoid timestamp range error by @shinga in https://github.com/Nike-Inc/brickflow/pull/176
• Add fail_on_error parameter to SnowflakeOperator by @shinga in https://github.com/Nike-Inc/brickflow/pull/179
• 150 - Support for yaml extension in project conf files by @riccamini in https://github.com/Nike-Inc/brickflow/pull/175
• Implement SLASensor by @chaelant in https://github.com/Nike-Inc/brickflow/pull/177
• [BUG] 182 - Fix brickflow deploy by @riccamini in https://github.com/Nike-Inc/brickflow/pull/183
• [FEATURE] 163 - For each task by @riccamini in https://github.com/Nike-Inc/brickflow/pull/185

New Contributors

• @riccamini made their first contribution in https://github.com/Nike-Inc/brickflow/pull/178
• @shinga made their first contribution in https://github.com/Nike-Inc/brickflow/pull/176
• @chaelant made their first contribution in https://github.com/Nike-Inc/brickflow/pull/177

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.2.1...v1.3.0

What's Changed

• [fix] Improve logic for local deployment of PythonScript by @mikita-sakalouski in https://github.com/Nike-Inc/brickflow/pull/168
• [fix] Adjust logic for applying outcome for IF_ELSE_CONDITION tasks by @mikita-sakalouski in https://github.com/Nike-Inc/brickflow/pull/173

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.2.0...v1.2.1

What's Changed

• Add spark_python_task support in Workflow by @mikita-sakalouski in https://github.com/Nike-Inc/brickflow/pull/156
• Add workflow parameters by @mikita-sakalouski in https://github.com/Nike-Inc/brickflow/pull/158
• propogate workflow tags to cluster tags by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/160
• Feature/support dbr 15.x by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/161
• Allow optional end-date in get_execution_stats() by @IMC07 in https://github.com/Nike-Inc/brickflow/pull/155

New Contributors

• @mikita-sakalouski made their first contribution in https://github.com/Nike-Inc/brickflow/pull/156
• @IMC07 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/155

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.1.1...v1.2.0

What's Changed

• Patch fix ENV variable setter by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/153

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.1.0...v1.1.1

What's Changed

• [FEATURE] Support remote workspace for RunJobTask by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/140
• Adding BoxToVolumesOperator, VolumesToBoxOperator and BoxOperator by @madhusudan3 in https://github.com/Nike-Inc/brickflow/pull/144
• add support for continuous schedule by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/142
• Support runtime project params by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/147
• support for selected workflows by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/146
• Feature/runtime workflow tags by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/149

New Contributors

• @madhusudan3 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/144

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v1.0.0...v1.1.0

What's Changed

• Feature run job task, sparkJAR and SQL task by @RajuGujjalapati in https://github.com/Nike-Inc/brickflow/pull/117
• Add sql file path to Snowflake Operator and add custom sql for UctoSnowflakeOperator by @bsangars in https://github.com/Nike-Inc/brickflow/pull/121
• [fix] WorkflowTaskDependecySensor fails if the task execution has not started by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/123
• Updating Databricks cli, bundles version to v0.220.0 by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/127
• updating dependencies by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/129
• Update proj-to-bundle method to include trigger. by @suarzac in https://github.com/Nike-Inc/brickflow/pull/130
• TaskDependencySensor: do not fail if upstream failed / derive execution time from context by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/132
• Feature if else task by @RajuGujjalapati in https://github.com/Nike-Inc/brickflow/pull/136

New Contributors

• @RajuGujjalapati made their first contribution in https://github.com/Nike-Inc/brickflow/pull/117
• @suarzac made their first contribution in https://github.com/Nike-Inc/brickflow/pull/130

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.11.4...v1.0.0

What's Changed

• [fix] AutosysSensor: Handle the situations where lastEndUTC is empty or None by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/113

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.11.3...v0.11.4

What's Changed

• fix ignore permission error due to most likely being caused by shared clusters by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/97
• added fix for snowflake operator to accept authenticator and make parameters optional based on load_type by @ckonuganti in https://github.com/Nike-Inc/brickflow/pull/92
• [BUG] Change date/time parsing in AutosysSensor by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/89
• [FEATURE] Optional auth for AutosysSensor by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/87
• TableauRefreshABCOperator, TableauRefreshDataSourceOperator and TableauRefreshWorkBookOperator by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/91
• Feature/bump plugin dependencies for pydantic 2.x by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/102
• Throw an error when tableauserverclient is not found with resolution instructions by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/104
• [FIX] Cluster policy_id converted to scientific notation by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/107
• [FEATURE] Sanitise input of the brickflow_internal_only_run_tasks parameter by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/108
• [FEATURE] Add WorkflowTaskDependencySensor by @maxim-mityutko in https://github.com/Nike-Inc/brickflow/pull/110

New Contributors

• @maxim-mityutko made their first contribution in https://github.com/Nike-Inc/brickflow/pull/89

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.11.2...v0.11.3

What's Changed

• Add timeout parameter to workflow class by @brent-johnson in https://github.com/Nike-Inc/brickflow/pull/83

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.11.1...v0.11.2

What's Changed

• added fix for import block type to support both str and int and added… by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/78
• fixed code gen for model to ensure that all tests pass by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/80

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.11.0...v0.11.1

What's Changed

• Pydantic 2.4 support by @dannymeijer in https://github.com/Nike-Inc/brickflow/pull/59
• updating contributors by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/62
• updated the model to the latest version of the bundles cli (0.210.2) and update… by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/51
• adding uc to snowflake operator by @ckonuganti in https://github.com/Nike-Inc/brickflow/pull/69

New Contributors

• @dannymeijer made their first contribution in https://github.com/Nike-Inc/brickflow/pull/59
• @ckonuganti made their first contribution in https://github.com/Nike-Inc/brickflow/pull/69

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.10.3...v0.11.0

What's Changed

• Fix: Title for autosys sensor in documentation by @rebecca-rajshree in https://github.com/Nike-Inc/brickflow/pull/50
• add run_if param to TaskSettings by @grigoriy835 in https://github.com/Nike-Inc/brickflow/pull/53
• Add ability to set timeout health check in workflows by @brent-johnson in https://github.com/Nike-Inc/brickflow/pull/58
• Fix issue 48 and 60 related to the WorkflowDependencySensor by @BrendBraeckmans in https://github.com/Nike-Inc/brickflow/pull/61

New Contributors

• @grigoriy835 made their first contribution in https://github.com/Nike-Inc/brickflow/pull/53
• @brent-johnson made their first contribution in https://github.com/Nike-Inc/brickflow/pull/58

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.10.2...v0.10.3

What's Changed

• Updated TaskDependency Sensor to account for execution delta and enabled logging for debugging purposes by @bsangars in https://github.com/Nike-Inc/brickflow/pull/38
• updating contributors and read me by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/41
• Feature support job schedule pause status by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/42
• Autosys Sensor by @rebecca-rajshree in https://github.com/Nike-Inc/brickflow/pull/33
• Make node_type_id optional as it's otherwise impossible to create a cluster from resources in a pool by @BrendBraeckmans in https://github.com/Nike-Inc/brickflow/pull/44

New Contributors

• @bsangars made their first contribution in https://github.com/Nike-Inc/brickflow/pull/38
• @rebecca-rajshree made their first contribution in https://github.com/Nike-Inc/brickflow/pull/33
• @BrendBraeckmans made their first contribution in https://github.com/Nike-Inc/brickflow/pull/44

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.10.1...v0.10.2

What's Changed

• made logos transparent by @jhollow in https://github.com/Nike-Inc/brickflow/pull/25
• Fix/brickflow examples by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/36
• Addressing the issue - #34 by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/37

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.10.0...v0.10.1

What's Changed

• Feature remove cdktf by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/18
• Adding pull request and issue templates by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/19
• fix: generate the bundle path including prefix/suffix if available by @pariksheet in https://github.com/Nike-Inc/brickflow/pull/20
• fixed documentation and refactored external task dependency sensor. by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/21

New Contributors

• @pariksheet made their first contribution in https://github.com/Nike-Inc/brickflow/pull/20

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.9.2...v0.10.0

What's Changed

• Adding code cov and github badges by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/13
• Fix/brickflow version by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/14

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.9.1...v0.9.2

What's Changed

• Multiple changes to move package to brickflows by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/11
  • Updating the package name to brickflows as pypi won't accept brickflow
  • Updating documentation and logo link
  • Updating codeowners
• modify all code to reference brickflows by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/12

Full Changelog: https://github.com/Nike-Inc/brickflow/compare/v0.9.0...v0.9.1

What's Changed

This is the first release for brickflow, which is a tool to build Databricks workflows using python. Right now brickflow supports CDKTF(to be deprecated soon) and Databricks Asset Bundles.

• Moved the source code from Internal Nike repository to the OSS public repo contributed by @asingamaneni & @stikkireddy
• Github actions initial setup by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/3
• Multiple changes by @stikkireddy in https://github.com/Nike-Inc/brickflow/pull/2
  • commit for multi projects and reimporting existing workflows with same name
  • fixed tests and added support for notebook native tasks
  • added some comments and examples of how to use parameters.
  • changes to help string for project add and also resolving recursively to root from notebook context as well.
  • added /Workspace root for getting notebook path
  • fixed removing projects correctly from the multi-project.yml file as well
  • fixed tests and added support for multiple project deployments in the same directory by making the environment names different values
  • be lax with development for local development if git commit yields nothing
  • added support for workflow level notifications and webhooks, task notification settings and support for photon in compute. also root, filepath and statepath will write posix paths. python binary will be fetched from entrypoint executable that is used
  • fixed unnecessary code in plugins and added a feature flag for using project name as part of the env
  • added synth command for projects
  • added local task settings at task level and also added workflow level settings for webhooks, emails and notifications
  • suffixed all the workflow level settings with Workflow due to import from brickflow making it confusing with task level objects
  • make sure import manager uses the new import folder syntax structure.
  • make sure imports only work with projects tagged with this brickflow project, permissive of duplicates still
  • clean up logging entry points to only log when project is kicked off or cli is called.
  • clear docs on resolver for imports and proper docs
  • remove resolver and properly load plugins for airflow and py4j (cronhelper) integration
  • made cronhelper lazy, added unit tests for plugins and added py4j as a dev dependency
  • added setup for java for github actions
  • automatically add the brickflow library to the cluster and flag to opt out. Run bundles cli from anywhere within the project root
  • set the monorepo path variable for correct bundle paths
  • updated the ascii art to be more legible
  • fixed bug related to suffix and prefix not being properly populated during runtime.
  • update databricks cli to 0.203.0
  • fixed library handling
  • update all deps
  • separate airflow installation from brickflow and remove it as optional
  • added deployment job to pypi and fixed issue with code coverage
• Add logos and update README.md by @jhollow in https://github.com/Nike-Inc/brickflow/pull/6
• Adding github badges by @asingamaneni in https://github.com/Nike-Inc/brickflow/pull/5

New Contributors

• @stikkireddy made their first contribution in https://github.com/Nike-Inc/brickflow/pull/3
• @jhollow made their first contribution in https://github.com/Nike-Inc/brickflow/pull/6
• @asingamaneni made their first contribution in https://github.com/Nike-Inc/brickflow/pull/5

Full Changelog: https://github.com/Nike-Inc/brickflow/commits/v0.9.0