github.com/smartcontractkit/chainlink

Added

New optional external logger added

AUDIT_LOGGER_FORWARD_TO_URL

• Default: none

When set, this environment variable configures and enables an optional HTTP logger which is used specifically to send audit log events. Audit logs events are emitted when specific actions are performed by any of the users through the node's API. The value of this variable should be a full URL. Log items will be sent via POST

There are audit log implemented for the following events:

• Auth & Sessions (new session, login success, login failed, 2FA enrolled, 2FA failed, password reset, password reset failed, etc.)
• CRUD actions for all resources (add/create/delete resources such as bridges, nodes, keys)
• Sensitive actions (keys exported/imported, config changed, log level changed, environment dumped)

A full list of audit log enum types can be found in the source within the audit package (audit_types.go).

The following AUDIT_LOGGER_* environment variables below configure this optional audit log HTTP forwarder.

AUDIT_LOGGER_HEADERS

• Default: none

An optional list of HTTP headers to be added for every optional audit log event. If the above AUDIT_LOGGER_FORWARD_TO_URL is set, audit log events will be POSTed to that URL, and will include headers specified in this environment variable. One example use case is auth for example: AUDIT_LOGGER_HEADERS="Authorization||{{token}}".

Header keys and values are delimited on ||, and multiple headers can be added with a forward slash delimiter ('\'). An example of multiple key value pairs: AUDIT_LOGGER_HEADERS="Authorization||{{token}}\Some-Other-Header||{{token2}}"

AUDIT_LOGGER_JSON_WRAPPER_KEY

• Default: none

When the audit log HTTP forwarder is enabled, if there is a value set for this optional environment variable then the POST body will be wrapped in a dictionary in a field specified by the value of set variable. This is to help enable specific logging service integrations that may require the event JSON in a special shape. For example: AUDIT_LOGGER_JSON_WRAPPER_KEY=event will create the POST body:

{
  "event": {
    "eventID":  EVENT_ID_ENUM,
    "data": ...
  }
}

Automatic connectivity detection; Chainlink will no longer bump excessively if the network is broken

This feature only applies on EVM chains when using BlockHistoryEstimator (the most common case).

Chainlink will now try to automatically detect if there is a transaction propagation/connectivity issue and prevent bumping in these cases. This can help avoid the situation where RPC nodes are not propagating transactions for some reason (e.g. go-ethereum bug, networking issue etc) and Chainlink responds in a suboptimal way by bumping transactions to a very high price in an effort to get them mined. This can lead to unnecessary expense when the connectivity issue is resolved and the transactions are finally propagated into the mempool.

This feature is enabled by default with fairly conservative settings: if a transaction has been priced above the 90th percentile of the past 12 blocks, but still wants to bump due to not being mined, a connectivity/propagation issue is assumed and all further bumping will be prevented for this transaction. In this situation, Chainlink will start firing the block_history_estimator_connectivity_failure_count prometheus counter and logging at critical level until the transaction is mined.

The default settings should work fine for most users. For advanced users, the values can be tweaked by changing BLOCK_HISTORY_ESTIMATOR_CHECK_INCLUSION_BLOCKS and BLOCK_HISTORY_ESTIMATOR_CHECK_INCLUSION_PERCENTILE.

To disable connectivity checking completely, set BLOCK_HISTORY_ESTIMATOR_CHECK_INCLUSION_BLOCKS=0.

Changed

• The default maximum gas price on most networks is now effectively unlimited.

  • Chainlink will bump as high as necessary to get a transaction included. The connectivity checker is relied on to prevent excessive bumping when there is a connectivity failure.
  • If you want to change this, you can manually set ETH_MAX_GAS_PRICE_WEI.

• EVMChainID field will be auto-added with default chain id to job specs of newly created OCR jobs, if not explicitly included.

  • Old OCR jobs missing EVMChainID will continue to run on any chain ETH_CHAIN_ID is set to (or first chain if unset), which may be changed after a restart.
  • Newly created OCR jobs will only run on a single fixed chain, unaffected by changes to ETH_CHAIN_ID after the job is added.
  • It should no longer be possible to end up with multiple OCR jobs for a single contract running on the same chain; only one job per contract per chain is allowed
  • If there are any existing duplicate jobs (per contract per chain), all but the job with the latest creation date will be pruned during upgrade.

Fixed

• Fixed minor bug where Chainlink would attempt (and fail) to estimate a tip cap higher than the maximum configured gas price in EIP1559 mode. It now caps the tipcap to the max instead of erroring.
• Fixed bug whereby it was impossible to remove eth keys that had extant transactions. Now, removing an eth key will drop all associated data automatically including past transactions.

1.9.0 - 2022-10-12

Added

• Added length and lessthan tasks (pipeline).
• Added gasUnlimited parameter to ethcall task.
• /keys page in Operator UI now exposes several admin commands, namely:
  • "abandon" to abandon all current txes
  • enable/disable a key for a given chain
  • manually set the nonce for a key See this PR for a screenshot example.

1.8.1 - 2022-09-29

Added

• New GAS_ESTIMATOR_MODE for Arbitrum to support Nitro's multi-dimensional gas model, with dynamic gas pricing and limits.
  • NOTE: It is recommended to remove GAS_ESTIMATOR_MODE as an env var if you have it set in order to use the new default.
  • This new, default estimator for Arbitrum networks uses the suggested gas price (up to ETH_MAX_GAS_PRICE_WEI, with 1000 gwei default) as well as an estimated gas limit (up to ETH_GAS_LIMIT_MAX, with 1,000,000,000 default).
• ETH_GAS_LIMIT_MAX to put a maximum on the gas limit returned by the Arbitrum estimator.

Changed

• EIP1559 is now enabled by default on Goerli network

1.8.0 - 2022-09-01

Added

• Added hexencode and base64encode tasks (pipeline).
• forwardingAllowed per job attribute to allow forwarding txs submitted by the job.
• Keypath now supports paths with any depth, instead of limiting it to 2
• Arbitrum chains are no longer restricted to only FixedPrice GAS_ESTIMATOR_MODE
• Updated Arbitrum Rinkeby & Mainnet & Mainnet configurationss for Nitro
• Add Arbitrum Goerli configuration
• It is now possible to use the same key across multiple chains.
• NODE_SELECTION_MODE (EVM.NodePool.SelectionMode) controls node picking strategy. Supported values: HighestHead (default) and RoundRobin:
  • RoundRobin mode simply iterates among available alive nodes. This was the default behavior prior to this release.
  • HighestHead mode picks a node having the highest reported head number among other alive nodes. When several nodes have the same latest head number, the strategy sticks to the last used node. For chains having NODE_NO_NEW_HEADS_THRESHOLD=0 (such as Arbitrum, Optimism), the implementation will fall back to RoundRobin mode.
• New keys eth chain command
  • This can also be accessed at /v2/keys/evm/chain.
  • Usage examples:
    • Manually (re)set a nonce:
      • chainlink keys eth chain --address "0xEXAMPLE" --evmChainID 99 --setNextNonce 42
    • Enable a key for a particular chain:
      • chainlink keys eth chain --address "0xEXAMPLE" --evmChainID 99 --enable
    • Disable a key for a particular chain:
      • chainlink keys eth chain --address "0xEXAMPLE" --evmChainID 99 --disable
    • Abandon all currently pending transactions (use with caution!):
      • chainlink evm keys chain --address "0xEXAMPLE" --evmChainID 99 --abandon
  • Commands can be combined e.g.
    • Reset nonce and abandon all currently pending transaction:
      • chainlink evm keys chain --address "0xEXAMPLE" --evmChainID 99 --setNextNonce 42 --abandon

Changed

• The setnextnonce local client command has been removed, and replaced by a more general key/chain client command.
• chainlink admin users update command is replaced with chainlink admin users chrole (only the role can be changed for a user)

1.7.1 - 2022-08-22

Added

• Arbitrum Nitro client error support

1.7.0 - 2022-08-08

Added

• p2pv2Bootstrappers has been added as a new optional property of OCR1 job specs; default may still be specified with P2PV2_BOOTSTRAPPERS config param
• Added official support for Sepolia chain
• Added hexdecode and base64decode tasks (pipeline).
• Added support for Besu execution client (note that while Chainlink supports Besu, Besu itself has multiple bugs that make it unreliable).
• Added the functionality to allow the root admin CLI user (and any additional admin users created) to create and assign tiers of role based access to new users. These new API users will be able to log in to the Operator UI independently, and can each have specific roles tied to their account. There are four roles: admin, edit, run, and view.
  • User management can be configured through the use of the new admin CLI command chainlink admin users. Be sure to run chainlink adamin login. For example, a readonly user can be created with: chainlink admin users create --email=operator-ui-read-only@test.com --role=view.
  • Updated documentation repo with a break down of actions to required role level
• Added per job spec and per job type gas limit control. The following rule of precedence is applied:

1. task-specific parameter gasLimit overrides anything else when specified (e.g. ethtx task has such a parameter).
2. job-spec attribute gasLimit has the scope of the current job spec only.
3. job-type limits ETH_GAS_LIMIT_*_JOB_TYPE affect any jobs of the corresponding type:

ETH_GAS_LIMIT_OCR_JOB_TYPE    # EVM.GasEstimator.LimitOCRJobType
ETH_GAS_LIMIT_DR_JOB_TYPE     # EVM.GasEstimator.LimitDRJobType
ETH_GAS_LIMIT_VRF_JOB_TYPE    # EVM.GasEstimator.LimitVRFJobType
ETH_GAS_LIMIT_FM_JOB_TYPE     # EVM.GasEstimator.LimitFMJobType
ETH_GAS_LIMIT_KEEPER_JOB_TYPE # EVM.GasEstimator.LimitKeeperJobType

1. global ETH_GAS_LIMIT_DEFAULT (EVM.GasEstimator.LimitDefault) value is the last resort.

Fixed

• Addressed a very rare bug where using multiple nodes with differently configured RPC tx fee caps could cause missed transaction. Reminder to everyone to ensure that your RPC nodes have no caps (for more information see the performance and tuning guide).
• Improved handling of unknown transaction error types, making Chainlink more robust in certain cases on unsupported chains/RPC clients

[1.6.0] - 2022-07-20

Changed

• After feedback from users, password complexity requirements have been simplified. These are the new, simplified requirements for any kind of password used with Chainlink:

1. Must be 16 characters or more
2. Must not contain leading or trailing whitespace
3. User passwords must not contain the user's API email

• Simplified the Keepers job spec by removing the observation source from the required parameters.

[1.5.1] - 2022-06-27

Fixed

• Fix rare out-of-sync to invalid-chain-id transaction
• Fix key-specific max gas limits for gas estimator and ensure we do not bump gas beyond key-specific limits
• Fix EVM_FINALITY_DEPTH => ETH_FINALITY_DEPTH

[1.5.0] - 2022-06-21

Changed

• Chainlink will now log a warning if the postgres database password is missing or too insecure. Passwords should conform to the following rules:

Must be longer than 12 characters
Must comprise at least 3 of:
	lowercase characters
	uppercase characters
	numbers
	symbols
Must not comprise:
	More than three identical consecutive characters
	Leading or trailing whitespace (note that a trailing newline in the password file, if present, will be ignored)

For backward compatibility all insecure passwords will continue to work, however in a future version of Chainlink insecure passwords will prevent application boot. To bypass this check at your own risk, you may set SKIP_DATABASE_PASSWORD_COMPLEXITY_CHECK=true.

• MIN_OUTGOING_CONFIRMATIONS has been removed and no longer has any effect. ETH_FINALITY_DEPTH is now used as the default for ethtx confirmations instead. You may override this on a per-task basis by setting minConfirmations in the task definition e.g. foo [type=ethtx minConfirmations=42 ...]. NOTE: This may have a minor impact on performance on very high throughput chains. If you don't care about reporting task status in the UI, it is recommended to set minConfirmations=0 in your job specs. For more details, see the relevant section of the performance tuning guide.

• The following ENV variables have been deprecated, and will be removed in a future release: INSECURE_SKIP_VERIFY, CLIENT_NODE_URL, ADMIN_CREDENTIALS_FILE. These vars only applied to Chainlink when running in client mode and have been replaced by command line args, notably: --insecure-skip-verify, --remote-node-url URL and --admin-credentials-file FILE respectively. More information can be found by running ./chainlink --help.

• The Optimism2 GAS_ESTIMATOR_MODE has been renamed to L2Suggested. The old name is still supported for now.

• The p2pBootstrapPeers property on OCR2 job specs has been renamed to p2pv2Bootstrappers.

Added

• Added ETH_USE_FORWARDERS config option to enable transactions forwarding contracts.
• In job pipeline (direct request) the three new block variables are exposed:
  • $(jobRun.blockReceiptsRoot) : the root of the receipts trie of the block (hash)
  • $(jobRun.blockTransactionsRoot) : the root of the transaction trie of the block (hash)
  • $(jobRun.blockStateRoot) : the root of the final state trie of the block (hash)
• ethtx tasks can now be configured to error if the transaction reverts on-chain. You must set failOnRevert=true on the task to enable this behavior, like so:

foo [type=ethtx failOnRevert=true ...]

So the ethtx task now works as follows:

If minConfirmations == 0, task always succeeds and nil is passed as output If minConfirmations > 0, the receipt is passed through as output If minConfirmations > 0 and failOnRevert=true then the ethtx task will error on revert

If minConfirmations is not set on the task, the chain default will be used which is usually 12 and always greater than 0.

• http task now allows specification of request headers. Use like so: foo [type=http headers="[\\"X-Header-1\\", \\"value1\\", \\"X-Header-2\\", \\"value2\\"]"].

Fixed

• Fixed max_unconfirmed_age metric. Previously this would incorrectly report the max time since the last rebroadcast, capping the upper limit to the EthResender interval. This now reports the correct value of total time elapsed since the first broadcast.
• Correctly handle the case where bumped gas would exceed the RPC node's configured maximum on Fantom (note that node operators should check their Fantom RPC node configuration and remove the fee cap if there is one)
• Fixed handling of Metis internal fee change

Removed

• The Optimism OVM 1.0 GAS_ESTIMATOR_MODE has been removed.

[1.4.1] - 2022-05-11

Fixed

• Ensure failed EthSubscribe didn't register a (*rpc.ClientSubscription)(nil) which would lead to a panic on Unsubscribe
• Fixes parsing of float values on job specs

[1.4.0] - 2022-05-02

Added

• JSON parse tasks (v2) now support a custom separator parameter to substitute for the default ,.
• Log slow SQL queries
• Fantom and avalanche block explorer urls
• Display requestTimeout in job UI
• Keeper upkeep order is shuffled

Fixed

• LOG_FILE_MAX_SIZE handling
• Improved websocket subscription management (fixes issues with multiple-primary-node failover from 1.3.x)
• VRFv2 fixes and enhancements
• UI support for minContractPaymentLinkJuels

[1.3.0] - 2022-04-18

Added

• Added support for Keeper registry v1.2 in keeper jobs
• Added disk rotating logs. Chainlink will now always log to disk at debug level. The default output directory for debug logs is Chainlink's root directory (ROOT_DIR) but can be configured by setting LOG_FILE_DIR. This makes it easier for node operators to report useful debugging information to Chainlink's team, since all the debug logs are conveniently located in one directory. Regular logging to STDOUT still works as before and respects the LOG_LEVEL env var. If you want to log in disk at a particular level, you can pipe STDOUT to disk. This automatic debug-logs-to-disk feature is enabled by default, and will remain enabled as long as the LOG_FILE_MAX_SIZE ENV var is set to a value greater than zero. The amount of disk space required for this feature to work can be calculated with the following formula: LOG_FILE_MAX_SIZE * (LOG_FILE_MAX_BACKUPS + 1). If your disk doesn't have enough disk space, the logging will pause and the application will log Errors until space is available again. New environment variables related to this feature:
  • LOG_FILE_MAX_SIZE (default: 5120mb) - this env var allows you to override the log file's max size (in megabytes) before file rotation.
  • LOG_FILE_MAX_AGE (default: 0) - if LOG_FILE_MAX_SIZE is set, this env var allows you to override the log file's max age (in days) before file rotation. Keeping this config with the default value means not to remove old log files.
  • LOG_FILE_MAX_BACKUPS (default: 1) - if LOG_FILE_MAX_SIZE is set, this env var allows you to override the max amount of old log files to retain. Keeping this config with the default value means to retain 1 old log file at most (though LOG_FILE_MAX_AGE may still cause them to get deleted). If this is set to 0, the node will retain all old log files instead.
• Added support for the force flag on chainlink blocks replay. If set to true, already consumed logs that would otherwise be skipped will be rebroadcasted.
• Added version compatibility check when using CLI to login to a remote node. flag bypass-version-check skips this check.
• Interrim solution to set multiple nodes/chains from ENV. This gives the ability to specify multiple RPCs that the Chainlink node will constantly monitor for health and sync status, detecting dead nodes and out of sync nodes, with automatic failover. This is a temporary stand-in until configuration is overhauled and will be removed in future in favor of a config file. Set as such: EVM_NODES='{...}' where the var is a JSON array containing the node specifications. This is not compatible with using any other way to specify node via env (e.g. ETH_URL, ETH_SECONDARY_URL, ETH_CHAIN_ID etc). WARNING: Setting this environment variable will COMPLETELY ERASE your evm_nodes table on every boot and repopulate from the given data, nullifying any runtime modifications. Make sure to carefully read the EVM performance configuration guide for best practices here.

For example:

export EVM_NODES='
[
	{
		"name": "primary_1",
		"evmChainId": "137",
		"wsUrl": "wss://endpoint-1.example.com/ws",
    "httpUrl": "http://endpoint-1.example.com/",
		"sendOnly": false
	},
	{
		"name": "primary_2",
		"evmChainId": "137",
		"wsUrl": "ws://endpoint-2.example.com/ws",
    "httpUrl": "http://endpoint-2.example.com/",
		"sendOnly": false
	},
	{
		"name": "primary_3",
		"evmChainId": "137",
		"wsUrl": "wss://endpoint-3.example.com/ws",
    "httpUrl": "http://endpoint-3.example.com/",
		"sendOnly": false
	},
	{
		"name": "sendonly_1",
		"evmChainId": "137",
		"httpUrl": "http://endpoint-4.example.com/",
		"sendOnly": true
	},
  {
		"name": "sendonly_2",
		"evmChainId": "137",
		"httpUrl": "http://endpoint-5.example.com/",
		"sendOnly": true
	}
]
'

Changed

• Changed default locking mode to "dual". Bugs in lease locking have been ironed out and this paves the way to making "lease" the default in the future. It is recommended to set DATABASE_LOCKING_MODE=lease, default is set to "dual" only for backwards compatibility.
• EIP-1559 is now enabled by default on mainnet. To disable (go back to legacy mode) set EVM_EIP1559_DYNAMIC_FEES=false. The default settings should work well, but if you wish to tune your gas controls, see the documentation.

Note that EIP-1559 can be manually enabled on other chains by setting EVM_EIP1559_DYNAMIC_FEES=true but we only support it for official Ethereum mainnet and testnets. It is not recommended enabling this setting on Polygon since during our testing process we found that the EIP-1559 fee market appears to be broken on all Polygon chains and EIP-1559 transactions are actually less likely to get included than legacy transactions.

See issue: https://github.com/maticnetwork/bor/issues/347

• The pipeline task runs have changed persistence protocol (database), which will result in inability to decode some existing task runs. All new runs should be working with no issues.

Removed

• LOG_TO_DISK ENV var.

[1.2.1] - 2022-03-17

This release hotfixes issues from moving a new CI/CD system. Feature-wise the functionality is the same as v1.2.0.

Fixed

• Fixed CI/CD issue where environment variables were not being passed into the underlying build

[1.2.0] - 2022-03-02

Added

• Added support for the Nethermind Ethereum client.
• Added support for batch sending telemetry to the ingress server to improve performance.
• Added v2 P2P networking support (alpha)

New ENV vars:

• ADVISORY_LOCK_CHECK_INTERVAL (default: 1s) - when advisory locking mode is enabled, this controls how often Chainlink checks to make sure it still holds the advisory lock. It is recommended to leave this at the default.
• ADVISORY_LOCK_ID (default: 1027321974924625846) - when advisory locking mode is enabled, the application advisory lock ID can be changed using this env var. All instances of Chainlink that might run on a particular database must share the same advisory lock ID. It is recommended to leave this at the default.
• LOG_FILE_DIR (default: chainlink root directory) - if LOG_FILE_MAX_SIZE is set, this env var allows you to override the output directory for logging.
• SHUTDOWN_GRACE_PERIOD (default: 5s) - when node is shutting down gracefully and exceeded this grace period, it terminates immediately (trying to close DB connection) to avoid being SIGKILLed.
• SOLANA_ENABLED (default: false) - set to true to enable Solana support
• TERRA_ENABLED (default: false) - set to true to enable Terra support
• BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS - if EIP1559 mode is enabled, this optional env var controls the buffer blocks to add to the current base fee when sending a transaction. By default, the gas bumping threshold + 1 block is used. It is not recommended to change this unless you know what you are doing.
• TELEMETRY_INGRESS_BUFFER_SIZE (default: 100) - the number of telemetry messages to buffer before dropping new ones
• TELEMETRY_INGRESS_MAX_BATCH_SIZE (default: 50) - the maximum number of messages to batch into one telemetry request
• TELEMETRY_INGRESS_SEND_INTERVAL (default: 500ms) - the cadence on which batched telemetry is sent to the ingress server
• TELEMETRY_INGRESS_SEND_TIMEOUT (default: 10s) - the max duration to wait for the request to complete when sending batch telemetry
• TELEMETRY_INGRESS_USE_BATCH_SEND (default: true) - toggles sending telemetry using the batch client to the ingress server
• NODE_NO_NEW_HEADS_THRESHOLD (default: 3m) - RPC node will be marked out-of-sync if it does not receive a new block for this length of time. Set to 0 to disable head monitoring for liveness checking,
• NODE_POLL_FAILURE_THRESHOLD (default: 5) - number of consecutive failed polls before an RPC node is marked dead. Set to 0 to disable poll liveness checking.
• NODE_POLL_INTERVAL (default: 10s) - how often to poll. Set to 0 to disable all polling.

Bootstrap job

Added a new bootstrap job type. This job removes the need for every job to implement their own bootstrapping logic. OCR2 jobs with isBootstrapPeer=true are automatically migrated to the new format. The spec parameters are similar to a basic OCR2 job, an example would be:

type            = "bootstrap"
name            = "bootstrap"
relay           = "evm"
schemaVersion	= 1
contractID      = "0xAb5801a7D398351b8bE11C439e05C5B3259aeC9B"
[relayConfig]
chainID	        = 4

EVM node hot failover and liveness checking

Chainlink now supports hot failover and liveness checking for EVM nodes. This completely supercedes and replaces the Fiews failover proxy and should remove the need for any kind of failover proxy between Chainlink and its RPC nodes.

In order to use this feature, you'll need to set multiple primary RPC nodes.

Removed

• deleteuser CLI command.

Changed

EVM_DISABLED has been deprecated and replaced by EVM_ENABLED for consistency with other feature flags. ETH_DISABLED has been deprecated and replaced by EVM_RPC_ENABLED for consistency, and because this was confusingly named. In most cases you want to set EVM_ENABLED=false and not EVM_RPC_ENABLED=false.

Log colorization is now disabled by default because it causes issues when piped to text files. To re-enable log colorization, set LOG_COLOR=true.

Polygon/matic defaults changed

Due to increasingly hostile network conditions on Polygon we have had to increase a number of default limits. This is to work around numerous and very deep re-orgs, high mempool pressure and a failure by the network to propagate transactions properly. These new limits are likely to increase load on both your Chainlink node and database, so please be sure to monitor CPU and memory usage on both and make sure they are adequately specced to handle the additional load.

[1.1.1] - 2022-02-14

Added

• BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS - if EIP1559 mode is enabled, this optional env var controls the buffer blocks to add to the current base fee when sending a transaction. By default, the gas bumping threshold + 1 block is used. It is not recommended to change this unless you know what you are doing.
• EVM_GAS_FEE_CAP_DEFAULT - if EIP1559 mode is enabled, and FixedPrice gas estimator is used, this env var controls the fixed initial fee cap.
• Allow dumping pprof even when not in dev mode, useful for debugging (go to /v2/debug/pprof as a logged in user)

Fixed

• Update timeout so we don’t exit early on very large log broadcaster backfills

EIP-1559 Fixes

Fixed issues with EIP-1559 related to gas bumping. Due to go-ethereum's implementation which introduces additional restrictions on top of the EIP-1559 spec, we must bump the FeeCap at least 10% each time in order for the gas bump to be accepted.

The new EIP-1559 implementation works as follows:

If you are using FixedPriceEstimator:

• With gas bumping disabled, it will submit all transactions with feecap=ETH_MAX_GAS_PRICE_WEI and tipcap=EVM_GAS_TIP_CAP_DEFAULT
• With gas bumping enabled, it will submit all transactions initially with feecap=EVM_GAS_FEE_CAP_DEFAULT and tipcap=EVM_GAS_TIP_CAP_DEFAULT.

If you are using BlockHistoryEstimator (default for most chains):

• With gas bumping disabled, it will submit all transactions with feecap=ETH_MAX_GAS_PRICE_WEI and tipcap=<calculated using past blocks>
• With gas bumping enabled (default for most chains) it will submit all transactions initially with feecap = ( current block base fee * (1.125 ^ N) + tipcap ) where N is configurable by setting BLOCK_HISTORY_ESTIMATOR_EIP1559_FEE_CAP_BUFFER_BLOCKS but defaults to gas bump threshold+1 and tipcap=<calculated using past blocks>

Bumping works as follows:

• Increase tipcap by max(tipcap * (1 + ETH_GAS_BUMP_PERCENT), tipcap + ETH_GAS_BUMP_WEI)
• Increase feecap by max(feecap * (1 + ETH_GAS_BUMP_PERCENT), feecap + ETH_GAS_BUMP_WEI)

[1.1.0] - 2022-01-25

Added

• Added support for Sentry error reporting. Set SENTRY_DSN at run-time to enable reporting.
• Added Prometheus counters: log_warn_count, log_error_count, log_critical_count, log_panic_count and log_fatal_count representing the corresponding number of warning/error/critical/panic/fatal messages in the log.
• The new prometheus metric tx_manager_tx_attempt_count is a Prometheus Gauge that should represent the total number of Transactions attempts that awaiting confirmation for this node.
• The new prometheus metric version that displays the node software version (tag) as well as the corresponding commit hash.
• CLI command keys eth list is updated to display key specific max gas prices.
• CLI command keys eth create now supports optional maxGasPriceGWei parameter.
• CLI command keys eth update is added to update key specific parameters like maxGasPriceGWei.
• Add partial support for Moonriver chain
• For OCR jobs, databaseTimeout, observationGracePeriod and contractTransmitterTransmitTimeout can be specified to override chain-specific default values.

Two new log levels have been added.

• [crit]: Critical level logs are more severe than [error] and require quick action from the node operator.
• [debug] [trace]: Trace level logs contain extra [debug] information for development, and must be compiled in via -tags trace.

[Beta] Multichain support added

As a beta feature, Chainlink now supports connecting to multiple different EVM chains simultaneously.

This means that one node can run jobs on Goerli, Kovan, BSC and Mainnet (for example). Note that you can still have as many eth keys as you like, but each eth key is pegged to one chain only.

Extensive efforts have been made to make migration for existing nops as seamless as possible. Generally speaking, you should not have to make any changes when upgrading your existing node to this version. All your jobs will continue to run as before.

The overall summary of changes is such:

Chains/Ethereum Nodes

EVM chains are now represented as a first class object within the chainlink node. You can create/delete/list them using the CLI or API.

At least one primary node is required in order for a chain to connect. You may additionally specify zero or more send-only nodes for a chain. It is recommended to use the CLI/API or GUI to add nodes to chain.

Creation

chainlink chains evm create -id 42 # creates an evm chain with chain ID 42 (see: https://chainlist.org/)
chainlink nodes create -chain-id 42 -name 'my-primary-kovan-full-node' -type primary -ws-url ws://node.example/ws -http-url http://node.example/rpc # http-url is optional but recommended for primaries
chainlink nodes create -chain-id 42 -name 'my-send-only-backup-kovan-node' -type sendonly -http-url http://some-public-node.example/rpc

Listing

chainlink chains evm list
chainlink nodes list

Deletion

chainlink nodes delete 'my-send-only-backup-kovan-node'
chainlink chains evm delete 42

Legacy eth ENV vars

The old way of specifying chains using environment variables is still supported but discouraged. It works as follows:

If you specify ETH_URL then the values of ETH_URL, ETH_CHAIN_ID, ETH_HTTP_URL and ETH_SECONDARY_URLS will be used to create/update chains and nodes representing these values in the database. If an existing chain/node is found it will be overwritten. This behavior is used mainly to ease the process of upgrading, and on subsequent runs (once your old settings have been written to the database) it is recommended to unset these ENV vars and use the API commands exclusively to administer chains and nodes.

Jobs/tasks

By default, all jobs/tasks will continue to use the default chain (specified by ETH_CHAIN_ID). However, the following jobs now allow an additional evmChainID key in their TOML:

• VRF
• DirectRequest
• Keeper
• OCR
• Fluxmonitor

You can pin individual jobs to a particular chain by specifying the evmChainID explicitly. Here is an example job to demonstrate:

type            = "keeper"
evmChainID      = 3
schemaVersion   = 1
name            = "example keeper spec"
contractAddress = "0x9E40733cC9df84636505f4e6Db28DCa0dC5D1bba"
externalJobID   = "0EEC7E1D-D0D2-476C-A1A8-72DFB6633F49"
fromAddress     = "0xa8037A20989AFcBC51798de9762b351D63ff462e"

The above keeper job will always run on chain ID 3 (Ropsten) regardless of the ETH_CHAIN_ID setting. If no chain matching this ID has been added to the chainlink node, the job cannot be created (you must create the chain first).

In addition, you can also specify evmChainID on certain pipeline tasks. This allows for cross-chain requests, for example:

type                = "directrequest"
schemaVersion       = 1
evmChainID          = 42
name                = "example cross chain spec"
contractAddress     = "0x613a38AC1659769640aaE063C651F48E0250454C"
externalJobID       = "0EEC7E1D-D0D2-476C-A1A8-72DFB6633F90"
observationSource   = """
    decode_log   [type=ethabidecodelog ... ]
    ...
    submit [type=ethtx to="0x613a38AC1659769640aaE063C651F48E0250454C" data="$(encode_tx)" minConfirmations="2" evmChainID="3"]
    decode_log-> ... ->submit;
"""

In the example above (which excludes irrelevant pipeline steps for brevity) a log can be read from the chain with ID 42 (Kovan) and a transaction emitted on chain with ID 3 (Ropsten).

Tasks that support the evmChainID parameter are as follows:

• ethcall
• estimategaslimit
• ethtx

Defaults

If the job- or task-specific evmChainID is not given, the job/task will simply use the default as specified by the ETH_CHAIN_ID env variable.

Generally speaking, the default config values for each chain are good enough. But in some cases it is necessary to be able to override the defaults on a per-chain basis.

This used to be done via environment variables e.g. MINIMUM_CONTRACT_PAYMENT_LINK_JUELS.

These still work, but if set they will override that value for all chains. This may not always be what you want. Consider a node that runs both Matic and Mainnet. You may want to set a higher value for MINIMUM_CONTRACT_PAYMENT on Mainnet, due to the more expensive gas costs. However, setting MINIMUM_CONTRACT_PAYMENT_LINK_JUELS using env variables will set that value for all chains including matic.

To help you work around this, Chainlink now supports setting per-chain configuration options.

Examples

To set initial configuration when creating a chain, pass in the full json string as an optional parameter at the end:

chainlink evm chains create -id 42 '{"BlockHistoryEstimatorBlockDelay": "100"}'

To set configuration on an existing chain, specify key values pairs as such:

chainlink evm chains configure -id 42 BlockHistoryEstimatorBlockDelay=100 GasEstimatorMode=FixedPrice

The full list of chain-specific configuration options can be found by looking at the ChainCfg struct in core/chains/evm/types/types.go.

Async support in external adapters

External Adapters making async callbacks can now error job runs. This required a slight change to format, the correct way to callback from an asynchronous EA is using the following JSON:

SUCCESS CASE:

{
    "value": < any valid json object >
}

ERROR CASE:

{
    "error": "some error string"
}

This only applies to EAs using the X-Chainlink-Pending header to signal that the result will be POSTed back to the Chainlink node sometime 'later'. Regular synchronous calls to EAs work just as they always have done.

(NOTE: Official documentation for EAs needs to be updated)

New optional VRF v2 field: requestedConfsDelay

Added a new optional field for VRF v2 jobs called requestedConfsDelay, which configures a number of blocks to wait in addition to the request specified requestConfirmations before servicing the randomness request, i.e. the Chainlink node will wait max(nodeMinConfs, requestConfirmations + requestedConfsDelay) blocks before servicing the request.

It can be used in the following way:

type = "vrf"
externalJobID = "123e4567-e89b-12d3-a456-426655440001"
schemaVersion = 1
name = "vrf-v2-secondary"
coordinatorAddress = "0xABA5eDc1a551E55b1A570c0e1f1055e5BE11eca7"
requestedConfsDelay = 10
# ... rest of job spec ...

Use of this field requires a database migration.

New locking mode: 'lease'

Chainlink now supports a new environment variable DATABASE_LOCKING_MODE. It can be set to one of the following values:

• dual (the default - uses both locking types for backwards and forwards compatibility)
• advisorylock (advisory lock only)
• lease (lease lock only)
• none (no locking at all - useful for advanced deployment environments when you can be sure that only one instance of chainlink will ever be running)

The database lock ensures that only one instance of Chainlink can be run on the database at a time. Running multiple instances of Chainlink on a single database at the same time would likely to lead to strange errors and possibly even data integrity failures and should not be allowed.

Ideally, node operators would be using a container orchestration system (e.g. Kubernetes) that ensures that only one instance of Chainlink ever runs on a particular postgres database.

However, we are aware that many node operators do not have the technical capacity to do this. So a common use case is to run multiple Chainlink instances in failover mode (as recommended by our official documentation, although this will be changing in future). The first instance will take some kind of lock on the database and subsequent instances will wait trying to take this lock in case the first instance disappears or dies.

Traditionally Chainlink has used an advisory lock to manage this. However, advisory locks come with several problems, notably:

• Postgres does not really like it when you hold locks open for a very long time (hours/days). It hampers certain internal cleanup tasks and is explicitly discouraged by the postgres maintainers.
• The advisory lock can silently disappear on postgres upgrade, meaning that a new instance can take over even while the old one is still running.
• Advisory locks do not play nicely with pooling tools such as pgbouncer.
• If the application crashes, the advisory lock can be left hanging around for a while (sometimes hours) and can require manual intervention to remove it before another instance of Chainlink will allow itself to boot.

For this reason, we have introduced a new locking mode, lease, which is likely to become the default in the future. lease-mode works as follows:

• Have one row in a database which is updated periodically with the client ID.
• CL node A will run a background process on start that updates this e.g. once per second.
• CL node B will spinlock, checking periodically to see if the update got too old. If it goes more than a set period without updating, it assumes that node A is dead and takes over. Now CL node B is the owner of the row, and it updates this every second.
• If CL node A comes back somehow, it will go to take out a lease and realise that the database has been leased to another process, so it will exit the entire application immediately.

The default is set to dual which used both advisory locking AND lease locking, for backwards compatibility. However, it is recommended that node operators who know what they are doing, or explicitly want to stop using the advisory locking mode set DATABASE_LOCKING_MODE=lease in their env.

Lease locking can be configured using the following ENV vars:

LEASE_LOCK_REFRESH_INTERVAL (default 1s) LEASE_LOCK_DURATION (default 30s)

It is recommended to leave these set to the default values.

Duplicate Job Configuration

When duplicating a job, the new job's configuration settings that have not been overridden by the user can still reflect the chainlink node configuration.

Nurse (automatic pprof profiler)

Added new automatic pprof profiling service. Profiling is triggered when the node exceeds certain resource thresholds (currently, memory and goroutine count). The following environment variables have been added to allow configuring this service:

• AUTO_PPROF_ENABLED: Set to true to enable the automatic profiling service. Defaults to false.
• AUTO_PPROF_PROFILE_ROOT: The location on disk where pprof profiles will be stored. Defaults to $CHAINLINK_ROOT.
• AUTO_PPROF_POLL_INTERVAL: The interval at which the node's resources are checked. Defaults to 10s.
• AUTO_PPROF_GATHER_DURATION: The duration for which profiles are gathered when profiling is kicked off. Defaults to 10s.
• AUTO_PPROF_GATHER_TRACE_DURATION: The duration for which traces are gathered when profiling is kicked off. This is separately configurable because traces are significantly larger than other types of profiles. Defaults to 5s.
• AUTO_PPROF_MAX_PROFILE_SIZE: The maximum amount of disk space that profiles may consume before profiling is disabled. Defaults to 100mb.
• AUTO_PPROF_CPU_PROFILE_RATE: See https://pkg.go.dev/runtime#SetCPUProfileRate. Defaults to 1.
• AUTO_PPROF_MEM_PROFILE_RATE: See https://pkg.go.dev/runtime#pkg-variables. Defaults to 1.
• AUTO_PPROF_BLOCK_PROFILE_RATE: See https://pkg.go.dev/runtime#SetBlockProfileRate. Defaults to 1.
• AUTO_PPROF_MUTEX_PROFILE_FRACTION: See https://pkg.go.dev/runtime#SetMutexProfileFraction. Defaults to 1.
• AUTO_PPROF_MEM_THRESHOLD: The maximum amount of memory the node can actively consume before profiling begins. Defaults to 4gb.
• AUTO_PPROF_GOROUTINE_THRESHOLD: The maximum number of actively-running goroutines the node can spawn before profiling begins. Defaults to 5000.

Adventurous node operators are encouraged to read this guide on how to analyze pprof profiles.

merge task type

A new task type has been added, called merge. It can be used to merge two maps/JSON values together. Merge direction is from right to left such that right will clobber values of left. If no left is provided, it uses the input of the previous task. Example usage as such:

decode_log   [type=ethabidecodelog ...]
merge        [type=merge right=<{"foo": 42}>];

decode_log -> merge;

Or, to reverse merge direction:

decode_log   [type=ethabidecodelog ...]
merge        [type=merge left=<{"foo": 42}> right="$(decode_log)"];

decode_log -> merge;

Enhanced ABI encoding support

The ethabiencode2 task supports ABI encoding using the abi specification generated by solc. e.g:

{
    "name": "call",
    "inputs": [
      {
        "name": "value",
        "type": "tuple",
        "components": [
          {
            "name": "first",
            "type": "bytes32"
          },
          {
            "name": "last",
            "type": "bool"
          }
        ]
      }
    ]
}

This would allow for calling of a function call with a tuple containing two values, the first a bytes32 and the second a bool. You can supply a named map or an array.

Transaction Simulation (Gas Savings)

Chainlink now supports transaction simulation for certain types of job. When this is enabled, transactions will be simulated using eth_call before initial send. If the transaction reverted, the tx is marked as errored without being broadcast, potentially avoiding an expensive on-chain revert.

This can add a tiny bit of latency (upper bound 2s, generally much shorter under good conditions) and will add marginally more load to the eth client, since it adds an extra call for every transaction sent. However, it may help to save gas in some cases especially during periods of high demand by avoiding unnecessary reverts (due to outdated round etc.).

This option is EXPERIMENTAL and disabled by default.

To enable for FM or OCR:

FM_SIMULATE_TRANSACTIONs=true OCR_SIMULATE_TRANSACTIONS=true

To enable in the pipeline, use the simulate=true option like so:

submit [type=ethtx to="0xDeadDeadDeadDeadDeadDeadDeadDead" data="0xDead" simulate=true]

Use at your own risk.

Misc

Chainlink now supports more than one primary eth node per chain. Requests are round-robined between available primaries.

Add CRUD functionality for EVM Chains and Nodes through Operator UI.

Non-fatal errors to a pipeline run are preserved including any run that succeeds but has more than one fatal error.

Chainlink now supports configuring max gas price on a per-key basis (allows implementation of keeper "lanes").

The Operator UI now supports login MFA with hardware security keys. MFA_RPID and MFA_RPORIGIN environment variables have been added to the config and are required if using the new MFA feature.

Keys and Configuration navigation links have been moved into a settings dropdown to make space for multichain navigation links.

Full EIP1559 Support (Gas Savings)

Chainlink now includes experimental support for submitting transactions using type 0x2 (EIP-1559) envelope.

EIP-1559 mode is off by default but can be enabled on a per-chain basis or globally.

This may help to save gas on spikes: Chainlink ought to react faster on the upleg and avoid overpaying on the downleg. It may also be possible to set BLOCK_HISTORY_ESTIMATOR_BATCH_SIZE to a smaller value e.g. 12 or even 6 because tip cap ought to be a more consistent indicator of inclusion time than total gas price. This would make Chainlink more responsive and ought to reduce response time variance. Some experimentation will be needed here to find optimum settings.

To enable globally, set EVM_EIP1559_DYNAMIC_FEES=true. Set with caution, if you set this on a chain that does not actually support EIP-1559 your node will be broken.

In EIP-1559 mode, the total price for the transaction is the minimum of base fee + tip cap and fee cap. More information can be found on the official EIP.

Chainlink's implementation of this is to set a large fee cap and modify the tip cap to control confirmation speed of transactions. So, when in EIP1559 mode, the tip cap takes the place of gas price roughly speaking, with the varying base price remaining a constant (we always pay it).

A quick note on terminology - Chainlink uses the same terms used internally by go-ethereum source code to describe various prices. This is not the same as the externally used terms. For reference:

Base Fee Per Gas = BaseFeePerGas Max Fee Per Gas = FeeCap Max Priority Fee Per Gas = TipCap

In EIP-1559 mode, the following changes occur to how configuration works:

• All new transactions will be sent as type 0x2 transactions specifying a TipCap and FeeCap (NOTE: existing pending legacy transactions will continue to be gas bumped in legacy mode)
• BlockHistoryEstimator will apply its calculations (gas percentile etc.) to the TipCap and this value will be used for new transactions (GasPrice will be ignored)
• FixedPriceEstimator will use EVM_GAS_TIP_CAP_DEFAULT instead of ETH_GAS_PRICE_DEFAULT
• ETH_GAS_PRICE_DEFAULT is ignored for new transactions and EVM_GAS_TIP_CAP_DEFAULT is used instead (default 20GWei)
• ETH_MIN_GAS_PRICE_WEI is ignored for new transactions and EVM_GAS_TIP_CAP_MINIMUM is used instead (default 0)
• ETH_MAX_GAS_PRICE_WEI controls the FeeCap
• KEEPER_GAS_PRICE_BUFFER_PERCENT is ignored in EIP-1559 mode and KEEPER_TIP_CAP_BUFFER_PERCENT is used instead

The default tip cap is configurable per-chain but can be specified for all chains using EVM_GAS_TIP_CAP_DEFAULT. The fee cap is derived from ETH_MAX_GAS_PRICE_WEI.

When using the FixedPriceEstimator, the default gas tip will be used for all transactions.

When using the BlockHistoryEstimator, Chainlink will calculate the tip cap based on transactions already included (in the same way it calculates gas price in legacy mode).

Enabling EIP1559 mode might lead to marginally faster transaction inclusion and make the node more responsive to sharp rises/falls in gas price, keeping response times more consistent.

In addition, ethcall tasks now accept gasTipCap and gasFeeCap parameters in addition to gasPrice. This is required for Keeper jobs, i.e.:

check_upkeep_tx          [type=ethcall
                          failEarly=true
                          extractRevertReason=true
                          contract="$(jobSpec.contractAddress)"
                          gas="$(jobSpec.checkUpkeepGasLimit)"
                          gasPrice="$(jobSpec.gasPrice)"
                          gasTipCap="$(jobSpec.gasTipCap)"
                          gasFeeCap="$(jobSpec.gasFeeCap)"
                          data="$(encode_check_upkeep_tx)"]

NOTE: AccessLists are part of the 0x2 transaction type spec and Chainlink also implements support for these internally. This is not currently exposed in any way, if there is demand for this it ought to be straightforward enough to do so.

Avalanche AP4 defaults have been added (you can remove manually set ENV vars controlling gas pricing).

New env vars

CHAIN_TYPE - Configure the type of chain (if not standard). Arbitrum, ExChain, Optimism, or XDai. Replaces LAYER_2_TYPE. NOTE: This is a global override, to set on a per-chain basis you must use the CLI/API or GUI to change the chain-specific config for that chain (ChainType).

BLOCK_EMISSION_IDLE_WARNING_THRESHOLD - Controls global override for the time after which node will start logging warnings if no heads are received.

ETH_DEFAULT_BATCH_SIZE - Controls the default number of items per batch when making batched RPC calls. It is unlikely that you will need to change this from the default value.

NOTE: ETH_URL used to default to "ws://localhost:8546" and ETH_CHAIN_ID used to default to 1. These defaults have now been removed. The env vars are no longer required, since node configuration is now done via CLI/API/GUI and stored in the database.

Removed

• belt/ and evm-test-helpers/ removed from the codebase.

Deprecated env vars

LAYER_2_TYPE - Use CHAIN_TYPE instead.

Removed env vars

FEATURE_CRON_V2, FEATURE_FLUX_MONITOR_V2, FEATURE_WEBHOOK_V2 - all V2 job types are now enabled by default.

Fixed

• Fixed a regression whereby the BlockHistoryEstimator would use a bumped value on old gas price even if the new current price was larger than the bumped value.
• Fixed a bug where creating lots of jobs very quickly in parallel would cause the node to hang
• Propagating evmChainID parameter in job specs supporting this parameter.

Fixed LOG_LEVEL behavior in respect to the corresponding UI setting: Operator can override LOG_LEVEL until the node is restarted.

Changed

• The default GAS_ESTIMATOR_MODE for Optimism chains has been changed to Optimism2.
• Default minimum payment on mainnet has been reduced from 1 LINK to 0.1 LINK.
• Logging timestamp output has been changed from unix to ISO8601 to aid in readability. To keep the old unix format, you may set LOG_UNIX_TS=true
• Added WebAuthn support for the Operator UI and corresponding support in the Go backend

Log to Disk

This feature has been disabled by default, turn on with LOG_TO_DISK. For most production uses this is not desirable.

[1.0.1] - 2021-11-23

Added

• Improved error reporting
• Panic and recovery improvements

Fixed

• Resolved config conversion errors for ETH_FINALITY_DEPTH, ETH_HEAD_TRACKER_HISTORY, and ETH_GAS_LIMIT_MULTIPLIER
• Proper handling for "nonce too low" errors on Avalanche

[1.0.0] - 2021-10-19

Added

• chainlink node db status will now display a table of applied and pending migrations.
• Add support for OKEx/ExChain.

Changed

Legacy job pipeline (JSON specs) are no longer supported

This version will refuse to migrate the database if job specs are still present. You must manually delete or migrate all V1 job specs before upgrading.

For more information on migrating, see the docs.

This release will DROP legacy job tables so please take a backup before upgrading.

KeyStore changes

• We no longer support "soft deleting", or archiving keys. From now on, keys can only be hard-deleted.
• Eth keys can no longer be imported directly to the database. If you with to import an eth key, you must start the node first and import through the remote client.

New env vars

LAYER_2_TYPE - For layer 2 chains only. Configure the type of chain, either Arbitrum or Optimism.

Misc

• Head sampling can now be optionally disabled by setting ETH_HEAD_TRACKER_SAMPLING_INTERVAL = "0s" - this will result in every new head being delivered to running jobs, regardless of the head frequency from the chain.
• When creating new FluxMonitor jobs, the validation logic now checks that only one of: drumbeat ticker or idle timer is enabled.
• Added a new Prometheus metric: uptime_seconds which measures the number of seconds the node has been running. It can be helpful in detecting potential crashes.

Fixed

Fixed a regression whereby the BlockHistoryEstimator would use a bumped value on old gas price even if the new current price was larger than the bumped value.

[0.10.15] - 2021-10-14

It is highly recommended upgrading to this version before upgrading to any newer versions to avoid any complications.

Fixed

• Prevent release from clobbering databases that have previously been upgraded

[0.10.14] - 2021-09-06

Added

• FMv2 spec now contains DrumbeatRandomDelay parameter that can be used to introduce variation between round of submits of different oracles, if drumbeat ticker is enabled.

• OCR Hibernation

Requesters/MinContractPaymentLinkJuels

V2 direct request specs now support two additional keys:

• "requesters" key which allows whitelisting requesters
• "minContractPaymentLinkJuels" key which allows to specify a job-specific minimum contract payment.

For example:

type                        = "directrequest"
schemaVersion               = 1
requesters                  = ["0xaaaa1F8ee20f5565510B84f9353F1E333E753B7a", "0xbbbb70F0e81C6F3430dfdC9fa02fB22BdD818C4e"] # optional
minContractPaymentLinkJuels = "100000000000000" # optional
name                        = "example eth request event spec with requesters"
contractAddress             = "..."
externalJobID               = "..."
observationSource           = """
...
"""

[0.10.13] - 2021-08-25

Fixed

• Resolved exiting Hibernation bug on FMv2

[0.10.12] - 2021-08-16

Fixed

• Resolved FMv2 stalling in Hibernation mode
• Resolved rare issue when the Gas Estimator fails on start
• Resolved the handling of nil values for gas price

[0.10.11] - 2021-08-09

A new configuration variable, BLOCK_BACKFILL_SKIP, can be optionally set to "true" in order to strongly limit the depth of the log backfill. This is useful if the node has been offline for a longer time and after startup should not be concerned with older events from the chain.

Three new configuration variables are added for the new telemetry ingress service support. TELEMETRY_INGRESS_URL sets the URL to connect to for telemetry ingress, TELEMETRY_INGRESS_SERVER_PUB_KEY sets the public key of the telemetry ingress server, and TELEMETRY_INGRESS_LOGGING toggles verbose logging of the raw telemetry messages being sent.

• Fixes the logging configuration form not displaying the current values
• Updates the design of the configuration cards to be easier on the eyes
• View Coordinator Service Authentication keys in the Operator UI. This is hidden behind a feature flag until usage is enabled.
• Adds support for the new telemetry ingress service.

Changed

The legacy job pipeline (JSON specs) has been officially deprecated and support for these jobs will be dropped in an upcoming release.

Any node operators still running jobs with JSON specs should migrate their jobs to TOML format instead.

The format for V2 Webhook job specs has changed. They now allow specifying 0 or more external initiators. Example below:

type            = "webhook"
schemaVersion   = 1
externalInitiators = [
    { name = "foo-ei", spec = '{"foo": 42}' },
    { name = "bar-ei", spec = '{"bar": 42}' }
]
observationSource   = """
ds          [type=http method=GET url="https://chain.link/ETH-USD"];
ds_parse    [type=jsonparse path="data,price"];
ds_multiply [type=multiply times=100];
ds -> ds_parse -> ds_multiply;
"""

These external initiators will be notified with the given spec after the job is created, and also at deletion time.

Only the External Initiators listed in the toml spec may trigger a run for that job. Logged-in users can always trigger a run for any job.

Migrating Jobs

• OCR All OCR jobs are already using v2 pipeline by default - no need to do anything here.

• Flux Monitor v1 We have created a tool to help you automigrate flux monitor specs in JSON format to the new TOML format. You can migrate a job like this:

chainlink jobs migrate <job id>

This can be automated by using the API like so:

POST http://yournode.example/v2/migrate/<job id>

• VRF v1 Automigration is not supported for VRF jobs. They must be manually converted into v2 format.

• Ethlog/Runlog/Cron/web All other job types must also be manually converted into v2 format.

Technical details

Why are we doing this?

To give some background, the legacy job pipeline has been around since before Chainlink went to mainnet and is getting quite long in the tooth. The code is brittle and difficult to understand and maintain. For a while now we have been developing a v2 job pipeline in parallel which uses the TOML format. The new job pipeline is simpler, more performant and more powerful. Every job that can be represented in the legacy pipeline should be able to be represented in the v2 pipeline - if it can't be, that's a bug, so please let us know ASAP.

The v2 pipeline has now been extensively tested in production and proved itself reliable. So, we made the decision to drop V1 support entirely in favour of focusing developer effort on new features like native multichain support, EIP1559-compatible fees, further gas saving measures and support for more blockchains. By dropping support for the old pipeline, we can deliver these features faster and better support our community.

KeyStore changes

• Key export files are changing format and will not be compatible between versions. Ex, a key exported in 0.10.12, will not be importable by a node running 1.0.0, and vice-versa.
• We no longer support "soft deleting", or archiving keys. From now on, keys can only be hard-deleted.
• Eth keys can no longer be imported directly to the database. If you with to import an eth key, you must start the node first and i