github.com/codenotary/immudb

Bug Fixes

• cmd/immuadmin: simplify logging when flushing and compacting current db
• pkg/database: return key not found when resolving a deleted entry
• pkg/database: Return correct error for verifiedGet on deleted entries

<a name="v1.2.3-RC1"></a>

Bug Fixes

• CI/CD: Golang compiler is not needed for building docker images
• CI/CD: Use CAS for notarization
• embedded/store: Fix early precondition checks
• embedded/store: Ensure up-to-date index on constrained writes
• embedded/tbtree: copy-on-write when increasing root ts
• immudb: Fix the name of signing key env var
• pkg: Fix tests after recent changes in API
• pkg/api: typo in kv metadata message
• pkg/api: Remove unused Sync field from IndexOptions
• pkg/api/schema: Use correct id for preconditions in SetRequest
• pkg/auth: Avoid unguarded read from user tokens map
• pkg/client: Adopt to EncodeReference changes
• pkg/client: Prevent updates with incorrect database settings
• pkg/client: Use correct response for UpdateDatabaseV2
• pkg/client/errors: Update the list of error codes
• pkg/client/errors: Ensure FromErrors works with ImmuError instance
• pkg/database: Better handling of invalid constraints
• pkg/database: Improve test coverage for KV constraints
• pkg/database: automatically set max score if not specified in desc order
• pkg/errors: Correct GRPC error mapping for precondition failure
• pkg/server: Use buffered channel for catching OS signals
• pkg/server: typo in log message
• pkg/server: adjust time to millis convertion
• pkg/server: ensure sessions locks get released
• pkg/server: override default settings with existent values
• tools/monitoring: Update grafana dashboards

Changes

• cleanup percentage as float value
• Fix typo in a comment
• Rename Constraints to Preconditions
• Update copyright to 2022
• Dockerfile: Improve dockerfile builds
• build: improve release instructions (#1100)
• cmd/immuadmin: add safety flag in delete database command
• cmd/immuclient: health command description
• embedded/ahtree: fix error message
• embedded/appendable: return io.EOF if there are not enough data for checksum calculation
• embedded/appendable: fix typo in error message
• embedded/appendable: discard capability
• embedded/appendable: appendable checksum calculation
• embedded/store: Add missing Copyright header
• embedded/store: add synced setting in index options
• embedded/store: do not skip expired entries when indexing
• embedded/store: verbose data corruption error
• embedded/store: parametrized index write buffer size
• embedded/store: syncThld at store options
• embedded/store: verbose logging during compaction
• embedded/store: index one tx per iteration
• embedded/store: improve compaction logging
• embedded/store: sync aht when syncing the store
• embedded/store: skip expired entries during indexing
• embedded/store: declare constants for all the options
• embedded/store: use store layer for constraint validations
• embedded/store: constraint validations with deletion and expiration support
• embedded/store/options: Simplify validation tests
• embedded/tbree: use shared writeOpts
• embedded/tbree: only insert nodes in cache when they were mutated
• embedded/tbree: remove obsolete property
• embedded/tbree: bump index version
• embedded/tbtree: middle node split
• embedded/tbtree: Add more internal metrics
• embedded/tbtree: min offset handling
• embedded/tbtree: validate compaction target path
• embedded/tbtree: data discarding with opened and older snapshots
• embedded/tbtree: Extend buckets for child node count histogram
• embedded/tbtree: synced flush if cleanup percentage is greater than zero
• embedded/tbtree: discard unreferenced data after sync
• embedded/tbtree: reduce allocs during flush
• embedded/tbtree: minOffset only for non-mutated nodes
• embedded/tbtree: ensure current snapshot is synced for compaction
• embedded/tbtree: validate input kv pairs before sorting
• embedded/tbtree: improve snapshot loading and discarding
• embedded/tbtree: discard unreferenced data when flushing index
• embedded/tbtree: discard unreferenced data
• embedded/tbtree: Add metrics for index data size
• embedded/tbtree: reduce allocations when flushing
• embedded/tbtree: positive compaction threshold
• embedded/tbtree: rebase non-indexed on kv syncthreshold
• embedded/tbtree: explicit error validation before loading
• embedded/tbtree: sort kv pairs in bulkInsert
• embedded/tbtree: checksum-based snapshot consistency validation
• embedded/tbtree: self-healing index
• embedded/tbtree: set initial offsets during initialization
• embedded/tbtree: validate data-format version
• embedded/tbtree: use double for min offset calculation
• embedded/tbtree: reduce fixed records length
• embedded/tbtree: open-ranged nodes
• embedded/tbtree: wip reduce node size
• embedded/tbtree: ensure sync on gracefully closing
• embedded/tbtree: fully replace sync with syncThld
• embedded/tbtree: use binary search during key lookups
• embedded/tbtree: Use KV entries count for sync threshold
• embedded/tbtree: no cache update during compaction reads
• makefile: fix cas sign instructions
• metrics: Add better flush / compaction metrics for btree
• pkg/api: use entries spec in verified and scan tx endpoints
• pkg/api: add synced param to flushindex endpoint
• pkg/api: non-indexable entries
• pkg/api: change proto schema toward db loading/unloading
• pkg/api: parametrized index cleanup percentage
• pkg/api: db loading and unloading
• pkg/api: uniform v2 endpoints
• pkg/api: prepare flushindex endpoint for future extensions
• pkg/api: use nullable prefix in db settings message
• pkg/client: optional client connection
• pkg/client: use txRequest in TxByIDWithSpec method
• pkg/client: synced flushing to enable physical data deletion
• pkg/database: parameters to resolve references at tx
• pkg/database: tx entries excluded by default if non-null spec is provided
• pkg/database: optional tx value resolution
• pkg/database: use shared tx holder when resolving tx entries
• pkg/database: remove db name from options
• pkg/integration: integrate non-indexed into grpc apis
• pkg/server: endpoint to retrieve settings of selected database
• pkg/server: expose max opened files for btree indexing
• pkg/server: use syncThreshold
• pkg/server: replication options for systemdb and defaultdb
• pkg/server: use previous store default values
• pkg/server: increase default max number of active snapshots
• pkg/server: tolerate failed user-created db loading
• pkg/server: expose flush index endpoint
• pkg/server: start/stop replicator when loading/unloading db
• pkg/server: convert time to milliseconds
• pkg/server: minor changes
• pkg/server: log web-console error on boot
• pkg/server: synced db runtime operations
• pkg/server: Dump used db options when loading databases
• pkg/serverr: validate request in deprecated database creation endpoint
• stats: Add btree cache prometheus stats

Features

• Improved API for database creation and update
• Improved validation of kv constraints
• Entries-independent constraints in GRPC api
• Move KV write constraints to OngoingTX member
• KV: Do not create unnecessary snapshots when checking KV constraints
• KV: Add constrained KV writes for Set operation
• KV: Add constrained KV writes for Reference operation
• KV: Add constrained KV writes for ExecAll operation
• KV: Move constraints validation to OngoingTx
• embedded/cache: dynamic cache resizing
• embedded/store: Fail to write metadata if proof version does not support it
• embedded/store: Add tests for generation of entries with metadata
• embedded/store: non-indexable entries
• embedded/store: Add max header version used during writes
• embedded/store: Allow changing tx header value using GRPC api.
• embedded/tbtree: decouple flush and sync by introducing syncThreshold attribute
• immuadmin: Allow changing proof compatibility from immuadmin
• kv: Update grpc protocol with KV set constraints
• pkg/api: tx api with entry filtering capabilities
• pkg/api: delete database endpoint
• pkg/client: new method to fetch tx entries in a single call
• pkg/database: Updated GRPC protocol for constrained writes
• pkg/database: add noWait attribute in get request
• pkg/database: Update code for new constrained write protocol
• pkg/server: database health endpoint
• pkg/server: support database loading/unloading
• pkg/server: new endpoint databaseSettings
• pkg/server: expose all database settings
• tools/monitoring: added datasource selection, added instance selection, labels include instance, fixed calculations
• tools/monitoring: Add immudb Grafana dashboard

<a name="v1.2.2"></a>

Bug Fixes

• registering connection in order to make possible conn recycling
• Dockerfile: Add ca-certificates.crt file to immudb image
• client/file_cache: Fix storing immudb state in file cache
• embedded/immustore: Avoid deadlock when acquire vLog lock
• embedded/sql: max key len validations
• embedded/sql: consider not null flag is on for auto incremental column
• pkg/server: validate if db is not replica then other replication attributes are not set
• pkg/stdlib: fix last insert id generation

Changes

• create code of conduct markdown file (#1051)
• cmd/immuclient: return actual login error
• embedded/sql: wip client provided auto-incremental values
• embedded/sql: change column constraints ordering
• embedded/sql: wip client provided auto-incremental values
• embedded/sql: add first and last insert pks retrivial methods
• embedded/sql: wip client provided auto-incremental values
• metrics: Add indexer metrics
• metrics: Add more s3-related metrics
• pkg/database: temporarily disable execall validations
• pkg/database: pre-validation of duplicated entries in execAll operation
• pkg/database: instantiate tx holder only in safe mode
• pkg/database: self-contained noWait execAll
• pkg/database: descriptive error messages
• pkg/replication: delay replication after failure
• pkg/stdlib: clean connection registration and leftovers

Features

• embedded/sql: support for basic insert conflict handling
• s3: Add support for AWS V4 signatures

<a name="v1.2.1"></a>

Bug Fixes

• fix interactive use database
• embedded/store: change already closed error message
• embedded/store: readonly tx entries to ensure no runtime modification
• embedded/store: reserve 4bytes in buffers for nentries
• embedded/tbtree: set fixed snapshot ts
• pkg/server/sessions: remove transaction on read conflict error
• pkg/server/sessions/internal/transactions: transaction is cleared after sqlExec error
• sql: Do not panic on error during delete
• tx: Remove summary from metadata

Changes

• embedded/store: txmetdata placeholder with zero len
• embedded/store: private readonly metadata is validated when reading data
• embedded/store: read-only kv metadata for committed entries
• embedded/store: rw and readonly kv metadata
• pkg/api: use new kvmetadata api
• pkg/client: tx read conflict error is mapped in an CodInFailedSqlTransaction
• pkg/server/sessions/internal/transactions: defer only when needed
• pkg/stdlib: clean tx after rollback
• pkg/stdlib: fix connection creation
• server/sessions: modify read conflict error message

Features

• pkg/stdlib: expose tx on std lib

<a name="v1.2.0"></a>

Bug Fixes

• database: Internal consistency check on data reads
• database/meta: Do not crash on history with deleted items
• pkg/database: history skipping not found entries
• protobuf: Fix compatibility with 1.1 version

Changes

• cmd/immuadmin/command: add super user login hint
• embedded/sql: use sql standard escaping with single quotes
• embedded/sql: support for escaped strings
• embedded/store: reduce attribute code size
• embedded/store: mandatory expiration filter
• embedded/store: fix expiration error declaration
• embedded/store: dedicated expiration error
• embedded/store: improve metadata serialization/deserialization methods
• embedded/store: validations during metadata deserialization
• embedded/store: return data corrupted error when deserialization cannot proceed
• embedded/store: easily extendable meta attributes
• embedded/store: use fixed time during the lifespan of a tx
• embedded/store: prevent value reading of expired entries
• makefile: remove windows binaries digital signature
• pkg/auth: require admin permission to export and replicate txs
• pkg/integration: remove useless compilation tag on tests
• pkg/server: deprecate GetAuth and WithAuth
• pkg/server/sessions: session max inactivity time set to 3m and minor stat collecting fix
• pkg/server/sessions: tuning sessions params
• pkg/server/sessions: session timeout set to 2 min

Features

• embedded/store: logical entries expiration
• pkg/api: logical entries expiration
• pkg/client: expirable set

<a name="v1.2.0-RC1"></a>

Bug Fixes

• Update jaswdr/faker to v1.4.3 to fix build on 32-bit systems
• Makefile: Use correct version of the grpc-gateway package
• Makefile: Fix building immudb for specific os/arch
• embedded/sql: normalize parameters with lower case identifiers
• embedded/sql: Do not modify value returned by colsBySelector
• embedded/sql: correct max key length validation based on specified col max length
• embedded/sql: fix rollback stmt
• embedded/sql: distinct row reader with limit argument
• embedded/sql: ensure determinism and no value overlaps distinct rows
• embedded/sql: Use correct statement for subquery
• embedded/sql: param substitution in LIKE expression
• embedded/sql: Fix SELECT * when joining with subquery
• embedded/sql: fix inserting calculated null values
• embedded/store: release lock when tx has a conflict
• embedded/store: read conflict validation
• embedded/store: typo in error message
• pkg/auth: Fix password tests
• pkg/client: fix database name saving on token service
• pkg/database: sql exec on provided tx
• pkg/server: fix keep alive session interceptor
• testing: using pointers for job channels
• webconsole: Fix html of the default missing page.

Changes

• Update build/RELEASING.md documentation.
• remove token service from client options and fix tests
• token is handled internally by sdk. Remove useless code
• refining sdk client constructor and add readOnly tx guard
• fix more tests
• decoupled token service
• cmd/immuadmin/command: fix immuadmin token name on client creation
• cmd/immuclient: deleteKeys functioin and updates after metadata-related changes
• cmd/immuclient: temporary disable displaying hash in non-verified methods
• embeddded/tbtree: leverage snapshot id to identify it's the current unflushed one
• embedded/multierr: minor code simplification
• embedded/sql: rollback token
• embedded/sql: changes on tx closing
• embedded/sql: postponing short-circuit evaluation for safetiness
• embedded/sql: set INNER as default join type
• embedded/sql: Better error messages when (up|in)serting data
• embedded/sql: minor code simplification
• embedded/sql: standardized datasource aliasing
• embedded/sql: remove opt_unique rule to ensure proper error message
• embedded/sql: fix nullable values handling
• embedded/sql: use order type in scanSpecs
• embedded/sql: kept last snapshot open
• embedded/sql: de-duplicate tx attributes using tx header struct
• embedded/sql: unsafe snapshot without flushing
• embedded/sql: wip sql tx preparation
• embedded/sql: set parsing verbose mode when instantiating sql engine
• embedded/sql: reusable index entries and ignore deleted index entries
• embedded/sql: limit row reader
• embedded/sql: delay index sync until fetching row by its pk
• embedded/sql: leverage metadata for logical deletion
• embedded/sql: Simplify row_reader key selection
• embedded/sql: use int type for limit arg
• embedded/sql: minor update after rebasing
• embedded/sql: cancel non-closed tx
• embedded/sql: expose Cancel method
• embedded/sql: sql engine options and validations
• embedded/sql: Alter index key prefixes
• embedded/sql: return map with last inserted pks
• embedded/sql: wip rw transactions
• embedded/sql: defer execution of onClose callback
• embedded/sql: wip sqlTx
• embedded/sql: standard count(*)
• embedded/sql: ddl stmts not counted in updatedRows
• embedded/sql: method to return sql catalog
• embedded/sql: non-thread safe tx
• embedded/sql: wip interactive sqltx
• embedded/sql: bound stmt execution to a single sqltx
• embedded/sql: use current db from ongoing tx
• embedded/store: expose ExistKeyWithPrefix in OngoingTx
• embedded/store: conservative read conflict validation
• embedded/store: non-thread safe ongoing tx
• embedded/store: reorder tx validations
• embedded/store: wip tx header versioning
• embedded/store: simplified ExistKeyWithPrefix in current snapshot
• embedded/store: set tx as closed even on failed attempts
• embedded/store: strengthen tx validations
• embedded/store: GetWith method accepting filters
• embedded/store: set header version at commit time
• embedded/store: remove currentShapshot method
• embedded/store: entryDigest calculation including key len
• embedded/store: threadsafe tx
• embedded/store: handle watchersHub closing error
• embedded/store: ongoing tx api
• embedded/store: tx header version validations and increased max number of entries
• embedded/store: early tx conflict checking
• embedded/store: filter out entries when filter evals to true
• embedded/tbtree: remove ts from snapshot struct
• embedded/tools: upgrade sql stress tool
• embedded/tools: upgrade stress tool using write-only txs
• embedded/tools: update stress_tool after metadata-related changes
• pkg/api: changes in specs to include new metadata records
• pkg/api: consider nil case during tx header serialization
• pkg/api/schema: increase supported types when converting to sql values
• pkg/client: check if token is present before injecting it
• pkg/client: omit deleted flag during value decoding
• pkg/client: updates after metadata-related changes
• pkg/client: avoid useless tokenservice call and add tests
• pkg/client/clienttest: fix immuclient mock
• pkg/client/tokenservice: handlig error properly on token interceptor and fix leftovers
• pkg/database: return a specific error in querying
• pkg/database: snapshots should be up to current committed tx
• pkg/database: improve readability of Database interface
• pkg/database: limit query len result
• pkg/database: updates after metadata-related changes
• pkg/database: use new transaction support
• pkg/database: implement current functionality with new tx supportt
• pkg/database: revised locking so to ensure gracefully closing
• pkg/database: enforce verifiableSQLGet param validation
• pkg/errors: useDatabase returns notFound code when error
• pkg/errors: invalid database name error converted to immuerror
• pkg/integration: updates after metadata-related changes
• pkg/server: use upgraded database apis
• pkg/server: updates after metadata-related changes
• pkg/server: error when tx are not closed
• pkg/server/sessions: polish logger call
• pkg/server/sessions: add sessions counter debug messages
• pkg/stdlib: remove context injection when query or exec
• pkg/stdlib: fix unit testing
• pkg/stdlib: increase pointer values handling and testing
• pkg/stdlib: general improvements and polishments
• pkg/stdlib: handling nil pointers when converting to immudb named params
• pkg/stdlib: improve connection handling and allow ssl mode in connection string
• stress_tool_sql: add sessions and transaction mode
• stress_tool_worker_pool: add long running stress tool
• test: test backward compatibility with previous release (v1.1.0)
• tsting: add index compactor in long running stress tool

Features

• helm chart for deploying immudb on kubernetes (#997)
• embedded/appendable: method for reading short unsigned integer
• embedded/sql: null values for secondary indexes
• embedded/sql: support for IN clause
• embedded/sql: support for not like
• embedded/sql: WIP un-restricted upsert
• embedded/sql: engine as tx executor
• embedded/sql: wip sqltx at engine with autocommit
• embedded/sql: increased expression power in LIKE and IN clauses
• embedded/sql: Detect ambigous selectons on joins
• embedded/sql: distinct row reader
• embedded/sql: delete from statement
• embedded/sql: sql update statement
• embedded/sql: support value expression in like pattern
• embedded/sql: create index if not exists
• embedded/store: initial commit towards full tx support
• embedded/store: wip enhanced tx support
• embedded/store: conservative tx invalidation
• embedded/store: included filters in key readers
• embedded/store: including metadata records
• embedded/store: logical key deletion api
• embedded/store: keyReader in tx scope
• embedded/store: functional constraints
• embedded/tbtree: read as before returns history count
• embedded/tbtree: implements ExistKeyWithPrefix in snapshots
• sql: Add support for IS NULL / IS NOT NULL expressions
• sql/index-on-nulls: Update on-disk format to support nullable values
• sql/timestamp: Add CAST from varchar and integer to timestamp
• sql/timestamp: Add timestamp support to embedded/sql
• sql/timestamp: Add timestamp to protobuf definition
• sql/timestamp: Add timestamp to stdlib

<a name="v1.1.0"></a>