github.com/dgraph-io/dgraph

Changed

• Breaking changes

  • uid schema type: The uid schema type now means a one-to-one relation, not a one-to-many relation as in Dgraph v1.1. To specify a one-to-many relation in Dgraph v1.0, use the [uid] schema type. (#2895, #3173, #2921)

  • _predicate_ is removed from the query language.

  • expand(_all_) only works for nodes with attached type information via the type system. The type system is used to determine the predicates to expand out from a node. (#3262)

  • S * * deletion only works for nodes with attached type information via the type system. The type system is used to determine the predicates to delete from a node. For S * * deletions, only the predicates specified by the type are deleted.

  • HTTP API: The HTTP API has been updated to replace the custom HTTP headers with standard headers.

    • Change /commit endpoint to accept a list of preds for conflict detection. (#3020)
    • Remove custom HTTP Headers, cleanup API. (#3365)
      • The startTs path parameter is now a query parameter startTs for the /query, /mutate, and /commit endpoints.
      • Dgraph custom HTTP Headers X-Dgraph-CommitNow, X-Dgraph-MutationType, and X-Dgraph-Vars are now ignored.
    • Update HTTP API Content-Type headers. (#3550) (#3532)
      • Queries over HTTP must have the Content-Type header application/graphql+- or application/json.
      • Queries over HTTP with GraphQL Variables (e.g., query queryName($a: string) { ... }) must use the query format via application/json to pass query variables.
      • Mutations over HTTP must have the Content-Type header set to application/rdf for RDF format or application/json for JSON format.
      • Commits over HTTP must have the startTs query parameter along with the JSON map of conflict keys and predicates.

  • Datetime index: Use UTC Hour, Day, Month, Year for datetime comparison. This is a bug fix that may result in different query results for existing queries involving the datetime index. (#3251)

  • Blank node name generation for JSON mutations. For JSON mutations that do not explicitly set the "uid" field, the blank name format has changed to contain randomly generated identifiers. This fixes a bug where two JSON objects within a single mutation are assigned the same blank node. (#3795)

• Improve hash index. (#2887)

• Use a stream connection for internal connection health checking. (#2956)

• Use defer statements to release locks. (#2962)

• VerifyUid should wait for membership information. (#2974)

• Switching to perfect use case of sync.Map and remove the locks. (#2976)

• Tablet move and group removal. (#2880)

• Delete tablets which don't belong after tablet move. (#3051)

• Alphas inform Zero about tablets in its postings directory when Alpha starts. (3271f64e0)

• Prevent alphas from asking zero to serve tablets during queries. (#3091)

• Put data before extensions in JSON response. (#3194)

• Always parse language tag. (#3243)

• Populate the StartTs for the commit gRPC call so that clients can double check the startTs still matches. (#3228)

• Replace MD5 with SHA-256 in dgraph cert ls. (#3254)

• Fix use of deprecated function grpc.WithTimeout(). (#3253)

• Introduce multi-part posting lists. (#3105)

• Fix format of the keys to support startUid for multi-part posting lists. (#3310)

• Access groupi.gid atomically. (#3402)

• Move Raft checkpoint key to w directory. (#3444)

• Remove list.SetForDeletion method, remnant of the global LRU cache. (#3481)

• Whitelist by hostname. (#2953)

• Use CIDR format for whitelists instead of the previous range format.

• Introduce Badger's DropPrefix API into Dgraph to simplify how predicate deletions and drop all work internally. (#3060)

• Replace integer compression in UID Pack with groupvarint algorithm. (#3527, #3650)

• Rebuild reverse index before count reverse. (#3688)

• Breaking change: Use one atomic variable to generate blank node ids for json objects. This changes the format of automatically generated blank node names in JSON mutations. (#3795)

• Print commit SHA256 when invoking "make install". (#3786)

• Print SHA-256 checksum of Dgraph binary in the version section logs. (#3828)

• Change anonynmous telemetry endpoint. (#3872)

• Add support for API required for multiple mutations within a single call. (#3839)

• Make lru_mb optional. (#3898)

• Allow glog flags to be set via config file. (#3062, #3077)

• Logging

  • Suppress logging before flag.Parse from glog. (#2970)
  • Move glog of missing value warning to verbosity level 3. (#3092)
  • Change time threshold for Raft.Ready warning logs. (#3901)
  • Add log prefix to stream used to rebuild indices. (#3696)
  • Add additional logs to show progress of reindexing operation. (#3746)

• Error messages

  • Output the line and column number in schema parsing error messages. (#2986)
  • Improve error of empty block queries. (#3015)
  • Update flag description and error messaging related to --query_edge_limit flag. (#2979)
  • Reports line-column numbers for lexer/parser errors. (#2914)
  • Replace fmt.Errorf with errors.Errorf (#3627)
  • Return GraphQL compliant "errors" field for HTTP requests. (#3728)

• Optimizations

  • Don't read posting lists from disk when mutating indices. (#3695, #3713)
  • Avoid preallocating uid slice. It was slowing down unpackBlock.
  • Reduce memory consumption in bulk loader. (#3724)
  • Reduce memory consumptino by reusing lexer for parsing RDF. (#3762)
  • Use the stream framework to rebuild indices. (#3686)
  • Use Stream Writer for full snapshot transfer. (#3442)
  • Reuse postings and avoid fmt.Sprintf to reduce mem allocations (#3767)
  • Speed up JSON chunker. (#3825)
  • Various optimizations for Geo queries. (#3805)

• Update various govendor dependencies

  • Add OpenCensus deps to vendor using govendor. (#2989)
  • Govendor in latest dgo. (#3078)
  • Vendor in the Jaeger and prometheus exporters from their own repos (#3322)
  • Vendor in Shopify/sarama to use its Kafka clients. (#3523)
  • Update dgo dependency in vendor. (#3412)
  • Update vendored dependencies. (#3357)
  • Bring in latest changes from badger and fix broken API calls. (#3502)
  • Vendor badger with the latest changes. (#3606)
  • Vendor in badger, dgo and regenerate protobufs. (#3747)
  • Vendor latest badger. (#3784)
  • Breaking change: Vendor in latest Badger with data-format changes. (#3906)

Dgraph Debug Tool

• When looking up a key, print if it's a multi-part list and its splits. (#3311)
• Diagnose Raft WAL via debug tool. (#3319)
• Allow truncating Raft logs via debug tool. (#3345)
• Allow modifying Raft snapshot and hardstate in debug tool. (#3364)

Dgraph Live Loader / Dgraph Bulk Loader

• Add --format flag to Dgraph Live Loader and Dgraph Bulk Loader to specify input data format type. (#2991)
• Update live loader flag help text. (#3278)
• Improve reporting of aborts and retries during live load. (#3313)
• Remove xidmap storage on disk from bulk loader.
• Optimize XidtoUID map used by live and bulk loader. (#2998)
• Export data contains UID literals instead of blank nodes. Using Live Loader or Bulk Loader to load exported data will result in the same UIDs as the original database. (#3004, #3045) To preserve the previous behavior, set the --new_uids flag in the live or bulk loader. (18277872f)
• Use StreamWriter in bulk loader. (#3542, #3635, #3649)
• Add timestamps during bulk/live load. (#3287)
• Use initial schema during bulk load. (#3333)
• Adding the verbose flag to suppress excessive logging in live loader. (#3560)
• Fix user meta of schema and type entries in bulk loader. (#3628)
• Check that all data files passed to bulk loader exist. (#3681)
• Handle non-list UIDs predicates in bulk loader. #3659
• Use sync.Pool for MapEntries in bulk loader. (#3763, 802ec4c39)

Dgraph Increment Tool

• Add server-side and client-side latency numbers to increment tool. (#3422)
• Add --retries flag to specify number of retry requests to set up a gRPC connection. (#3584)
• Add TLS support to dgraph increment command. (#3257)

Added

• Add bash and zsh shell completion. See dgraph completion bash --help or dgraph completion zsh --help for usage instructions. (#3084)

• Add support for ECDSA in dgraph cert. (#3269)

• Add support for JSON export via /admin/export?format=json. (#3309)

• Add the SQL-to-Dgraph migration tool dgraph migrate. (#3295)

• Add assign_timestamp_ns latency field to fix encoding_ns calculation. Fixes #3668. (#3692, #3711)

• Adding draining mode to Alpha. (#3880)

• Enterprise features

  • Support applying a license using /enterpriseLicense endpoint in Zero. (#3824)
  • Don't apply license state for oss builds. (#3847)

Query

• Type system

  • Add type function to query types. (#2933)
  • Parser for type declaration. (#2950)
  • Add @type directive to enforce type constraints. (#3003)
  • Store and query types. (#3018)
  • Rename type predicate to dgraph.type (#3204)
  • Change definition of dgraph.type pred to [string]. (#3235)
  • Use type when available to resolve expand predicates. (#3214)
  • Include types in results of export operation. (#3493)
  • Support types in the bulk loader. (#3506)

• Add the upsert block to send "query-mutate-commit" updates as a single call to Dgraph. This is especially helpful to do upserts with the @upsert schema directive. Addresses #3059. (#3412)

  • Add support for conditional mutation in Upsert Block. (#3612)

• Allow querying all lang values of a predicate. (#2910)

• Allow regexp() in @filter even for predicates without the trigram index. (#2913)

• Add minweight and maxweight arguments to k-shortest path algorithm. (#2915)

• Allow variable assignment of count(uid). (#2947)

• Reserved predicates

  • During startup, don't upsert initial schema if it already exists. (#3374)
  • Use all reserved predicates in IsReservedPredicateChanged. (#3531)

• Fuzzy match support via the match() function using the trigram index. (#2916)

• Support for GraphQL variables in arrays. (#2981)

• Show total weight of path in shortest path algorithm. (#2954)

• Rename dgraph --dgraph option to --alpha. (#3273)

• Support uid variables in from and to arguments for shortest path query. Fixes #1243. (#3710)

• Add support for len() function in query language. The len() function is only used in the @if directive for upsert blocks. len(v) It returns the length of a variable v. (#3756, #3769)

Mutation

• Add ability to delete triples of scalar non-list predicates. (#2899, #3843)
• Allow deletion of specific language. (#3242)

Alter

• Add DropData operation to delete data without deleting schema. (#3271)

Schema

• Breaking change: Add ability to set schema to a single UID schema. Fixes #2511. (#2895, #3173, #2921)
  • If you wish to create one-to-one edges, use the schema type uid. The uid schema type in v1.0.x must be changed to [uid] to denote a one-to-many uid edge.
• Prevent dropping or altering reserved predicates. (#2967) (#2997)
  • Reserved predicate names start with dgraph. .
• Support comments in schema. (#3133)
• Reserved predicates
  • Reserved predicates are prefixed with "dgraph.", e.g., dgraph.type.
  • Ensure reserved predicates cannot be moved. (#3137)
  • Allow schema updates to reserved preds if the update is the same. (#3143)

Enterprise feature: Access Control Lists (ACLs)

Enterprise ACLs provide read/write/admin permissions to defined users and groups at the predicate-level.

• Enforcing ACLs for query, mutation and alter requests. (#2862)
• Don't create ACL predicates when the ACL feature is not turned on. (#2924)
• Add HTTP API for ACL commands, pinning ACL predicates to group 1. (#2951)
• ACL: Using type to distinguish user and group. (#3124)
• Reduce the value of ACL TTLs to reduce the test running time. (#3164)
  • Adds --acl_cache_ttl flag.
• Fix panic when deleting a user or group that does not exist. (#3218)
• ACL over TLS. (#3207)
• Using read-only queries for ACL refreshes. (#3256)
• When HttpLogin response context error, unmarshal and return the response context. (#3275)
• Refactor: avoid double parsing of mutation string in ACL. (#3494)
• Security fix: prevent the HmacSecret from being logged. (#3734)

Enterprise feature: Backups

Enterprise backups are Dgraph backups in a binary format designed to be restored to a cluster of the same version and configuration. Backups can be stored on local disk or stored directly to the cloud via AWS S3 or any Minio-compatible backend.

• Fixed bug with backup fan-out code. (#2973)
• Incremental backups / partial restore. (#2963)
• Turn obsolete error into warning. (#3172)
• Add dgraph lsbackup command to list backups. (#3219)
• Add option to override credentials and use public buckets. (#3227)
• Add field to backup requests to force a full backup. (#3387)
• More refactoring of backup code. (#3515)
• Use gzip compression in backups. (#3536)
• Allow partial restores and restoring different backup series. (#3547)
• Store group to predicate mapping as part of the backup manifest. (#3570)
• Only backup the predicates belonging to a group. (#3621)
• Introduce backup data formats for cross-version compatibility. (#3575)
• Add series and backup number information to manifest. (#3559)
• Use backwards-compatible formats during backup (#3629)
• Use manifest to only restore preds assigned to each group. (#3648)
• Fixes the toBackupList function by removing the loop. (#3869)
• Add field to backup requests to force a full backup. (#3387)

Dgraph Zero

• Zero server shutdown endpoint /shutdown at Zero's HTTP port. (#2928)

Dgraph Live Loader

• Support live loading JSON files or stdin streams. (#2961) (#3106)
• Support live loading N-Quads from stdin streams. (#3266)

Dgraph Bulk Loader

• Add --replace_out option to bulk command. (#3089)

Tracing

• Support exporting tracing data to oc_agent, then to datadog agent. (#3398)
• Measure latency of Alpha's Raft loop. (63f545568)

Removed

• Breaking change: Remove _predicate_ predicate within queries. (#3262)

• Remove --debug_mode option. (#3441)

• Remove deprecated and unused IgnoreIndexConflict field in mutations. This functionality is superceded by the @upsert schema directive since v1.0.4. (#3854)

• Enterprise features

  • Remove --enterprise_feature flag. Enterprise license can be applied via /enterpriseLicense endpoint in Zero. (#3824)

Fixed

• Fix anyofterms() query for facets from mutations in JSON format. Fixes #2867. (#2885)

• Fixes error found by gofuzz. (#2914)

• Fix int/float conversion to bool. (#2893)

• Handling of empty string to datetime conversion. (#2891)

• Fix schema export with special chars. Fixes #2925. (#2929)

• Default value should not be nil. (#2995)

• Sanity check for empty variables. (#3021)

• Panic due to nil maps. (#3042)

• ValidateAddress should return true if IPv6 is valid. (#3027)

• Throw error when @recurse queries contain nested fields. (#3182)

• Fix panic in fillVars. (#3505)

• Fix race condition in numShutDownSig in Alpha. (#3402)

• Fix race condition in oracle.go. (#3417)

• Fix tautological condition in zero.go. (#3516)

• Correctness fix: Block before proposing mutations and improve conflict key generation. Fixes #3528. (#3565)

• Reject requests with predicates larger than the max size allowed (longer than 65,535 characters). (#3052)

• Upgrade raft lib and fix group checksum. (#3085)

• Check that uid is not used as function attribute. (#3112)

• Do not retrieve facets when max recurse depth has been reached. (#3190)

• Remove obsolete error message. (#3172)

• Remove an unnecessary warning log. (#3216)

• Fix bug triggered by nested expand predicates. (#3205)

• Empty datetime will fail when returning results. (#3169)

• Fix bug with pagination using after. (#3149)

• Fix tablet error handling. (#3323)

• Fix crash when trying to use shortest path with a password predicate. Fixes #3657. (#3662)

• Fix crash for @groupby queries. Fixes #3642. (#3670)

• Fix crash when calling drop all during a query. Fixes #3645. (#3664)

• Fix data races in queries. Fixes #3685. (#3749)

• Bulk Loader: Fix memory usage by JSON parser. (#3794)

• Fixing issues in export. Fixes #3610. (#3682)

• Bug Fix: Use txn.Get in addReverseMutation if needed for count index (#3874)

• Bug Fix: Remove Check2 at writeResponse. (#3900)

• Bug Fix: Do not call posting.List.release.

Fixed

• Preserve the order of entries in a mutation if multiple versions of the same edge are found. This addresses the mutation re-ordering change ([#2987][]) from v1.0.15.
• Fixing the zero client in live loader to avoid using TLS. Fixes #3919. (#3936)
• Remove query cache which is causing contention. (#4071).
• Fix bug when querying with nested levels of expand(_all_). Fixes #3807. (#4143).
• Vendor in Badger to fix a vlog bug "Unable to find log file". (#4212)
• Change lexer to allow unicode escape sequences. Fixes [#4157][]. (#4252)

Changed

• Increase max trace logs per span in Alpha. (#3886)
• Include line and column numbers in lexer errors. Fixes #2900. (#3772)
• Release binaries built with Go 1.12.7.

Fixed

• Decrease rate of Raft heartbeat messages. (#3708, #3753)
• Fix bug when exporting a predicate name to the schema. Fixes #3699. (#3701)
• Return error instead of asserting in handleCompareFunction. (#3665)
• Fix bug where aliases in a query incorrectly alias the response depending on alias order. Fixes #3814. (#3837)
• Fix for panic in fillGroupedVars. Fixes #3768. (#3781)

Changed

• Vendor in prometheus/client_golang/prometheus v0.9.4. (#3653)

Fixed

• Fix panic with value variables in queries. Fixes #3470. (#3554)
• Remove unused reserved predicates in the schema. Fixes #3535. (#3557)
• Vendor in Badger v1.6.0 for StreamWriter bug fixes. (#3631)

Fixed

• Fix bug that can cause a Dgraph cluster to get stuck in infinite leader election. (#3391)
• Fix bug in bulk loader that prevented loading data from JSON files. (#3464)
• Fix bug with a potential deadlock by breaking circular lock acquisition. (#3393)
• Properly escape strings containing Unicode control characters for data exports. Fixes #3383. (#3429)
• Initialize tablets map when creating a group. (#3360)
• Fix queries with offset not working with multiple orderasc or orderdesc statements. Fixes #3366. (#3455)
• Vendor in bug fixes from badger. (#3348, #3371, #3460)

Changed

• Use Go v1.12.5 to build Dgraph release binaries.
• Truncate Raft logs even when no txn commits are happening. (3be380b8a)
• Reduce memory usage by setting a limit on the size of committed entries that can be served per Ready. (#3308)
• Reduce memory usage of pending txns by only keeping deltas in memory. (#3349)
• Reduce memory usage by limiting the number of pending proposals in apply channel. (#3340)
• Reduce memory usage when calculating snapshots by retrieving entries in batches. (#3409)
• Allow snapshot calculations during snapshot streaming. (ecb454754)
• Allow quick recovery from partitions by shortening the deadline of sending Raft messages to 10s. (77b52aca1)
• Take snapshots less frequently so straggling Alpha followers can catch up to the leader. Snapshot frequency is configurable via a flag (see Added section). (#3367)
• Allow partial snapshot streams to reduce the amount of data needed to be transferred between Alphas. (#3454)
• Use Badger's StreamWriter to improve write speeds during snapshot streaming. (#3457) ([#3442][])
• Call file sync explicitly at the end of TxnWriter to improve performance. (#3418)
• Optimize mutation and delta application. Breaking: With these changes, the mutations within a single call are rearranged. So, no assumptions must be made about the order in which they get executed. (#2987)
• Add logs to show Dgraph config options. (#3337)
• Add -v=3 logs for reporting Raft communication for debugging. These logs start with RaftComm:. (9cd628f6f)

Added

• Add Alpha flag --snapshot_after (default: 10000) to configure the number of Raft entries to keep before taking a snapshot. (#3367)
• Add Alpha flag --abort_older_than (default: 5m) to configure the amount of time since a pending txn's last mutation until it is aborted. (#3367)
• Add Alpha flag --normalize_node_limit (default: 10000) to configure the limit for the maximum number of nodes that can be returned in a query that uses the @normalize directive. Fixes #3335. (#3467)
• Add Prometheus metrics for latest Raft applied index (dgraph_raft_applied_index) and the max assigned txn timestamp (dgraph_max_assigned_ts). These are useful to track cluster progress. (#3338)
• Add Raft checkpoint index to WAL for quicker recovery after restart. ([#3444][])

Removed

• Remove size calculation in posting list. (0716dc4e1)
• Remove a -v=2 log which can be too noisy during Raft replay. (2377d9f56).
• Remove dgraph_conf from /debug/vars. Dgraph config options are available via logs. (#3337)