What's New in v20.2.0-alpha.1

June 17, 2020

Warning:

A denial-of-service (DoS) vulnerability is present in CockroachDB v20.2.0 - v20.2.3 due to a bug in protobuf. This is resolved in CockroachDB v20.2.4 and later releases. When upgrading is not an option, users should audit their network configuration to verify that the CockroachDB HTTP port is not available to untrusted clients. We recommend blocking the HTTP port behind a firewall.

For more information, including other affected versions, see Technical Advisory 58932.

Get future release notes emailed to you:

Downloads

Docker image

icon/buttons/copy 
$ docker pull cockroachdb/cockroach-unstable:v20.2.0-alpha.1

Backward-incompatible changes

  • The textual error and warning messages displayed by cockroach quit under various circumstances have been updated. Meanwhile, the message "ok" remains as an indicator that the operation has likely succeeded. #45149
  • cockroach quit now prints out progress details on its standard error stream, even when --logtostderr is not specified. Previously, nothing was printed on standard error. Scripts that wish to ignore this output can redirect the stderr stream. #45149
  • The copy of system and crdb_internal tables extracted by cockroach debug zip is now written using the TSV format (inside the zip file), instead of an ASCII-art table as previously. #48096
  • CockroachDB v20.1 introduced a new rule for the --join flag to cockroach start, causing it to prefer SRV records, if present in DNS, to look up the peer nodes to join. This feature is experimental, and has been found to cause disruption in in certain deployments. To reduce this disruption and the resulting UX surprise, the feature is now gated behind a new command-line flag --experimental-dns-srv which must now be explicitly passed to cockroach start to enable it. #49077
  • Prior to this patch, the phase of server shutdown responsible for range lease transfers to other nodes would give up after 10000 attempts of transferring replica leases away, regardless of the value of server.shutdown.lease_transfer_wait. The limit of 10000 attempts has been removed, so that now only the maximum duration server.shutdown.lease_transfer_wait applies. #45149

General changes

  • Transactions reading a lot of data behave better when exceeding the memory limit set by kv.transaction.max_refresh_spans_bytes. Such transactions now attempt to resolve the conflicts they run into instead of being forced to always retry. Increasing kv.transaction.max_refresh_spans_bytes should no longer be necessary for most workloads. #46275
  • Schema changes started in 19.2 will now be automatically migrated in the background in 20.1 after upgrade finalization so that they can run to completion. #46504
  • Prior to this patch, the phase of server shutdown responsible for range lease transfers to other nodes had a hard timeout of 5 seconds. This patch makes this timeout configurable via the new cluster setting server.shutdown.lease_transfer_wait. #45149
  • Statement diagnostics zip bundles now contain a representation of the statement trace that can be imported into Jaeger for visualization. #47303
  • The system.replication_constraint_stats report now treats conjunctions of replication constraints as a single unit of reporting, whereas before there was a report entry for each constraint in the conjunction. For example, given a zone with a replication constraint like {"+region=us,+az=az1":1} (meaning, one replica in a node with locality region=us,az=az1), now there will be a report entry for "+region=us,+az=az1" where before there was a separate entry for "+region=us" and another one for "+az=az1". #47389
  • The Kubernetes manifests now generate node certificates with additional addresses 'cockroachdb-public.<namespace>' and 'cockroachdb-public.<namespace>.svc'. #47610
  • Changed the default engine type for new storage directories from RocksDB to Pebble. Existing stores will continue to use the previously specified storage engine, and an explicit specification (via --storage-engine=...) will override the default. #48145
  • Removed the publication of musl libc CockroachDB builds. #49685
  • Improved error reporting when trying to access non-existent Google Cloud storage blobs. #49089
  • Improved reliability by handling interrupted downloads from Google Cloud external storage more gracefully. #48229
  • Treat errors due to draining nodes as retryable when starting CDC. #49743

Enterprise edition changes

  • BACKUP and RESTORE now collect some anonymous telemetry on throughput and feature usage. #46716
  • SHOW BACKUP now shows whether a BACKUP is a full cluster backup or not. #45860
  • The privileges assigned to a given table/database are now visible through SHOW BACKUP. SHOW BACKUP will list which users and roles had which privileges on each table/database in the backup if the WITH privileges option is specified. #45862
  • The new appended incremental backup syntax does not allow converting a full cluster backup to a specific table or database backup. #46864
  • BACKUP and RESTORE to HTTP storage locations now require explicitly specifying incremental storage locations (i.e. cannot use the new automatically appended incremental syntax). #46887
  • Fixed a bug where the job ID of a lagging changefeed would be omitted and instead it would be reported as sinkless. #48493
  • Changefeeds now retry after encountering transient errors contacting the Confluent Schema Registry. #48759

SQL language changes

Geospatial SQL updates

  • Introduced the ability to parse, store and display PostGIS style GEOGRAPHY and GEOMETRY types. #47171
  • When doing name resolution via search path, the pg_extension schema (containing tables such as geometry_columns, geography_columns and spatial_ref_sys) will now have an attempted resolution before the public schema. This mimics PostGIS behavior where the aforementioned tables are in the public schema, and so by default are discoverable tables with a new CLI session. #49611
  • Added support for the AddGeometryColumn function, which adds a new GEOMETRY column to an existing table and returns metadata about the column created. This improves compatibility with PostGIS. #49456
  • Added support for ST_Covers, ST_CoveredBy, ST_Contains, ST_Crosses, ST_Equals, ST_Intersects, ST_Overlaps, ST_Touches and ST_Within for GEOMETRY geospatial types. #47584
  • Added the following geospatial functions: ST_GeomFromText, ST_GeometryFromText, ST_GeogFromText, ST_GeographyFromText, ST_GeomFromWKB, ST_GeomFromEWKB, ST_GeomFromEWKT, ST_GeomFromGeoJSON, ST_GeogFromWKB, ST_GeogFromEWKB, ST_GeogFromEWKT, ST_GeogFromGeoJSON, ST_AsText, ST_AsBinary, ST_AsEWKB, ST_AsEWKT, ST_AsKML, ST_AsGeoJSON. #48015
  • Added geospatial functions ST_Area, ST_Length, ST_Distance, and ST_Perimeter, which operate on GEOMETRY data types. #48074
  • Added builtins allowing parsing text and WKB for all supported shapes, e.g. ST_PointFromWKB and ST_PointFromText, which return NULL if the WKT/EWKT/WKB do not match the shape provided. #48441
  • Implemented the GEOMETRY based builtins ST_Relate and ST_ContainsProperly. #48552
  • Introduced the following functions that work on GEOGRAPHY types: ST_Covers, ST_CoveredBy, ST_Intersects, ST_Distance, ST_DWithin, ST_Perimeter, ST_Area, and ST_Length. #48529
  • Implemented the GEOMETRY based builtin ST_SRID. #49169
  • Implemented the ST_Point and ST_MakePoint functions. #49194
  • Implemented the ST_SetSRID function on GEOMETRY and GEOGRAPHY types. #49222
  • Implemented the ST_MaxDistance and ST_DFullyWithin functions for geometries. #49094
  • Implemented the ST_WKBToSQL and ST_WKTToSQL geospatial functions. #49237
  • Implemented the GEOMETRY based builtin ST_AsBinary with encoding. #49481
  • Implemented the following geospatial builtins: ST_X (resolves #49069), ST_Y (resolves #49070), ST_NDims (resolves #48992), ST_NumPoints (resolves #49001), ST_NPoints (resolves #48995), ST_StartPoint (resolves #49047), ST_EndPoint (resolves #48924), ST_NumInteriorRings (resolves #48999), ST_NumInteriorRing (resolves #48998), ST_InteriorRingN (resolves #48949), ST_ExteriorRing (resolves #48930), ST_NumGeometries (resolves #48997), ST_GeometryN (resolves #48945). #49341
  • Implemented the ST_Centroid builtin function which works for GEOMETRY and STRING arguments. #49613
  • Implemented the GEOMETRY builtin ST_AsHexEWKB with encoding. #49644
  • Implemented the following builtins available on GEOMETRY types: ST_Area2D (resolves #48869), ST_NRings (resolves #48996), ST_PointN (resolves #49008), and ST_GeometryType (resolves #48946). #49649
  • Implemented the GEOMETRY based builtin ST_Summary. #49738
  • Implemented ST_Buffer for GEOMETRY and STRING variants. #49722
  • Implemented the following built-in functions: ST_LineInterpolatePoint, ST_LineInterpolatePoints. #49742
  • Implemented DWithin for GEOMETRY types. #49085
  • Populated the pg_extension.geometry_columns and pg_extension.geography_columns virtual table with metadata for tables which store have columns relevant to GEOMETRY or GEOGRAPHY data types. #49424

General SQL updates

  • Updated the type checking code to prefer aggregate overloads with string inputs if there are multiple possible candidates due to arguments of unknown type. #46649
  • Added syntax for DECLARE and CLOSE. CLOSE ALL is a no-op, as there are no cursors to close. CLOSE <cursor> and DECLARE raise unimplemented errors. #45662
  • Modified the substring() function to allow it support bit and byte arrays. #46710
  • Added support for CREATE INDEX CONCURRENTLY and DROP INDEX CONCURRENTLY syntax. These statements are no-ops, as all indexes are created concurrently. #46695
  • Previously, we implemented the timezone builtin as timezone(<time type>, <timezone>). However, this is the opposite from the Postgres implementation (which allowed timezone(<timezone>, <time type>). In 20.1, we supported both permutations to aid with this migration, but with this change we will only support the timezone first. You can also use AT TIME ZONE, which is compatible with the timezone function in all versions. #46748
  • The IMPORT statement now records some additional anonymous telemetry about its performance and reliability. #46731
  • Removed the output showing the number of rows reported as affected by the CREATE/ALTER/DROP ROLE statements. The number doesn't make sense to include. Since the rows affected include some rows in system tables such as system.role_options rows, this number can be misleading (e.g., CREATE ROLE returning 3). This behavior matches Postgres, where no number is returned for these commands. #46795
  • Added a hint to use ALTER ROLE when trying to "grant" a role option directly to a user (using the GRANT ROLE syntax). #46795
  • Added a new unimplemented error that is signalled when attempting to ADD CONSTRAINT with the EXCLUDE USING syntax. #46787
  • Added support for using CREATE INDEX .... INCLUDE (col1, col2, ...), which is an alias that PostgreSQL uses that is analogous to our STORING (col1, col2, ...) syntax. #46785
  • Added support for parsing the REINDEX syntax, which results in an unimplemented error that explains that REINDEXing is not required in CockroachDB. #46850
  • The vectorized execution engine will only run queries with streaming operators. Issue SET vectorize=on to enable the vectorized execution engine for buffering operators. #46845
  • Added support for compressing output files using gzip. #46932
  • If setting an invalid cluster setting, the invalid values are now returned. #46932
  • CockroachDB will now be more verbose about full scans when executing EXPLAIN. Previously, we would output SPANS | ALL, now it will be SPANS | FULL SCAN or (if there is a limit) SPANS | LIMITED SCAN. #46708
  • The bit_and() aggregate function now supports BIT and VARBIT data types. #46954
  • The bit_or() aggregate function now supports BIT and VARBIT data types. #47023
  • The error messages for duplicate columns in NATURAL and USING joins now match the error messages used by Postgres. #47099
  • Modified the strpos() function to allow it to support BIT and BYTES arrays. #46875
  • Improved the error message for ALTER COLUMN ... SET DATA TYPE ... for data type conversions that involve overwriting existing values. #47155
  • Modified SHOW TABLES to return schema and table type. Furthermore, sequences will now appear in SHOW TABLES. Any user who relies on SHOW TABLES to return one column can use SELECT table_name FROM [SHOW TABLES] to get compatible behavior between previous versions of CockroachDB. #46756
  • Made small UX improvements to hash sharded index creation. #47017
  • Added CREATE TABLE LIKE specifiers. #47017
  • Added support for the CREATE OR REPLACE VIEW command. #47051
  • We previously allowed mixed type 20.1/19.2 upgrades to add precision types. However, the 19.2 nodes would disrespect the precision component and we documented that. To make this cleaner, we instead only allow precision to be specified in time-related types when the version upgrade is complete. #47425
  • Histogram collection with CREATE STATISTICS is no longer supported on columns with type ARRAY. Only row count, distinct count, and NULL count are collected for array-type columns. #47281
  • A new default_transaction_priority session variable is now supported, which configures the priority that SQL transactions use by default. The accepted options for the setting are LOW, NORMAL, and HIGH. #47641
  • Removed two public cluster settings sql.distsql.temp_storage.joins and sql.distsql.temp_storage.sorts that can disable usage of temporary on disk storage of intermediate results of query execution. #47357
  • Added support for collection of multi-column statistics. By default, statistics are now collected on all prefixes of each index, in addition to 100 other individual columns. This feature can be disabled by setting the cluster setting sql.stats.multi_column_collection.enabled to false. #47729
  • Added square root and cube root unary operators |/ and ||/. #47680
  • We previously supported any TIMESTAMP(TZ) representable in Go. However, this is incompatible with the Postgres protocol format, as well as resulting in sorting errors when using values such as (-9223372036854775808)::TIMESTAMP. We have changed our bounds for timestamps to be the same as Postgres - between 4714-11-24 00:00:00+00 BC and 294276-12-31 23:59:59.999999 inclusive. Users who have previously stored these values will be able to access them, but the WHERE clause must use < '4714-11-24 00:00:00+00 BC' or > 294276-12-31 23:59:59.999999. #47077
  • Users now have the ability to control the level of NOTICE outputs with the session variable client_min_messages. #46650
  • ROLLBACK TO SAVEPOINT is no longer permitted after miscellaneous internal errors. #47724
  • Common table expressions (CTEs) were extended to support the MATERIALIZED/NOT MATERIALIZED syntax added in PostgreSQL 12. #47341
  • The pg_collation, pg_proc, and pg_type tables in pg_catalog no longer require privileges on any database in order for the data to be visible. #47996
  • Added two new session variables and corresponding cluster settings to control whether the optimizer uses histograms and multi-column statistics for cardinality estimation. The session variables are optimizer_use_histograms and optimizer_use_multicol_stats, with corresponding cluster settings sql.defaults.optimizer_use_histograms.enabled and sql.defaults.optimizer_use_multicol_stats.enabled. Both settings are enabled by default. #48105
  • Reduced virtual table memory use. #48066
  • Added virtual schema and populated the pg_catalog.pg_aggregate table. #48126
  • The default vectorized execution mode is now "on", which means that all queries that can use vectorized execution will do so. To revert to previous behavior SET CLUSTER SETTING sql.defaults.vectorize='201auto'. #48059
  • Added support for the ALTER TABLE ... ADD COLUMN ... REFERENCES ... syntax for tables that are empty. #47082
  • Caused primary key changes to not create a copy of the old primary key if the primary key change only changes the bucket count of a hash sharded index. #45894
  • Populated the pg_proc table's provolatile field based on the internal builtin volatility definition. This value used to always be NULL. #48491
  • The RowDescription message of the pgwire protocol now contains the table OID and column ID for each column in the result set. These values correspond to pg_attribute.attrelid and pg_attribute.attnum. If a result column does not refer to a simple table or view, these values will be zero, as they were before. #48417
  • Added support for indexing and ordering of arrays of indexable and orderable inner types. #48045
  • Correctly report type length for the CHAR data type. #48602
  • Changed the statistics collection logic to only include a row in the NULL count for a column statistic if all columns in the statistic are NULL. Previously, the row was included if any of the columns were NULL. This change makes it easier to use multi-column statistics for accurate cardinality estimation in the optimizer. #48528
  • Fixed an issue with optimizing subqueries involving set operations that could prevent queries from executing. #48658
  • The pg_database table in pg_catalog no longer require privileges on any database in order for the data to be visible. #48744
  • Improved the optimizer's ability to inline constant values in filter conditions. #48629
  • Improved compatibility of the pg_typeof builtin function. #48614
  • Introduced the pg_extension virtual schema, which contains tables which are on the public schema when using Postgres style extensions. #48754
  • Renamed the experimental_serial_normalization session variable to serial_normalization since the setting is no longer experimental. #48796
  • The RowDescription message of the pgwire protocol now contains the type modifier for each column in the result set. This corresponds to pg_attribute.atttypmod. If it is not available, the value is -1, as it was before. #48762
  • Added support for the percentile_disc and percentile_cont ordered-set aggregation functions. #47668
  • Added support for Postgres ENUM types to be used in queries and stored in tables. #48556
  • Added the Postgres supported ENUM builtins enum_first, enum_last, and enum_range. #49086
  • The catalog table pg_type is now populated with rows for user defined types (like ENUMs). #48693
  • Populated the catalog table pg_catalog.pg_enum. #49213
  • 24:00 time now displays correctly in the CLI, returning 0001-01-02 00:00:00. Furthermore, backups correctly emit and read in 24:00 time properly. #49330
  • Vectorized execution engine now supports (although not "natively" - meaning we don't have optimized representations for the newly-supported types) all data types that CockroachDB supports. #48422
  • Vectorized execution engine now supports set operation joins (queries with INTERSECT ALL and EXCEPT ALL). #49159
  • Added the Postgres syntax GENERATED ALWAYS alias for computed column construction. #49525
  • Correctly populated the rolcanlogin value for roles in pg_roles and pg_catalog. #49389
  • Added support for the max/min aggregation functions on collated strings. #49530
  • Added support for referencing static data types under the pg_catalog qualification like pg_catalog.int. #49307
  • Added support for the max/min aggregation functions on ENUM types. #49558
  • Added experimental support for the ALTER COLUMN TYPE online schema change for changing a column type's data. #46933
  • Previously, using NULL (or tuples containing _NULL_s) as the left-hand-side of an IN operator would not typecheck unless the _NULL_s were explicitly casted. Now, the casting is not required. #49723
  • The optimizer can now fold two grouping operators together when they are aggregating over functions like sum. #49627
  • The optimizer can now eliminate an unnecessary join that is the input to a GroupBy operator. #49683
  • Added the crdb_internal.create_type_statements virtual table. It holds create statements for user defined types. #49771
  • The vectorized execution engine now supports the "Concat" ("||") operator. #49758

Command-line changes

  • The cockroach demo command now displays a connection URL to the demo cluster, and in a multi-node cluster it displays connection strings for all of the nodes in the cluster. #46913
  • It is now possible to pre-configure the secure mode of cockroach demo using the environment variable COCKROACH_INSECURE like other client commands. #46922
  • The client-side commands specific to cockroach demo, starting with \demo, are now advertised in the output of \?. Note that this feature is currently experimental. #46962
  • The cockroach CLI commands that internally use SQL, including cockroach sql, now can connect to a server using a unix datagram socket. The syntax for this is --url 'postgres://user@?host=/path/to/directory?port=NNNN'. #46968
  • The cockroach demo command now displays a connection URL using a unix datagram socket before the TCP-based URL, as this may deliver better performance in clients running on the same system. #46979
  • The SQL URL generated by cockroach demo, when running in the secure mode, now embeds the password. This way, a command ran by copy-pasting the URL is able to run without asking for a password first. #46979
  • The SQL URL generated by cockroach demo is simplified to no longer require TLS client certificates in particular directory locations. #46979
  • The new client-side command \demo ls (experimental) can now (re-)display the connection parameters for every node in the simulated cluster. #46979
  • Added a new cockroach start option --clock-device that allows CockroachDB's hybrid logical clock (HLC) to use an IEEE 1588 PTP clock device for the HLC current time. #46942
  • The time that cockroach quit waits client-side for the node to drain (i.e., remove existing clients and push range leases away) is now configurable via the command-line flag --drain-wait. Note that separate server-side timeouts also apply separately, check the server.shutdown.* cluster settings for details. #45149
  • It is now possible to drain a node without shutting down the process, using cockroach node drain. This makes it easier to integrate with service managers and orchestration: it now becomes safe to issue cockroach node drain and then separately stop the service via a process manager or orchestrator. Previously, there was a risk to misconfigure the service manager to auto-restart the node after it had shut down via quit, in a way that was surprising or unwanted. The new command node drain also recognizes the new --drain-wait flag. #45149
  • The commands cockroach quit and cockroach node drain now report a "work remaining" metric on their standard error stream. The value reduces until it reaches 0 to indicate that a graceful shutdown has completed server-side. An operator can now rely on cockroach node drain to obtain confidence of a graceful shutdown prior to terminating the server process. #45149
  • The default value of the parameter --drain-wait for cockroach quit has been increased from 1 minute to 10 minutes, to give more time for nodes with thousands of ranges to migrate their leases away. #46396
  • The error message displayed upon cockroach start / cockroach start-single-node when manual intervention is needed in the store directory is now clearer. #47756
  • Client commands such as cockroach init and cockroach quit now support the --cluster-name and --disable-cluster-name-verification flags in order to support running them on clusters that have been configured to use a cluster name. Previously, it was impossible to run such commands against nodes configured with the --cluster-name flag. #48000
  • Added support to cockroach cert list for certificates which require --cert-principal-map to pass validation. #48013
  • cockroach debug zip now properly skips over fully decommissioned nodes. #48073
  • cockroach debug zip now tries multiple times to retrieve data using SQL if it encounters retry errors. #48099
  • Enabled cockroach dump to dump all databases with a single command using the new --all flag. #47387
  • cockroach debug zip now supports two command-line parameters: --nodes and --exclude-nodes. When specified, they control which nodes are inspected when gathering the data. This makes it possible to focus on a group of nodes of interest in a large cluster, or to exclude nodes that debug zip would have trouble reaching otherwise. Both flags accept a list of individual node IDs or ranges of node IDs, e.g. --nodes=1,10,13-15. #48150
  • Added support for the --cert-principal-map flag in the cockroach cert command, and various "client" commands such as cockroach sql, cockroach init, and cockroach quit. #47449
  • CockroachDB's own SQL shell (cockroach sql and/or cockroach demo) no longer prompts for more lines of input after the user enters BEGIN before sending the input to the server. Instead, full lines of input are always sent to the server immediately. The corresponding client-side option smart_prompt thus becomes ineffective and is deprecated. It will be removed in a later version. #48433
  • Made the --storage-engine flag to cockroach start sticky (i.e. resolve to the last used engine type when unspecified), even when specified stores are encrypted at rest. #48721
  • The --decommission flag for cockroach quit is now removed. It was previously deprecated in CockroachDB v20.1. Users should be using cockroach node decommission instead. #49350

DB Console changes

  • Dismissing the Release note signup form is now permanent (per user). For insecure mode, it is now dismissed globally. #46522
  • Added a vertical scroll bar on the Cluster Overview main panel to allow: scroll down to Node Map section, and scroll down Node list table. #46741
  • Changed the default sorting column on the Statements page to Execution Count. #46780
  • Tooltips showing statements and jobs are now limited in size for very long statements. #46557
  • The default timescale on the metrics page is now always 10m. Previously, it defaulted to the age of the longest running node. #46190
  • Added a tracking call to Analytics, sending tracking data for Table sorts. #45920
  • Added Analytics tracking for searches on the Statements page. #45920
  • Added Analytics tracking for pagination on the Statements page. #45920
  • Added a tracking call for diagnostics activation on the Statements page. #45920
  • Changed a statement parameter type back to string from AggregateStatics. #45920
  • Moved the diagnostics activation tracking call to a more accurate place. #45920
  • Fixed a bug where sort columns were only being applied per-page instead of for the entire multi-page list of statements. #46416
  • Added analytics tracking for the statement details sub-navigation. #46789
  • Changed the download link for statement diagnostics to point to the diagnostics bundle zip file. #46790
  • Improved tooltips for existing capacity and storage metrics. #46801
  • Removed the Statements tab from the Databases > Table Details page. #47089
  • Fixed and cleaned up bar charts code on the Statements page. #46574
  • Fixed a bug on the Statements page that would not reset pagination state to page 1 after the app selection was changed. #46972
  • Updated layout sizes. #46627
  • Updated colors according to new designs. #45979
  • Use new SourceSansPro font across all of the DB Console. #45979
  • Updated font styles for monospaced text (SQL statements, etc.). #45979
  • Added links to statement details from the Statement Diagnostics history page. #46923
  • Added tooltips with full length statements on the Statement Diagnostics history page. #46923
  • Updated the Database loading state design. #46857
  • Updated the design of empty state on the Jobs, Diagnostics, Statements, and Statements Details pages. #45981
  • Fixed a bug where "Raft log too large" was reported incorrectly for replicas for which the raft log size is not to be trusted. #48032
  • Fixed a bug where a multi-node cluster without localities defined wouldn't be able to render the Network Latency page. #49138
  • Fixed a bug where the link to specific problem ranges had an incorrect path. Problem ranges are now linked correctly again. #49122
  • Fixed a bug where the metrics timepicker was always defaulting to the "Custom" date pane. Access to time window presets has been restored and is always shown first. The "Custom" pane is accessed by selecting "Custom" in the dropdown as before. #49133
  • Fixed the incorrect display of the table replica count on the Table Details page. #49185
  • Fixed the Jobs page header overlapping with scrollbars. #49552
  • Updated the tooltip design on the Statements page. #46626
  • Removed Capacity and Logical Bytes per Store graph names from storage tooltips to avoid redundancy. #47475
  • The Databases page now shows a default error message about restricted permissions for non-admin users. #49592

Bug fixes

  • Index and table GCs happen closer to their GC deadline. #46715
  • A rare bug causing an assertion failure was fixed. The assertion error message was "caller violated contract: discovered non-conflicting lock". The bug could have theoretically allowed isolation violations between transactions without hitting the assertion, though this was never observed in practice. #46625
  • Fixed an incorrect query result that could occur when a scalar aggregate was called with a NULL input. #46649
  • Benign "Outbox Next" errors from the vectorized runtime are now only logged when log verbosity is set to 1 or greater. #46724
  • Statement diagnostics created through EXPLAIN ANALYZE (DEBUG) now show up in the Statement Diagnostics page of the DB Console. #46759
  • RESTORE will now properly clean up partially imported data after having failed or being cancelled. #46766
  • Fixed a bug where failed or cancelled IMPORTs may not have cleaned up partially imported data. #46727
  • Fixed a rare bug causing errors to be returned for successfully committed transactions. The most common error message was TransactionStatusError: already committed. #46596
  • Fixed a bug for statement diagnostics that led to tooltips always showing, instead of only on hover. #46854
  • The "insecure cluster" indicator is now displayed again for insecure clusters in the DB Console, at the top right of the screen. #46844
  • A rare assertion failure that contained the text "existing lock cannot be acquired by different transaction" was fixed. This assertion was only present in earlier v20.1 releases and not in any earlier releases. This bug created a risk of servers crashing unexpectedly due to improper synchronization of KV requests. #46830
  • Fixed incorrect result with count(*) when grouping on constant columns. #46879
  • Fixed a bug where CockroachDB could hit an internal error when queries with projections only of INT2 and/or INT4 columns were executed via the vectorized engine. #46712
  • cockroach demo now properly cleans up its temporary files if the background license acquisition fails. #46991
  • Fixed a bug where, when using the TIME data type as an index, comparisons using the != operator would return incorrect results if '24:00:00' was located in the index. #46993
  • Fixed a bug where CockroachDB was using an existing table name 100% of the time when creating a new table, which resulted in no tables being created. #47056
  • Fixed a bug where CockroachDB could incorrectly consider a non-NULL value from an interleaved parent table to be NULL when the interleaved child had a NULL value in the row with the corresponding index key. #47035
  • Fixed an error that could incorrectly occur when planning queries with NATURAL and USING joins containing multiple aliases for the same column. #47099
  • CockroachDB now returns a proper error for index creation statements using a column which does not exist. #47090
  • CDC no longer combines with long running transactions to trigger an assertion with the text "lock timestamp regression". This bug could crash a server if the right sequence of events occurred. This was typically rare, but was much more common when CDC was in use. #47101
  • Fixed a bug where in some cases where system tables have changed, incremental, full-cluster BACKUPs with revision history were sometimes incorrectly disallowed. #47092
  • As part of migrating to the new schema change job implementation in 20.1, failed IMPORT and RESTORE jobs which left behind table data in 19.2 that had not been completely garbage collected by the time the cluster was upgraded to 20.1 will now have GC jobs automatically created for them. #47136
  • Fixed a data race on AST nodes for SELECT statements that include a WINDOW clause. It is unclear whether this could have resulted in incorrect results being returned for these queries. #47146
  • Fixed a bug preventing clusters from creating TIMETZ columns before they accept an upgrade to 20.1 without downgrading. #47156
  • Fixed an infinite loop when adding a self-referencing foreign key constraint in the same transaction which creates a table. #47113
  • Fixed a bug where it was possible to drop a column that was referenced in a computed column expression. #47085
  • Fixed a bug where a mechanism to detect stalled disks was erroneously left disabled. It is now enabled: nodes will verify that they can write to their engines before each attempt to announce themselves as live to the cluster. #46975
  • Fixed the behavior of crdb_internal.zones in mixed-version clusters. #47173
  • Fixed reads from system.namespace and crdb_internal.zones in 19.2 nodes in mixed cluster settings. #47173
  • Fixed a bug where restoring a backup from 2.1 to 20.1 with a TIMESTAMP column would result in incomplete type data. This would crash commands such as SHOW COLUMNS FROM <new_table>. #47234
  • Fixed an incompatibility with 19.2 nodes for tables with computed columns. #47269
  • Fixed an internal "no output column equivalent to X" error that could occur in some very rare cases. #47159
  • Fixed a bug that could cause CockroachDB processes to crash due to an assertion failure with the text "expected latches held, found none". The bug became louder (and crashed servers) due to recent changes that added new assertions to the code. #47247
  • Fixed an incorrect error that occurred when executing UNION statements with hidden and non-hidden columns. #47094
  • Fixed a bug where in some cases limits were applied incorrectly when pushed down into scans (resulting in some queries returning more results than they should). #47287
  • Fixed a bug in the new schema change GC job implementation causing unnecessary table descriptor lookups whenever any table was updated. #47313
  • Fixed a critical bug that would otherwise cause "invalid truncation decision" panics. #47143
  • Fixed a bug preventing NULL index members from being added to hash sharded indexes. #47311
  • Fixed a backwards incompatibility between RocksDB and Pebble that prevented RocksDB from opening a Pebble-created WAL file under certain conditions. #47350
  • Fixed a mishandling of truncated WAL records in Pebble which could prevent Pebble from opening a DB after a crash. #47350
  • Fixed a short (3s) period of unavailability that could occur on a range after removing a replica. #41122
  • Fixed an error that occurred when statistics collection was explicitly requested on a column with type ARRAY. #47281
  • Avoid a condition during rapid version upgrades where a node would refuse to start, claiming "a store is too old for running version". Before this bug fix, the workaround was to decommission the node, delete the store directory, and re-add it to the cluster as a new node. #47358
  • TRUNCATE can now run on temporary tables, fixing a bug in 20.1.0 where temporary tables could not be truncated, resulting in an error unexpected value: nil. #47482
  • Fixed a bug where instead of returning a parsing error in queries with count(*), CockroachDB could incorrectly return no output (when the query was executed via the row-by-row engine). #47036
  • Fixed a bug that could cause a workload to stall under heavy load. This stall was due to a deadlock that was introduced in an earlier v20.1 release. #47465
  • Fixed a bug where columns which were members of hash sharded indexes could not be renamed. Indexes created in prior releases will need to be dropped and recreated to resolve this limitation. #47431
  • Fixed a bug that could could trigger an assertion with the text "received X results, limit was Y". The underlying bug was only performance related and could not cause user-visible correctness violations. The bug could result in an assertion failure and a node crashing. Even though this was an old bug (present in many releases before v20.1), it became a lot easier to hit in v20.1 because we started performing ranged intent resolution more often due to implicit SELECT FOR UPDATE. #47492
  • Fixed incorrect results that could occur when casting negative INTERVALs or TIMESTAMPs to type DECIMAL. #47483
  • Fixed a bug introduced with the new schema change job implementation that caused errors when rolling back a schema change to be swallowed. #47446
  • Fixed a panic that could occur when SHOW RANGES or SHOW RANGE FOR ROW was called with a virtual table. #47500
  • Setting kv.closed_timestamp.target_duration to 0 now disables the closed timestamp as was previously documented. #47480
  • Fixed a bug where CockroachDB could not dump tables that have no visible columns. #46406
  • Fixed a bug causing some schema change rollbacks to fail permanently even on transient errors. #47553
  • Fixed a bug introduced in 20.1 that could cause multiple index GC jobs to be created for the same schema change in rare cases. #46929
  • Fixed an issue in RocksDB that could result in missing data in some very specific instances that are very unlikely to be encountered in practice. #47448
  • Fixed an incompatibility between Pebble and RocksDB bloom filters that could result in keys disappearing or reappearing when switching storage engines. #47612
  • Fixed a bug where concurrent schema change interactions could lead to schema changes being blocked for minutes. #47462
  • Fixed a bug where PARTITION BY and ORDER BY columns in window specifications were losing qualifications when used inside views. #47709
  • Removed redundant metadata information for subqueries and postqueries in EXPLAIN (VERBOSE) output. #47951
  • Return a proper Postgres error code from DROP INDEX statements when dropping UNIQUE indices. #47585
  • Fix a rare bug where stats were not automatically generated for a new table. #47718
  • Fixed a planning error that could happen in rare cases when a histogram was used for a descending indexed column. #48033
  • Fixed a bug where CockroachDB could return an internal error when performing a query with CASE, AND, OR operators in some cases when it was executed via the vectorized engine. #47938
  • cockroach debug zip can now successfully avoid out-of-memory errors when extracting very large system or crdb_internal tables. #48096
  • Return a proper SQLSTATE error for index creation statements using an existing index name. #48035
  • cockroach debug zip will now properly report an error if some error is encountered while writing the end of the output zip file. #48109
  • Fix cockroach debug encryption-status and the DB Console display of encryption status when using Pebble. #48151
  • Fixed "non-values node passed as fixed value to zigzag join" internal error. #48128
  • Return a proper CockroachDB-specific Postgres error code for trying to modify a table without a primary key instead of the generic XXUUU error code. #48125
  • Fixed a bug where a read operation in a transaction would error out for exceeding the maximum count of results returned. #48160
  • Fixed a bug in which (tuple).* was only expanded to the first column in the tuple and the remaining elements were dropped. #48225
  • CockroachDB will now avoid producing a severe "internal error" upon certain privilege check failures via pg_catalog built-in functions. #48216
  • Fixed a bug where more than 1 job was created from a DROP COLUMN statement, which was confusing clients. Now, we better distinguish between the delete jobs for columns and dependent jobs for deleting indices, views and sequences. #48163
  • Fixed a bug where vectorized queries on composite data types could sometimes return invalid data. #48052
  • Fixed a rare bug causing a range to deadlock and all the writes to the respective range to timeout. #48082
  • Fixed a bug that could lead to data corruption or data loss if a replica was both the source of a snapshot and was being concurrently removed from the range, and certain specific conditions existed inside RocksDB. This scenario is rare, but possible. #48320
  • Made SRV resolution non-fatal for join list records, to align with the standard and improve reliability of node startup. #47735
  • Fixed a long-standing bug where HTTP requests would start to fail with error 503 "transport: authentication handshake failed: io: read/write on closed pipe" and never become possible again until the node is restarted. This bug existed since v2.1 or some time prior. #48369
  • When processing the --join flag to cockroach start, invalid SRV records with port number 0 are now properly ignored. #48325
  • Fixed a bug where SHOW STATISTICS USING JSON contained incorrect single quotes for strings with spaces inside histograms. #48514
  • Return a proper SQLSTATE error when trying to ADDor RENAME a column with an existing name. #48120
  • Prevent dropping of databases which contain tables which are currently offline due to IMPORT or RESTORE. Previously, dropping a database in this state could lead to a corrupted schema which prevented running backups. #48596
  • The two settings kv.range_split.by_load_enabled and kv.range_split.load_qps_threshold are documented but were incorrectly marked as non-public in the output of SHOW CLUSTER SETTINGS. This has been corrected. This bug was introduced in v20.1. #48582
  • Fixed a bug preventing timestamps from being closed, which could result in failed follower reads or failure to observe resolved timestamps in changefeeds. #48561
  • Fixed a bug which falsely indicated that kv.closed_timestamp.max_behind_nanos was almost always growing. #48521
  • Previously, when an IMPORT failed, it created a GC job which would behave as a no-op. Now the partially imported data after an IMPORT fails or is cancelled should be deleted. #48593
  • Fixed a bug where CockroachDB could encounter an internal error when a query with LEFT SEMI or LEFT ANTI join was performed via the vectorized execution engine in some cases. This was likely to occur only with the vectorize=on setting. #48659
  • Fixed a bug where the SHOW CREATE statement would sometimes show a partitioning step for an index that had been dropped. #48624
  • Fixed a bug where cockroach dump on tables with interleaved primary keys would erroneously include an extra CREATE UNIQUE INDEX "primary" ... INTERLEAVE IN PARENT statement in the dump output. This made it impossible to re-import the dumped data without manual editing. #48688
  • Re-allow diagnostics.forced_sql_stat_reset.interval, diagnostics.sql_stat_reset.interval and external.graphite.interval to set to their maximum values (24hr, 24hr and 15min respectively). This previously only excluded these values to be allowed. #48758
  • Fixed a bug where changing the primary key of a table that had partitioned indexes could cause indexes to lose their zone configurations. In particular, the indexes that got rebuilt as part of a primary key change would keep their partitions, but lose the zone configurations attached to those partitions. #48510
  • Fixed a bug where cockroach dump on a table with collated strings would omit the collation clause for the data insertion statements. #48687
  • Manually writing a NULL value into the system.users table for the "hashedPassword" column will no longer cause a server crash during user authentication. #48773
  • Fixed a bug introduced in v20.1.0 where the migration for ongoing schema change jobs would cause the node to panic with an index out of bounds error upon encountering a malformed table descriptor with no schema change mutation corresponding to the job to be migrated. #48823
  • Properly support restoring tables that were backed up while they were in the middle of a schema change. #47588
  • Fixed a bug where columns of a table could not be dropped after a primary key change. #49081
  • Fixed an error that could occur when using NULL in some array indirections. #48830
  • Fixed incorrect logic for IS NULL and IS NOT NULL operators with tuples, correctly differentiating them from IS NOT DISTINCT FROM NULL and IS DISTINCT FROM NULL, respectively. #48299
  • Fixed costing of lookup join with a limit on top, resulting in better plans (that utilize lookup join) in some cases. #48862
  • SHOW BACKUP SCHEMAS no longer shows table comments, as they may be inaccurate. #48167
  • Fixed a bug where when a database was dropped, it would not drop the entry for its public schema in the system.namespace table. #49125
  • Fixed a possible deadlock during vectorized query execution. #49106
  • Fixed a memory leak which could affect CHANGEFEEDs performing scans of large tables. #49101
  • Improved accuracy of column types returned from queries to improve PostgreSQL compatibility. #48619
  • Fixed a bug where use of arrays combined with window functions was causing a query failure. #49234
  • Fixed a small race in the vectorized execution engine. #49202
  • Fixed a bug where contended DROP INDEX queries returned an assertion failure error rather than a retryable error. #49221
  • Some benign errors were previously reported as unexpected internal errors by the vectorized execution engine; this is now fixed. #49260
  • Prevent namespace orphans (manifesting as database "" not found errors) when migrating from 19.2. #49099
  • Removed duplicate SQL memory metrics. #47750
  • When the value passed to --drain-wait is very small, but non-zero, cockroach quit in certain cases would not proceed to perform a hard shutdown. This has been corrected. This bug existed since v19.1.9, v19.2.7 and v20.1.1. #49362
  • Fixed a bug where in some cases an internal error could occur when queries that have columns of BYTES type in the output were executed via the vectorized engine. #49223
  • CockroachDB now correctly reports engine type in bug reports when using cockroach demo. #49372
  • Fixed a bug where \demo node restart would not work due to an invalid certificate directory. #49382
  • Abandoned intents due to failed transaction coordinators are now cleaned up much faster. This resolves a regression in v20.1.0 compared to prior releases. #49218
  • Fixed a bug where if one attempted to perform a full cluster RESTORE on a backup that did not contain any user data, it would fail. #49591
  • CockroachDB no longer leaks file descriptors during GSS authentication. #49572
  • Fixed a bug where CockroachDB could return an internal error or incorrect results on queries when they were run via the vectorized execution engine and had a hash router in the DistSQL plan. This could only occur with the vectorize=on setting. #49333
  • Fixed a bug where executing a large number of statements in a transaction without committing could previously crash a CockroachDB server. #48859
  • The schema change workload is meant for testing the behavior of schema changes on clusters with nodes with minimum version 19.2. It will deadlock on earlier versions. #49662
  • Fixed a panic observed as "unexpected arg type tree.DOidWrapper". #49601
  • Fixed a bug where if ctx.err() is nil, the CLI will panic when a workload returns an error. #48338

Performance improvements

  • Some queries which contain a division by a constant have improved execution plans. #46809
  • The query planner can now decorrelate correlated exists subqueries with LIMIT expressions, leading to better query plans. #46540
  • Make virtual tables that access all table descriptors make fewer round trips. #46872
  • Reduce time needed to run a BACKUP command when it is built on a lot of previous incremental backups. #47158
  • The query optimizer now produces faster query plans for some disjunctions (OR expressions) by utilizing multiple indexes. #47094
  • The query optimizer is now more selective in splitting disjunctions into unions, avoiding the overhead of unnecessary memo expansion. #47161
  • Improve performance of UPDATE on tables with inverted indexed columns. #47354
  • Query optimization for disjunctions was extended to support queries which do not select strict keys. #47342
  • The query optimizer is now smarter when splitting disjunctions into unions by grouping disjunctions based on the columns referenced. #47361
  • The query optimizer now uses a more efficient plan when splitting disjunctions into multiple sub-queries. #47454
  • Query optimization for disjunctions was extended to support scans over inverted indexes. #47542
  • Histograms are now used by the optimizer to estimate the cost of index scans with multiple constrained columns or descending columns. This enables better query plan selection in some cases. #47565
  • Improved execution plans by removing unnecessary remaining filters in some cases where the filters are reflected by an index constraint. #47623
  • The cleanup job which runs after a primary key change to remove old indexes, which blocks other schema changes from running, now starts immediately after the primary key swap is complete. This reduces the amount of waiting time before subsequent schema changes can run. #47624
  • The query optimizer now uses a more efficient plan when splitting disjunctions into multiple sub-queries. #47617
  • The query optimizer is now more efficient when generating plans for queries with many OR expressions. #48025
  • Filtered scans over virtual tables have improved performance in common cases. #47316
  • SHOW CREATE is much more efficient. #47316
  • The query optimizer can now split disjunctions in more complex expressions into a union over two index scans. #48036
  • Disabled the Go runtime block profile by default, resulting in a small but measurable reduction in CPU usage. The block profile has diminished in utility with the advent of mutex contention profiles and is almost never used during performance investigations. #48133
  • The optimizer can now detect when an EXISTS subquery can be eliminated because the input has zero rows. This leads to better plans in some cases. #48162
  • Before this change, a simple schema change could take 30s+. The reason was that if the schema change is not first in line in the table mutation queue it would return a retryable error and the jobs framework would re-adopt and run it later. The problem is that the job adoption loop timer is 30s. Instead of returning on retryable errors, we retry with an exponential backoff in the schema change code. This pattern of dealing with retryable errors in client job code is encouraged vs. relying on the registry, because the latter leads to slowness and additionally to more complicated test fixtures that rely on hacking with the internals of the job registry. #48608
  • Fixed a bug in the histogram filtering logic in the optimizer which was causing inaccurate cardinality estimates for queries with equality predicates on UUIDs and strings, as well as some other types. This bug has existed since histograms were first introduced into the optimizer in the 19.2.0 release. Fixing it improves the optimizer's cardinality estimates and results in better query plans in some cases. #48580
  • Histograms used by the optimizer for query planning now have more accurate row counts per histogram bucket, particularly for columns that have many NULL values. This results in better plans in some cases. #48528
  • DROP TABLE performance is improved due to doing fewer roundtrips for object comment maintenance. #48265
  • The optimizer and execution engine can now plan lookup joins into virtual indexes, avoiding full scans against virtual tables. #48226
  • Lookup joins with no required ordering are now more efficient. #48439
  • A performance inefficiency has been fixed in the vectorized execution engine which results in speed ups on all queries when run via the vectorized engine, with most noticeable gains on the queries that output many rows. #48669
  • The optimizer now normalizes a comparison operator (=, >=, <=, <, >) with a nested "timezone" function applied to a variable. This results in the generation of constrained index scans in more cases, and therefore, better query plans. #48724
  • Reduce memory allocations and improve performance of low-level KV scan operations on Pebble. #49121
  • COUNT is now converted to COUNT(*) (aka COUNT_ROWS) in more cases. #49126
  • Lookup join performance has been improved. #48058
  • The optimizer can now determine more accurate costs for query plans with a combination of disjunctions and contradictions. #49462
  • Default replication configurations have been tuned to support higher replication throughput in high-latency replication quorums. #49619
  • Fixed a bug in the histogram logic in the optimizer which was causing an over-estimate for the cardinality of constrained index scans in some cases when multiple columns of the index were constrained. This problem was introduced early in the development for the 20.2 release, so should not have ever been part of a release. The fix improves the optimizer's cardinality estimates and may result in better query plan selection. #49134
  • Fixed the optimizer's distinct count estimate for columns constrained by an index constraint, which was too low in some cases. The fix improves the optimizer's cardinality estimates, which can lead to better query plan selection. #49134
  • Fixed the optimizer's estimated distinct count for a multi-column statistic when all of the columns in the statistic are constrained by a filter predicate. The fix can lead to improved cardinality estimates, leading to better query plan selection in some cases. #49134
  • Added support for calculating the selectivity of filter predicates in the optimizer using multi-column statistics. This improves the cardinality estimates of the optimizer when a query has filter predicates constraining multiple columns. As a result, the optimizer may choose a better query plan in some cases. #49134
  • Remove compaction-debt based SSTable ingestion backpressure which was artificially slowing down IMPORTs and RESTOREs on Pebble and not providing any utility on RocksDB. Removed the private rocksdb.ingest_backpressure.pending_compaction_threshold cluster setting. #49726

Build changes

  • It's now possible to build CockroachDB with the Clang++ v10 compiler. #46859
  • Release Docker images are now built on Debian 9.12. #49593

Contributors

This release includes 899 merged PRs by 68 authors. We would like to thank the following contributors from the CockroachDB community:

  • Amit Sadaphule
  • Andrii Vorobiov
  • Anthony Huang (first-time contributor)
  • Artem Barger
  • BurtonQin (first-time contributor)
  • Girish Ramnani (first-time contributor)
  • Helen He (first-time contributor)
  • Jaewan Park
  • Jaime Soriano Pastor (first-time contributor)
  • Petr Jediný (first-time contributor)
  • Richard Tweed (first-time contributor)
  • Yongyang Lai (first-time contributor)
  • abhishek20123g
  • lancerutkin (first-time contributor)
YesYes NoNo

Product

Resources

Learn

Support Channels

Company

Get developer news