Commit Graph

1760 Commits

Author SHA1 Message Date
drh 3a799b3e4e Fix to the previous checkin: The colUsed parameter should have high-order bit
set if any of the 64-th or greater columns of the virtual table is used.  The
lower 63 bits of colUsed always show the usage of the first 63 columns of the
virtual table.

FossilOrigin-Name: 90ac8457750ace1d76d7bc957af7877e86e6301bb07361e19beaf5c7b3b6085a
2024-08-03 23:00:17 +00:00
dan 0a83dac31c Ensure the sqlite3_index_info.colUsed mask always includes the PK fields of WITHOUT ROWID vtabs in cases where they may be used.
FossilOrigin-Name: c327c0c02cfefdba373cfb15933a9cdfddb578b6582f2ce7c08929203743ffe9
2024-08-03 18:45:48 +00:00
drh c855df67a4 Improved debugging output when using PRAGMA vdbe_addoptrace:
Show the range of opcodes being checked for Column-to-Copy conversion.

FossilOrigin-Name: 2c122026db979a099a3d1a14ae1e01a0a64a559905ea414b2d80ed7dc3e04846
2024-07-21 23:10:55 +00:00
drh 9175b18226 Do not omit the ORDER BY clause from a recursive CTE just because the
query that contains the CTE also contains an ORDER BY clause.  Plus
other changes imported from the recursive-cte-orderby-fix branch.

FossilOrigin-Name: 13242289c5d412b706f50fc7e1553032ea3a52d41a3e34e155432adaf0551481
2024-06-14 23:13:54 +00:00
drh 2fbb3fb532 Disable the omit-noop-join optimization when there are 64 or more terms in
the ORDER BY clause.

FossilOrigin-Name: 40de3939792e17df25598b3e60d1cebcecde2b00832acd55604f14b21398a9a7
2024-06-09 17:34:03 +00:00
drh 4703b7d092 Better optimize queries that use parameters in the LIMIT clause.
FossilOrigin-Name: e58cb304d1e0ec6e30260a165aaccc2cf096ce6c999efb06683c4ef600ee12ef
2024-06-06 15:03:16 +00:00
dan 7d24afa0de Better handle WHERE terms that are common to two or more OR branches when planning virtual table queries.
FossilOrigin-Name: 1976c3f7e1fe77cf3367710e8ada230a3672ed374e316425164e42b2622526c7
2024-06-04 17:26:15 +00:00
drh 0507e6e388 Be more aggressive about freeing memory associated with the
sqlite3_index_info.idxStr field.

FossilOrigin-Name: 85dcd0a8479a658203833cfd75f22813faa26d4793ebfbb8843035d683bee105
2024-06-02 10:52:35 +00:00
dan cadfdd4e09 Better handle WHERE terms that are common to two or more OR branches when planning virtual table queries.
FossilOrigin-Name: 4edd9b29f58621335b8a562280c991c34804bbba090f90c951261d043cff1965
2024-05-31 19:26:22 +00:00
drh 5ef9c89977 Do not even try star-schema detection if the join has fewer than 5 tables,
since 5 is the minimum for a positive detection.

FossilOrigin-Name: a07ec16bbc056fbc23a7bd58e5e32ef691c13e9babeb542918cf9a01cac40c20
2024-05-31 15:39:00 +00:00
drh d72ca8f810 Document the OP_Explain opcode. Add the WhereLoop.rRun value as P3 in
OP_Explain opcodes associated with WhereLoops, for testing purposes.

FossilOrigin-Name: 996c46e61d9a53a54018672dd407b8ba8c480dd6795393428f9d5fcb81b47ab5
2024-05-31 14:39:42 +00:00
drh 1344b540d9 Ensure that the second pass call to wherePathSolver() always reports a positive
value for the estimated number of output rows.  This fixes a long-standing
(though exceedingly minor) problem that was only revealed by a new assert()
added yesterday.  dbsqlfuzz 545d6debc2da496ab2b915e33ea34d9a35a935df

FossilOrigin-Name: a47c644fef71f3ab3dc584ea917eaab9a8e5b4c9dcb57bdd29747ba32108e85f
2024-05-30 13:27:29 +00:00
drh 1085eee098 Change an assert() (incorrectly) added by the previous check-in into a
testcase().

FossilOrigin-Name: 5e64b541d175c10a5df67ef3419ecd74fcb51392af5f5458973cea85ee6a3246
2024-05-29 14:32:22 +00:00
drh eccb42eb0e Small performance enhancement in wherePathSolver()
FossilOrigin-Name: bdab69641df4704f7b080cf7841bbf39da9160222f7b664a79ed662b70ca7b0a
2024-05-29 14:04:52 +00:00
drh bb24c75b41 Improvements to comments and debugging output.
FossilOrigin-Name: 85164ee155e42809fe34e6c6b6fbe0a2baa9d616326a811173a0b0c8a885fcce
2024-05-29 13:29:39 +00:00
drh 54bfc9435f Remember the amount of the heuristic cost adjustment associated with
star schemas and compensate when computing whether or not to use Bloom
filters.

FossilOrigin-Name: 21daf2463ef70e6b5dc73ba5bf62b3d2fb504e9189f645ac74b513d3d8b979c2
2024-05-29 10:40:53 +00:00
drh c34ff12c59 Use a heuristic in the query planner to help it better cope with large
star schema queries.

FossilOrigin-Name: a98be0f548f277fab8f38a2dec6ddcbe7a7fff27856ba19e76ad8c5641894b7b
2024-05-29 00:38:12 +00:00
drh 05665e9de9 Tweaks to the query planner so that it is better able to deal with star
queries with many dimensions and bad indexes.

FossilOrigin-Name: 1e49081001f93b75ee499536f8a61a0f49225a1745117bb9267249c38c4bf803
2024-05-28 18:53:36 +00:00
drh efe474af97 Increase the number of parallel paths in the query solver from 12 to 20.
In the .wheretrace output, sort the parallel paths in order of increasing
cost.

FossilOrigin-Name: 8ba2c2f5cb31f7bcc426bec78457316ef11d0b5debf24e8da8c6fc2f95878b1e
2024-05-28 12:41:11 +00:00
drh 18970dac18 Improve the automatic-index decision such that no attempt is made to create
an automatic index on a column that is known to be non-selective because if
its use in other indexes that have been analyzed.  See
[src:/tktview/8ff324e120|ticket 8ff324e120] and
[forum:/forumpost/b21c2101a559be0a|forum post b21c2101a559be0a].

FossilOrigin-Name: 2a7603c327d11d5e6bc4d40c29151fea544d650f51200ff76efb921adb0c38df
2024-05-28 00:16:52 +00:00
drh 38af125238 Query planner tuning: Increase the maximum number of simultaneous solutions
to track int eh solver from 10 to 12.

FossilOrigin-Name: fe2e1dadbacbe6392ceed44fd287a2ed82189cb8055f35631d37967d9a7a5d1d
2024-05-28 00:01:07 +00:00
drh b23f92b286 Avoid creating an automatic index on a column that is known to be not very
selective.

FossilOrigin-Name: 70409763f70faa2a0f4f072fd318a687b109a0905cc57906ad7f80d2885fe6d9
2024-05-27 21:55:01 +00:00
drh 294cd87afc Merge various enhancements that were being held on a branch until after the
3.46.0 release.  (1) Allow arbitrary expressions in the second argument to
the RAISE() function. (2) Performance optimization to the OP_OpenRead and
OP_OpenWrite bytecode operators. (3) Allow FTS5 tables to be dropped even if
the associated tokenizer is not available.  (4) Performance optimizations in
FTS5.  (5) Generate better bytecode to improve performance of ONEROW scans.

FossilOrigin-Name: 1e5385ffc30743412ade1536583504eec36790cac5355042aa1e12d8d2271d1f
2024-05-23 15:05:57 +00:00
drh 0462499051 Fix the definition of sqlite3_vtab_distinct() such that return codes 2 and
3 mean that all rows must be distinct over "colUsed" which is a superset of
"aOrderBy".  Also, disallow return codes 2 and 3 if the rowid of the virtual
table is accessed.

FossilOrigin-Name: 922731ce98c0ce7837784ff7966049e59fa73da2aa04abf3506503b6fc4aa048
2024-05-18 20:00:08 +00:00
dan 0445cd1020 Fix a problem in where.c preventing SQLite from identifying some ONEROW scans, leading to (slightly) sub-optimal byte-code.
FossilOrigin-Name: b41e71208abcdf988f57c3fd7e6a372bbaf442c32a8e2c051beb8d45c8001e0f
2024-05-17 13:56:38 +00:00
drh 2eb9adb564 Assume that a function is able to return a subtype if either (1) it is itself
marked with SQLITE_RESULT_SUBTYPE, or (2) one of its arguments is a function
that is able to return a subtype.  This check-in backs out the code changes
from the previous two on this same branch, but keeps the test cases from
the previous two.

FossilOrigin-Name: f16b200f25a0ec59ad765d254d81c3ffdba21f79e6e82807a7b80d00627952e2
2024-05-04 15:04:45 +00:00
dan 297472a2fc Have where.c ignore any plan from a virtual table that tries to use LIMIT/OFFSET without also using all WHERE constraints.
FossilOrigin-Name: 7d30596496c6a7a37b925f13d8d94d5de224ec31bb86594fa4cc07b10082e776
2024-04-26 17:19:59 +00:00
drh 14602eafa7 Fix a problem with vector IN operators used with an index where the affinities
and collations for the various vector terms are different.

FossilOrigin-Name: 86e8c782e7494377de3c27b750cd83a7eb2302c1182ee2004d3864db50fca557
2024-04-23 12:02:03 +00:00
dan 0309fb919a Fix an uninitialized variable problem in the new code on this branch.
FossilOrigin-Name: 85625b38718c6e4cf7aa97fa3c52153bd25c810b68f19e7924e1f4b709c6a05d
2024-04-23 11:56:20 +00:00
drh 4f2f6c74ca Extra robustness in the code that causes cursors to return NULL when they
are participating in an OUTER JOIN.

FossilOrigin-Name: 672c2869ef48e08447d37b0d76a1850cdafbe30ca1906ec98c55e3ab496fd9a6
2024-04-22 13:31:24 +00:00
dan 5749e1ee43 Slight performance improvement for the new code on this branch.
FossilOrigin-Name: 500c67f1341fe2a7e7333d525c90df201cc73a683b943ad5c1e41d4a4f639043
2024-04-20 19:19:30 +00:00
dan d737feeacf Fix a problem where an expression like (a, b) IN (SELECT ...) might not use an index on (a, b) if the affinities and collation sequences of "a" and "b" are not identical.
FossilOrigin-Name: 4d870fd8b5450047a7486fc023d1ac9439642e8ed91eadfd5026c4cda7cc9179
2024-04-20 15:14:06 +00:00
drh 0526db32e2 Correct handling of OUTER JOIN when on or the other operand is a subquery
implemented using the VALUES-as-coroutine optimization.
dbsqlfuzz bde3bf80aedf25afa56e2997a0545a314765d3f8.

FossilOrigin-Name: 8c0f69e0e4ae0a446838cc193bfd4395fd251f3c7659b35ac388e5a0a7650a66
2024-04-18 16:11:01 +00:00
drh 6951c49779 Do not allow changes to sqlite3ExprIsTableConstant() that support pushdown of
subqueries interfere with the hash-join logic.

FossilOrigin-Name: 8682931f9c6b40e1b09139c10bbe932c38330b5eb0c5c84f2432ad19a6793e37
2024-04-06 18:30:09 +00:00
drh b411c4d69e Check-in [a9657c87c53c1922] is wrong: the IndexedExpr.bMaybeNullRow flag is
required for virtual columns if they are part of an outer join.  Add a
test case (derived from dbsqlfuzz b9e65e2f110df998f1306571fae7af6c01e4d92b)
to prove it.

FossilOrigin-Name: 4484ec6d26b31305e31de89bdbae26344d8083a7e7de20861430d31737d9979c
2024-04-05 13:56:05 +00:00
drh 8ce73ce15b Fix typos in comments. Provided ".wheretrace" debugging output for the
interstage heuristic module.  Do omit automatic index loops in the
interstage heuristic.

FossilOrigin-Name: 186dcae19e249db36de15f295999cff25063b54ee3d5d481cd2ba99b6d13148e
2024-04-02 14:12:29 +00:00
drh 7f7d0b19f8 Add a heuristic in between the two solver() passes of the query planner that
tries to prevent a very slow query plan in cases where the output row count
estimate is imprecise.

FossilOrigin-Name: 8018417b0143ea11535f2457bf3e4b3755717c554a17df1076425b4251b5f2c6
2024-04-02 11:44:44 +00:00
drh 9352cfa30b Improved comments in the query planner logic that computes the cost for a
particular step in a query plan.  No code changes.

FossilOrigin-Name: 0b2ac2cdc767db764e3ea8bbc33898cac4e1ec27fe8c9b60ce08a1785f921e6d
2024-04-01 15:38:15 +00:00
drh ed6c9aa6ab Fix an inaccuracy in automatic indexes that was exposed by the fix
at [80c4223098c1827f].

FossilOrigin-Name: fdc9406f1c8ba4a7341c1e408f6042ddc788cf65f98e1de2ee101390bfb0abae
2024-03-25 14:44:10 +00:00
drh 4a402b80c5 Improved "wheretrace" debugging output for calls to xBestIndex. No changes
to release code.

FossilOrigin-Name: 33966bb06ce9a8e90734de59a8d43c523f60870d20be64789638da4c84a8b81b
2024-03-24 20:26:59 +00:00
drh 27a5ee855d Improvements to EXPLAIN QUERY PLAN output for multi-row VALUES claues.
FossilOrigin-Name: ac6f095e13e43d66c06552c8b01f6bec3407c9d41a34c4cdb0be57b0b828ad0d
2024-03-18 12:49:30 +00:00
drh f696591dd4 Enhance the sqlite3ExprIsConstant() function so that it recognizes
constant functions.  So far the enhancement only applies to the multi-row
VALUES clause, but it could possibly be applied in many other places.

FossilOrigin-Name: c9e0488c6c0135932b6e76b0f3f3acd69ef65327e0a54daa59777f35da1aef26
2024-03-16 13:18:48 +00:00
dan 815e055bff Attempt to reduce the memory used by VALUES clauses in as many statements as possible, not just INSERT. This branch still has problems.
FossilOrigin-Name: 17d1f7cfabc7593d0725051b0c7c9619a23a482265f30f15ab9493fef5caeeb0
2024-03-11 17:27:19 +00:00
dan 626d619218 Consider using "=" and IS operators with even low-quality indexes in cases where they are selected explicitly using an INDEXED BY clause.
FossilOrigin-Name: 43cbbea82132db2d0ddb4f34cc2b6910b3a1243ae6d4e837b1b27bfe91b84834
2024-02-20 16:04:27 +00:00
drh 9d00aba895 Rename the TF_StatsUsed flag to TF_MaybeReanalyze, to more accurately
reflect its usage.  No logic changes.

FossilOrigin-Name: b4790da5e0d6f0f617e6c3a17fc2606842e41cab14339392abd335410fec0aa3
2024-02-16 12:57:04 +00:00
drh ed0a614c21 Replace an conditional assignment that was made obsolete by [d4c193f0b49f4950]
with an assert().  The conditional was added by [d6fd512f50513ab7] as
a fix for tickets [c36cdb4afd504dc1], [4051a7f931d9ba24], and
[d6fd512f50513ab7] which means now [d4c193f0b49f4950] is the correct fix
for those tickets.
that check-in

FossilOrigin-Name: 44b5524d522e749ad6bf76c94d754ff16c309c32439ec46802924663f64e8b09
2024-01-31 20:11:54 +00:00
dan 581bf002ce Consider an index for queries like "SELECT count(DISTINCT col) FROM ...", even if the index records are not smaller than the table records.
FossilOrigin-Name: ad06868807a27f0c96fa5649ebd981e07909eb2c5e03781377c876fdec49abdc
2024-01-29 19:30:16 +00:00
drh 5b480ead99 Automatically disable the DISTINCT optimization during query planning if the
ORDER BY clause exceeds 63 terms.
dbsqlfuzz afd1d61fc27d14938a0d78a50970fa1e9fbfee58.

FossilOrigin-Name: d4c193f0b49f4950b20c2f0e6aa037d2ed7d8c0b4687c14923b3a0d0d4a1b3fd
2024-01-28 17:39:31 +00:00
drh 4ccf707c6e Extra steps taken to avoid using low-quality indexes in a query plan.
This branch accomplishes the same end as the nearby enhanced-stat1 branch,
but with much less change and hence less risk.

FossilOrigin-Name: c030e646262fee43a59b45fdc1630d972f8bf88ac3c142b6bdaf4cbb36695a4f
2024-01-01 17:48:02 +00:00
drh 5a85cf5879 Fix harmless compiler warnings associated with [5db30bcc338aac1c]
FossilOrigin-Name: e55d1c2333f35fc20615aa83a7843d08cae7945710a2156d44eee0cc37d90ade
2023-12-24 11:43:49 +00:00