Compare commits

...

566 Commits

Author SHA1 Message Date
drh 0cb576f306 Fix harmless compiler warnings about unused variable assignments.
FossilOrigin-Name: afe7e9fd1c476c7cdcefab0adf0191a8dc3283811f72a685fff3eb6539b72dcc
2026-03-29 22:41:12 +00:00
drh 6d27844bbe Merge trunk performance enhancements and fixes into the direct-printf branch.
FossilOrigin-Name: 9fa9cd11f8f5f7591168870c6e41121221b58a4a3094e708e3fa5bab1502f190
2026-03-29 19:23:47 +00:00
drh 03f13d77a5 Avoid using unsigned 64-bit integer division on platforms that do not support
it in hardware.

FossilOrigin-Name: 2197677491dfc5ec87b57bbf807776875248a250f80ce9a5ce94ae385bb1d2bc
2026-03-29 19:06:36 +00:00
drh 445fbc61c8 The _umulh() intrinsic function is not available on 32-bit windows. So don't
use it there.

FossilOrigin-Name: e75f30c7e66e725a2f3f46c7d4d69a2ff925e2de9a86c3a6ec38897545b43a58
2026-03-29 10:32:25 +00:00
drh eb0cb3f53b In sqlite3_str_vappendf(), write floating-point renderings directly into
the output buffer, saving a memcpy().

FossilOrigin-Name: e4e040e7aaaf5bfccbf0414fee3d08c5fdc24e03d1118dce92d2dd16c614ce63
2026-03-28 20:24:47 +00:00
drh 5239e69fb1 Remove a harmless unused variable from the fp-speed-2.c benchmark.
FossilOrigin-Name: ff9008e7615d877e6cc852c60097777006b274971646d368eaa56b99a7acf6c7
2026-03-28 14:21:53 +00:00
drh 20d93e16d7 Enhancements to the fp-speed-N.c programs, suggested by Andreas Kupries.
FossilOrigin-Name: 7d81ff4c509d4039ef9e2f1066284ec560131422db0cbffa7e1826d71bb8f8e1
2026-03-28 14:17:53 +00:00
drh c409eaaf14 Reduce the compiled size of the quote() SQL function.
FossilOrigin-Name: 9ea980a35628f7d0741ab6c65840ab9dab8abb576c6c60109147336452f9637a
2026-03-28 13:59:34 +00:00
drh 2e01416762 Turn an sqlite3VMPrintf() into an sqlite3DbStrDup().
FossilOrigin-Name: 75e8703664ae94244284819d9021877a5fc33a268fb86f3e70de20875a766d80
2026-03-28 13:30:47 +00:00
drh ff8a0895d6 Fix up the floating-point speed tests so that they only need to be run once
to get a performance comparision.  Provide the fp-speed-test makefile target
to build and run them both.

FossilOrigin-Name: 09e0e78a8265c2e2bc36fef1fa88bd5bf414f9b776966c1c91a4ed4b7866331d
2026-03-27 20:18:06 +00:00
drh a2f78f665c Add a comment to test/fptest01.sql to remind us that failures in that
script might not be the fault of SQLite.

FossilOrigin-Name: 65efcb0a2ed49fa2fd747b5f034d976ed3cb4a01a185bcada70bf8a5208f3841
2026-03-27 19:25:25 +00:00
drh c4cedd7ffe New test cases for text→float conversions.
FossilOrigin-Name: bbf2d648376a965605d5c641a889ab7ef365bc53788d984c054468e9889deee6
2026-03-27 18:58:50 +00:00
drh 00c6eca32c Enable the use of intrinsic high-precision multiplies on Windows ARM.
Make the SQLITE_DISABLE_INTRINSIC option effective on high-precision
multiple subroutines, for testing purposes.

FossilOrigin-Name: efb6212de4ca998546329b06b43122f0fffa3759e4615ca3357ef9fa13606d44
2026-03-27 16:09:10 +00:00
drh be2242d2ee Change the print resolution the fp-speed-1.c test to 17 digits.
FossilOrigin-Name: c687b32a956315ddf4336df42deb4037c37d51b536b8a24c5cef4c5a0bc4c79b
2026-03-27 14:54:01 +00:00
drh 2dd3b4a646 Update fp-speed-1.c and fp-speed-2.c to include its own timers, so that it
works on platforms that lack the "time" command.

FossilOrigin-Name: 9b083070de2d782fe1db62652af8f3cd48abe16b55b974f3927b738e3d914887
2026-03-27 14:19:01 +00:00
drh 4570aeb7bc Add the fp-speed-2.c performance test program.
FossilOrigin-Name: b788256f1980531e7abb15c5fbcdd40ae166f091aea28f449c0407105ddf1aa4
2026-03-27 14:03:41 +00:00
drh 1283969f26 Add a comment showing reserved file-control numbers to sqlite3.h
FossilOrigin-Name: cd472d7c85f3fda54bba5166eae72b5f9b86861c4a1160583e8ee902ba992255
2026-03-27 11:46:11 +00:00
stephan 772ea2f01c Add SQLITE_STRICT_SUBTYPE to PRAGMA compile_options, an omission reported in [forum:3906a71526|forum post 3906a71526].
FossilOrigin-Name: c5af6a10245b6b847d30002806c1577b020c36ab27f7b1cf202ade136aa4779c
2026-03-26 19:11:57 +00:00
drh d4af925e34 Enhance the return value from sqlite3AtoF() so that it reports if the input
includes more than 19.25 significant digits.

FossilOrigin-Name: 1721c8da769816ca16104cd636299d84f682fc004938a50056b76b269e49e915
2026-03-26 18:19:47 +00:00
drh 61d3bc8ddc Revamp the return code from sqlite3AtoF() for additional information and
for small speed improvement and size reduction.

FossilOrigin-Name: f10a431eade20ecd4c4d501fc534e2dae6ab2fe67302ea7bb3cd6eba127848fc
2026-03-26 16:33:03 +00:00
drh f9a3b78561 Improvements to the sqlite3AtoF() interface. Add a test-control for
direct testing of sqlite3AtoF().

FossilOrigin-Name: 97ee48b6ef65fb55633196282c9e75b3b0a3b81ba9755f63493a38e3f1a5610c
2026-03-26 14:19:34 +00:00
drh f855727e45 Back out the sqlite3_atof() API, for now.
FossilOrigin-Name: 7568d264be3988df195401158d8d07105b8e9e018eccf98ac948219bee9388ac
2026-03-26 12:09:37 +00:00
drh 3d58f022a8 Fix an off-by-one error that cause a loop to (harmlessly) run more times
than necessary.  Performance optimization, not a bug fix.

FossilOrigin-Name: 58f4e058f1cbb798b8a48208c6310fd10ca9cd1fc7c550f3ed283cf38ae45c2f
2026-03-25 21:23:23 +00:00
drh 7e9eaf3c1f Performance optimization when zero-padding in printf().
FossilOrigin-Name: dbd81abdc6633124e72999321db6fc8640851fa5eb063e6317a4c3e57b48dbaa
2026-03-25 21:05:18 +00:00
drh 0944b3efad Small size reduction and performance increase in sqlite3Atoi64().
FossilOrigin-Name: c818165fd709b813ab07788662bf69205ece2dcea41346609b3439950f900057
2026-03-25 19:36:16 +00:00
drh f3f194c385 Signification optimizations to layout of text for floating point
conversions in sqlite3_str_vappendf().

FossilOrigin-Name: 6a74df6d6ecfda8e0ff72ca0f4167190e989973ad124412892f7d36f1666ccfc
2026-03-25 17:59:22 +00:00
dan 183a73c352 Handle the case where the compile-time default page-size is greater than 8192 bytes.
FossilOrigin-Name: 3aea251f238277c68d7a80a4fa643bb0f80b274cef77ef32c4eb833cfaf46740
2026-03-25 11:19:38 +00:00
drh 53197b7940 Prevent overly long JSON paths from using too much stack space.
FossilOrigin-Name: 82a0794814bbc095235f8158a4129a5d83f6af1b0c27cb56449ea32feb44faf5
2026-03-24 19:26:05 +00:00
drh f1f1784f4b Limit JSONB recursion depth in the json_patch() function.
FossilOrigin-Name: f8659b700c0088abe19ce65fec032c1005a1f750566488f4578256d41558770c
2026-03-24 12:39:05 +00:00
drh 814f592cf0 Improved error messages for JSONB nested too deep.
FossilOrigin-Name: 82036fafa175e798fd32b3dc4ad043fc2bbe2bb2af17781f12dcb4d23a233865
2026-03-24 11:54:55 +00:00
stephan 9b805c6248 Remove a stale reference to the long-removed SQLITE_MUTEX_APPDEF, as pointed out in [forum:348453389b|forum post 348453389b].
FossilOrigin-Name: cb8e55b59113d0102c50db0a43b1b59d531cdcb12a7cce4c4c9a428c7c6e1adf
2026-03-24 11:33:22 +00:00
drh 3c2ccd2903 Minor simplification to JSON path error reporting.
FossilOrigin-Name: 1cd30c4827e382861cf12af48579eedc9685b9612dedb6a2844f20355b910e7e
2026-03-24 11:25:44 +00:00
drh 7bf4d708dc Recognize artifical JSONB blobs with deep recursion as being ill-formed.
FossilOrigin-Name: adac4ed1f13a9a1c4a34c8f59438eac68867981748629ca3957055ad9f940299
2026-03-24 00:31:28 +00:00
drh 5239e821e8 Move datatype and structure definitions up near the top in the CLI source code,
to facilitate future changes.  No changes to the actual logic (yet).

FossilOrigin-Name: 6b3d84d34d1a84eb8ddff08cd85d9dff6e71099e7a4b2c19db8774d942d3a040
2026-03-23 21:03:34 +00:00
stephan a217a4af09 Require a button click to start the OPFS concurrency tester, rather than simply launching all N workers immediately on page load.
FossilOrigin-Name: 3342d6786e07d3d8dd4b1f2291edea9f8354d384ff21242cc513c218ce6f1630
2026-03-23 13:17:13 +00:00
drh 3e3c31ff0e Use sqlite3_atof() instead of atof() in the CLI. Dogfooding.
FossilOrigin-Name: 276c350313a1ac2ebc70c1e8e50ed4baecd4be4d4c93ba04cf5e0078da18700e
2026-03-23 13:00:56 +00:00
drh dc32016ee0 Move the definition of the new sqlite3_atof() function in sqlite3.h so
that it is adjacent to sqlite3_mprintf().

FossilOrigin-Name: 1553a60506fa29b5f00535ae0f7aeb8cc94ebe84015386b3248fd8491108fc60
2026-03-23 12:25:52 +00:00
drh 5700f8c471 Add the sqlite3_atof() interface.
FossilOrigin-Name: 52f76c59f27152dd62069bd833e2409d57d0cd36ccaa568b2ec4ec823e98cb44
2026-03-23 10:58:29 +00:00
drh 80b88b2482 Proposed new API routine: sqlite3_atof().
FossilOrigin-Name: db370f0f628895279a1c1db79ba7428cb4e31b56cafc191859c2f1a5a924435b
2026-03-22 20:52:23 +00:00
drh 8e7f122c7d Performance enhancements and improved documentation for sqlite3AtoF().
FossilOrigin-Name: 62f011509600ab8cda28888ebd32e5faa4e865702755bc32dee7a12bf2f4425d
2026-03-22 18:26:43 +00:00
drh dfd31e158c Additional performance improvements in sqlite3AtoF(). Also update the
header comments.

FossilOrigin-Name: 8c3a1b450d6d5778004b9fb0e92b4f3f90d3e06ff87276370d01a45b7d607d23
2026-03-22 17:50:48 +00:00
drh cc19c1dafd Further simplification and performance improvement
FossilOrigin-Name: 6c3dc2430adef061b458b6d9159628bb8debfd4d63876d5d5e7493afb799d97f
2026-03-22 02:51:18 +00:00
drh 99571bf3c3 Performance improvements in the floating-point number parser.
FossilOrigin-Name: aea648f3668219a2539160eb68c77ddd06aa8eac235c1326fe7b1547288b11ee
2026-03-22 02:26:38 +00:00
drh 3c1f727603 Because version 3.52.0 was withdrawn, that means QRF has never been
officially released.  Hence the new nMultiInsert field of the spec
can be moved out of the extensions area and into the main part of the
spec, and the version number can drop back from 2 to 1.

FossilOrigin-Name: cf19982241afb17e3db4148b8257f6737141935a291c340ab085c42ed946af1a
2026-03-21 19:49:38 +00:00
drh d9e70f2b90 Try to fix harmless compiler warnings from legacy C compilers in the CLI.
FossilOrigin-Name: 8e1d0c4eac303cd0834449f4792605c0ed983db1f360af42300f0048dadbbef8
2026-03-21 13:21:41 +00:00
drh 4c78799bb0 Add support for REINDEX EXPRESSIONS
FossilOrigin-Name: d56954146ce5eccd63b65242abe2e6556dbceda39a58eab39f5884884afea337
2026-03-21 13:05:54 +00:00
drh 958b6a52e5 Undo "indices"→"indexes" changes in comments, to avoid diff clutter.
Fix a couple of unreachable branches.

FossilOrigin-Name: 0042ff9afd00eca0a837e9bbe57540ff2810fcd58b4f7e3f4e3b83aad5699757
2026-03-21 12:55:57 +00:00
drh f62cff6a9e Fix harmless compiler warnings in the csv.c.
FossilOrigin-Name: 6ac701f89e2ac8dfd18d041eb8c36a0c2b9fedc5618d3a95be794aecce5c59cf
2026-03-20 20:35:21 +00:00
drh f163bfba4e If the argument to REINDEX is EXPRESSIONS (with no schema prefix) then it
updates both expression indexes, and any indexes named "expressions" or
all indexes of any tables name "expressions".

FossilOrigin-Name: 72650dc152279a6da1937f377b1e16b79f4cd664841e12222dc5459838408d22
2026-03-20 19:30:58 +00:00
drh b0a4b69b95 If there are not collating sequences, tables, or indexes named "EXPRESSIONS"
then the "REINDEX EXPRESSIONS" command rebuilds all expression indexes.

FossilOrigin-Name: df5c5aa26758e0dc00a9ccba29eac83071176d257e121207e0a13c54833b18c0
2026-03-20 14:37:39 +00:00
dan bd580270a5 Ensure that collation sequences on non-PK columns of a WITHOUT ROWID table are used correctly when they are part of a row value comparison. Fix for forum post [forum:7a308e933d | 7a308e933d].
FossilOrigin-Name: 078b3162d0d3d3035f4d3ad88664066d99c218a731ac481b9f1172529e26e4eb
2026-03-20 11:35:15 +00:00
drh de4ef2dc70 Ensure that sqlite3RCStr values returned by json_group_array() and
json_group_object() are zero-terminated.

FossilOrigin-Name: 88a931ea2ae734478928c2a6021b00b0f2be490328260630e176cf3cc7111472
2026-03-19 23:49:29 +00:00
dan 0cb7603d4c Update fts5 scan costs to take into account that a prefix query combined with a rowid equality constraint is still quite expensive.
FossilOrigin-Name: 539769b63e4044018be2993894b1962df6a6ae58636682b7fb7a2dd5aaf4075c
2026-03-19 17:56:10 +00:00
drh 28cfb1e4a7 Restructure some internal subroutine signatures for improved
performance of affinity type conversions.

FossilOrigin-Name: 2bc6ce87b39c6ae324c3641c5af60e000df7d7c6d5fd2dd1a318c94cbb4aef84
2026-03-19 17:08:08 +00:00
drh 915a55444b Use compiler intrinsics to improve the performance of
sqlite3Multiply160().

FossilOrigin-Name: 3c64f31392328f51e205fde52f19cf70c193e90897ccb7e93c685bad4f342cac
2026-03-19 13:40:40 +00:00
dan 58012b85b0 Do not assume that a recursive CTE emits rows in ORDER BY order. Fix for forum post [forum:deb1eadf4d677bd5 | deb1eadf4d677bd5].
FossilOrigin-Name: cadb6cdd32c3c9abd7e063f25acd39dddc9ff764a16019ffd5aa11063175326f
2026-03-19 10:51:40 +00:00
dan a2d80200b1 Reverse the if() condition added by this branch.
FossilOrigin-Name: a9a0dd521c7c24b0c8b22abf2a131028f4dc9cd233678dc1f658da81a4a70e4b
2026-03-19 10:39:24 +00:00
drh 08eb5e9e51 Continue enforcing the legacy declaration-before-statement C-language rule,
since some downstream still uses it.

FossilOrigin-Name: 5c237f1f863a32cf229010d2024d0d1e76a07a4d8b9492b26503b959f1c32485
2026-03-18 22:31:56 +00:00
drh b425ebdd4f Add logic to silently and transparently deal with stale expression index
values when deleting or update a table row, instead of reporting a corrupt
database.  Enhance PRAGMA integrity_check to identify non-serious
stale expression index entries as such, rather than proclaiming that the
indexes is broken.

FossilOrigin-Name: 0ed040128b7ebb5be438b1841607ed9c525b271cac81021bc356cf573d1cd516
2026-03-18 20:17:48 +00:00
drh ac53ce155a Minor optimization to error message generation in integrity_check.
FossilOrigin-Name: e0fa7c500052af9d36cb2d232131ef93361403187f4a60f53279b102c7a95860
2026-03-18 18:14:14 +00:00
drh 5a6772eaa5 Comment improvements. Ensure that tests in shellB.test that require
virtual tables do not run when compiling without virtual table support.

FossilOrigin-Name: a270c6cc0eea8a8a2cf1ce30d724b2f12b11a1c343cd8520bd7843f4e62878f8
2026-03-18 17:16:18 +00:00
drh acc213b42f Improve the integrity-check error message when floating point index values
are off by one or two ULPs.  Change the name of the eiib1.test test module
to expridx1.test.

FossilOrigin-Name: 4e996c183ea8d270b4448b471d4d032e13cef89f537674599d1e7c367cc2d0f8
2026-03-18 15:51:15 +00:00
dan 345bc91673 Do not assume that a recursive CTE emits rows in ORDER BY order. Forum [forum:deb1eadf4d677bd5 | deb1eadf4d677bd5].
FossilOrigin-Name: 971aa34b3fd86ba30fe170886d9f83c17159b1638c4bd4fb6cdef79b1c9a88e2
2026-03-18 15:40:26 +00:00
drh 1539575e05 Take care not to overread the record header when decoding the record
in vdbeIsMatchingIndexKey().

FossilOrigin-Name: 3a275b5848767b61011b9d1d3d1a62328a80906386375d1b1e13fd92b6983e05
2026-03-18 14:47:20 +00:00
dan f0648f6924 Pad the allocation in vdbeIsMatchingKey() a little to avoid undefined behaviour if the record is corrupt and getVarint32() reads past the end of it.
FossilOrigin-Name: 9b0671a4f58098948d530f5e238b483a0e9f1309021aff0d6b5ea90e6c8f4e7b
2026-03-18 14:01:21 +00:00
drh 15a3b2d4e1 Adjust the fix at [f7389cdb129d3386] so that integrity_check output still
appears prior to the SQLITE_CORRUPT return.

FossilOrigin-Name: efd9a7a6c862f778da9cd74e38f674e5d1094aa1c566ea3e68553e83f59502d3
2026-03-18 13:49:06 +00:00
drh 765ff2189a Simplification to the logic that decides whether or not an index
field matches when floating point values are close but not exact.

FossilOrigin-Name: 3897017c0e871c676d44786a6f11205fd4b7a947d9d1359109037a123eda6b9b
2026-03-18 12:36:01 +00:00
drh 3ca448dee5 Fix harmless signed/unsigned compiler warnings.
FossilOrigin-Name: aa211fa1d49b6734913e957f9e4b9825709a1493d275f7dfaf9123dc8c7232dc
2026-03-18 11:15:57 +00:00
dan 3f2d7d4aaa Fix some cases where integrity-check would fail to identify real values in indexed expressions that are with 2 ULPs of each other.
FossilOrigin-Name: 40c5966f7b784f181365c81fc559aee9463c374ee56677cf6e0aecaed39d2b28
2026-03-18 11:06:10 +00:00
drh 00989bcd9f Fix harmless "unused variable" compiler warning.
FossilOrigin-Name: 3380eb1ea71d6bf8325dbe49f4bac7623b3b6193f3b36dedd9411188cfc80b89
2026-03-18 11:04:15 +00:00
dan 362b9cecdb Modify integrity-check so that it is only tolerant of very small distortions of real values in indexed expression fields.
FossilOrigin-Name: 82e6de835bde306778425d18ab1a15ae80fdf01af7f577ead89c09f9d53b5b2f
2026-03-17 21:17:42 +00:00
drh f4e6483cda Improved error handling in OP_IFindKey and in vdbeIsMatchingIndexKey().
FossilOrigin-Name: f7389cdb129d3386b7dfb8acacf84816cf10864c6800a9bd9a61c6364b850a31
2026-03-17 18:55:24 +00:00
dan ae2820ab28 When integrity-check finds a missing index entry, search a small range (currently 10) of entries around the missing key for an entry that matches all non-virtual and non-expression fields.
FossilOrigin-Name: 978cedcbd33165dadc7dffc118bde359eff6751ec9842230880cb827d9f7845f
2026-03-17 17:05:45 +00:00
drh bdbad43f33 Do not run index healing on an invalid cursor.
FossilOrigin-Name: eb6403f02c113f58030c0b4143d91b6327ee21dcb324d3f235c2bf6bf3d6577c
2026-03-17 13:13:02 +00:00
drh d6965b1dab Merge recent trunk enhancements into the idxdelete-tolerance branch
FossilOrigin-Name: 9d84fdb7819a9850919e2c1977811bcf514b3baad29eb8cc9fc78451d500c8b6
2026-03-17 12:16:47 +00:00
drh ccf2babd42 Fix an assertion fault that could occur in RTree when given a corrupt
database.

FossilOrigin-Name: ed29fe973e6cbf291eff6fc582aeb34556ea4f8df105ec61b361e4cdda017e62
2026-03-17 12:07:49 +00:00
dan 2953e950c8 Fix display of P4_INDEX values in EXPLAIN output.
FossilOrigin-Name: e717017f0213441da2ef8a9e1f1e88ebfdefa26fc4773711968417c0bd6ebe5f
2026-03-17 11:26:57 +00:00
dan 8736da94bc Fix compiler warning an inefficiency introduced by previous commit.
FossilOrigin-Name: 03058d73d3a88ab30e874b4231cc5b263b874a577f8bf5189a452d7376e25a47
2026-03-17 11:22:51 +00:00
drh b302c4d39c Typo fix in sqlite3Multiply128(): The datatype should be *unsigned*.
FossilOrigin-Name: b459f6ff63325f71d4056dad2ebb536e0d414c973f3c6e167e8f67e94e6f0e2d
2026-03-17 11:19:16 +00:00
dan 022f2f96bc Change the sqlite3VdbeFindDeleteKey() routine so that all index fields that are not expressions or virtual columns must match the table for the delete to succeed.
FossilOrigin-Name: eccd9fcfa7e34ddcdeb671544c6afac7aa0d4665c5e1df7c8e53c41387ed0386
2026-03-17 11:17:54 +00:00
drh 5a73386511 Fix a harmless compiler warning.
FossilOrigin-Name: aa23d6cc69471eacb3fb0d789c34169e047e4a09a50beb430eb8e8e63b5bce78
2026-03-17 10:19:11 +00:00
drh 1c1f7c259e Merge trunk enhancements into the idxdelete-tolerance branch
FossilOrigin-Name: 5f5ab3eb3618b6a2799355df50bfdf94121161f7cadba6253963036d9f284f08
2026-03-17 09:53:02 +00:00
drh cf93c51790 Enhance text→float conversions for exact rounding in all cases.
FossilOrigin-Name: 704ddfd0df57c2b137af88264df20a540e3f8f114a58ae5e31b40ab8259bb63e
2026-03-17 09:49:07 +00:00
drh b50ce08591 Include the float-point table generator utility in the tools directory.
FossilOrigin-Name: 4ab10b251d870af8297b9e505452393f6000f83784c822ff128e7aec06f8c72f
2026-03-17 09:45:47 +00:00
drh e9018e5ff5 Reorder bounds checking in sqlite3Fp10Convert2()
FossilOrigin-Name: 23ad656edb3a63e7e6602770e1f2c4a5ef1f3d41565e7f13408b5305619dfa3d
2026-03-17 09:18:36 +00:00
dan 4cc023bd01 Remove a couple of unused variables.
FossilOrigin-Name: f66cb3dc275d3f31e57576466e697a6c7c5179144babab08d75b624e9846ecc3
2026-03-16 19:14:07 +00:00
dan f6308e68ff Further tests for the code on this branch.
FossilOrigin-Name: f928a38091977268aa6a096104797158883ed89b9e026b4e4b98b9b5da967de1
2026-03-16 16:44:14 +00:00
dan 066a5bcdd7 If OP_IdxDelete cannot find an index entry to delete, first search 10 entries in either direction of where the entry should be in the index. If that fails, search the entire index for an entry with matching PK fields.
FossilOrigin-Name: 2cb9f4e4178549144acd82e317a1646e71c9281ac9fc4aa26bf129f1068ded45
2026-03-16 16:21:43 +00:00
dan b2006499c7 Fix a problem with the find-delete-key method on this branch.
FossilOrigin-Name: 6f49b5ecad5d924999d045a1ade93d95e731182be1781aefa5bc763b6202e9c3
2026-03-16 15:49:58 +00:00
drh bf057073c3 Rework the ".indexes" command of the CLI. (1) The PATTERN now matches
the name of the index itself, not the name of the table containing
the index, thus making the pattern actually useful.  (2)  System-generated
indexes (for UNIQUE constraints and similar) are omitted unless
the -a|-all or --sys options are added.  (3) The --expr option causes
only expression indexes to be shown.

FossilOrigin-Name: 13e2cad810bfd59a9e18eb38bda1241715b76756ffec7d617e9ce99b89f19284
2026-03-16 15:35:41 +00:00
dan d21413adcb Make OP_IdxDelete tolerant of small variations in index key values.
FossilOrigin-Name: c4b0081f08bd0458dbcf269b43a4051941eec8067393aa8e6810d4b8422ce44a
2026-03-16 15:16:13 +00:00
dan 061459bf76 Properly fix temp triggers created as part of FK processing to their schemas. Otherwise they may become confused by similarly named child tables in other attached databases. Fix for forum post [forum:636bd0180a | 636bd0180a].
FossilOrigin-Name: 80bc5bc07e221f837c28066f0a438f11c8ab6be4c8ba93615439eb1667967003
2026-03-16 11:14:26 +00:00
drh a6640846dd Improve accuracy of text→floating-point conversions by moving
from 64-bit to 96-bit approximations to powers of ten.

FossilOrigin-Name: a19c0785dcfb9fb74963c45b161b12ac8f4379e2b990f6c5de0d7b959c5be98d
2026-03-16 09:40:12 +00:00
stephan 7539d2f1ba Add a missing return for the invalid arguments error case in sqlite3_test_control_fault_install. No current scripts trigger this error.
FossilOrigin-Name: dc78d258745a1350685b37da56f0a2ac2e437b422415d634522dc61d340d06eb
2026-03-14 16:30:13 +00:00
drh 0f08d9586c Changes some legacy testing targets in the makefiles into aliases for
"devtest" and "releasetest"

FossilOrigin-Name: 4c54f22f7abbf5403e554fbf3dd70ddee97352d7de1f7fe19f540bdf681f4e75
2026-03-13 15:03:55 +00:00
stephan 6c446015bd Update the canonical JimTcl home page link (it was moved earlier today).
FossilOrigin-Name: 568573f872733bf32c2fde6a3a43100bdac17fc762e5c9a6d7a47b988fdcea4e
2026-03-12 08:33:14 +00:00
stephan 9e572064f3 Replace several comment-based switch-fall-through notices with attribute based ones, as suggested in [forum:b6ab8dc547|forum post b6ab8dc547].
FossilOrigin-Name: 2b0209a3bd3fb19af424386dcd13ff49a0c86edfd5d283fd5e5c42c266872100
2026-03-11 15:17:14 +00:00
drh 169f68ed88 Improved defenses against mischief.
FossilOrigin-Name: 8bdc0d485e3ad0c7a1e818da66f106951d496b05cbe61d12c2c448f2f24b6d5d
2026-03-11 14:43:29 +00:00
drh c739d13217 Fix harmless compiler warnings.
FossilOrigin-Name: d7b4a4a3d3d6296a012cc14fc2d95fe810b5bdb625209a2d28bf2c7a026749bf
2026-03-10 22:43:39 +00:00
drh 3bd23b0d78 Fix the behavior dot-commands that appear as command line arguments to the
CLI so that their effect persists into subsequent commands on the command
line.  This fix also improves error messages caused by command line
arguments. 
[forum:/forumpost/2026-03-10T19:26:06z|Forum post 2026-03-10T19:26:06z].

FossilOrigin-Name: ff084ae341eab5c4ce9403defee03adc448c05f77648a954f0942a38f640080b
2026-03-10 21:01:59 +00:00
drh abc82c49c2 Revamp the QRF insert-mode optimization of [659ff6ab55802507]. The
control value is now nMultiInsert.  Multiple rows are added to each INSERT
statement until the number of bytes in that statement exceeds the
nMultiInsert value.  The CLI uses a default value of 3000, which provides
a good balance between speed and prepare-statement size.  The output
from ".dump" now loads nearly 2x faster in some cases.

FossilOrigin-Name: 55b0ce9f93d68b17a18423f0f751b778573127743a2bac67abfea074ac32d41a
2026-03-10 18:13:13 +00:00
drh bd10a18f00 Bug fix to the ".dump" command in the CLI: Temporarily disable character
limit to QRF so as not to truncate values in the generated output.

FossilOrigin-Name: f6787e4b50af50663d8fa00ebe0694440354a7ad0509837a6b249e090e6cffca
2026-03-10 15:43:57 +00:00
drh efa8633fd8 Enhance QRF with the mxTuple field, which if greater than 1 causes the
QRF_STYLE_Insert style to group as many as mxTuple adjacent rows into
a single INSERT statement.  The field is accessible using the -mxtuple
option in the TCL interface, and the --mxtuple option in ".mode".  The
output of the ".dump" and ".fullschema" commands responds to the current
--mxtuple setting.

FossilOrigin-Name: 659ff6ab5580250707908af003ecd093bd5313f03f21f0efd000a7aff6638b3c
2026-03-10 14:02:03 +00:00
drh 140cbff0d2 Remove an obsolete pragraph from the header comment on the fileio.c extension.
FossilOrigin-Name: 17613b72f34162ce22864621c519e883930c9f51a0a86afbf30ce664f95d7be1
2026-03-09 14:04:34 +00:00
drh e795890d7e In QRF, fix line-wrap indentation for "line" mode when using a
non-standard column separator.

FossilOrigin-Name: 00c96cca3de709e40207feb9c10a94e38e7857c9bc01938849c6c5daee7148c7
2026-03-09 13:53:58 +00:00
dan 5ed6e4ccd1 Fix a problem with RIGHT JOIN and expressions on indexes. [forum:e3dba5426a | Forum post e3dba5426a].
FossilOrigin-Name: f27d578edac06eb28739f23351ab1cf07531bc7b32f4077e4dc9c8166f551d95
2026-03-09 11:49:51 +00:00
drh 016a0ebedc Fix the completion.c TVF so that the xNext method reports errors it encounters
while running queries.

FossilOrigin-Name: eebcdcceef8a436e5fb1397cad6eb5bcf0722060fe1cd6acb26cdf1e86f07293
2026-03-08 19:31:40 +00:00
drh 456c7ea715 In the decimal.c extension, set a default MAX_DIGITS to 100 million.
Do not allow integer overflow of the N parameter to the internal
decimal_round() function.

FossilOrigin-Name: d0e23423d1bdd0482db4b74ef0fde2fbc2bbad02a7b92ba45a27ca57f4740e16
2026-03-08 18:50:35 +00:00
stephan 5236bad923 Consolidate the opfs and opfs-wl tester1 tests using the preprocessor.
FossilOrigin-Name: 77846c8c7aa4625729a5e113c12e7d16efc88c1306c01ba36f47240aac450b0f
2026-03-08 17:07:01 +00:00
stephan 46a692c08b Upgrade ext/wasm/c-pp-lite.c to its newer sibling because we've reached the older one's limits. This renames c-pp-lite.c to libcmpp.c to maintain both the SCM- and code lineage but that may end up looking a bit weird because the diff between the two is vast.
FossilOrigin-Name: 2e2339bd9e4293bad04ece7673a3048b99c2143cf9573ade2ec082d95744b981
2026-03-08 16:03:08 +00:00
stephan 02cbc96d09 JS doc updates.
FossilOrigin-Name: 699295f84ef2c4da363f13126abde6af07697d0959896cd868b14513d4278964
2026-03-08 12:42:34 +00:00
drh 1eb335e313 Increase the version number on trunk to 3.53.0
FossilOrigin-Name: 0152b445ff0cdf560fc028938e1d43a56685d7c046fad84fa90166cd01d71279
2026-03-08 10:51:47 +00:00
stephan 976b50620a Merge opfs-wl into trunk, not so much for the "opfs-wl" VFS, which is stil tentative (working fine but with little perceived benefit), but for the "opfs" VFS cleanups which necessarily happened around it.
FossilOrigin-Name: bcaa876c6dafee0995d239f59b6ce7810b6f5faa219b9c392966211ae3dec17a
2026-03-08 08:32:22 +00:00
stephan 303fad1f91 Update the tester1 SEE tests to account for opfs-wl.
FossilOrigin-Name: 616d32c8380ef47c5a7c2eb765e7b6fbec9500d4e16c837f4ee11d71c239a55f
2026-03-08 08:09:09 +00:00
stephan 03b0f8fc9c Refactor tester1 OPFS test to support both the "opfs" and "opfs-wl" VFSes.
FossilOrigin-Name: 17ac8d7bdfaf96510abea8c11019045c1703b7040655d9c4c8d7001636260b50
2026-03-08 07:58:02 +00:00
stephan 84ad6f302f Add parens around an expression in the CLI shell to account for operator precedence, as reported in [forum:856ff7a2f1|forum post 856ff7a2f1].
FossilOrigin-Name: da3ad170aebf57f2202e56673ac149bc51077d126ab5928f1db16492e403d5b9
2026-03-08 07:02:46 +00:00
stephan 9358070cb3 Expose sqlite3_bind_zeroblob() to JS/WASM.
FossilOrigin-Name: 0c5d4fe0bd322b4c8dbd0800001317a21cd511b00244de74b4db3fd051153879
2026-03-08 05:53:47 +00:00
drh eac7cd634d Use the UINT64_C() macro rather than the LLU suffix on integer literals
for portability to older Microsoft compilers.
[forum:/forumpost/0a9ffaff6fcdedb3|Forum thread 0a9ffaff6fcdedb3].

FossilOrigin-Name: 27be26566fdb4d01c44dc111f1c18c835797de0990d4fcc876e91edc216e9e93
2026-03-07 22:37:28 +00:00
drh 13a494fddd Give the fuzzcheck test program access to all the same extensions that
dbsqlfuzz has access to.

FossilOrigin-Name: d9c23b49e355233bbfcdc8305dcae21640e66a0bbf1daed0dd3f9d132c1ef69f
2026-03-07 21:07:18 +00:00
drh 85ee7e15e3 Fix harmless compiler warnings on Windows.
FossilOrigin-Name: 358d78da75033d4d1fcb27b5f95c940dcc306cdf34ef999262da901e71f2b306
2026-03-07 17:50:32 +00:00
stephan ac26170191 Minor typo fixes from [forum:8fc8bc34291d6f45|forum post 8fc8bc34291d6f45].
FossilOrigin-Name: c8fb88c19ed91dc5e205f719aedb895f7ceb62b113f61ec94cc6e435051da15c
2026-03-07 15:15:37 +00:00
drh cde22e8806 An attempt to get the CLI build to work with mingw32.
[forum:/forumpost/086cb1ef66|Forum post 086cb1ef66].

FossilOrigin-Name: d95b9e7c1746b9cbd067aa65806b1f74d0dfe77ab6084cb3da9ef2242cac9134
2026-03-07 12:00:56 +00:00
drh b4f257f5e0 Fix another memory leak following OOM in the decimal extension.
FossilOrigin-Name: 414f6dbc09c2a3cf04ccdfd8f11eecb7ef781facff979ae4236a452913001391
2026-03-07 11:23:11 +00:00
stephan 237fce8205 Extend the JS pre-bootstrapping configuration to include an option to disable inclusion of any given extension VFS (not the default VFSes, like memdb). The primary motivation for this is to give people who don't use OPFS to a way to keep the OPFS VFSes from loading their proxy workers, which are expensive.
FossilOrigin-Name: 9a07eb7a941479510891d1444aacbeb440efaad3e9a13f186833618d8b60a8da
2026-03-07 06:10:29 +00:00
stephan 78649bb421 Fix the mangling of the opfs async proxy script's URI caused by both OPFS VFSes modifying it to accound for sqlite3.dir. Add a link to speedtest1 with opfw-wl to index.html.
FossilOrigin-Name: c1cb61f473a11a325ef421ba8edfb20257203688565f7db62309044be183af5f
2026-03-07 04:19:08 +00:00
stephan 7d10030d75 Cleanups and docs in the opfs vfs and its concurrency tester. Experimentally add sqlite3.capi.sqlite3_js_retry_busy(), which runs a callback repeatedly until it _stops_ returning (or throwing) an SQLITE_BUSY error.
FossilOrigin-Name: 45c02ed21635f7ef45214ab5ec6230b6b8bd89e35ef5889db318523e1c679fea
2026-03-07 03:32:17 +00:00
stephan f20c4e650d Elide more of the 64-bit was stuff from the default build target.
FossilOrigin-Name: 3b74ad9081daee560bf9b400e69a75abcc62dae7c5f55dcf46de84daae50ed9f
2026-03-07 03:29:26 +00:00
drh a0851f1483 Enhancements and fixes to some of the extensions used in the CLI so that
they can be fuzzed using dbsqlfuzz.

FossilOrigin-Name: d800ea06d231957cf571327d9eaf81564d5439aeee099987101050a41f9676c0
2026-03-07 02:42:27 +00:00
drh 6142524b70 The optional SQLITE_DECIMAL_MAX_DIGIT compile-time option limits the
maximum number of digits that the decimal extension will operate on.

FossilOrigin-Name: 4214485b2fe4b95d9440008158d458b4b957a111f201965de1a518623589fd79
2026-03-07 02:36:39 +00:00
stephan 9be458e611 Cleanups and docs in the new opfs code structure.
FossilOrigin-Name: 3b470c4c7a1fcc710e6b9eae32134c7f6c3f6008b24c7351257f66f5e8f70311
2026-03-07 02:19:23 +00:00
stephan 57d0fe254a For backwards compatibility, ensure that the "opfs" VFS does not specifically require Atomics.waitAsync() (a new requirement of "opfs-wl"), but make use of it if available. Only apply jitter to the concurrency test runs at random intervals.
FossilOrigin-Name: f2175f526c00cfe562e8f332eb197b5ef2c3d6be1fff2aab1566c2c533a293ac
2026-03-07 01:01:13 +00:00
drh 3866d01907 Reenable the is_base85() extension function that was disabled
by [af18b68978ec9b1f].

FossilOrigin-Name: be07c16e3a7d28b95bea4b987a880d5b4b2b44d9a31ac5b48caa5b6c6dc5bec3
2026-03-07 00:14:36 +00:00
drh 02d9860c8e Fix another NULL ptr deref following OOM in the sha1 extension.
FossilOrigin-Name: c4843e4cce71acb1e3743c45e0ee8cf62e91c39b57cec4f356bf7a4a10f81352
2026-03-07 00:10:34 +00:00
drh 7f6236bed1 Fix a NULL pointer dereference following OOM in the sha1 extension.
FossilOrigin-Name: 530b3f97ca4e8b1d263266855743b0d57d048104a6c3bed6c4d7958985767bc6
2026-03-07 00:03:47 +00:00
drh 56b571780e Fix a NULL pointer dereference following OOM in the decimal extension.
FossilOrigin-Name: de37f30b2100e0c249e5c0853d6af90cd3c2069899747dcf0889e6a3d6b94061
2026-03-06 23:58:58 +00:00
drh 84334c941a Fix a memory leak following OOM in the completion extension.
FossilOrigin-Name: ff5b1efbf80875a82b3707b56440551eb6f68a899ade27363bc62872fe46fd75
2026-03-06 23:56:03 +00:00
stephan ac63b524ac Add some jitter and duration info to the opfs concurrency tester. Ensure that Atomics.waitAsync() is available for opfs-wl. Further testing has shown that Web opfs-wl is consistently fairer about doling out contested locks but it's dog slow compared to the original VFS under moderate concurrency. In single-connection use they're effectively on par.
FossilOrigin-Name: a9aecc987512d60f2663973f43c769cf086fc14149edfbcb18c0aec9f5aa3dbf
2026-03-06 23:50:44 +00:00
drh 8f65da4574 Fix a memory leak in the decimal extension that follows an OOM.
FossilOrigin-Name: 8d59e47e7ae0aae01cb25c6c054cf9a4d9651e858bc0dd1a0e2a013ec6579012
2026-03-06 23:37:45 +00:00
stephan 2251f0c4cc Get "opfs" and "opfs-wl" properly split. Speedtest1 and the concurrency tester are happy with opfs-wl but it is not yet plugged in to tester1.
FossilOrigin-Name: 12dcd5425967b6306f3e3178babeac130962edb78e6298d545a429a1a20457cc
2026-03-06 22:21:06 +00:00
drh 1bf79ca445 Rename the initializer functions for base64() and base85() to include the
number at the end.

FossilOrigin-Name: af18b68978ec9b1ff732505bfb92bb65e4f7381b904123f42c638b747569fa28
2026-03-06 22:09:39 +00:00
drh 3ed9043c71 Increase the version number to 3.52.1.
FossilOrigin-Name: 94a650b169bfd1056995c1cfe584d3d92667bce5e9cf93e6381607ed87251bf3
2026-03-06 21:34:49 +00:00
drh 7c483444a1 Fix a NULL pointer dereference in the (undocumented) two-argument version
of the decimal() extension SQL function.
[forum:/forumpost/fa9c2dc83f|Forum post fa9c2dc83f].

FossilOrigin-Name: a0266351df8700383d50c2ef2d179dd82b0e0fd7fc70d05a91d5e4e845983b46
2026-03-06 21:31:07 +00:00
stephan 7cc8db2f5d Small typo fix in the FP_DIGITS API docs.
FossilOrigin-Name: 0286f97f34fd560d4ba423bd28d18ac080c918b0e99b8d16c4a213cf205b5e5b
2026-03-06 19:39:03 +00:00
stephan a85d09f1a4 Minor cleanups and docs. Teach the OPFS concurrency tester to deal with SQLITE_BUSY instead of failing.
FossilOrigin-Name: 247ffed141f66a6a5a396a3e002995a9f00c70333271199200530066e77956c4
2026-03-06 19:33:25 +00:00
drh 275e7c7caa Merge fixes that were deferred until after the 3.52.0 release, including:
(1) Fix handling of DISTINCT with OUTER JOIN, (2) Enhance configure to
auto-detect linenoise, (3) Add recent configuration settings to WASM.

FossilOrigin-Name: 9d51bec1b17925f5cac0d590c0d70bb34e77fdb7253d18007f52711d42f6f5d0
2026-03-06 18:51:49 +00:00
drh 88e552f1b9 Fix the linenoise auto-detection added by the previous check-in.
FossilOrigin-Name: e36958584511d83666590abef223c2edc3da15b5b32e280baf08159ddae80e50
2026-03-06 18:26:50 +00:00
stephan a98f278f58 This one reliably runs 5 workers. Checking in before subsequent cleanups and debug output removal break it.
FossilOrigin-Name: ba81d95febc5fd0f9bbb2685fef5b1b10f9991751f2bdfafba80c15877af1cef
2026-03-06 17:10:28 +00:00
stephan c008b8fe3e Correct an internal doc falsehood.
FossilOrigin-Name: 53aa080e357d7a2ffeab68a3584fda43d51ecef3dc8a1d46dd32392ae4f9740c
2026-03-06 16:15:32 +00:00
stephan 7b9cd03c0a This Web Lock impl can reliably run a single OPFS connection but rather unreliably 'loses' workers with higher counts, presumably due to deadlock or deadly embrace (how _all_ of them can deadlock at once is unclear, but clearly a bug).
FossilOrigin-Name: d4e8583e2e80665adfe4e814adb6c219936af1dcac4105795045cb1a7b1e4864
2026-03-06 16:04:21 +00:00
drh 257fdf8496 Version 3.52.0
FossilOrigin-Name: 557aeb43869d3585137b17690cb3b64f7de6921774daae9e56403c3717dceab6
2026-03-06 16:01:44 +00:00
stephan ad2fe531d0 Expose SQLITE_{DBCONFIG_FP_DIGITS,LIMIT_PARSER_DEPTH,PREPARE_FROM_DDL} to WASM and add support for DBCONFIG_FP_DIGITS to the variadic wrapper for sqlite3_db_config(). These unfortunately went overlooked before the 3.52 release.
FossilOrigin-Name: 837069635e53849cbca0aa876bad7c7ff44e17578ae492e07acf354067b7e16d
2026-03-06 15:41:17 +00:00
drh 93353c0bc2 Add the new realpath() extension function to the list of functions
prohibited in --safe mode of the CLI.

FossilOrigin-Name: 5fa49c4d592778fb82c4e25c77cf0442d3dc23cc7f8d91d25952c722af866930
2026-03-06 14:33:12 +00:00
stephan 544ef18bbd Remove an extraneous OPFS metrics increment.
FossilOrigin-Name: bf3548a37712e848c7a9cadfdc1669a2be572ea0a0c28d84c157ab30f8c30c44
2026-03-06 11:49:36 +00:00
dan ea595494e2 Uninstall faultsim at the end of the walrestart.test script.
FossilOrigin-Name: 798905a1277ecb17245f2a7378fdc8dc608374fde50bdfe44d09195a7c78cb5c
2026-03-06 11:18:12 +00:00
stephan 891f5a176b A slight simplification/optimization in the OPFS proxy's waitLoop().
FossilOrigin-Name: 521bb140b7ed237c118ac9094732d06907229a6ff385502e850c679bd623fd58
2026-03-06 09:11:47 +00:00
stephan 831e233202 Reimplement the OPFS async proxy's wait-forever loop to be more async-friendly for upcoming changes.
FossilOrigin-Name: fd775772fbb50b04c8c37977b90a708784eefed403e3668196c9d95559d3a5e1
2026-03-06 09:06:24 +00:00
drh 7a125b97d7 Fix the build for -DSQLITE_OMIT_FLOATING_POINT.
FossilOrigin-Name: f41b9ad98c474755c6701eb6ab3dc9864ace9b52a81581f73ca0912cedbc3c37
2026-03-05 18:28:43 +00:00
drh d61ecd83b8 Enhance ./configure to automatically detect linenoise in ../linenoise
relative to the build directory, or ../linenoise relative to the source
directory, or at $HOME/linenoise.

FossilOrigin-Name: eb6b47c667e52d5c8682d650dbd114c08d2ecce4bb7bce6fdf2589d1ea09f5c0
2026-03-05 18:16:25 +00:00
dan 21bdb75ac8 Ensure xMutexInit() is called in SQLITE_THREAD_MISUSE_WARNINGS builds.
FossilOrigin-Name: ac6a389e09a0de1b622af4a7136dc66875e8009c27950f2260a96f17fec4f9bc
2026-03-05 16:23:04 +00:00
dan 46686ac681 Fix a long-standing problem with DISTINCT LEFT JOIN queries.
FossilOrigin-Name: f43294a5582b540a33c584ec8c69b6a5006a4d243ad5cf36125b2b0806e3518b
2026-03-05 14:39:27 +00:00
stephan 14b5eadefe Strip the opfs-wl "back to formula", removing the current false starts so that this can be tried again without tripping over any cruft. The current impl is subject, with no obvious way out of it, to starvation in the async proxy.
FossilOrigin-Name: d022f1f5e74dedae044801330eb099022498f359f408e69b3574885641b312f5
2026-03-05 11:51:27 +00:00
stephan cef3ff74cd The opfs-wl lock hang has been traced to starvation between the WebLock and the tight wait-on-VFS-calls Atomics.wait() loop. This can reportedly be resolved with another level of indirection in which the WebLock takes over the wait-on-VFS-calls part until it's unlocked, returning to the global loop when it's done. That exceeds this morning's ambitions but is next to try out.
FossilOrigin-Name: 113bd910e12fea17f9f4a0a3baf706f15627c08cfa6b47a960a83eee761ef4dd
2026-03-05 06:30:57 +00:00
drh bef2a1e33e Add a little flexibility to checkpoint sizes in the walrestart.test script.
FossilOrigin-Name: e438b564ca84377746464034d770e9c1f5899d935c6a4dffc546acb92afb800e
2026-03-05 00:30:05 +00:00
stephan 671a6f83f4 Add some debugging output to opfs and track down the breakage to the initial WebLock request, which is never reaching its callback. Still broken, but this is progress.
FossilOrigin-Name: 62fc8b35aeec75f5648b3daa24162c638999d447aa874bdfcbac1431c5c97b6f
2026-03-04 21:16:36 +00:00
stephan 232446b532 Get opfs-wl plugged in to the concurrency tester. Somewhat ironically, all competing workers fail with locking errors while worker 1 is busy running off the rails somewhere. Stashing for closer investigation later.
FossilOrigin-Name: 3b27310aa29ea84f459974981a600301abac5c705029a289d3872ecacf231da3
2026-03-04 20:33:21 +00:00
stephan b85067c568 Consolidate the last 200 lines of common OPFS VFS code. "opfs" still works, "opfs-wl" registers fine but is still otherwise untested.
FossilOrigin-Name: 5978ee4902e4223fed6b95bd2d8f489bb300af8b762650e7113d1f3e97519d88
2026-03-04 19:21:09 +00:00
stephan 080da5f56d Factor out about 300 lines of common OPFS VFS bootstrapping code.
FossilOrigin-Name: 57adecbab71795b62b1c2e4570ff504f35681e81dd8c94f78ad8e05ef39d36fd
2026-03-04 17:54:02 +00:00
stephan 691494a93b Consolidate the OPFS VFS's metrics-tracking code.
FossilOrigin-Name: b71c79ef9672c77a72a976ffcd7cbebfaf0ff314dff97b274f7d092de6a7773f
2026-03-04 16:30:51 +00:00
stephan 6ff9fcfad2 Baby steps in consolidating common OPFS VFS code.
FossilOrigin-Name: b0dd23299e97ff975f213cb3a8b051f4d7b785b29def82e01f53427fdf77ecb6
2026-03-04 14:33:33 +00:00
drh 0f7f9ab007 Remove dead (commented-out) code that came to my attention because it
ws nearby the fix of the previous check-in.

FossilOrigin-Name: 61f8a28591a833b1f5834a347feefeba8414fecc7ff154f1b6ef19963f181812
2026-03-04 13:08:14 +00:00
drh 76e562e222 Fix a code-generator bug introduced by the UNION/INTERSECT/EXISTS
optimization of check-in [132ba781b031972c] and reported in
[forum:/forumpost/2026-03-04T05:06:26Z|forum post 2026-03-04T05:06:26Z].

FossilOrigin-Name: 6ba7797cac7dc873b62a12be5712511c9199b379342dac4a1887bfd65227a298
2026-03-04 13:02:08 +00:00
stephan 96daad10fa Consolidate much of the OPFS utility code into a new file for use by two of the OPFS VFSes.
FossilOrigin-Name: db19a6e9663c3a44996178cb8c35dc4ccd60f48cb4b81b6c214411a56c57def7
2026-03-04 11:37:39 +00:00
stephan 6d6bf30735 A potential fix for the probable breakage of bundler-friendly builds in the previous check-in. Pending review from someone who uses those tools.
FossilOrigin-Name: 8ea85776116521526d684f221d67e288126e62931d4a0ea7fc7f164cd2d5b2ec
2026-03-04 00:48:03 +00:00
stephan 9ac95d9ce1 opfs-wl is now loading and registering but it's still untested.
FossilOrigin-Name: 9a471f7491a371052bf1785098ba966dd0d03503e7d8b9fbcd65f07b038e5021
2026-03-04 00:37:55 +00:00
stephan a2d3a54f80 An initial attempt to plugging opfs-wl into the build but its initial handshake with the async half collides with the opfs VFS's handshake, causing bootstrapping to fail miserably. We'll need to either devise a handshake which can differentiate between the two instances or we'll need to preprocess sqlite3-opfs-async-proxy into two copies. Move the (now) three copies of some common code shared by the opfs pieces into a preprocessor #include.
FossilOrigin-Name: 1e0b72631aecb0bb72f4089116da221e6c4abf962db589de08132cd52c2be0e2
2026-03-03 23:43:40 +00:00
stephan 0b7b802d83 Add a missing reset of the lock handshake SharedArrayBuffer slot.
FossilOrigin-Name: 5910ed7ac843aa1fdb103ddcfaf6270d12cb39199205fc0b9f721fbf7fa2f851
2026-03-03 21:46:44 +00:00
stephan e2309eebd7 Implementations of WebLock-based opfs::xLock() and xUnlock(). Still completely untested and not yet integrated into the build.
FossilOrigin-Name: 3343e3aabe465f4ab91dd148dfc5a60346e9afb560c10d1f92aeae98763ec3ce
2026-03-03 21:07:46 +00:00
dan 72a14a575e Avoid an obscure race condition between a checkpointer and a writer wrapping around to the start of the wal file.
FossilOrigin-Name: 7168988acbec2d8d51106a263e553f8942b8b23d983dbbe5028e0f9be68cbb83
2026-03-03 19:43:19 +00:00
stephan d016fad5a8 Merge trunk into the opfs-wl branch.
FossilOrigin-Name: 81bc4b3b6abc19f98d1f3b1065c4b39a42620a0d7abebe98605dca62dd2d6f9a
2026-03-03 18:34:59 +00:00
dan 8ba2a38363 Do not run test script walrestart.test with the "memsubsys1" permutation.
FossilOrigin-Name: 703cbb0f760515eac1e6f72a5e5cd928258c71378e4a976082b6518c90128135
2026-03-03 18:34:38 +00:00
drh bb80d688ac Minor tweaks to the previous, for testability.
FossilOrigin-Name: decb69015457b4f70cf574aeef87d40f9de60b94965f4a915e8d0cd7559ef2ae
2026-03-03 18:21:22 +00:00
dan fe57e14b49 Avoid an obscure race condition between a checkpointer and a writer wrapping around to the start of the wal file.
FossilOrigin-Name: 053bd3930f827156fd67ea4546a36227cffbb6f8bada3b5d1a7cf5f1867ac624
2026-03-03 17:34:01 +00:00
stephan 5c4692b6a9 Add JS tests for sqlite3_bind_zeroblob().
FossilOrigin-Name: 0473eeb3cefa1d7882375e3f89d239d570649349c5a0a8ddd90b0a336a9bb493
2026-03-03 14:13:34 +00:00
stephan a2fef84b19 Expose sqlite3_bind_zeroblob() to JS/WASM, per request in [forum:e7acf225f9|forum post e7acf225f9]. This is currently missing tests and is not planned for merge until the 3.53 dev cycle starts.
FossilOrigin-Name: b0c2ef6046b53b6133fed6a890d2ab5e47eeebd7b0b32849b5f69ae54a03014c
2026-03-03 01:48:18 +00:00
drh 18f99f43bc Convert many sqlite3_realloc() calls to sqlite3_realloc64().
FossilOrigin-Name: 88dce64242552e7443d9fb496f6f3ad71dc5b4a882ce21b7ef1d5ea4e26f1e61
2026-03-02 17:11:44 +00:00
drh f6c8adf3a2 Omit explain_i calls from TCL test scripts.
FossilOrigin-Name: 204f25c882832058c3291d4d6a0a6ac2711c081cb91b19feb2cf94a25ed61ecf
2026-03-02 16:40:16 +00:00
drh 72afb50a6d Fix TEMP INSTEAD OF triggers so that they work on TEMP views.
[forum:/forumpost/2026-02-28T23:33:04z|Forum post 2026-02-28T23:33:04z].

FossilOrigin-Name: 0596bd508f0bc548158858a1dc113af2235d02632ce8c41cd27d1029e1a7e48d
2026-03-02 15:34:16 +00:00
drh 7fa89ea907 Change some sqlite3_realloc() calls into sqlite3_realloc64().
FossilOrigin-Name: a391f5646926787fd9a004225ea406b61d20f282c852c0282fd26cada644b601
2026-03-02 15:06:59 +00:00
drh c1454ee833 Remove the experimental and incomplete ext/repair extension, to prevent AIs
from scanning the (incomplete) code and reporting bugs against it.

FossilOrigin-Name: 213b1c6608af4b3e9d6e0d8de6432cc6857931427baf4beac1e0a4294e4dc6ce
2026-03-02 13:44:04 +00:00
drh ea6b2a87d1 Disable test shell1-5.0 as it is causing a use of initialized deep inside
of TCL.

FossilOrigin-Name: 440bd6091e3767f0a2f42ffdc92ca5e4736c0a73324fdd15397c3b5dbbc31fb8
2026-03-02 13:43:08 +00:00
drh f270460366 Use only 64-bit memory allocation in FTS5. Fix for UAF reported by
Zijie Zhao.

FossilOrigin-Name: e8976d5041c929675772039b7a8fc4ff0b609537d86f9aa6e445ecd512a10673
2026-03-02 11:41:48 +00:00
dan 33478099c9 Fix a possible 9-byte buffer overread in the zipfile extension hit when processing a corrupt zip file. Forum post [forum:/forumpost/721a05d2c5 | 721a05d2c5].
FossilOrigin-Name: 5db21813d126554d80db903be6d36ab6c1f73f9135a54af6dcfcfce0bcc18e68
2026-03-02 11:22:28 +00:00
drh 93ad2e083e The intck01.sql test should only be run if virtual tables work.
FossilOrigin-Name: 641d6f31a7f7b4901061e24d4e624da5ed92282e79771bb019b82a882e5d1ae9
2026-03-02 00:06:02 +00:00
drh ffaf9110f6 More floating point conversion test cases.
FossilOrigin-Name: 44a736a700ab12b398873400dc06cd334817842b4fa08bf8070a9ad6dbff0b51
2026-03-01 23:23:52 +00:00
drh 7e4e866baf New floating-point test case.
FossilOrigin-Name: f353399932e9797198e635405a594c2987850070e3f9aee87a05c5a65d4c1e00
2026-03-01 23:01:17 +00:00
drh e293b2d835 Fix an OOB read in the incremental integrity-check extension.
FossilOrigin-Name: abecc8e388e294311aa0b572e0a984b8ddad2afbf829c1246e1682fa549c8fac
2026-03-01 22:36:02 +00:00
drh 83de60bd7b Simplified rounding logic in sqlite3Fp10Convert2().
FossilOrigin-Name: 8ac63ebc5c04ba555bbf0d878a70e25deba5fcc75ff44c464600b92c27e5dcb0
2026-03-01 20:22:08 +00:00
drh a000db9754 Code changes in sqlite3Fp10Convert2() for easier maintenance. Does not
affect the generated machine code.

FossilOrigin-Name: 21c8fc7100e23b632b77934cbbafb98dfd3e6e73bab702446ef6345b378c9d36
2026-02-28 20:22:10 +00:00
drh 7ac310d0c6 Improvements to power-of-ten computations used for text ↔ binary64
conversions.

FossilOrigin-Name: 0780bce854b962fb2d4a1a19c55c9b5790a9669f26e1ff8b5f1f1733cfc647e0
2026-02-27 22:59:46 +00:00
drh afaf7ea2e4 Minor simplification to the text→binary64 conversion algorithm.
FossilOrigin-Name: 6632d51f1673e9a6b6e26baebc7aaa1ae27024ad5db1baff6ee5fcf865099f43
2026-02-27 20:37:03 +00:00
stephan f707e2ec5d Two code comment typo fixes reported in the forum. No code changes.
FossilOrigin-Name: 3fbd28fabc5509a0bf2026cb60891d6d3efe251910baa3a3934225d0a7187bfa
2026-02-27 18:32:12 +00:00
drh 7d9fbe6476 Fix an off-by-one error in the indentation of multi-line outputs in
QRF_STYLE_Line mode.

FossilOrigin-Name: d47499b21c893cc8749a1cafe1cf712dacfeaa6cfe2cdb75d1fc450f3cb6879f
2026-02-27 13:25:42 +00:00
drh c67a538168 Add an assert() to sqlite3RegisterLikeFunctions() to hopefully prevent
future false-positive bug reports coming out of CodeQL.

FossilOrigin-Name: 7c5f4dcd748baa60097bbf68d7aca99cc959bb1f7da92bd9ad86a4425a37d391
2026-02-27 11:36:43 +00:00
stephan 57d7a20ce7 Another nano-optimization in kvvfs v2.
FossilOrigin-Name: a9d60190d693504742a894918f145111908d7d6cb9e853ba7a2356529046372a
2026-02-27 06:33:28 +00:00
stephan d22d0edc48 JS: slight performance improvements in KVVfsStorage.
FossilOrigin-Name: 56edf2addb155dbb60269fe563ecb5f6132672c6c61f6f9d18ebdff72111298d
2026-02-27 03:21:47 +00:00
stephan 8c6bf948b7 An initial copy/paste stub for experimenting with using WebLocks for a new copy of the OPFS vfs. This is not yet functional, or even loading, but needs stashing to avoid potential loss.
FossilOrigin-Name: 372f18ae909bafd7167b70051935832e3107ede72aaeccbf5c042db7ba791912
2026-02-27 03:11:40 +00:00
drh 5dfccfb0bb Fix typos in the SQLITE_PREPARE_FROM_DDL documentation.
FossilOrigin-Name: 9174e7dacf867c1f80e73ccf5e0ee7b9e84fbe9f2e53559d06b72206b1cde3c6
2026-02-26 16:19:21 +00:00
drh 5a6eef2fb2 Improvements to the SQLITE_PREPARE_FROM_DDL documentation.
FossilOrigin-Name: c3288f16848866a2c846221c33631785f7b39938078bb95c61895f789395aa1d
2026-02-26 13:35:38 +00:00
drh 8bc78bebfb Fix an inconsequential typo in the output of the datedebug() SQL function.
FossilOrigin-Name: 3a92a53d8cf77822c22618dbf35851a20d27c4a659d23db2d24e7ccb3f61c7fc
2026-02-26 12:17:19 +00:00
dan a86c6017e7 Fix a problem where the wrong collation could be used as part of a row-value comparison between columns with different collation sequences. [forum:/forumpost/6ceca07fc3 | Forum post 6ceca07fc3].
FossilOrigin-Name: 212c68249cc0e8904fc36d8e90646d04604f2b3b9d4c32eaeac7d920fa0fba99
2026-02-26 11:03:22 +00:00
drh 1a152ea92a The GCC bug is fixed in GCC-15.
FossilOrigin-Name: 21f9abe6c9d1e2ce99ed75c2984103dcd2356b5d53bca3f3732176bb856bf3f7
2026-02-25 21:19:20 +00:00
drh f45c695471 Reinstant the work-around for [https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96270|GCC compiler bug 96270]
which is apparently still an issue as of gcc 13.3.0 with -m32 on Mint Linux.
The test case in the bug report is fixed, but the vdbeMemRenderNum() routine
in SQLite still shows the problem.

FossilOrigin-Name: 7fae321095ebec775de1b39b974fb1f295d5efbd5723e9cb29a981c97a4262c8
2026-02-25 12:44:19 +00:00
drh 46bf39e396 Use a 64-bit hash for the testing option SQLITE_CHECK_PAGES to maintain
page alignment.

FossilOrigin-Name: 75a35cd8f97f8a6cd2311c829685bd3b1cfb394de15174aef347f15cfb1913a1
2026-02-25 01:04:22 +00:00
drh b1a73ba34d Adjust the changes to shell8.test from [5da9bf09cc00faf9] so that they
work with older versions of TCL.

FossilOrigin-Name: c3b9e2cee3565da50bcc92766f700c31142391a697cbab7465774cf20a307dd8
2026-02-24 19:50:27 +00:00
drh 9d65290207 Rename SQLITE_ENABLE_MULTITHREADED_CHECKS to SQLITE_THREAD_MISUSE_WARNINGS.
Run test cases with that option. Also add the
SQLITE_THREAD_MISUSE_ABORT option.

FossilOrigin-Name: be8c8b9cb7b618a1571a988bc1cfdc15d99a8bf144d699385c0076e940b9f7f5
2026-02-24 19:37:16 +00:00
drh da67225f04 Fix an assert() that is incorrect when the SQLITE_DBCONFIG_FP_DIGITS setting
is on a non-default value.

FossilOrigin-Name: 3ca1ed81c4fa41f5f9fdbebf0929dd8421a4e29f95764fe1027d4d8706a41480
2026-02-23 19:51:54 +00:00
drh d2e27e5d54 Add test cases for the symlink defenses of the previous check-in.
FossilOrigin-Name: 5da9bf09cc00faf98cc515fb5a10a0af325b8f7608893808d031dfef62380be2
2026-02-23 13:29:29 +00:00
drh ffd9892a02 Change the SQLAR archive extraction algorithm in the CLI so that it
uses the newly enhanced realpath() SQL function to guard against
attacks that use symlinks to try to write files outside of the
destination directory.
[forum:/forumpost/641b09daa17d9086|Forum post 641b09daa17d9086].

FossilOrigin-Name: 7cced53e8c508fbf1816162c5358c77a712f76a38fd18f07171efc3c028a3c57
2026-02-23 12:19:05 +00:00
drh 9235f9b049 Enhance the realpath() SQL function in the fileio.c extension
so that it works ever for pathnames that do not exist.

FossilOrigin-Name: 27a5735fb1e194d763ab9fdb933fad4f694fb2f8ad19205d17ac81caebd82548
2026-02-23 11:44:30 +00:00
drh 9dac613f04 When doing an SQLAR archive extraction in the CLI, postpone creating symlinks until after
all files and directories have been created.  This prevents a hostile archive from
creating a symlink through which it can subsequently write content outside of the target
directory.  [forum:forumpost/9e176adfef91c207|Forum post 9e176adfef91c207].

FossilOrigin-Name: 9719034d4d3becda127dc294f7f58ded9c982509c690dd55b56310912957eb51
2026-02-23 01:34:14 +00:00
drh 95fe5b9c38 Improved implementation of realpath() in the fileio.c extension that does not require
the last element of the path to actually exist.

FossilOrigin-Name: 4df4999484d9008d8af3c9c340810e0cf5f57161ba053ed5501276b450577039
2026-02-23 00:57:00 +00:00
drh 9bc017423c Add the realpath() SQL function to the fileio.c extension. And clean up
the UTF8 handling on the Windows side of that extension while we are at it.

FossilOrigin-Name: 8bb8941930378b436f1353603be194644568b55fe347475be0caddddad40efa3
2026-02-22 20:44:52 +00:00
drh 60f132bf12 Fix typos and include wording in the documentation for
SQLITE_DBCONFIG_FP_DIGITS.

FossilOrigin-Name: c30cee1224904eb2092daa0ad9494aec7c7a8c6c1661f5b91b62c3ef0c5ea95b
2026-02-21 21:19:41 +00:00
drh f8681fb638 Fix a harmless warning in the decimal extension.
FossilOrigin-Name: f16c7c4000f2b992f7245dcaf669f13ed464579f8894f3c16842eebf41285c70
2026-02-21 20:20:06 +00:00
drh 283f33233b Improved rounding of double→text conversions for rounding digits
of 17.  Add the new sqlite3_db_config(), SQLITE_DBCONFIG_FP_DIGITS,
to specify the number of significant digits to preserve on conversions.

FossilOrigin-Name: 78c12b4f6ae869c3ea5e368c97b45991442bdd2d3a60935f1e6fc02a8e01cd41
2026-02-21 20:03:47 +00:00
drh 8642ca2025 New sqlite3_db_config(SQLITE_DBCONFIG_FP_DIGITS) that let's the application
specify the number of significant digits that double→text conversions
will attempt to preserve.

FossilOrigin-Name: 3f16985dcc47a366b54164c5024f920a79dddd76faeac5e36b4770732ed72c0a
2026-02-21 19:26:58 +00:00
drh 699959ff25 New test cases for floating-point conversions.
FossilOrigin-Name: 3033fe97b14ba0531278d4aa444bc5340e044b87a72b3a4341032ddee442000f
2026-02-21 13:57:40 +00:00
drh 7aba858597 Remove an unreachable branch.
FossilOrigin-Name: 93f90eacc0c5b2ae0042ec525359298883f8473e24967208feef4029d9fa2d08
2026-02-21 13:40:07 +00:00
drh 4428aa842c Increase the precision of double → text conversions that happen in
sqlite3_column_text(), or sqlite3_value_text(), or in the CAST() operator,
or similar, so that round-tripping the value back to double results in
exactly the same value.

FossilOrigin-Name: ef0049e3ade304f64e1ab97e52e1e21379e4db461f3fa80dac8e544fb90d622f
2026-02-21 10:57:02 +00:00
drh 24ef9fbb19 Further simplification, performance gain, and size reduction in
sqlite3FpDecode().

FossilOrigin-Name: 5954f22ceb6b43ed160ec085cd86b66299d8aab3e349a41941a4f1c2eade7457
2026-02-21 00:56:11 +00:00
drh aec283385e Small performance increase and size reduction in sqlite3FpDecode() by using
local variables instead of structure elements.

FossilOrigin-Name: b4c378bba582205aa676e45b21ffa17ad6199e2a017ec73cf41a0243f693b589
2026-02-21 00:20:29 +00:00
drh ac5ee556eb Inconsequential changes to floating-point conversion, amounting to mere
code cleanup to aid comprehension.

FossilOrigin-Name: 6d9c29123b6b143b0f7c8f5d018f170c72edfc5b1a4d67edd45e5552def2af6c
2026-02-20 20:43:25 +00:00
drh 62546d4b1f In the decimal extension, an optional second argument to the
decimal() and decimal_exp() functions rounds the value to the
number of significant digits specified by that argument.

FossilOrigin-Name: cb24edf1afc3f9083a4963c5fe232933eccc7c0cb8872aa5fcd336d226b885ef
2026-02-20 16:21:58 +00:00
stephan 9e395efec6 Fix a cse in the fiddle.debug build where it could not overwrite its read-only/generated index.html.
FossilOrigin-Name: 7adb2c0f438a97d377760436b1ac81fffab36e541a2b5ee733bfc7108e0180e5
2026-02-20 14:22:09 +00:00
drh d447df4205 Add new functions ieee754_to_int() and ieee754_from_int() to the
ieee754 extension.  No changes to the core SQLite.

FossilOrigin-Name: 8660d3e94cbe892693554df282bce0fa8c7aedbc5e020cab647cbbff3d7e55b7
2026-02-20 14:20:33 +00:00
drh 20c65785cd Improved rounding of floating-point conversions in some corner cases.
FossilOrigin-Name: 436dccd42650a6ab0a74459ae42fdb3e36d853f934324acc2bc961f87c001852
2026-02-20 12:27:52 +00:00
drh e8750019c2 In the CLI, the ".quit" command in a --cmd argument causes an immediate
exit.  [forum:/forumpost/f057122b68|Forum post f057122b68].

FossilOrigin-Name: 4731cc8856aa4dcfad5db501648bfd0afef43d8c5f339e7b11a1e3ee64348b87
2026-02-19 17:20:40 +00:00
drh cac09127e1 Improvements to rounding behavior in the new floating-point conversion logic.
FossilOrigin-Name: e540f6c370675ae043af8cdbb80f7eb17c08e50f7634e0b78f0b1dccf7bd4b18
2026-02-19 12:59:42 +00:00
drh 37ee141d25 Restore the CLI function dtostr() back to its original purpose of converting
floating point values to decimal using the host computer's C-library.

FossilOrigin-Name: f3cc05ba7fa9cb77573de5cd28ab90f10844567692ed57a706c5dc35b1348009
2026-02-17 20:28:25 +00:00
drh 7549e784a8 Minor fixes to tmstmpvfs.c.
FossilOrigin-Name: 091bd90ed5a6a326fd532e93bd17903e052476da14dc6aee2599a9acf75a3276
2026-02-17 14:28:18 +00:00
drh 8c3e8dce41 Increase the number of floating-point precision test cases by 9x, and
increase the require precision to 16 digits.

FossilOrigin-Name: 62c2cf74db5e47f975d1f47818f82f060ddecad4b68c4d4dbc6190cc9a8fc397
2026-02-17 14:02:27 +00:00
drh 31d4e56b29 Fix a harmless compiler warning from MSVC.
FossilOrigin-Name: 12f3a6d0aaff8818c427d79e2b9394328625b3a60ccaa7b628d064084c65cfaf
2026-02-17 12:23:12 +00:00
drh 530f536ce7 New assert()s for the return value from sqlite3SchemaToIndex().
FossilOrigin-Name: 2610105a439e25c050b2deb32953861187c81b1d97407f41dc188e6627e0ac4d
2026-02-17 11:28:48 +00:00
drh eb8c4c5742 Further simplifications, performance improvements, and size reductions
to the numeric to text conversion logic.

FossilOrigin-Name: dd5af703e1082951a4295a3453611db12b23cfbcfee4258ec3985abe96ab54ba
2026-02-17 01:04:23 +00:00
drh 0b5e944480 Faster conversion of integers to text, by converting two digits at a time.
FossilOrigin-Name: 90300e02f0de45b57d53976cae72928b3e56532d90b8b8cdc988e14762777930
2026-02-17 00:37:12 +00:00
drh 63338e3ba8 Simplifications to sqlite3AtoF(), resulting in slightly faster performance
and slightly smaller size.

FossilOrigin-Name: 6b9ab641763bc1620a01beec71589506b3b0827bff1c6cd98a5d17c6070050b3
2026-02-16 23:52:51 +00:00
drh 6315919ba8 Fix a bad assert() in the previous check-in.
FossilOrigin-Name: a4540582b5b704af13b570b3dc609dbacada719302372a038cf74eee3688d5e7
2026-02-16 17:22:17 +00:00
drh 5c977eaa0a Refactor the sqlite3AtoF() routine so that it requires a zero-terminated
UTF-8 input.  When the need arises to convert UTF16 or non-terminated
strings, wrapper functions are used.  Together, this makes the code slightly
smaller and faster.

FossilOrigin-Name: 67c7c72e9bdf04c920d77006538a202c923fa74b47c81dc3014c2929dac7277d
2026-02-16 16:56:59 +00:00
dan 1105c33900 Fix another case where SQLite might fail to identify an ON clause on a RIGHT or FULL join that refers to FROM-clause elements to its right.
FossilOrigin-Name: e956b36063e77b5ad0d8b8afb5dc942665f570d762929ff277e320c06ded8ce6
2026-02-16 16:56:25 +00:00
dan 7b93f1a387 Fix a problem with identifying ON clauses attached to RIGHT or FULL JOINs (which is an error) if the join appeared in a flattened sub-query.
FossilOrigin-Name: cf2dc6dfad275dad8fef763a57baaaf6301b0d3bf1916be90f22200cbe0115d0
2026-02-16 11:59:14 +00:00
drh c47bc0ffca Performance improvement in floating-point conversions.
FossilOrigin-Name: b5ebbd004183f81902fa79a143222204b33dbe1cacb918194556b8dac67bd567
2026-02-16 11:14:59 +00:00
drh adbb756155 Avoid the big power-of-ten lookup table.
FossilOrigin-Name: 245ac7d9ec61e14fcef13a731e290fc5e8979efef6f4345f875eab9b882b0713
2026-02-15 18:41:29 +00:00
drh 658908a32f Fix to [9d3a12d3926d55ef] to force the static const string literal to
have two-byte alignment.  Necessary to get that patch to work in Mac-ARM.

FossilOrigin-Name: a80c847169f307a4d2e524b6732ea557bf7f312f1a319d6c6c2e81adb0907596
2026-02-14 22:04:53 +00:00
drh b3b2695d4b Add the "LLU" suffix to 64-bit integer constants, for portability to older
compilers.

FossilOrigin-Name: 4bf08110662c4b8db9ee90ce34271890a62a469166ea292f7f782b84236531d4
2026-02-14 21:47:08 +00:00
drh 775e4b7073 Fix to the generic version of the 128-bit multiple subroutine.
FossilOrigin-Name: 9c3d09ee75beb26447161c0bf5ea252520a6ec051774f1a4e9f18204281d504f
2026-02-14 21:31:31 +00:00
drh 4bee65dcb9 Another minor performance refinement.
FossilOrigin-Name: 9d3a12d3926d55efffdc84bff342bd1dbccd08426104aeb2d339b064bf6f02f4
2026-02-14 20:48:58 +00:00
drh 4d0611c1eb Minor corrections.
FossilOrigin-Name: 93fdee20021b8cc2a113ea042fedbccd8b53b6aceeeb34c0810e6d3a5f106f07
2026-02-14 19:42:55 +00:00
drh 10fb1e3077 Improved comment. No code changes.
FossilOrigin-Name: c5a4da1bf7af8f8ec4b3273f88b09c01ce90986c8a5ce3d03319bd094706c411
2026-02-14 19:36:41 +00:00
drh db84cdb747 Use a precomputed table of 696 powers of 10 to improve the performance
of floating-point to decimal and decimal to floating-point conversions.

FossilOrigin-Name: 64d9b7becf2da8927024adda278e837a46837e46af3cff51bd54a183d1716750
2026-02-14 17:58:46 +00:00
drh 990d861038 Add new assert() statements to demonstrate that the complaint given in
[forum:/forumpost/56ad8b632c|forum post 56ad8b632c] is not a real bug.

FossilOrigin-Name: d62999907d5f5987fe0030e1a4a7144c898e55595ac116eec966741a5099322b
2026-02-13 16:02:27 +00:00
drh 23e57bc7cd Use only a single bit, rather than a whole byte, to store infrequently
accessed boolean values in the Parse object.

FossilOrigin-Name: e1bcd7e1cf6d6b1add49deac65e1b64bcae52fbbad094e561846bb92959db76a
2026-02-13 12:20:57 +00:00
drh 7d49eb1dff Add a "const" to avoid a harmless false-positive compiler warning.
FossilOrigin-Name: d44af6cc093ab6b5a196249a84681148bbae3f624abf4bc8cc42c88f3f7320b2
2026-02-13 11:40:28 +00:00
drh df59c5da78 Respect "const" strings in the sqlite3ShadowTableName() function.
Patch moved to trunk, where it was intended.

FossilOrigin-Name: 51b5afdfac401cbf74af6383d4559439cb6defe2ce23ebcf7ed9a5b526a07b51
2026-02-12 18:15:26 +00:00
drh 643fb6cf4b Minor performance enhancement to floating-point rendering.
FossilOrigin-Name: 38d8c0d8a0b0e9990ba7bdcce979f2824ffee22a083cb788a75917628b1eb559
2026-02-11 19:42:46 +00:00
drh c652e34317 Correction to date/time computations associated with the timestamp-vfs.
FossilOrigin-Name: b67889e4f17c3280f839ee7045256cc47d6ce3ed60d880925e3d30f9ebbcf3ff
2026-02-10 19:33:11 +00:00
drh c94f97fac7 Minor tweaks to the star-query optimization in the query planner.
FossilOrigin-Name: 5021e114b251467fee6a39749c88ec581ee1f7d50093b02a44dc300db0c0a1c5
2026-02-10 18:28:41 +00:00
drh 1f9ae1f780 Immediately reset all pending prepared statements on any call to
sqlite3_set_authorizer(), even if the authorizer is being disabled.

FossilOrigin-Name: 5c0468acd1a12f7fd01b9974250fb42c5811939dc2319729a1ef93c073dc0071
2026-02-09 19:58:04 +00:00
dan 612bee531f Do not allow table-function argument expressions on the RHS of an OUTER join to refer to FROM clause elements to their right.
FossilOrigin-Name: 021e48e28931afd781e76db6293e60fd2ffb8377cb1051e8b802f2743a10fa5d
2026-02-09 14:45:17 +00:00
stephan de702aebb9 Fix a method name typo in test-only code which could hide the triggering error condition.
FossilOrigin-Name: b2c1a4184c3de5a26524906ba04ce6c9c80b27f8d89c40fde7bbc8f5a9f99e34
2026-02-08 12:13:19 +00:00
stephan e45418c5f5 JS doc touchups.
FossilOrigin-Name: dcd22fd6d135892852358c8cfb5049dce4a1410dbc0f28cafc8b340f69350165
2026-02-08 05:08:30 +00:00
drh a0857f321c Improved output from calls to sqlite3ShowSrcList() made from a symbolic debugger.
FossilOrigin-Name: 241d23f8d36a0d4d0bcf058c5c1ff6144cfe7dd33460e8d94de5f0daf56aaf50
2026-02-07 13:41:05 +00:00
drh 450a90097f Do a better job of not generating unnecessary Bloom filters associated
with IN operators.

FossilOrigin-Name: 3861df5c127f82becea80c48c547de8ed0d01eebcf6a6a9d7f234dde31f50394
2026-02-05 21:14:34 +00:00
drh e58c1a987c Fix harmless compiler warning when the CLI is compiled with
SQLITE_OMIT_PROGRESS_CALLBACK.

FossilOrigin-Name: d6150c813ce43e15285a07d24135b10a7c254b3c175dee4f61c3b6a6ce91c29f
2026-02-05 13:03:04 +00:00
drh d0c4a5853b Display scanstatus results in neat columns.
FossilOrigin-Name: c476d956d0bd3065cf894de6f9d393b999ff7d2268a35f01a6d88804789ab58f
2026-02-04 20:51:27 +00:00
drh b55d6f3754 Improved output for ".scanstatus est" in the CLI.
FossilOrigin-Name: e6902937ecdbeb449986469859b46631272fb0a9e7e1c31adea14cff072b6d67
2026-02-04 18:10:49 +00:00
drh c9ccd27045 Improved display of scan-status estimates.
FossilOrigin-Name: 70ef095740c62081090147bb1900c843cb31864e93a8af518ff48855f1175717
2026-02-04 17:52:50 +00:00
drh 5e842d0ece Render scan-status values with 3 or 4 significant digits only, for
improved readability.

FossilOrigin-Name: d9102ff0c43a496e782b0a3ec86ba7cd45f8118a07f15a4f3d58b399466d6cf2
2026-02-04 13:35:32 +00:00
drh 3fab5d2a40 Fix scanstatus output in QRF so that it shows the total number of cycles
on the "QUERY PLAN" line.

FossilOrigin-Name: 611cef0ec281ae06eac97df8048d617fd445c348c2fc1673202b9cc081a365db
2026-02-04 11:51:20 +00:00
drh 761118497a Improve the query planner by reducing the estimated number of output rows
for each stage of a join if there are LIKE, GLOB, REGEXP, or MATCH
constraints on the table where the pattern string is a constant.  The
reduction in the output row estimate is proportional to the length of
the pattern string.

FossilOrigin-Name: 6852843e6aa23051de0884593ce64b96a16f16c4d63b222bd324297581f7cf75
2026-02-03 17:21:59 +00:00
drh e54142009c Minor change to test/dotcmd01.test so that it works with STAT4.
FossilOrigin-Name: 8aaf5b6b6c3d71789eebd2b8f440dd2fdaad2ef2adc9c7b2b019f08853d0b195
2026-02-03 17:07:28 +00:00
drh 1131eddbf9 Improved comments on this enhancement. No code changes.
FossilOrigin-Name: 510a6738e5d186a74a45fa7385f4722f4e090d85562f259b02c49393ca56860b
2026-02-03 16:29:14 +00:00
drh 3212ab33d4 Refactor the implementation of this enhancement to keep all the code inside
the query planner, not leaking out into SQL function implementations.
Expand the enhancement to cover MATCH and REGEXP operators and overloads
of LIKE and GLOB.

FossilOrigin-Name: 96f8ce225ee863c6dfe0df2d10da27a35407128d7c4691ddb927401465047a6e
2026-02-03 14:00:28 +00:00
drh 49c70b88bc When reducing the truth probability of a LIKE/GLOB constraint, only consider
non-wildcard characters in the pattern.

FossilOrigin-Name: 2ba8fc290dbc80d159a217d745d8bd62371c305443d5aed10bfcf34cc98d3985
2026-02-02 20:54:28 +00:00
drh 77c5456d7f Do not reduce the estimated truth probability of LIKE and GLOB operators
by quite so much.  Fix a single test case whose output changed.

FossilOrigin-Name: fdebbedbd9a99165359761106cdc76d327f74043b22806b7bd96402fbed36f14
2026-02-02 16:03:24 +00:00
drh 5b23d027cf Merge the latest trunk enhancements into the prune-by-like-glob branch.
FossilOrigin-Name: d9d3d3a807710957f35b023ddc89cb80eea529f654ce3aa0f9aafe6c15b3ce88
2026-02-02 14:24:47 +00:00
dan 71848d362d Update session module test code to correctly use Tcl_Size instead of int.
FossilOrigin-Name: 619060c2267a889089bf5328ba83640806688164a096df8582a5cf91b173741d
2026-02-02 11:42:19 +00:00
drh 1ac0d9bbf9 Add the "--timeout SECONDS" option to the ".progress" command in the CLI.
FossilOrigin-Name: 8eb5c88aeb37279f68f5bb0b268339a8f6fb0b927b1e398ac4c83bf42c281273
2026-02-01 21:10:04 +00:00
drh 213262fa53 Enhance the ".timer" command in the CLI to accept the "once" argument, and
so that it leaves its last real-time result in the $TIMER variable.  Also
fix a harmless warning from an earlier check-in.

FossilOrigin-Name: 8ad7dffc261cbb3f7c9b4d38b04282c805c1c047fb431076f85e577c81f21574
2026-02-01 15:58:37 +00:00
drh 5c8cd31246 An experimental query-planner change that reduces the estimated number
of output rows for FROM clause terms that are restricted by a LIKE or
GLOB operator, based on the number of bytes in the pattern.  The idea is
that longer patterns will match fewer records and hence should reduce the
estimated output count.  The implementation is not workable as it stands
now.  This is just a crazy idea, saved for future reference.

FossilOrigin-Name: 97bcb56a208af0687750a6438981bbfe36a8d516d996178e62551d302ef811ee
2026-02-01 00:37:04 +00:00
drh 1d8eb1d9b6 Fix a missed optimization opportunity due to a typo in check-in [898bfa1afd8260ea].
Reported by [forum:/forumpost/2026-01-31T00:49:53z|forum post 2026-01-31T00:49:53z].

FossilOrigin-Name: 3a4f9a323da90611d7eda51b90cb058175ddde0a128e1ff00ce58cc83af0f376
2026-01-31 02:17:55 +00:00
drh 4fd4e8f066 Make CROSS JOIN a join reorder barrier again, as the SQLite documentation says it is. It
mistakenly stopped being a join barrier with check-in [c1ea064948ba08c4].

FossilOrigin-Name: d294106ecb78d765305ab740007f4a678e28baa13ae403fe57ea9cbfc259620f
2026-01-30 17:10:22 +00:00
drh c79b1f7d6c Fix the new CLI so that the ".eqp" setting does not affect the output of other dot
commands such as .schema or .tables.

FossilOrigin-Name: a5a1b69b3feedb507ecfe2e9ddc039d7c01e1bb54e3e881df5010481886cdef9
2026-01-30 16:03:22 +00:00
drh 4b4df2ee1e Change the new escape-character mechanism for CSV import so that there are
separate options for an escape character for quoted and unquoted fields of
the CSV.

FossilOrigin-Name: 293ec20e29e46d822a98302f8a0ba52ad6f74a7e93f780536d393730c2edbf61
2026-01-30 13:11:20 +00:00
drh 1d1cd36428 Undo the previous check-in. In its place, add the --escape option to
the ".import" command.

FossilOrigin-Name: f11a05d52a1eb933f394d48c0d4ee0353f07ad725b7378c40afd01632e67f72d
2026-01-30 12:27:13 +00:00
drh a5cc01681e Enhance the CSV import capability in the CLI so that it understands
backslash-escaped double-quotes.

FossilOrigin-Name: 7ba0594d3c24531bc014dd029093b0503118a29cfdd13af2a2ebdd456855ce94
2026-01-30 12:15:25 +00:00
stephan 561e28b75b kvvfs fix for [https://github.com/sqlite/sqlite-wasm/issues/146|npm ticket #146]: use of a test-mode-only symbol in non-test runs leads to a null deref in xFileControl().
FossilOrigin-Name: 407724c4e80efdf93d885e95b5209a100a3f470fe0298138be57201f65f9817e
2026-01-30 06:37:34 +00:00
stephan bfb33d476b Capture sqlite3_rsync's remote-end result code so the local side can exit with non-0 if, e.g., the remote sqlite3_rsync binary is found but fails to start. [forum:43eb1cd1c3979817|Confirmation received] that it resolves the motivating problem report.
FossilOrigin-Name: 971d51374e3bf0d0e0b106750dc1e499d1fdbd3233cf8264a534138b27f8d0a1
2026-01-28 17:25:18 +00:00
drh fabd481c76 Improved (faster) bytecode for the merge algorithm.
FossilOrigin-Name: 0b7f4b97e68f17ae2fec86017cdb170f3b318a71cda34b5c82ec2d9df780564f
2026-01-28 16:59:34 +00:00
dan 2c49785000 Add a comment to sqlite3Stat4Value() explaining that it read a few bytes past the end of the specified buffer.
FossilOrigin-Name: b95644eafdd42293096a3760af8b2110f3c7d83feecdeff5ff9f008d9748e874
2026-01-28 10:52:53 +00:00
drh 60229c2971 Avoid unsigned integer overflow when evaluating an array index in a
JSON path expression.
[forum:/forumpost/2026-01-27T14:18:49z|Forum post 2026-01-27T14:18:49z].

FossilOrigin-Name: 631c8d44cd1624ddc45babd3f0d810908c2a368784744262042c63506ff2e333
2026-01-27 23:33:42 +00:00
drh 836231691e Improved byte-code coverage testing verification in the merge algorithm.
FossilOrigin-Name: e49ec0e24b4765a6a819f3e980b680b9604f6db21aa8112ec4e9b54d82220c00
2026-01-27 22:27:14 +00:00
drh c4e7964fd6 The tmstmpvfs.c extension should not modify the content of pages going into
the WAL file, as that would corrupt the page checksum used for recovery.
Instead, only insert timestamp information as content is written into the
database file.

FossilOrigin-Name: dabaeeb1dab82eaea2449658b9c4c74af0058e5a64a076dab046902c6efac292
2026-01-27 16:18:35 +00:00
stephan f5d29d6424 Teach sqlite3_rsync to increment its error count when a child process fails, based on [forum:8fe404e547faa42e|forum post 8fe404e547faa42e]. This passes basic sanity tests but requires more testing and needs a review of the final 'else' block in the new code.
FossilOrigin-Name: 8be55d405f4ce424760150a2b59b5f663041fdcb377d1b9893b4f4e8e1940246
2026-01-27 16:15:22 +00:00
drh b4142c491e For CSV import in the CLI, when the table is created automatically
because it does not previously exist, make the column types "ANY"
instead of "TEXT" so that they will automatically adjust to different
datatypes in the input text.

FossilOrigin-Name: aff74e71ea734e1ad8fa893652885fd8c6c432fb60e92913a799e6df5e980b47
2026-01-27 14:59:08 +00:00
drh a943c0e973 Add the --csv option to the showtmlog utility program.
FossilOrigin-Name: 2e96e8efa9a331b0f7d928c55e7d85be64e7c60c85032c1e972f77fe790bdec4
2026-01-27 14:27:19 +00:00
dan 6f8f9fbe25 Fix trivial buffer overreads in the sessions module that could occur when parsing changeset blobs.
FossilOrigin-Name: 661878a62870023f7f54e8c591a0823dc457cb89780ab40c1891fb3d5e8f095f
2026-01-27 14:00:59 +00:00
drh 87efa1bd79 Additional usage notes added to the header comment of the tmstmpvfs.c
source file.  No code changes.

FossilOrigin-Name: d1b8e7740bee13a8cf199c6477ee20a4f8bcbbd9ec4096bcdc05a996fadf0d56
2026-01-27 12:50:52 +00:00
drh 389926bd3c Enhanced documentation in the header comment of tmstmpvfs.c.
FossilOrigin-Name: 3d37da3cb5943a61f528e3002c4c3ac3d41e871d742d8e603bffcc4bc5bd42fd
2026-01-27 12:02:02 +00:00
drh f202df5857 Fix frame number computation in the log generated by tmstmpvfs.c.
FossilOrigin-Name: 57fda9dd85f0de8c5d4a4d8015e67982d73ab96ad6f1cd590a48d8995dc28140
2026-01-27 11:19:49 +00:00
drh e17a469d74 Enhancements to tmstmpvfs.c: (1) Use the exact same timestamp on logfile
entries as on the pages that control, where appropriate.  (2) Include the
WAL frame number in ELOG_CKPT_PAGE logfile entries.

FossilOrigin-Name: fcf16297850abf5186b0d414b4f31d4b515925ef5cca5e7d7d84984c26160d00
2026-01-26 21:20:28 +00:00
dan 8f03e92c41 Add test for fts3 compress= and uncompress= options.
FossilOrigin-Name: bace9de67db426017d98f31685da1fc6ab10c7755681a41ca978cc97cb2ad5df
2026-01-26 20:17:10 +00:00
drh 583644e660 Improve the comments on the generated byte-code for the merge algorithm.
FossilOrigin-Name: 4468c9e1a262d3d7efd8edff19e22e13b64ebed606ce2f18c9cabce93ef257d3
2026-01-26 19:37:57 +00:00
drh 15357aaf10 Fix the initialization of the sqlite3Api[] constant to conform
with the previous check-in.

FossilOrigin-Name: 0a31a740843404958f61798aa17f94b5162bcf467c1c863d745346a316160d13
2026-01-26 13:54:05 +00:00
drh 49bf78685e Add alternative sqlite3_carray_bind_v2() interface.
FossilOrigin-Name: 925384d7711867428644595c0903682e6e365eb974c57e6680b51d511c980054
2026-01-26 13:44:42 +00:00
stephan 840199ee18 Fix speedtest1.wasm build regression introduced with Emscripten 5.0.0.
FossilOrigin-Name: 4733d351ec2376291f093ba8d2ba71d82c6f100c68dc860eee0532986c154e71
2026-01-26 10:53:24 +00:00
drh 713ee08d33 Fix a possible problem following OOM in a WITH statement.
FossilOrigin-Name: c38797d400dc4135108a6e303f2c10779ba9adf0daae28f1789f9106a68d21d5
2026-01-26 00:11:01 +00:00
drh 9b7005feea Always use the sort-and-merge algorithm for EXCEPT, INTERSECT, and UNION,
even if there is no ORDER BY clause.

FossilOrigin-Name: 132ba781b031972c6ac564ff0d1d6b2dfaeee77fe503ad6e9e8647bb5ba28bd3
2026-01-25 23:28:42 +00:00
drh 33db5b73e5 The Select.addrOpenEphm field is no longer needed. Remove it and
simplify related code.

FossilOrigin-Name: 1fe5a1ca72892987716b71b9419265f25c8c426e52320acf56724fd64b838be6
2026-01-25 20:15:00 +00:00
drh 3b440ce573 Refactor one function name. Comment improvements. No functional changes.
FossilOrigin-Name: 5b4c0a5e8609e16a4b08b7f95d55e513ee4300b22d044ac8f42e60b96cff5be6
2026-01-25 18:21:53 +00:00
drh 8e272dd9a7 Fix the jsonb_group_array() and jsonb_group_object() functions so that they return
JSONB as they should, and not text JSON, when the array or object is empty.
[forum:/forumpost/2026-01-25T00:47:06z|Forum post 2026-01-25T00:47:06z].

FossilOrigin-Name: 8b53b97833afe27c0c3782c5fbc0437976215b571579f73a94c33e28d3fedb41
2026-01-25 15:18:31 +00:00
drh 46a395ff3d Additional simplifications to achieve 100% MC/DC.
FossilOrigin-Name: 781b7ec0bcaa5d94e8c5fd31b4fe0203393d5209a5f518848f057ec3d2b8d7ac
2026-01-25 13:26:19 +00:00
drh 619f1f7083 Fix harmless compiler warnings in FTS5.
FossilOrigin-Name: 29d9132a768e0460e22e41abb8b6188181ba0fc6c4665161d3d0af28d7583e19
2026-01-25 11:56:25 +00:00
drh 84075044de Remove additional unnecessary code associated with the setup phase of a recursive CTE.
FossilOrigin-Name: 82735513762c8adfe7ad4865dcb76c625cbd8f6976bb98f2c342cf98cfdf3b36
2026-01-25 11:41:49 +00:00
drh fb288a432c Assert() statements added to sqlite3_result_text64() and
sqlite3_bind_text64() to detect misuse of SQLITE_UTF8_ZT that can
result in undefined behavior.

FossilOrigin-Name: e761fcc6838aee1fa5e098ad167866845a8ee140a3346fb089a1238abb58d998
2026-01-24 17:18:56 +00:00
drh 40b6eace0e Expound upon the meaning of SQLITE_UTF8_ZT, in an attempt to head off
confusion.

FossilOrigin-Name: 9a74d0c3e78eef9a32775f445ecb3fae76e2f3886b9173a16971d7c7506967fe
2026-01-24 12:19:00 +00:00
stephan a5070d083c Expose SQLITE_UTF_ZT to WASM for completeness's sake, but the WASM build does not expose sqlite3_bind_text64() or sqlite3_result_text64() because it only supports UTF8 encoding, so ZT does not currently have a genuine use there.
FossilOrigin-Name: d931831ce9d2dea3730f33afc046e159ad886842114831a6bbe56d5b746ab0af
2026-01-24 01:17:06 +00:00
drh ca77c585be Fix an incorrect assert().
FossilOrigin-Name: b9ce349e1e4166c3d1c97d2f6fb1792b89e3316c8ec182daa032b6655e9c2c17
2026-01-24 00:33:41 +00:00
drh e4445f04c5 In the cost metrics of the query planner, the estimated number of output rows
from an EXISTS-to-JOIN loop should not be more than 1.
[forum:/forumpost/989880d0aa|Forum post 989880d0aa].

FossilOrigin-Name: fc4e93a381dfb99dddfd2ef001bf56b92ab89ee3c7044c7ce13ad32f38b3f418
2026-01-23 20:46:54 +00:00
drh 4b736ce2f8 More code simplifications.
FossilOrigin-Name: 3d76732a45333f6f3e21667bb018a859d20815243d276fad0dc126d3c667d4aa
2026-01-23 18:52:45 +00:00
drh 447fd1bf29 Fix a minor EXPLAIN QUERY PLAN formatting problem. Remove a NEVER() that is
reachable.

FossilOrigin-Name: 423173079ee0911f86a76c9b905d98328f7aae45d43a38a13957c5c510f87670
2026-01-23 17:06:54 +00:00
drh 1f567aef4c Remove code that is no longer used.
FossilOrigin-Name: 8d7f91e07b0c7573f94fddd64f39671e608969764615bd7f5c060e7ee7540bc6
2026-01-23 16:07:51 +00:00
drh 40573642e8 Continuing work toward using sort-and-merge for UNION.
FossilOrigin-Name: 94bf43634bbb1fd8e67235b8be0d9744cca1e364ab3cceecca4528e7b78780d1
2026-01-23 14:37:08 +00:00
drh c46074808e Merge the latest trunk enhancements into the union-by-merge branch.
FossilOrigin-Name: bdf4d6a9398266c98a0d4ad3c60b431cd498bfa80add3e81d717c687634c2602
2026-01-23 11:44:40 +00:00
drh 8fd91361e6 Add the SQLITE_UTF8_ZT value which can be the encoding argument to
sqlite3_result_text64() or sqlite3_bind_test64() to indicate that the
argument is UTF-8 text that is zero-terminated.

FossilOrigin-Name: cef0eed8054f3f760cf3142a897c7eead20869157da8b2a0ee9443733eecd371
2026-01-23 00:55:36 +00:00
drh cc11fb3bb2 Merge the latest trunk enhancements into the utf8-zt branch.
FossilOrigin-Name: 0dd1633d34a0c6a6d56e32ca3b05bc0aa55e47097f600d1fc9f00022a53681b3
2026-01-22 19:29:34 +00:00
drh cb3d6d14fe Enhance the [/info/e33da6d5dc964db8|EXISTS-to-JOIN optimization] so that
the inserted JOIN terms are not required to be on the inner-most loops,
as long as all dependencies for the EXISTS-to-JOIN loops are in outer
loops.  This addresses the performance concern of
[forum:/forumpost/2026-01-21T19:49:04z|forum post 2026-01-21T19:49:04z].
Test cases in TH3.

FossilOrigin-Name: 298d5c8fa6207afb6cdcca3b312a1eeddda0edeb6d840aa5476a7195047a2158
2026-01-22 19:02:32 +00:00
drh d3285e67a8 For the ".eqp full" show both the QUERY PLAN and the full byte code, just as
it has in the past.

FossilOrigin-Name: 756484fcaa7efed0691814f3affeb2449d3301c5c764530f2ec8e290caea567f
2026-01-22 15:59:45 +00:00
drh e0e67f1a73 Add a new encoding constant SQLITE_UTF8_ZT, which if used with
sqlite3_result_text64() or sqlite3_bind_text64() declares that the
string provided is UTF8 and zero-terminated at the length specified.

FossilOrigin-Name: 2d84ce88fed12766272f6b1293927eb4c7ce92da0334b09b4875ed2dfdd00ade
2026-01-21 19:24:07 +00:00
drh 2d98df1536 Add a new assert().
FossilOrigin-Name: bc450aba674a8934d718a88af3c2343555b43c2d8403bd5f6d14e177fce6bc9f
2026-01-21 11:29:20 +00:00
drh e702822708 Strive to use sort-and-merge for compounds in CTEs.
FossilOrigin-Name: 73ed1c790bb386a2981bf7076d23d37e09fa9db3966a7a164b8964fdb14fdad2
2026-01-20 20:10:41 +00:00
dan ca0b14a490 Update the shell tool to work with SQLITE_OMIT_AUTOINIT builds.
FossilOrigin-Name: 2b3b36da9d60c265dceec5964ea51c752d81f41459fb6849c8faea658b253552
2026-01-20 18:30:48 +00:00
drh e66806de33 Fix another test case to work with union-by-merge.
FossilOrigin-Name: 8279c10e2465838010508f6018aa6fdb0e07ca3639f65e727a03a4222573ea57
2026-01-20 18:20:26 +00:00
stephan 815905544d Tiny JS doc touchup.
FossilOrigin-Name: 6d9ec0a21a2b33a7bb3cf9c3677c9ef7a9738d318cd2d8159b33583b2bd4abec
2026-01-20 18:07:27 +00:00
stephan e822005684 wasm: make pre-js.c-pp.js a no-op in bundler-friendly builds.
FossilOrigin-Name: 982a91abc0c97e7e785d3ba69a4d0516a899f4d6cd462027ebdf7115e577e8c3
2026-01-20 18:01:56 +00:00
stephan bb56ddc4e3 wasm: a potential workaround for cases such as [forum:1eec339854c935bd|forum post 1eec339854c935bd], where client-local needs require overriding Emscripten config defaults (which we very intentionally do not document or support for reasons explained in that thread). This is untested because it requires a special-needs setup, but it doesn't break anything for run-of-the-mill cases.
FossilOrigin-Name: c8e6be9241e3e17898c451baad109b75dc8a97a91ff1e43c67a6c962b75af8f3
2026-01-20 17:10:22 +00:00
drh ec13fcecd2 Minor fixes. Tests are passing now.
FossilOrigin-Name: 305ae331f5e791941c12b04b727656ac8d80f7c5cb6343277b979a456eb3b834
2026-01-20 15:07:28 +00:00
stephan d4dc275ebc Add an SQLITE_API_ARMOR check to sqlite3_get_clientdata(), per request in [forum:a011a001f4|forum post a011a001f4].
FossilOrigin-Name: 346ad366a8ebed1e7936c59f8a40e9c8e7e31d0153bc4f654a47b2ddc39b18ca
2026-01-20 10:57:44 +00:00
stephan f4a613aa46 Minor doc updates in mkwasmbuilds.c.
FossilOrigin-Name: 81840bd4b2132c1c701bb6fd5a2ac48a4ed33d2b68a3f7a826fa32dc0fc16483
2026-01-19 22:20:12 +00:00
stephan d50161a3cd wasm: remove the bundler-friend builds from the 'all' target, as they're only needed for the 'npm' target.
FossilOrigin-Name: dfe4b788ca6596ea9f26445c864f56f81c7816798f5fedfc4e89ba1eb097b0aa
2026-01-19 22:01:10 +00:00
stephan 4eb307cf46 Minor doc touchups in the opfs-sahpool VFS. No functional changes.
FossilOrigin-Name: 67c0d11701842baaa5227ccce02483ead07bec8426b9861f2e2a0465414ae382
2026-01-19 21:57:32 +00:00
drh 0af76f6caa Be more aggressive about using a sort-and-merge to compute the UNION of
two SELECT statements.  Experimental branch.  This check-in is failing some
tests.

FossilOrigin-Name: e534af8a78f5458bd656f70eb9f68084c0ed3f97360f5a033fba3564849e970c
2026-01-19 19:51:03 +00:00
drh 8c4e315e19 Add two new JSON functions: json_array_insert() and jsonb_array_insert().
Patterned after the json_array_insert() function of MySQL.
[forum:/forumpost/2026-01-17T10:40:39z|Forum thread 2026-01-17T10:40:39z].

FossilOrigin-Name: c1e55132410a80d37bad30e4ddecc7a98fad92a68b436880324998af1cee37ae
2026-01-19 11:44:59 +00:00
drh 4720c16223 Raise an error if any PATH argument to json_array_insert() is not the path
of an array element.  Add some test cases for json_array_insert().

FossilOrigin-Name: 96b7d2c60a3f8a9f9d69313bad599a3a5ad335cb382263e4254e2d20929d5e55
2026-01-17 19:01:37 +00:00
drh 6f3a679a7d Add new SQL function: json_array_insert(). Suggested in
[forum:/forumpost/2026-01-17T10:40:39z|forum thread 2026-01-17T10:40:39z].
This check-in seems to work, but contains no test cases.  Also, no error
is raised if the PATH does not reference an array.  Prototype only.

FossilOrigin-Name: d6c82c8248f8b1ce3d2c01a92e7d1058beac2b2ccac3b122a2cd385c77dc76df
2026-01-17 14:23:28 +00:00
dan 4b9def1903 Fix further problems with deleting from corrupt fts5 databases in secure-delete mode.
FossilOrigin-Name: f8f89d2e10f06b54acf58a3b0543aef749c3d2b65670d4cc71530dea7997651e
2026-01-16 20:04:04 +00:00
drh 9dff6ac96e Try to prevent false positives in fuzzinvariants.c due to scalar subqueries.
FossilOrigin-Name: c20be882a62e75abc50fd2dcae26ebbe83a01f09369525392fa22223627a04ea
2026-01-16 17:48:46 +00:00
drh 4dd1a202de The optimization at [8bdda827a3d26800] is incorrect for the case where one
of the two expressions has been commuted such that the collating sequence
has changed.  This check-in fixes the problem.  Problem reported by
[forum:/forumpost/2026-01-16T11:35:28Z|forum post 2026-01-16T11:35:28Z].

FossilOrigin-Name: 9d8fb7a9e23d5ef76d05824401976247debe51be6f658f132609257a068a73de
2026-01-16 15:53:20 +00:00
drh cd8aee7946 Change the copyright on the tool/dbtotxt.c source code file to the
standard SQLite blessing.

FossilOrigin-Name: 766c89e009578aa5c1a92cbb2dcc34b3529cf4dd6157f07c84f08470d81942a9
2026-01-16 12:13:42 +00:00
dan 90942efd21 Fix an error in new test script fts5corrupt9.test.
FossilOrigin-Name: a5918835e172fd4eda6511d297b61368952d617eac33950b67368671e3bd40fe
2026-01-15 17:17:00 +00:00
dan bcc496e526 Add extra test for handling corrupt delta records in rbu.
FossilOrigin-Name: 7dba05100fbef3614e6ee8eff39dab7a4713550e5672c2f6cf8f4b490a96aa25
2026-01-15 17:00:58 +00:00
dan ce83fd80ad Fix a buffer overrun that could occur when deleting rows in secure-delete mode from a strategically corrupted fts5 database.
FossilOrigin-Name: 1e66d6e3276cc6aab4d002a1df13b044e61f3946322bf97cac06d98dbb68e433
2026-01-15 16:09:35 +00:00
drh 43eb1d4d22 Adjust the output of the fuzzcheck test program so that testrunner.tcl can
capture the number of tests run even when the --slice option is used.

FossilOrigin-Name: dddaeff4ce552f3aa57cc6e18ed35051138b591338242df5f3af39e24006a834
2026-01-14 19:22:36 +00:00
dan abdfd3312a Update sessions module so that it takes the db mutex before, instead of just after, opening a savepoint when generating a changeset.
FossilOrigin-Name: 41bbd13d737d2b7879d906b6fa76df42c6ee142f1a7acdafd26da0063bc90baf
2026-01-14 17:52:27 +00:00
drh e5fba7f094 Guard against oversized index entries in databases larger than 2GiB.
FossilOrigin-Name: ed17a878e5a2e0cd1e9b69d528f5ac2ba8452d7c83deaf3cc72ecbff054f5ca3
2026-01-14 13:43:32 +00:00
drh 35b9709d46 Fix integer overflow problems and error reporting memory leaks in
the zipfile extension.
[forum:/forumpost/2026-01-13T00:09:06Z|Forum post 2026-01-13T00:09:06Z].

FossilOrigin-Name: 8e656a483557bc5385219d560b9155c232e7dc9d62642249abc879fb37bacd2a
2026-01-13 02:35:19 +00:00
dan 4b12a38ffb Fix an OOB read in the sessions module that could occur while processing a corrupt changeset.
FossilOrigin-Name: 3c46295487f089a891f566cae43b67ce97794bb60645d5806285600e05eff456
2026-01-12 20:16:04 +00:00
drh 9a341c1edb Avoid unsigned integer overflow in the delta_apply() extension function.
Not part of any standard deliverable.  
[forum:/forumpost/d41879b367c7f7ec|Forum thread d41879b367c7f7ec].

FossilOrigin-Name: b354dd12c25c820c04b08e0be4ba8c095fc648dfb4b71345aacce50a17fd269a
2026-01-12 19:56:00 +00:00
dan 9f78d22870 Fix an infinite loop in the sessions module that could occur when processing a corrupt changeset blob.
FossilOrigin-Name: d98653bdbc9781970f1c5d66f69c81c93ad14549223ceae02e74c1b99ab05377
2026-01-12 19:13:09 +00:00
stephan bc2101b44e wasm: filter the custom Module.instantiateWasm() out of node builds, per request from the npm project.
FossilOrigin-Name: b57a8215f4259a0aae188b7ee5060f8ff48919303179aae80b58b43ed3b991f5
2026-01-12 15:43:18 +00:00
drh 6444f694a6 Improved documentation (in the header comment) for tmstmpvfs.c.
FossilOrigin-Name: 70b1da718c176b8eb154fe087af4352eb6f55c9c0d1f09fc625d073d9f8075f4
2026-01-12 14:45:55 +00:00
stephan a87d7c00f4 Add sqlite3_carray_bind() to sqlite3_api_routines. [forum:257e642a4c|Forum post 257e642a4c].
FossilOrigin-Name: 7c40dbcec8843f6189549a25dcfa8475f53212de8b0de81e10fd6687ef8fae9e
2026-01-11 21:18:23 +00:00
stephan 428c2656ff Add the @preserve tag to the standalone file sqlite3-opfs-async-proxy.js.
FossilOrigin-Name: 334b2be57e752672cfcaf764e3bc73e3d00a0a5174e8ef7c3d83c26136bccaf5
2026-01-11 16:21:32 +00:00
stephan 9ab2a2af1b wasm: add @preserve tags to two comment blocks (license header and build version info) so that the npm tools know to retain those comments when minifying.
FossilOrigin-Name: 596f1ea084e78843aa668d3a197690929848d474e55ff7b1444f78bd11f116b1
2026-01-11 14:14:16 +00:00
stephan 2f47a58c5d Update speedtest.md to point out xdg-open on Linux.
FossilOrigin-Name: 47c7fbcb46df5a657ec76dff9a370aea9e7d0d95406b1e124e9fe527cf85350d
2026-01-10 14:12:40 +00:00
stephan 644c42f3a4 Fix a help text typo in speedtest1.tcl, pointed out off-list by BrickViking.
FossilOrigin-Name: 1fd02a75a914f131948fe06fc99f851d3e226f9f1270e25393d79c493c28a347
2026-01-10 00:38:00 +00:00
stephan aef151ada7 Add the 'npm' ext/wasm makefile target to create npm-bundle.zip for use by the downstream npm package. This target is intended to become their path for pulling npm-relevant deliverables, rather than the canonical downloads, and they have our/my commitment not to break it without involving them first.
FossilOrigin-Name: a4752c18a85bee0cef61ee5ac7911661c1894fb861412c957692069195a001a5
2026-01-09 19:44:31 +00:00
stephan 0ad6c6c440 Fix the wasm snapshot build's zipfile name to include '-snapshot' and correct the update-docs target for recent build refactoring.
FossilOrigin-Name: 0f53dbf06010f66ba23941ee4269d507efa370f1a8cc0099d274d8baadebdb46
2026-01-09 18:42:27 +00:00
drh 4b93124b6b Fix potential OOB read on the undocumented test function rtreenode() that
is part of the RTREE extension, as described in
[forum:/forumpost/2026-01-08T23:32:19Z|forum post 2026-01-08T23:32:19Z].
The problem is almost certainly harmless since any memory allocation will
be a multiple of 8 bytes, and once the input buffer size gets rounded up to
the next multiple of 8 bytes, the access will still be within bounds.
Nevertheless, it still needs to be fixed.

FossilOrigin-Name: 9adab8b2bef4130abd358d53384cb5f4dd691b808336bb7102793b0165b1c516
2026-01-09 00:41:35 +00:00
drh 1d49bbc25e Enhance VACUUM INTO so that if a URI filename is used as the target and that
filename as a reserve=N query parameter with N between 0 and 255, then the
reserve amount for the generated database copy is set to N.  This simplifies
making a copy of a database file with a reduced or reset reserve.

FossilOrigin-Name: a482f9836597de55a9b58fddd3ca2963b8c67ecefef1e34a8c079a2d76f287d0
2026-01-08 20:29:02 +00:00
drh 4b4d07253f In the reserve=N query parameter on the VACUUM INTO target file, the
parameter is a silent no-op if N is out of range.

FossilOrigin-Name: 48e59a27330932cf29bcbd01080757b82ea4a03f5a9e1fa7da076dbfcb7f60e3
2026-01-08 20:20:03 +00:00
drh c34d561940 Change the name of the log file for tmstmpvfs.c to be an ISO8601 date/time,
for improved readability and situational awareness.

FossilOrigin-Name: 66018b85a92db2f3b6b15f7c5c0d54bac890f8e50a1839c08857507c39ec4af7
2026-01-08 18:55:49 +00:00
stephan 22c99e01ff Fix the sqlite3-worker1.mjs (ESM, non-bundler) build and have demo-worker1-promiser use that file rather than the vanilla one, as a test case. Delete the global symbol installed by that file in ESM and bundler builds, as it's unnecessary there. The first change should resolve [https://github.com/sqlite/sqlite-wasm/issues/123|downstream sqlite-wasm/issues/123], with thanks to Jure Rotar for his help.
FossilOrigin-Name: 11f5ab0dd8efc68112c3dc5ea0ed486c32bcdb9fde85352180f40a77c6354cac
2026-01-08 18:15:23 +00:00
drh 0a10cdb663 Merge all the latest trunk changes into the reset-reserve branch.
FossilOrigin-Name: 6e5f1fbaec775db5c5e90b29cfa0820c9f4d560b36c063b504ef55a6c3a65bdc
2026-01-08 18:14:29 +00:00
drh 5df3efe6f4 Fix an off-by-one error in the page number of ELOG_CKPT_PAGE log
messages in tmstmpvfs.c

FossilOrigin-Name: 6fafd010de2d9820f77ee6d2a5f0a7ab1260872d09e95b6932c959ef017aaa77
2026-01-08 17:52:42 +00:00
stephan cc4a404048 Add a missing #if check for SQLITE_OMIT_LOAD_EXTENSION to shell.c.in, trigged by the fiddle build.
FossilOrigin-Name: 1091f3371f4af75344c2e6da26656c259fa96c8a8da9acde3db79df8aa95fd0a
2026-01-08 15:26:19 +00:00
drh 403c153dfa Minor bug fixes in tmstmpvfs.c, showdb.c, and showtmlog.c.
FossilOrigin-Name: 1fee8265f7eab649ee8b28456c37d95198f5cde54e905154e3eccc0b328ece9a
2026-01-08 15:25:04 +00:00
stephan 784bceae33 Remove an obsolete '32-bit' label from tester1-worker.
FossilOrigin-Name: e14d9a980351a8d5e935e6991ea02a1c40c9ea8f0ad0fadcabe07d575a9b289c
2026-01-08 15:06:28 +00:00
stephan 92730d21ff Fix the --size X flag for speedtest1.html when X=0 and force the job size to 5 for kvvfs so that it doesn't block the main thread for too awful long.
FossilOrigin-Name: e2634e500cce44650ad5dd95b722b993b1f999b9f8b2396ab15154e67cb065c7
2026-01-08 15:03:00 +00:00
drh 22b08034aa Enhance the --vfs command-line option to the CLI such that if the argument
is not a valid VFS name, but is the name of a file, attempt to load that
file as an extension, which presumably adds the desired VFS.

FossilOrigin-Name: 509954b44eb305a4401ddcbcce24d8f9ed5534717290888899d716b9df40c981
2026-01-08 13:43:15 +00:00
drh c219d3002f In the showdb utility, the -tmstmp option works without -csv for the pgidx
command.

FossilOrigin-Name: c0aecd2189b431d82eca1c1d6718920cf928d910d443ea506e7b3bf3c5d7e246
2026-01-08 13:20:36 +00:00
stephan 9f9ee82b62 Fix ESCALE typo in shell.c, reported in [forum:b8f6a41c8b|forum post b8f6a41c8b].
FossilOrigin-Name: 6fa09449738af03d096208e26447b795f2825d6aaa397de09590cf18b065f82d
2026-01-08 12:43:21 +00:00
drh 67429703bf Do not show LIKE wildcards on table names in the output of the ".schema"
command in the CLI.
[forum:/info/6ebb57b41691293d|Forum post 6ebb57b41691293d].

FossilOrigin-Name: 4c6f646bd2910e6dc80dfec7546a3ed7b47bb9d00402e7c3871ece2c185f4ec0
2026-01-08 12:39:49 +00:00
drh c0420cd8e4 Add the showtmlog utility for decoding the tmstmpvfs log files.
Fix tmstmpvfs so that it records timestamps in milliseconds since the
Unix epoch, not in JDN milliseconds.

FossilOrigin-Name: 762d3cbf06218f1010425a98a71ebf7623b84b45a90e6419b7dec7f10ad0c5a5
2026-01-08 12:22:50 +00:00
drh e46b51e234 Improve showdb so that it does a better job of handling reserve-bytes.
Add the --tmstmp option that causes pgidx to interpret tmstmpvfs tags
if they are available.

FossilOrigin-Name: d7e6e9a5781f467a5b5f02f46134099cf01607a0cf55510155533d57d4a4618a
2026-01-08 01:37:13 +00:00
drh 8a2be4b739 Add the tmstmpvfs.c extension to trunk. It is not a part of any deliverable
and can easily be removed later, if found to not be desirable.

FossilOrigin-Name: 869f306592a86d2a78f00266c615f033cd2318a7bff0a97b3dcdd3348d99fc66
2026-01-08 00:41:12 +00:00
drh ad4b08dcab Add the --csv option to the "pgidx" option to the "showdb" utility.
FossilOrigin-Name: b6ad93f62eb883ee8f4c43dd90c69b31398be6b13186866a7b05d85fe588d967
2026-01-07 23:46:44 +00:00
drh bba40a9c9b Enhance the VACUUM INTO command such that if the target file is a URI
with a reserve=N query parameter, then the reserve_bytes value is set to
N provided that N is within the allowed range.  This can even reduce the
size of the reserve, all the way to zero.  This provides a way to remove
reserve from a database without having to do a complete dump/restore.

FossilOrigin-Name: 9cec3b67d95c145c968c560e8b3d2c33896b4b5b6c86c61350263764bb616c43
2026-01-07 21:19:42 +00:00
drh ef5d9952eb Fix the static build.
FossilOrigin-Name: 4e1a1b6aeffaa5b8c11e165803c6d33becabd1f646f39e4309dfba6892fd4bd6
2026-01-07 19:06:11 +00:00
drh 0b5f8766fd Improvements to logging. Better log-file names. Log files are in a
subdirectory named after the database.  Only connections that write to
the database create logs.  Better documentation in the header comment.

FossilOrigin-Name: 5b1cf0fe73ba1a40fe4cd8f5f6e8ee5b471f96b60aff52c03c045ad6514c363a
2026-01-07 18:45:23 +00:00
drh 03c0ac435e Enhance QRF so that it terminates more quickly if it encounters an OOM
or oversize string error, and fix it so that it reports such errors.

FossilOrigin-Name: ebedeb6169cc3bc3708130d4061065af600a8851e896fc3eeaf33c8c83fe3b95
2026-01-07 16:19:18 +00:00
drh 242a9347f8 Add the -p or --port option to sqlite3_rsync.
FossilOrigin-Name: 53f806fdab7161e439a165af47bd45332c15233b87bc35c2272621256a2dc337
2026-01-06 21:39:12 +00:00
drh 8fe0385661 Documentation tweaks.
FossilOrigin-Name: b10b0aa7658a04b9354ca09a2dcb807eecd9421451537f3230ba5f47269ec5c1
2026-01-06 21:18:42 +00:00
drh d3c9cce43c Enhance the timestamp-vfs prototype so that it also keeps an event log.
FossilOrigin-Name: f087c6c0cdd542ec24b4668fff9cb434f46caf554f8856a49fd062ab3f72014e
2026-01-06 20:51:06 +00:00
drh bcf1c8d6e3 Update to the sqlite3_get_clientdata() documentation. No code changes.
FossilOrigin-Name: 395b7168b873390c05ea2cc71fe14bafb5bdee77f32a047bec8d7ed6feb9ed90
2026-01-06 13:00:24 +00:00
dan 55f258bd27 Update the regexp extension so that the "-" character may be escaped with a backslash. Forum post [forum:c5e7358f82 | c5e7358f82].
FossilOrigin-Name: 80013129361fb81ce5db1f6390cad640c2cb547e5ba16724dfb55d0aceecdf6d
2026-01-05 18:29:07 +00:00
drh bb4428e605 Add the experimental tmstmpvfs.c extension that adds a timestamp and other
debugging information on each page of a database as that page is written,
if the database is configured for exactly 16 bytes of reserve space.

FossilOrigin-Name: 703e593dfc92676a2d44c67ca8282a78a1943b9c1e3f2447c0a9917d70a383c4
2026-01-05 17:28:33 +00:00
stephan b1e9111249 Fix an uninitialized var use reported in [forum:88561a4a1e|forum post 88561a4a1e].
FossilOrigin-Name: 707c0f6442e946f23de061ee2753eb5994ab55d411c49b232799f309ba0f10cf
2026-01-04 11:05:05 +00:00
drh 904abc85cf Fix yet another bug in the [/info/e33da6d5dc964db8|EXISTS-to-JOIN]
optimization, reported by
[forum:/forumpost/2026-01-03T14:05:48z|forum post 2026-01-03T14:05:48z].

FossilOrigin-Name: f60e863e0ca2d8ab853fa5f48d3cd7b062b13167fcddffc4563bde9285b92320
2026-01-03 15:11:07 +00:00
drh 0ed2541168 Minor change to an #if in the Windows VFS, which I am told will allow
SQLite to be compiled for 64-bit Windows7.

FossilOrigin-Name: 2afc54b91fe94a37bd3754b5beae6b1f378488b33f7d23eeeebe96d78d120fe3
2026-01-02 19:42:55 +00:00
dan a0cc06b66b Ensure that loops created by the exists-to-join optimization use unique cursor numbers, even if the EXISTS expression in the WHERE clause is a copy of an expression from the result-set of the SELECT. dbsqlfuzz 4f6a5213cf74fcf2a8a3e48abc2333100348c38e.
FossilOrigin-Name: 19e57a18cbfad44bb1cb1e62d4485635d26059cb2319f4b7f1244df2638bea43
2026-01-02 17:10:40 +00:00
drh 2b112e0907 Change the shelltest target on the windows makefile to use testrunner.tcl
FossilOrigin-Name: db4d07623500325153c3254c2ad9eaf5e0f1d056637c6799643fcc9bb3a5e84e
2026-01-02 00:23:22 +00:00
drh b0a1ed2980 Further improvements to testrunner.tcl documentation.
FossilOrigin-Name: c71402031ce81b4415a793afa3152f5691b197dce634c1322948f39086b8c819
2026-01-01 21:35:25 +00:00
drh 6e8afa79e7 Updates to the testrunner documentation.
FossilOrigin-Name: bcac99f962c6403fddfcaeef309ea9f825fa661fee0b3e3fd24f04c861be1d1b
2026-01-01 15:41:50 +00:00
drh db4d8946ef Verify the effect of "--titles on" in the "insert" mode in the CLI.
FossilOrigin-Name: a5f922b0832d9544b083f3d55d4029aa80e320bf5505547c5a1ddb21f4685e3e
2025-12-31 20:43:31 +00:00
drh f395df1307 The header setting from the legacy ".headers" dot-command takes precedence
over the current ".mode" setting, for backwards compatibility.
[forum:/forumpost/2025-12-31T19:14:24z|Forum post 2025-12-31T19:14:24z].

FossilOrigin-Name: d2c5df41285a9c8a66e9bdea4e28d86fe10665ad01cfbab5a7ce43b8e39b4e34
2025-12-31 20:35:16 +00:00
drh 02e38b4d21 In the (completely unused, as far as I know) btreeinfo.c extension (not
a part of any standard build) do a better job of detecting corrupt database
files and provide better error messages when corruption is detected.
[forum:/forumpost/e66a458339|Forum post e66a458339].

FossilOrigin-Name: 326d29ac7ecd933d6fca80d3d538783ed46939e7bc6e3aa60ded4b619aa73730
2025-12-31 19:45:45 +00:00
drh 6789770822 Use 64-bit integers to track memory allocation sizes in the CSV extension.
[forum:/forumpost/e66a458339|Forum post e66a458339].

FossilOrigin-Name: f293a2b3ffcfe20fcda954bd31e88bbfcba18d007e9b5f29a473983f6030f2ce
2025-12-31 18:30:45 +00:00
drh d964b8cfa3 Add the new "retest" command to testrunner.tcl, which reruns any tests that
failed or never completed in the previous invocation.  New makefile target
"retest" for both unix and Windows that runs testrunner.tcl with "retest".

FossilOrigin-Name: 9b668d56bf0eca3eb826df3edd6705cc809d920494c192a9949cafda791d83f9
2025-12-31 17:21:30 +00:00
drh 634a915132 Enhance testrunner.tcl so that it can always be run from an ordinary tclsh
as long as that tclsh can load the sqlite3 package.

FossilOrigin-Name: 1a08b85a862e8c9ac56c11e05f76abc25ad98b510fb13ad0716206ba797e53a8
2025-12-31 14:17:51 +00:00
drh 31f6052f03 Fix test/shellB.test so that it works under Device-One. Also bring the
header comment of shellB up to date.

FossilOrigin-Name: 836d02468c7749fa9cdb656c9f80cfef9a41e2eb1321d4bb56c0e0cbc5f37df4
2025-12-31 11:46:26 +00:00
drh f8bd7df996 Use sqlite3_malloc64() in all places in the compress.c extension to avoid
integer overflow.
[forum:/forumpost/2025-12-31T01:04:00z|Forum post 2025-12-31T01:04:00z].

FossilOrigin-Name: 7ceaf1f0c3d9b96d2224ce6526fb3092b9e1673573e3d7dd898b2a135e23027d
2025-12-31 01:19:51 +00:00
drh 03380a55a5 Fix a typo in [0819fe670f9ceec7] that lead to incomplete protection against
buffer overflow in the zipfile extension.
[forum:/forumpost/2025-12-30T23:57:19z|Forum post 2025-12-30T23:57:19z]

FossilOrigin-Name: a6abbadacbb88c1ddcc236b40fb34eddf3bb0891189bd00a5af8d34b42871967
2025-12-31 01:01:14 +00:00
drh d32e21ac95 Fix the output from the (undocumented) --test-argv option of the CLi so
that it correctly displays UTF8 text on Windows.

FossilOrigin-Name: 39e15ce3d6c42b13d0c1164a161e7b5d5d8382050c2729961733ddef695c9116
2025-12-30 13:10:10 +00:00
drh 9f44bc5ea0 Add missing free() to winmain.c
FossilOrigin-Name: 44dff1a1bb986cd08b729d4b73f09c995cbc67f884caa297674b31fb36805c8b
2025-12-30 13:01:05 +00:00
drh 02ceb91875 Improved handling of the process entry point in Windows, to translate
arguments into UTF8.

FossilOrigin-Name: 1d4ee6c35e69d43e2b801c425308de0ef5d5d9d2d0af9b406b664bf0f9cfa337
2025-12-30 12:38:43 +00:00
dan 918efe3cce Avoid signed integer overflow when an fts5 'merge' command is passed -2147483648 as a parameter.
FossilOrigin-Name: 52738908b04848b93d54431def37ffaa9723043270ff8ba1e2fa59ab0040afc5
2025-12-30 11:36:23 +00:00
dan 01f2907d48 Fix the change on this branch so that it works with amalgamation builds.
FossilOrigin-Name: 7f0b9e7a8bba1b73ef16257e72a055ea84fb10eb5cf0e6431fca81b2f040c5ee
2025-12-30 11:33:30 +00:00
dan 6e92494955 Avoid signed integer overflow when an fts5 'merge' command is passed -2147483648 as a parameter.
FossilOrigin-Name: 4c5d18d7f5bdcf95efdf96062c7434d5d10584a1fdfd65b18d331b8aae42e26a
2025-12-30 11:19:27 +00:00
drh 9924b07cc2 Fix the new .import logic in the CLI so that it honors ".bail on".
[forum:/forumpost/6c1c0e213d|Forum post 6c1c0e213d].

FossilOrigin-Name: d58846e74b6971a5fd80e5f030985273b7dfa0087c2f3c9d0c088c248e183f8a
2025-12-29 23:56:05 +00:00
drh fb795dbe73 Fix error messages associated with .import, along with corresponding
test cases.

FossilOrigin-Name: 2084e805d0d9da8f419b8506fccd652a7d20262bd95729250a2d2eabfdd079ad
2025-12-29 21:39:48 +00:00
drh 7dfb1431dc Incremental improvements to .import, and new test cases.
FossilOrigin-Name: 5a1e931b19c346bdc8d1a51fa6ef8ee04b7371cdf5df2ee32e1333226c42d327
2025-12-29 19:04:41 +00:00
drh bd91ba71d5 Update to the unix build documentation.
FossilOrigin-Name: 004cf4ad3cb0c05e501a464bee55e8ed168c4825cf419902890593468ca4d588
2025-12-29 12:46:35 +00:00
drh ce36cccd36 Fix the "testfixture" makefile target so that it builds with a static TCL
library on Mac-ARM.  Fix the "tool-zip" target so that it uses ordinary
tclsh rather than testfixture to run the tool/mmktoolzip.tcl script that
builds the archive.

FossilOrigin-Name: adf5d624cf109ebd05952e5df62f143f40829070cbcbe43280d371a60db7470b
2025-12-29 12:34:37 +00:00
drh 6693e78772 Test cases for .import, as well as minor enhancements to .import and to .check.
FossilOrigin-Name: 0252ac1c946cd2101f0084f11951b1aeea330dea23b707934c8852d9abe136b9
2025-12-28 21:37:58 +00:00
drh c00c0a8392 If the filename argument to the .import command is of the form
"<<endmark" then the content is read from the script (or stdin) until
the first line that begins with "endmark".

FossilOrigin-Name: 6e8ab51fbe0ee918bce6ab960c9fb078764aa9721ef41d39818127eb98045d46
2025-12-28 14:32:28 +00:00
drh 38c8014940 Improvements to the .import command of the CLI.
FossilOrigin-Name: 436ed7937bcd3b5781539d883ff2957b81d74abccb75f65a2ffb7446a5944522
2025-12-28 14:01:52 +00:00
drh 35f37ec959 Change a single integer variable to 64-bit in the tokenizer to avoid
any possibility of an integer overflow.

FossilOrigin-Name: 69cf692a24d714305b2e4c8c4c40f70dc6510c26b8db7e5249f32a53e44c7e5c
2025-12-27 19:57:59 +00:00
drh 7f56d23f9c Work around the Windows "Worst-Fit" bug in the "sqldiff.exe" program.
FossilOrigin-Name: f41851587328f8b434e66987765324b7bf5746ecdf3a9f2869000276e0f79401
2025-12-26 20:13:32 +00:00
dan 4056656047 To ensure that the sqlite3_hard_heap_limit() call it makes is effective, add "sqlite3_config(SQLITE_CONFIG_MEMSTATUS, 1);" to fuzzcheck.c.
FossilOrigin-Name: f7736af849d910f04d8622ed7c6e705d84cf22f0a73dca0f30eead9bd1f28bc7
2025-12-26 17:23:25 +00:00
stephan 91d0ffa576 Doc typo fixes in the configure script pieces.
FossilOrigin-Name: 8ab37d9fb909dc6bfbede004f342e6c1625c99df9b316807dc77a9a636ee6873
2025-12-24 18:12:08 +00:00
drh 683df60dbf Enhance the omit-noop-join optimization so that it is able to omit a
chain of noop-joins.  Optimization improvement suggested by
[forum:/forumpost/11a53f2bad|forum post 11a53f2bad].

FossilOrigin-Name: 5a85b83b9f21ad88938e273bb35b398e91133424683e915836f322b69f277673
2025-12-24 14:03:15 +00:00
drh 337fce82f0 Additional changes to support the ".schema" command on builds that omit
virtual tables.

FossilOrigin-Name: 6b5c0374fa81cf069263bebf9b740573de4f82fde3c4e5796af80dcbe6a1a2e7
2025-12-23 17:24:10 +00:00
drh f256886b8e Fix the ".schema" command in the CLI so that it works on builds that
omit virtual table support.

FossilOrigin-Name: b4e3c8170a107b2018fe5374495dbec14c851ccbf7419cd382991b2fe68c45e4
2025-12-23 16:51:49 +00:00
drh 8511ee1e19 Harmless typo fixes in shell.c
FossilOrigin-Name: 519877cdbde80071ac08f2e467ef8b9679816dc486cf6eb87ec2d9a38c819f88
2025-12-23 11:13:52 +00:00
drh 766389bd65 API documentation enhancements.
FossilOrigin-Name: 99ac94cb9a7a76312cda2d236dcc0c92b2ca269456daed06a1c2674c4f326867
2025-12-22 17:06:37 +00:00
dan add73c831b Add the SQLITE_PREPARE_FROM_DDL flag to sqlite3_prepare_v3(). Use this to prevent fts3/4 tables in non-trusted schemas from calling unsafe SQL functions.
FossilOrigin-Name: 26f39ac806a5582fab1497e38a4ce68dad2b79df8c8aade43c4e21eed7576931
2025-12-22 15:22:02 +00:00
drh 8d41b94bb4 Fix spurious error log reports caused by [d25c8a6222d4e3f2]. See
[forum:/forumpost/d2326687662430c1|forum post d2326687662].

FossilOrigin-Name: 403dafc7790a2e8d1772537800760addc47d3e3555bda343c3991d56c1b36e70
2025-12-22 12:33:27 +00:00
drh 3b81f93532 Fix a harmless signed integer overflow warning in the ieee754() extension
function.

FossilOrigin-Name: f43d3339c1f06aabcb8e678ed4b91a490cfd0bc880c642ce9c1138c7e08cb360
2025-12-22 12:00:57 +00:00
drh 19a3f74bdc Remove unused thunk cases from os_win.c, so that it will (hopefully) compile
again on WinXP.

FossilOrigin-Name: eb2c020481dd591f459cf7191d18c2499aeec210fc3fc27bb35e5abc0d9cb6d8
2025-12-22 11:28:16 +00:00
drh 174dabe096 Add experimental compile-time option SQLITE_ALLOW_TRIGGERS_ON_SYSTEM_TABLES.
FossilOrigin-Name: f916bcc8657a4db2d069018751815f2ad91851499add761b1eb119f265b72a09
2025-12-19 13:17:40 +00:00
drh 683c563241 Bug fix in the vt02.c test virtual table.
FossilOrigin-Name: a7a73d30c552195c95c282688eb3d067746c722b702c7f2ba151996ec92136f5
2025-12-18 20:44:48 +00:00
dan 65bc1f0ff4 Allow queries that use "GROUP BY e1 ORDER BY e2" where e1 and e2 are identical aprt from ASC/DESC sort-orders to be optimized using a single index. Also allow virtual tables to optimize DISTINCT in cases where the result-set of a query does not exactly match the ORDER BY clause.
FossilOrigin-Name: fba29a8b560d839e42c2a46421ab870da6e6ed6e15a17aa51f223e32ec3aa8ae
2025-12-18 18:47:49 +00:00
drh ae66a8d760 New options for vt02.c that allow tests to configure it to partially
de-duplicate a DISTINCT query.

FossilOrigin-Name: 1c65bb6e5d92cf39c409b7f870253571ec96b48abf87e5ed931cabd86f582069
2025-12-18 16:14:38 +00:00
dan ce46f82bc4 Improve tests in bestindexF.test. No changes to non-test code.
FossilOrigin-Name: 347d4d34c1815827e7049e57830c1fff67f6eb16ae5cc00839e35d94bac81e92
2025-12-18 16:05:05 +00:00
drh 4461335b00 Better orderByConsumed logic in the vt02.c test virtual table.
FossilOrigin-Name: 612c1ece6752e3318fc688717ad6a82219b85df7d32e4e07dbc2f361a5aeeee4
2025-12-18 15:35:50 +00:00
drh 1b7b75679a Enhance the sqlite3_load_extension() so that when the entry point is
not specified, after trying sqlite3_extension_init() and
sqlite3_X_init() where X is the alphabetic part of the base filename,
if neither or found it tries again with X as the alpha*numeric* part
of the base filename.  Hence, a shared library named "vt02.so" can be
loaded with just ".load vt02" and without having to specify the entry
point separately.

FossilOrigin-Name: acc95f24cc034408de1e9225aef4dae8bf76beec812c152bea32db95f7d2b854
2025-12-18 14:01:43 +00:00
drh 93c6b19f92 Enhance the vt02.c test virtual table with the ability to process
ORDER BY DESC.

FossilOrigin-Name: 35461a8d5a103e373e5b38b3974a3bb1074e914e862825108713b36bef04b6bd
2025-12-18 12:49:44 +00:00
dan 6c2f5dabf4 Fix the return value of sqlite3_vtab_distinct() in cases where a virtual table query has similar GROUP BY and ORDER BY expression lists.
FossilOrigin-Name: d017ae8640f24543ad5bf37a3e514c1e919730b8c686255c61890448ba22b3e1
2025-12-17 21:00:06 +00:00
drh 89bc8e8a34 Fix a FILE* leak in the CLI
FossilOrigin-Name: 7986a35d27fe0cfd7d9a3bfec0ee0d1c5556b24d571bb68c35dd5eb7251e423c
2025-12-17 13:10:23 +00:00
drh 93ecf30068 CLI: The .imposter command matches the index name regardless of case.
Minor fixes to .check.

FossilOrigin-Name: 989ebba9ad263abafeddaf4a0e3e1275e83a3a12d7f95c7a451ecf8cbd64a583
2025-12-16 13:43:35 +00:00
drh 6c26b5db71 CLI improvements. In the .check command, ignore leading and trailing
whitespace unless the --exact option is used.  In the .limit command, do
not print the new limit when changing the limit, but only when requesting
the current value of the limit.

FossilOrigin-Name: d72cb66d672d514822b2fbb0be8880377628599f1f8e2a219d36791c125c6242
2025-12-16 13:25:45 +00:00
drh 28ec7be607 Stricter enforcement of the LIKE_PATTERN_LENGTH for the regexp extension.
FossilOrigin-Name: c94b1e9c5674f7d0ad75d91085262f38cba6036979e8a0f9d39edc7db841915f
2025-12-16 13:13:21 +00:00
dan a1dbb7a93e Fix for optimizing "SELECT DISTINCT ...cols... FROM vtab" in cases where the virtual table implementation is able to deliver results sorted by "...cols...".
FossilOrigin-Name: 80c0d922937615422e813593ac4eb83c3040a417fad97dba18a549de50a91b72
2025-12-15 18:45:22 +00:00
drh 97601244e2 Enhance the vtablog extension so that it shows the value of
sqlite3_vtab_distinct() in xBestIndex, and so that provides the new
consume_order_by option that can cause xBestIndex to set the
orderByConsumed flag.

FossilOrigin-Name: b158fe929929ae209f2603b11a2c4f44ad9147f6b2ce09a10ec1f92429402631
2025-12-15 17:32:56 +00:00
drh 22546e1d7e Clarify the affect of nCharLimit, nLineLimit, and nTitleLimit on the
xRender output from QRF.

FossilOrigin-Name: 881534858de8bccffca53e7256a725378a98a354e792374d972b120880c2ae78
2025-12-15 13:58:41 +00:00
drh 586a649698 Fix the test/vt100-a.sql script to use .testcase and .check.
FossilOrigin-Name: 5cf9e99ff8279530cb73a24794107591f8dab755fe78acd6c11429e532f3d1bf
2025-12-15 13:26:27 +00:00
drh e2bb4cf412 Enable an ORDER BY clause to refer to columns in outer queries.
FossilOrigin-Name: ef9b4b1e4690c889c48f4e725d42127c46f60c6bf2023ae79bf1e9c6c83368ec
2025-12-15 11:59:47 +00:00
drh 76f1fe2942 Use the correct wildcard character in sqlite3_strlike() in the previous
check-in.

FossilOrigin-Name: 87bca1250c5e660debbb29c5fabb89082c2d2059fc8b5851e1b1a519c3be7fc1
2025-12-13 23:18:04 +00:00
drh 5266818aed In the CLI, if a command-line argument names a file and the filename ends
with .sql or .txt, then process that file as an SQL script.  Use this
feature for better testing.

FossilOrigin-Name: 19cc0522e2255f21f6fb6af1442c2ba122fdd4deacd9ea2c181ce9e597b88770
2025-12-13 21:11:49 +00:00
drh ce43bf85fa Back out the --compat option from the CLI.
FossilOrigin-Name: 850d5dbfb0f1eacd1e5213759810ec7e5eba4fcc0b2718dabccd5b269b126f1c
2025-12-13 00:13:40 +00:00
drh 4f3ab29aab Get the .testcase and .check commands working in the CLI.
FossilOrigin-Name: 1ca31e1a297c0d53d068afb658ce6602887cda0f3eaf42cd629c4d1b7204f0b0
2025-12-12 18:22:07 +00:00
drh d031267c17 Add new assert() statements designed to detect and prevent the kind
of system interface numbering errors that happened in [108691a3cb2f296f]
and that were not fixed until [fe49703034bd23fa].

FossilOrigin-Name: 4e6e551c89c4f18299b3d60369f439e80f2fe2b0e70199f649821b1dbf20c668
2025-12-12 12:06:42 +00:00
drh 48915e7c7d Omit unused entries in the aSysCall array for the Windows driver. Fix
incorrect numbering in alter entries of that array.

FossilOrigin-Name: fe49703034bd23fa8f43ffdf0e23ec9613688d957adf96b3f1ea86fb02a8d86b
2025-12-12 11:50:37 +00:00
dan 462289df52 Remove from os_win.c a reference to unused system call CreateEventExW() left over from the now removed winrt support.
FossilOrigin-Name: c68ef69ec97da0312b95375b3af159f8a06ce1a69376d042c557dcca18d3f77e
2025-12-12 11:02:18 +00:00
drh 63f1fb40be Fix an inaccuracy in the qrfRelaxable() routine in QRF when the input is
an empty string.

FossilOrigin-Name: eb8a519571f7d35b81cbdbe668da45b33e5505d9a038305129df7ba2c16fd3b0
2025-12-12 10:21:52 +00:00
drh 5eba66ad66 Include the latest Makefile.msc changes and the new "make.bat" script
in the amalgamation tarballs.

FossilOrigin-Name: 6deab75874acfa183e51b8ae8d9305e8d09072bb6c646859ed40beb46e91fe0f
2025-12-12 00:21:10 +00:00
drh c2cd606f8a Prevent a malicious delta from causing an integer overflow in the
fossildelta extension.  This code is not used in the SQLite core.

FossilOrigin-Name: 01409738afc2c0d5bdaa248ffb508aa5f36a66390f6b8e4834734529ee8ed2fa
2025-12-11 23:24:05 +00:00
drh b304b0d165 Update build instructions for Windows to explain how to link against
ZLIB.

FossilOrigin-Name: e785a80e4100c368dca8d73cb662cff4d0fd76734fa0f3fa9b5754a380f7c746
2025-12-11 18:16:39 +00:00
drh e29acd77c3 Adjust the MSVC Makefile so that when it invokes the zlib makefile, it
changes CFLAGS to use the correct -M compiler option to match SQLite.

FossilOrigin-Name: ba003c7c74fb4dd665a8ec6dea7d030f1e7e9cbb13e60bb728860ebffdff5aaf
2025-12-11 17:49:17 +00:00
dan 9334b9f1b8 Fix calls to sqlite3_wal_checkpoint_v2(SQLITE_CHECKPOINT_NOOP) so that they do not return SQLITE_MISUSE.
FossilOrigin-Name: 117f4093f08e0abbde2041f70898220e237e250664f6ede6d4eb2a8a44050466
2025-12-11 11:05:26 +00:00
drh c0160fa11a Use macros for the default values for QRF limits in the CLI, rather than
hard-coded magic numbers.

FossilOrigin-Name: 4ef66da15bfe69d940cfbeff2be2b7e13da156495b349b453d68e70f4dcca36a
2025-12-10 19:43:06 +00:00
drh 85fb541ac0 Various improvements to the .mode command of the CLI, and especially to how
it handles the --limits option.

FossilOrigin-Name: f883b854976d2aa679c87edce3b577f38b5fa518f707706320d1ddb38365e2dd
2025-12-10 19:04:49 +00:00
stephan 3262f769ee In fiddle, remove the default .dot-commands it runs, as they do not reflect current shell defaults.
FossilOrigin-Name: 27b6aeda5d70e3480b1448b06277a19395b30a205f3f3bac5ce0e4102e3849ed
2025-12-10 16:20:44 +00:00
drh 2f2b5d7b20 Add the --titlelimit option to CLI, defaulting to 20. In "line" mode of
the CLI, the default seperator is changed from " = " to ": ".  This can be
overridden using the --colsep option.

FossilOrigin-Name: 291677ebb3e5c2fe00e4340d8d3661fde6bd18494f1d4e56da1efcf613ab21b6
2025-12-10 16:10:26 +00:00
drh 9d5f00706f Do not show the --titlelimit setting in ".mode" setting output unless -v
is used or unless the current style is tabular or line.

FossilOrigin-Name: a9b0a3ce043fc910465d676c72bcf44c2ff6fbc1b0a832106cd2d3f97c7fb16a
2025-12-10 15:55:44 +00:00
drh 1debc25ef8 Test cases and documentation tweaks.
FossilOrigin-Name: cc9e47a205883aedf24b5e85e8e0499567f514780e3e6ec58069a39ea6270133
2025-12-10 15:50:23 +00:00
drh 9bf37cc3eb Add the nTitleLimit option to QRF. Also change "line" style so that it
uses zColumnSep as the divider between the title and the value.

FossilOrigin-Name: d67a299c86ff80ed67857d2546b0edaf852cb23dbfbc15e24793d1c4860a2e50
2025-12-10 15:24:46 +00:00
dan b4f6c09e60 In fts5, avoid holding an open blob handle after an INSERT/UPDATE/DELETE statement, as this can interfere with sqlite3_interrupt(). Forum post [forum:/forumpost/95413eb410 | 95413eb410].
FossilOrigin-Name: f88e1d03573f394fc7caf60704c96152717b3fe8be4eef0caeb64737054f0fd2
2025-12-09 13:41:01 +00:00
stephan 567dcbb5db Update the added-in-version stamp to some kvvfs JS APIs.
FossilOrigin-Name: 939275c6c574a3354e0626164aab1a005998f811c53a66b078249d6b2fcbd68e
2025-12-08 13:11:17 +00:00
stephan 91479c6b10 Introducing kvvfs v2 for the JS bindings. Summary: no longer hard-coded to session/localStorage, available outside of the main UI thread (non-persistently), a simpler-to-use import/export API, and adds an asynchronous event interface intended for creating page-by-page db backups.
FossilOrigin-Name: ec866b04d088e53b09764d00f5403902785328fd8f3cbf80d3affa166477711d
2025-12-08 13:06:27 +00:00
stephan 50ef2f7813 Replace some double-quotes with single quotes an SQL doc snippet in the csv example virtual table.
FossilOrigin-Name: 1781e5e8d0cd2b508f7992987257416bd48841ea7e30bc9294784a7c1a402be7
2025-12-08 12:24:23 +00:00
drh 1241fe1450 Minor tweaks to the QRF documentation. No changes to code.
FossilOrigin-Name: b2517d01e65b34ea4ca52c9149d7b255a36a45a50b332cb8ccfdacf22e629be2
2025-12-07 18:19:22 +00:00
drh ee9f4acb10 Do not return too many bytes from the zipfile extension given a malformed
ZIP file.

FossilOrigin-Name: 9766b47beb9ec72f55bfe9fa6e4dadf1829848389388aa30e97094a325de17fa
2025-12-06 23:58:09 +00:00
stephan b6306586ec kvvfs: make the filename suffixes -wal and -shm illegal and teach xAccess() to silently not fail for illegal names (else it breaks things when a -wal file is checked for).
FossilOrigin-Name: 2ea29c77a85236ca4126c05c7fd1d45a80dfe3f653af9b3ed86d6e62877ec5af
2025-12-02 18:28:22 +00:00
stephan b58b45d48e Get kvvfs::xAccess() fleshed out.
FossilOrigin-Name: d947e9bed5affee3f6e54b0fe3e2b1a0ac3771865b93c06a4087318f26d87f7f
2025-12-02 18:10:53 +00:00
stephan 6993f52d94 Consolidate kvvfs's page size workaround into a central flag and add some logging to facilitate testing and debugging.
FossilOrigin-Name: 8d683130d2f2aaf6300a8c858802eb671e831ed520621af59c9e830e87a9d753
2025-12-02 16:29:28 +00:00
stephan 22b1bc9997 Add more logging to kvvfs to try to trace down why it cannot recover from a page size change.
FossilOrigin-Name: 0bf0b8a98a2cc5128aa0e510ef2fe411a6859ce807d6159175f5eaf3bc14183b
2025-12-02 15:47:32 +00:00
stephan 9d6b913098 Begrudingly allow sqlite3_js_kvvfs_clear() to work for opened storage only for the local/session storage cases and only (one of the demo apps reminds me) for backwards compatiblity. Likewise, the JsStorageDb.clearStorage() inherited method is now deprecated because it requires an opened db (but continues to work for those two stores), whereas its 'static' method of the same name is unaffected. Rename kvvfs.size() to kvvfs.estimateSize().
FossilOrigin-Name: 09e9255828ed6a7ccbe18f701f4a88b859cfdbd1ddca7ac5dac09744542c25fa
2025-12-01 22:20:54 +00:00
stephan f1d5e9e913 Harden a change to SqliteTestUtil.assert() in the previous checkin to handle a null second argument.
FossilOrigin-Name: 9cbd5be803aba28916440cfa70c54588e7891d2073caebfa2b4dfcdd434b5c49
2025-12-01 21:14:11 +00:00
stephan 2463c80b54 Minor JS docs and cleanups. Add some #if'd-out kvvfs and oo1 pieces to experiment with later.
FossilOrigin-Name: 3db29b00db8d75c6b6a909a46a5c71515d3407af27ff9cb79091f293e4005ce3
2025-12-01 21:10:16 +00:00
stephan db94b3c0d7 Relax the name limits on kvvfs dbs.
FossilOrigin-Name: 9901cf8e4a00ea9a199a3fb54bd58bd66cff4d02c55433d55f2b417e188e49e0
2025-12-01 16:28:55 +00:00
stephan c281beded3 Reformulate sqlite3-wasm.c's exports so that they export properly with the wask-sdk compiler.
FossilOrigin-Name: d71849958aabdb05225be18d6bc46699cfda9de67c7105b11c3f79d1d01f47d4
2025-12-01 16:25:53 +00:00
stephan b8f8d021c9 Teach kvvfs to handle a NULL db file name by generating a random name, and to honor the SQLITE_OPEN_CREATE flag.
FossilOrigin-Name: 104291469169bef0c2ec5aee9c1cc505447541801bc822e6d5fe5360aef6a2e4
2025-12-01 15:49:31 +00:00
stephan 7c74a3fba5 Fix filtering of the 64-bit speedtest1 wasm test pages to again load the 64-bit wasm files.
FossilOrigin-Name: f507a8af41228d4ca332ab27e842767016ca18adda92e27a1e02e78f12c9be79
2025-11-30 18:59:44 +00:00
stephan 8ff58b4f41 kvvfs docs and 64-bit fixes.
FossilOrigin-Name: cf58e17fa2dc2e183a6ea1d41795c701efb303c9b378aa9b90953c9b568c621a
2025-11-30 18:43:59 +00:00
stephan 4694b91988 Optimize out a kvvfs event notification call in the common case where there are no listeners.
FossilOrigin-Name: 8405c19d32f1e8b7273953a038f8bdfd4ea1a6548300bac5421cdf6fc6840285
2025-11-30 16:37:47 +00:00
stephan 316020580e Rename kvvfs's 'localThread' storage object to '.'. Swap kvvfs.listen()'s elideJournal(=false) option with includeJournal(=false) (i.e. opt in instead of opt out).
FossilOrigin-Name: 7f0eca9d0aeb49c86a785ae930d235902bbd0f15877cc8f6083daac8efb2d1c1
2025-11-30 08:32:52 +00:00
stephan 9dd6979f3e Add sync events to kvvfs.listen() so that streaming can maybe get a better sense of when it's okay to process the stream.
FossilOrigin-Name: a4b6dadffa1b43aadebfa06a1de14d763efd18b16c4a3e87f1bd039fbff2524d
2025-11-30 07:08:45 +00:00
stephan 3dbe1f2f96 More work on the kvvfs events. Demonstrate re-importing a streamed-out set of db pages.
FossilOrigin-Name: b8fee839b0199d1d2714c99612ea4b4d943d8b0033d742eaa66e1575ce46f0d2
2025-11-30 06:20:28 +00:00
stephan b4397ec831 Give kvvfs.unlisten() a useful result value.
FossilOrigin-Name: bbd08b67ceeaae2a3a82655da5a3983f4e9057424de4223f1505ba74c36238af
2025-11-30 05:25:44 +00:00
stephan 9f83f4c563 Extend the kvvfs.listen() config to enable posting of raw binary db pages instead of the kvvfs-encoding. This is much more expensive but was added to... Demonstrate basic async streaming of kvvfs db page-level changes via logging of kvvfs write/delete ops.
FossilOrigin-Name: 0f2bad285577c26f1185dcafd3b8ca2f16e74aa9dc40e6e23867150bccee4602
2025-11-30 05:20:17 +00:00
stephan 768c364aaa Extend kvvfs export to optionally export the raw binary db pages as a list of Uint8Array instead of kvvfs-encoded strings. This is typically much larger but the pages can then be used as-is.
FossilOrigin-Name: a4f59496a53a079f8f73e4cde68f47dbd13d2d74de2ad11bc716e7e5c00f1ec0
2025-11-30 03:02:06 +00:00
stephan 6a59bb6057 Add a comment explaining the new kvvfsDecode() return -1 from [cd81cb70525e].
FossilOrigin-Name: ed9ab366d1c1880d3c06edce6c0c33ad30c7ae59725c1ec1fe3f620be1835630
2025-11-29 23:41:34 +00:00
stephan e8416526f0 Fix kvvfs event notification to notify _all_ listeners and optimize better for the common case of no listeners. Squelch some debug output.
FossilOrigin-Name: 989d097b1324cf712107bb81697fa8e9044aea0e7feacf0e6b6561e216f07989
2025-11-29 23:29:30 +00:00
stephan f2ec1d2560 Add a WASM-side export for kvvfsDecode() for pending private use in the kvvfs export bits and ensure that it consistently returns -1 for malformed input.
FossilOrigin-Name: cd81cb70525edeb25c4ab287cd5fd03db08ec45fd9e4001cad5c3eadf11505ae
2025-11-29 23:29:07 +00:00
stephan 4719c67753 More tests for the kvvfs vtab.
FossilOrigin-Name: 0358a99eedb503aaf8eff32a3143a2591081dfd3fd9eeecc51c601c98c714d6f
2025-11-29 21:19:35 +00:00
stephan facbf34cbf Add a kvvfs vtab for inspecting the storage state. This is for testing and my own edification, not part of the public API.
FossilOrigin-Name: eaff2e8b7f63575e85fb9be800a2c59163e402d6a25fc6a955b9df15afe13b92
2025-11-29 16:41:27 +00:00
stephan c4a22d6c14 Merge trunk into the kvvfs-v2 branch.
FossilOrigin-Name: 0e1a6ca0f1123da680918027b16608bf87593a9d91f7b1f3fc0c3a454185740d
2025-11-28 22:17:02 +00:00
stephan 973340c681 Minor kvvfs error handling cleanup.
FossilOrigin-Name: 5ec20ebe057af297f5e0ccc0d7184ab0ed720f91db0f470a7cf827df0f2aa530
2025-11-26 20:20:40 +00:00
stephan 154e1082f1 Teach kvvf's xClose() and xFileControl() to return more specific result codes for exceptions.
FossilOrigin-Name: 6b48375972e6e4112efd51ffea0e54d33a4245bf0155b7fa3ca1e3f147eb043c
2025-11-26 20:18:11 +00:00
stephan a4cb56e425 kvvfs: silently reject attempts to change the page size, as changing it corrupts this vfs for reasons not yet understood. Add more kvvfs tests.
FossilOrigin-Name: 65272adfab333565719bde4f6c5bbe4f0b5415c17a8de43e6b2d7b0518cca4c7
2025-11-26 20:12:29 +00:00
stephan a5f1fc25e3 Factor out an obsolete malloc failure check and an snprintf() for keys in transient kvvfs storage.
FossilOrigin-Name: e2e94f9094f8bbe93cdf7d2a2e72e92462b94e18319603c5a364f183cb780be1
2025-11-26 17:19:44 +00:00
stephan 77e1e0b4ef Remove a JS-specific block from os_kv.c.
FossilOrigin-Name: 3c1d5eac270e8afe6196ccb11a6d7bb0d1f262c882ce390a16b998bd2f55cb3d
2025-11-26 17:17:22 +00:00
stephan 2348682e4e Fix tests broken by the previous commit.
FossilOrigin-Name: 430d4d668fc2d151f20a02ddbbac7f884258af7fd1268d3aafb5f9e8eadf844b
2025-11-26 16:40:23 +00:00
stephan 4ad869aa3f Remove some dead code. Resolve the remaining kvvfs v1/v2 API compatibility confusion by eliding the v1 API from Workers like we've always done. Add some tests.
FossilOrigin-Name: 450b4bcca6038be58a73ff15ff18b9d70df661fe0bd9777273e0db6fbce5f296
2025-11-26 16:27:11 +00:00
stephan 0e4ea1821f Wasm speedtest1: replace some 'self' with 'globalThis' and expose the sqlite3 object to the global scope for inspection via the dev console (for inspecting resulting kvvfs storage objects).
FossilOrigin-Name: c18154bc9a7cf07d3902d74d27b4ca696929aae9e1abe031679bd5a42d759046
2025-11-26 16:26:27 +00:00
stephan 0baab04f8b Minor optimization in kvvfs event notification.
FossilOrigin-Name: fd85b96b12d164f7700624dba478d35e886b23f5c31c853715f783639ea95f23
2025-11-26 14:22:51 +00:00
stephan a781caa9ad Internal kvvfs docs and cleanups. Wasm speedtest1: remove the hard-coded --size=2 limit for kvvfs and use a transient storage object instead of session storage (which is limited to --size=2). For small job sizes kvvfs is roughly 3x the speed of memdb and that gap grows as the --size increases, with kvvfs _really_ bogging down on the --size=50 jsonb tests.
FossilOrigin-Name: 43173267849f7f4b77a9ee15ab586e9deb0e290071149857fe18421fb948a715
2025-11-26 13:55:20 +00:00
stephan 1da8e7ac4f Remove some debug output.
FossilOrigin-Name: 9fc6b811d6b0e268be18b768b6a69381606557a1702064b29f0ff4ca7c505cac
2025-11-25 19:06:05 +00:00
stephan 20e1f8a289 Remove the ostensible support for empty kvvfs names, as those names apparently never reach kvvfs.xOpen (the empty name is special). That actually simplifies things with regards to backwards compatibility.
FossilOrigin-Name: a5c61fa56770a6fffcb4c76db2256c71b2d5f0beee6fc2c4800c40bf7ef995c8
2025-11-25 19:05:13 +00:00
stephan f0ca498c61 Improve kvvfs file name validation. Add sqlite3.kvvfs.unlink().
FossilOrigin-Name: 0dfdf4681cf63541de971a20be21b33d0d3b38e8281f302d20aca9492df3da42
2025-11-25 18:52:08 +00:00
stephan e62055f274 Add the sqlite3.kvvfs namespace for the new kvvfs APIs instead of adding more sqlite3_js_kvvfs_...() methods. Reinstate that clearing kvvfs storage is illegal when db handles are opened, solely for sanity's sake (they can actually recover from that but supporting such use feels ill-advised).
FossilOrigin-Name: 02793c5905e6b99379cd5ad6bfe1eb6cccf839da081fc174dce7b06245e212fb
2025-11-25 16:41:56 +00:00
stephan 27aec2d1c1 Add an explicit rollback test for kvvfs.
FossilOrigin-Name: 2bf31ef8027a3e15887d4dcd26fe09463b5f8852c5ce443f7d07c23d29c37311
2025-11-25 16:07:05 +00:00
stephan 30f9c25d9a A more compact formulation of [ac8af73c16ab821] which uncovered two remaining direct calls to kvrecordRead(), now replaced with calls through the thunk. Fix a typo which caused the storage object name to not be passed on to event listener.
FossilOrigin-Name: d391f1ce691fca4504c21a7e3e05a029f247da23d432ee34b0dae14be51d909c
2025-11-25 15:59:58 +00:00
stephan a63538027c In wasm builds, #if out the impls for the 3 core kvvfs i/o routines which get overwritten from JS (and thus aren't used) to reduce dependencies on POSIX APIs.
FossilOrigin-Name: ac8af73c16ab821a7aa7d28a5fde8aeba0bc4d03c0af0dd2cff288c8fdabc49e
2025-11-25 15:35:53 +00:00
stephan 68230ef670 kvvfs internal cleanups. Experimentally add async event listeners to kvvfs to explore backing up a kvvfs a page at a time.
FossilOrigin-Name: f355fd484947a645206c9b9c2fd6fe691455dece7fb1aa5b72cb51a86b39474f
2025-11-25 05:44:03 +00:00
stephan 76dc170938 Factor out some now-superfluous JS-side kvvfs code. Factor out a superfluous allocation. Shorten the public API names of the new methods.
FossilOrigin-Name: be435b668f1aee56fc4965592c207de25283de238fe89002f1a68ba0567aca65
2025-11-25 04:07:50 +00:00
stephan c14460f466 Disallow a filename of '-journal' for db files in kvvfs, as that name has a special meaning there.
FossilOrigin-Name: 357cc42633efb85c3ca9bc3d6d46430e1ecaf2825e6bdd7d7b4e0f6865d0b599
2025-11-25 03:09:45 +00:00
stephan d3a0fc5473 Add xGetLastError() to kvvfs but it does not appear to be called for xOpen() failures.
FossilOrigin-Name: 7e7944f00792c5e7a9ea013f9c5d1f6bb6313070de20ec4ce262ff0b430f9801
2025-11-25 02:08:49 +00:00
stephan 45d8831cad Fix a 32-/64-bit incompatibility in opfs-sahpool.
FossilOrigin-Name: 256c1dd2c367dbc3b3df5028f2004bb1126851109efc71a1699f60f493514fcb
2025-11-25 02:08:11 +00:00
stephan b7ed25a86f Do not allow the kvvfs transient flag to flag a previously-non-transient storage as transient. i.e. only applies when first opening a db. Rename the transient flag to delete-on-close to avoid confusion with other meanings of transient in that context.
FossilOrigin-Name: 46d6b636307d98d17344d995afd99204e16eb06b6b19021a824dab95267d5377
2025-11-24 20:43:14 +00:00
stephan 91c8b974e5 Add the sqlite3_db_filename() to the sqlite3.oo1.DB tracing output. Document an unfortunate design choice which was only recently recognized as such.
FossilOrigin-Name: dec0c6073cc130ba445ec68bc49a410001d778c4aa21624dcb0a68b88fce251f
2025-11-24 20:36:49 +00:00
stephan c4d6f31747 Add kvvfs URI flag transient=1 to cause the storage object to be removed when the last handle to the db is closed. By default it behaves like a filesystem would, retaining the storage objects.
FossilOrigin-Name: c6183261be77d03451aed563a5e4c19aa69706aa311cfdaf5db5e9b0a167d814
2025-11-24 20:14:23 +00:00
stephan d8b323f652 Fix a typeof check so that it works in Worker threads.
FossilOrigin-Name: 3f2ed39e40e9c76c7d035b819d06c316ec4fba4e112b8a293822a8648147594b
2025-11-24 18:17:07 +00:00
stephan c340a0f717 Have kvvfs enforce ASCII-only names to avoid truncating multibyte characters in kvvfs's key buffer.
FossilOrigin-Name: 03cb2a13c7f9081132390b5169cf22ffeced6ebdc5f01654360f951720310916
2025-11-24 18:08:25 +00:00
stephan 72b4d638a5 Get kvvfs v2 storage import working by disallowing it when there are opened db/journal handles. Move import/export out of the JsStorageDb class and into sqlite3_js_kvvfs_import/export_storage().
FossilOrigin-Name: c9e0b32278290baf987b5a46bd47358439e3d0f190b2879a965d6e4262ea7baf
2025-11-24 15:17:52 +00:00
stephan 853335af49 More work on kvvfs v2. Db imports work, in the sense that the storage is properly replaced, but the native side is not yet able to recover from that (and how to make it able to do so is not clear).
FossilOrigin-Name: 4857c9d2fe428c19319244bf0589eaf93c124f020a633d6b7d40d35aaf969d24
2025-11-23 22:13:42 +00:00
stephan 8679c33355 Add the long-missing sqlite3_next_stmt() binding to JS/WASM.
FossilOrigin-Name: 53a99ff4ed5ef5f8620bf324a4f7a1d0812f5c80311228eb820039430ca04bd5
2025-11-23 22:09:16 +00:00
stephan 606b1bd153 Docs, cleanups, and dead code removal.
FossilOrigin-Name: 49db59aa9c74e49d878adc8671b0d32db8f1f898bde29d046ce0e368d8987868
2025-11-23 20:05:21 +00:00
stephan da0d872270 Get the kvvfs v2 working in 64-bit wasm builds (a 0 vs null vs BigInt issue).
FossilOrigin-Name: 4811742688bd8ae847c0a13ca9b395b2d2edd24ce4e7760a5d1b0adf1c7f332d
2025-11-23 16:54:16 +00:00
stephan 274db1fd84 Eliminate the superfluous 'kvvfs-' part of keys for transient storage and cleanup their export a bit. Reminder for later: we could potentially use the dbpage vtab to serialize any db to this JSON-friendly form.
FossilOrigin-Name: f63014f70febf82976a99f9f1ce6d793e2ca8d1dd2f72622152b64b2d65f8adc
2025-11-23 16:35:18 +00:00
stephan d855948547 Initial work on eliminating the superfluous-for-transient-storage kvvfs key prefixes and implementing kvvfs db import.
FossilOrigin-Name: 6bc64059410b1868b7a3576e16d03c02e7c007a2be8b313e386eeb2e2a35c258
2025-11-23 15:38:34 +00:00
stephan 56a4a2173e Add tests demonstrating basic concurrent use of kvvfs storage and that kvvfs still works after wiping the storage out from under it).
FossilOrigin-Name: cbbb9e61ae81097625488d78c5cbc0065c0acc3c8a7fd819490bc697f9d808c5
2025-11-23 13:33:56 +00:00
stephan ea375f1320 Bump kvvfs's sqlite3_vfs::iVersion to 2 so that its xCurrentTimeInt64() can be used. Add JsStorageDb.exportToObject() to serialize a kvvfs to a JSON-friendly form.
FossilOrigin-Name: 393c1beee00ec3b3e98eb385cae0abeb1b34475140c0e8c3d57541d349904436
2025-11-23 12:14:18 +00:00
stephan 48c8405f99 Replace an alloc/free on every read of a JS kvvfs key with a one-time alloc and a long paragraph explaining why we have to leak it.
FossilOrigin-Name: 66fb9978f0a63887d214e895343adedfa46ee7aefccb8389d6d3af727e0a65ea
2025-11-23 11:28:27 +00:00
stephan 44fdf80c28 This combination of kvvfs callbacks seems to work okay for both persistent and transient storage, but it's unclear why/how, as xAccess() does not appear to be doing anything useful (and misbehaves when told to report that the storage object in question exists). These tests fail in 64-bit builds for as-yet-unknown reasons (other tests continue to pass in 64-bit).
FossilOrigin-Name: 69e591d0054218ead789cee199e5258f1c378a89e4b7b0e38fe74e834e23156b
2025-11-22 19:21:25 +00:00
stephan 05478c7149 Fix a misinteraction of CREATE/READONLY flags in the OPFS VFS.
FossilOrigin-Name: 0b53be562f1e1a5b20ffe8d72df64e753a8d759b580d949a0f32409144769bb0
2025-11-22 17:27:16 +00:00
stephan 7ae80c0bc1 Fix storage-size counting of non-local/non-session kvvfs storage. Rename the 'global' buildin kvvfs storage object to 'localThread'.
FossilOrigin-Name: 61606be2ae2b0d73cdcd7947a77c7ad87cdf850bba90b0c3e3cdf8c02177db73
2025-11-22 16:29:45 +00:00
stephan ec274603f1 Incremental work on kvvfs v2.
FossilOrigin-Name: 60d61cf383b63b25dcfbf8da9539aaec253b6618ec83403f6690b7a32c13363d
2025-11-22 07:35:36 +00:00
stephan 16d127eb45 Factor KVVfsFile::zName back out. Remove the extraneous part of the storage keys for non-local/non-session storage.
FossilOrigin-Name: d2bf96d68c6cb2ae68558722edb22192fb1dbbf08fefdb2fd0a4827688e082a8
2025-11-22 05:37:21 +00:00
stephan fc6096b013 Demonstrate completely transient and a semi-transient (until page reload) kvvfs instances.
FossilOrigin-Name: 3f9ff9873303c3900dd3cba6e922bfb8cdb1f595353b692796b62e3025013517
2025-11-22 02:43:56 +00:00
stephan 7845d312b1 Latest side-stream jaccwabyt/wasmutil.
FossilOrigin-Name: e0b33b51229a977cc3fa8a5a6c8ea59669f8bf566b2a6330fd24da1ad886a716
2025-11-22 02:23:02 +00:00
stephan 92fa31bc5f Get initial JS overrides of the kvvfs sqlite3_vfs and sqlite3_io_methods in place. It now tracks a distinct Storage-ish object per sqlite3_file instance.
FossilOrigin-Name: 19a3349a2031e2b7fae67847b55643e4f70f8dae863ebc1ace3b09d1f482c8eb
2025-11-21 22:13:35 +00:00
stephan 7c07260353 More work towards using JS generic Storage objects as db page storage via kvvfs.
FossilOrigin-Name: 90a33941c69b3581feaed271542f0238ca81ee34fe5b353ca7da48b81ac73a5f
2025-11-21 18:55:33 +00:00
stephan 6690972855 Remove kvvfs-specific filename validation from the oo1.DB ctor.
FossilOrigin-Name: 206275292217be4ff317d4c9186ecaf863ca69295e2f995ed175aa65d9ad11dc
2025-11-21 18:49:58 +00:00
stephan 81b7e49d2d Initial restructuring of kvvfs working towards generic support of JS's Strorage interface as storage rather than hard-coding it to just window.localStorage and window.sessionStorage. The eventual goal of that is to have a storage solution which works without the POSIX I/O APIs, getting us one step closer to a wasi-sdk build. It would be transient, like Emscripten's virtual filesystem, and could hypothetically replace our dependency on that particular feature.
FossilOrigin-Name: 190ef4a94005b0feebe865a960d846a1b60ec1b267b15d5a24749f174a6169bc
2025-11-21 15:23:52 +00:00
253 changed files with 34301 additions and 11186 deletions
+52 -54
View File
@@ -788,17 +788,21 @@ BCC = $(BCC) /d2guard4 -D_ARM_WINAPI_PARTITION_DESKTOP_SDK_AVAILABLE
!IF $(DEBUG)>1
TCC = $(TCC) -MDd
BCC = $(BCC) -MDd
ZLIBCFLAGS = -nologo -MDd -W3 -O2 -Oy- -Zi
!ELSE
TCC = $(TCC) -MD
BCC = $(BCC) -MD
ZLIBCFLAGS = -nologo -MD -W3 -O2 -Oy- -Zi
!ENDIF
!ELSE
!IF $(DEBUG)>1
TCC = $(TCC) -MTd
BCC = $(BCC) -MTd
ZLIBCFLAGS = -nologo -MTd -W3 -O2 -Oy- -Zi
!ELSE
TCC = $(TCC) -MT
BCC = $(BCC) -MT
ZLIBCFLAGS = -nologo -MT -W3 -O2 -Oy- -Zi
!ENDIF
!ENDIF
@@ -1715,6 +1719,7 @@ FUZZERSHELL_COMPILE_OPTS =
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -I$(TOP)\test -I$(TOP)\ext\recover
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_ENABLE_MEMSYS5
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_OSS_FUZZ
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_DECIMAL_MAX_DIGIT=1000
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_ENABLE_BYTECODE_VTAB
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_ENABLE_CARRAY
FUZZCHECK_OPTS = $(FUZZCHECK_OPTS) -DSQLITE_ENABLE_DBPAGE_VTAB
@@ -1761,7 +1766,17 @@ FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\test\fuzzinvariants.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\test\vt02.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\recover\dbdata.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\recover\sqlite3recover.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\base64.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\base85.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\completion.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\decimal.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\ieee754.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\randomjson.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\regexp.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\series.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\shathree.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\sha1.c
FUZZCHECK_SRC = $(FUZZCHECK_SRC) $(TOP)\ext\misc\stmtrand.c
OSSSHELL_SRC = $(TOP)\test\ossshell.c $(TOP)\test\ossfuzz.c
DBFUZZ_COMPILE_OPTS = -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION
@@ -1862,8 +1877,8 @@ $(SQLITE3EXE): shell.c $(SHELL_CORE_DEP) $(LIBRESOBJS) $(SHELL_CORE_SRC) $(SQLIT
/link $(SQLITE3EXEPDB) $(LDFLAGS) $(LTLINKOPTS) $(SHELL_LINK_OPTS) $(LTLIBPATHS) $(LIBRESOBJS) $(LIBREADLINE) $(LTLIBS) $(TLIBS)
# <<mark>>
sqldiff.exe: $(TOP)\tool\sqldiff.c $(TOP)\ext\misc\sqlite3_stdio.h $(TOP)\ext\misc\sqlite3_stdio.c $(SQLITE3C) $(SQLITE3H) $(LIBRESOBJS)
$(LTLINK) $(NO_WARN) -I$(TOP)\ext\misc $(TOP)\tool\sqldiff.c $(TOP)\ext\misc\sqlite3_stdio.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS) $(LIBRESOBJS)
sqldiff.exe: $(TOP)\tool\sqldiff.c $(TOP)\ext\misc\sqlite3_stdio.h $(TOP)\ext\misc\sqlite3_stdio.c $(SQLITE3C) $(SQLITE3H) $(LIBRESOBJS) $(TOP)\tool\winmain.c
$(LTLINK) $(NO_WARN) -I$(TOP)\ext\misc $(TOP)\tool\sqldiff.c $(TOP)\ext\misc\sqlite3_stdio.c $(TOP)\tool\winmain.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS) $(LIBRESOBJS)
dbhash.exe: $(TOP)\tool\dbhash.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) $(TOP)\tool\dbhash.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
@@ -2362,7 +2377,7 @@ shell.c: $(SHELL_DEP) $(TOP)\tool\mkshellc.tcl $(JIM_TCLSH)
$(JIM_TCLSH) $(TOP)\tool\mkshellc.tcl shell.c
zlib:
pushd $(ZLIBDIR) && $(MAKE) /f win32\Makefile.msc clean $(ZLIBLIB) && popd
pushd $(ZLIBDIR) && $(MAKE) /f win32\Makefile.msc clean $(ZLIBLIB) CFLAGS="$(ZLIBCFLAGS)" && popd
# Rules to build the extension objects.
#
@@ -2536,43 +2551,19 @@ coretestprogs: testfixture.exe sqlite3.exe
testprogs: $(TESTPROGS) srcck1.exe fuzzcheck.exe sessionfuzz.exe
fulltest: alltest fuzztest
alltest: $(TESTPROGS)
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\all.test $(TESTOPTS)
soaktest: $(TESTPROGS)
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\all.test -soak=1 $(TESTOPTS)
fulltestonly: $(TESTPROGS) fuzztest
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\full.test
queryplantest: testfixture.exe shell
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\permutations.test queryplanner $(TESTOPTS)
fuzztest: fuzzcheck.exe
.\fuzzcheck.exe $(FUZZDATA)
# Legacy testing target for third-party integrators. The SQLite
# developers seldom use this target themselves. Instead
# they use "nmake /f Makefile.msc devtest" which runs tests on
# a standard set of options
#
test: $(TESTPROGS) sourcetest fuzztest tcltest
# Minimal testing that runs in less than 3 minutes (on a fast machine)
#
quicktest: testfixture.exe sourcetest
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\extraquick.test $(TESTOPTS)
# This is the common case. Run many tests that do not take too long,
# including fuzzcheck, sqlite3_analyzer, and sqldiff tests.
# Legacy testing targets, no longer used by the developers and
# now aliased to one of the commonly used testing targets.
#
quicktest: devtest
test: devtest
fulltest: releasetest
alltest: releasetest
soaktest: releasetest
fulltestonly: releasetest
# The veryquick.test TCL tests.
#
@@ -2597,6 +2588,11 @@ devtest: srctree-check sourcetest
mdevtest:
$(TCLSH_CMD) $(TOP)\test\testrunner.tcl mdevtest
# Rerun test cases that failed on the previous testrunner invocation
#
retest:
$(TCLSH_CMD) $(TOP)\test\testrunner.tcl retest
# Show all errors from the most reason testrunner invocation
#
errors:
@@ -2631,8 +2627,14 @@ smoketest: $(TESTPROGS)
@set PATH=$(LIBTCLPATH);$(PATH)
.\testfixture.exe $(TOP)\test\main.test $(TESTOPTS)
shelltest: $(TESTPROGS)
.\testfixture.exe $(TOP)\test\permutations.test shell
# Measure the performance of floating-point conversions.
#
fp-speed-test: fp-speed-1.exe fp-speed-2.exe
fp-speed-1 1000000
fp-speed-2 1000000
shelltest:
$(TCLSH_CMD) $(TOP)\test\testrunner.tcl release shell
sqlite3_analyzer.c: $(SQLITE3C) $(SQLITE3H) tclsqlite-ex.c $(TOP)\tool\spaceanal.tcl $(TOP)\tool\mkccode.tcl $(TOP)\tool\sqlite3_analyzer.c.in $(TOP)\ext\misc\sqlite3_stdio.h $(TOP)\ext\misc\sqlite3_stdio.c $(SQLITE_TCL_DEP)
$(TCLSH_CMD) $(TOP)\tool\mkccode.tcl -DINCLUDE_SQLITE3_C $(TOP)\tool\sqlite3_analyzer.c.in > $@
@@ -2651,23 +2653,6 @@ sqltclsh.exe: sqltclsh.c $(SHELL_CORE_DEP) $(LIBRESOBJS)
sqlite3_expert.exe: $(SQLITE3C) $(TOP)\ext\expert\sqlite3expert.h $(TOP)\ext\expert\sqlite3expert.c $(TOP)\ext\expert\expert.c
$(LTLINK) $(NO_WARN) $(TOP)\ext\expert\sqlite3expert.c $(TOP)\ext\expert\expert.c $(SQLITE3C) $(TLIBS)
CHECKER_DEPS =\
$(TOP)\tool\mkccode.tcl \
sqlite3.c \
tclsqlite-ex.c \
$(TOP)\ext\repair\sqlite3_checker.tcl \
$(TOP)\ext\repair\checkindex.c \
$(TOP)\ext\repair\checkfreelist.c \
$(TOP)\ext\misc\btreeinfo.c \
$(TOP)\ext\repair\sqlite3_checker.c.in
sqlite3_checker.c: $(CHECKER_DEPS)
$(TCLSH_CMD) $(TOP)\tool\mkccode.tcl $(TOP)\ext\repair\sqlite3_checker.c.in > $@
sqlite3_checker.exe: sqlite3_checker.c $(LIBRESOBJS)
$(LTLINK) $(NO_WARN) -DBUILD_sqlite -I$(TCLINCDIR) sqlite3_checker.c \
/link $(LDFLAGS) $(LTLINKOPTS) $(TCLLIBPATHS) $(LTLIBPATHS) $(LIBRESOBJS) $(TCLLIBS) $(LTLIBS) $(TLIBS)
dbdump.exe: $(TOP)\ext\misc\dbdump.c $(SQLITE3C) $(SQLITE3H) $(LIBRESOBJS)
$(LTLINK) $(NO_WARN) -DDBDUMP_STANDALONE $(TOP)\ext\misc\dbdump.c $(SQLITE3C) \
/link $(LDFLAGS) $(LTLINKOPTS) $(LTLIBPATHS) $(LIBRESOBJS) $(LTLIBS)
@@ -2700,6 +2685,9 @@ showwal.exe: $(TOP)\tool\showwal.c $(SQLITE3C) $(SQLITE3H)
showshm.exe: $(TOP)\tool\showshm.c
$(LTLINK) $(NO_WARN) $(TOP)\tool\showshm.c /link $(LDFLAGS) $(LTLINKOPTS)
showtmlog.exe: $(TOP)\tool\showtmlog.c
$(LTLINK) $(NO_WARN) $(TOP)\tool\showtmlog.c /link $(LDFLAGS) $(LTLINKOPTS)
index_usage.exe: $(TOP)\tool\index_usage.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION \
$(TOP)\tool\index_usage.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
@@ -2737,6 +2725,14 @@ speedtest1.exe: $(TOP)\test\speedtest1.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) $(ST_COMPILE_OPTS) -DSQLITE_OMIT_LOAD_EXTENSION \
$(TOP)\test\speedtest1.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
fp-speed-1.exe: $(TOP)\test\fp-speed-1.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) $(ST_COMPILE_OPTS) -DSQLITE_OMIT_LOAD_EXTENSION \
$(TOP)\test\fp-speed-1.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
fp-speed-2.exe: $(TOP)\test\fp-speed-2.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) $(ST_COMPILE_OPTS) -DSQLITE_OMIT_LOAD_EXTENSION \
$(TOP)\test\fp-speed-2.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
kvtest.exe: $(TOP)\test\kvtest.c $(SQLITE3C) $(SQLITE3H)
$(LTLINK) $(NO_WARN) $(KV_COMPILE_OPTS) \
$(TOP)\test\kvtest.c $(SQLITE3C) /link $(LDFLAGS) $(LTLINKOPTS)
@@ -2891,6 +2887,7 @@ env:
@echo USE_STDCALL = $(USE_STDCALL)
@echo USE_ZLIB = $(USE_ZLIB)
@echo XCOMPILE = $(XCOMPILE)
@echo ZLIBCFLAGS = $(ZLIBCFLAGS)
@echo ZLIBDIR = $(ZLIBDIR)
@echo ZLIBINCDIR = $(ZLIBINCDIR)
@echo ZLIBLIBDIR = $(ZLIBLIBDIR)
@@ -2925,6 +2922,7 @@ clean:
del /Q LogEst.exe fts3view.exe rollback-test.exe showdb.exe dbdump.exe 2>NUL
del /Q changeset.exe 2>NUL
del /Q showjournal.exe showstat4.exe showwal.exe speedtest1.exe 2>NUL
del /Q fp-speed-1.exe fp-speed-2.exe 2>NUL
del /Q mptester.exe wordcount.exe rbu.exe srcck1.exe 2>NUL
del /Q sqlite3.c sqlite3-*.c sqlite3.h 2>NUL
del /Q sqlite3rc.h 2>NUL
+1 -1
View File
@@ -1 +1 @@
3.52.0
3.53.0
+1 -1
View File
@@ -282,7 +282,7 @@ DIST_FILES := \
README.txt VERSION \
auto.def autosetup configure tea \
sqlite3.h sqlite3.c shell.c sqlite3ext.h \
Makefile.in Makefile.msc Makefile.fallback \
Makefile.in Makefile.msc Makefile.fallback make.bat \
sqlite3.rc sqlite3rc.h Replace.cs \
sqlite3.pc.in sqlite3.1
+4
View File
@@ -672,17 +672,21 @@ BCC = $(BCC) /d2guard4 -D_ARM_WINAPI_PARTITION_DESKTOP_SDK_AVAILABLE
!IF $(DEBUG)>1
TCC = $(TCC) -MDd
BCC = $(BCC) -MDd
ZLIBCFLAGS = -nologo -MDd -W3 -O2 -Oy- -Zi
!ELSE
TCC = $(TCC) -MD
BCC = $(BCC) -MD
ZLIBCFLAGS = -nologo -MD -W3 -O2 -Oy- -Zi
!ENDIF
!ELSE
!IF $(DEBUG)>1
TCC = $(TCC) -MTd
BCC = $(BCC) -MTd
ZLIBCFLAGS = -nologo -MTd -W3 -O2 -Oy- -Zi
!ELSE
TCC = $(TCC) -MT
BCC = $(BCC) -MT
ZLIBCFLAGS = -nologo -MT -W3 -O2 -Oy- -Zi
!ENDIF
!ENDIF
+1 -1
View File
@@ -461,4 +461,4 @@ all other significant processing.
[sqlite-config.tcl]: /file/autosetup/sqlite-config.tcl
[Makefile.in]: /file/Makefile.in
[main.mk]: /file/main.mk
[JimTCL]: https://jim.tcl.tk
[JimTCL]: https://msteveb.github.io/jimtcl/
+2 -2
View File
@@ -1842,7 +1842,7 @@ proc proj-setup-autoreconfig {defName} {
}
#
# @prop-append-to defineName args...
# @prop-define-append defineName args...
#
# A proxy for Autosetup's [define-append]. Appends all non-empty $args
# to [define-append $defineName].
@@ -1873,7 +1873,7 @@ proc proj-define-append {defineName args} {
# but it is technically correct and still relevant on some
# environments.
#
# See: proj-append-to
# See: proj-define-append
#
proc proj-define-amend {args} {
set defName ""
+23
View File
@@ -1138,6 +1138,27 @@ proc sqlite-check-line-editing {} {
set editLibName "readline" ; # "readline" or "editline"
set editLibDef "HAVE_READLINE" ; # "HAVE_READLINE" or "HAVE_EDITLINE"
set dirLn [opt-val with-linenoise]
# If none of --with-linenoise, --enable-readline, or --enable-editline
# are provided, but there exists a directory "linenoise" at $HOME or
# a sibling of the build or source directory, then try to use that linenoise
# direcctory.
#
if {"" eq $dirLn
&& ![proj-opt-was-provided readline]
&& ![proj-opt-was-provided editline]
} {
set dirlist ../linenoise
catch {lappend dirlist [file normalize $::autosetup(srcdir)/../linenoise]}
catch {lappend dirlist $::env(HOME)/linenoise}
foreach d $dirlist {
if {[file exists $d/linenoise.c] && [file exists $d/linenoise.h]} {
set dirLn $d
break
}
}
}
if {"" ne $dirLn} {
# Use linenoise from a copy of its sources (not a library)...
if {![file isdir $dirLn]} {
@@ -1173,6 +1194,8 @@ proc sqlite-check-line-editing {} {
if {$::sqliteConfig(use-jim-for-codegen) && 2 == $lnVal} {
define-append CFLAGS_JIMSH -DUSE_LINENOISE [get-define CFLAGS_READLINE]
user-notice "Adding linenoise support to jimsh."
} else {
msg-result "Using linenoise at [file normalize $dirLn]"
}
return "linenoise ($flavor)"
} elseif {[opt-bool editline]} {
+10 -3
View File
@@ -22,9 +22,10 @@ guidance on building for Windows.
or <https://sqlite.org/tmp/tcl9.0.0.tar.gz>.
<li>Untar the source archive. CD into the "unix/" subfolder
of the source tree.
<li>Run: `mkdir $HOME/local`
<li>Run: `./configure --prefix=$HOME/local`
<li>Run: `make install`
<li>Run: &ensp; `mkdir $HOME/local`
<li>Run: &ensp; `./configure --prefix=$HOME/local`<br>
SQLite deliverable builds add: &emsp; `--static CFLAGS=-Os`
<li>Run: &ensp; `make install`
</ol>
<p>
As of 2024-10-25, TCL is not longer required for many
@@ -41,6 +42,12 @@ guidance on building for Windows.
You do not need to use --with-tclsh if the tclsh you want to use is the
first one on your PATH or if you are building without TCL.
The SQLite developers typically add the
&ensp; `--with-linenoise=$HOME/linenoise` &ensp; option
to provide command-line editing. "$HOME/linenoise" is a directory
that contains [linenoise](https://github.com/antirez/linenoise) source
code files, `linenoise.c` and `linenoise.h`.
6. Run the "`Makefile`" makefile with an appropriate target.
Examples:
<ul>
+36
View File
@@ -180,3 +180,39 @@ statically linked so that it does not depend on separate DLL:
6. After your executable is built, you can verify that it does not
depend on the TCL DLL by running:
<blockquote><pre>dumpbin /dependents sqlite3_analyzer.exe</pre></blockquote>
## Linking Against ZLIB
Some feature (such as zip-file support in the CLI) require the ZLIB
compression library. That library is more or less universally available
on unix platforms, but is seldom provided on Windows. You will probably
need to provide it yourself. Here the the steps needed:
1. Download the zlib-1.3.1.tar.gz tarball (or a similar version).
Unpack the tarball sources. You can put them wherever you like.
For the purposes of this document, let's assume you put the source
tree in c:\\zlib-64. Note: If you are building for both x64 and
x86, you will need separate builds of ZLIB for each, thus separate
build directories.
2. Before building SQLite (as described above) first make these
environment changes. The lead-programmer for SQLite (who writes
these words) has BAT files named "env-x64.bat" and "env-x32.bat"
and "env-arm64.bat" that make these changes, and he runs those
BAT file whenever he starts a new shell. These are the settings
needed:
<blockquote>
set USE_ZLIB=1<br>
set BUILD_ZLIB=0<br>
set ZLIBDIR=c:\\zlib-64
</blockquote>
3. Because the settings in step 2 specify "BUILD_ZLIB=0", you will need
to build the library at least once. I recommand:
<blockquote>
make clean sqlite3.exe BUILD_ZLIB=1
</blockquote>
4. After making the environment changes specified in steps 1 through 3
above, you then build and test SQLite as you normally would. The
environment variable changes will cause ZLIB to be linked automatically.
+102 -57
View File
@@ -4,6 +4,12 @@
<ul type=none>
<li> 1. <a href=#overview>Overview</a>
<ul type=none>
<li> 1.1. <a href=#runtr>Running testrunner.tcl</a>
<li> 1.2. <a href=#runviamake>Run using "make"</a>
<li> 1.3. <a href=#outputs>Outputs from testrunner.tcl</a>
<li> 1.4. <a href=#help>Built-in help</a>
</ul>
<li> 2. <a href=#binary_tests>Binary Tests</a>
<ul type=none>
<li> 2.1. <a href=#organization_tests>Organization of Tcl Tests</a>
@@ -26,17 +32,40 @@
The testrunner.tcl program is a Tcl script used to run multiple SQLite
tests in parallel, thus reducing testing time on multi-core machines.
It supports the following types of tests:
The testrunner.tcl supports running tests that based on `testfixture`,
`sqlite3`, and `fuzzcheck`.
* Tcl test scripts.
<a name="runtr"></a>
## 1.1 Running testrunner.tcl
* Fuzzcheck tests, including using an external fuzzcheck database.
The testrunner.tcl script is located in the "test" subdirectory of the
SQLite source tree. So if your shell is current positioned at the top
of the source tree, you would normally run the script using the command:
"<tt>test/testrunner.tcl</tt>". On Windows, you have to specify the
`tclsh` interpreter command first, like this:
"<tt>tclsh&nbsp;test/testrunner.tcl</tt>".
* Tests run with `make` commands. Examples:
- `make devtest`
- `make releasetest`
- `make sdevtest`
- `make testrunner`
In this document, we will assume that you are on a unix-like OS
(not on Windows) and that your current directory is the root
of the SQLite source tree, and so all invocations of the testrunner.tcl
script will be of the form "<tt>test/testrunner.tcl</tt>". If you
are in a different directory, then make appropriate adjustments to
the path. On Windows, add the "<tt>tclsh</tt>" interpreter command
up front.
<a name="runviamake"></a>
## 1.2 Run using make
The standard Makefiles for SQLite include targets that invoke
testrunner.tcl. So the following commands also run testrunner.tcl:
* `make devtest`
* `make releasetest`
* `make sdevtest`
* `make testrunner`
<a name="outputs"></a>
## 1.3 Outputs from testrunner.tcl
The testrunner.tcl program stores output of all tests and builds run in
log file **testrunner.log**, created in the current working directory.
@@ -54,17 +83,19 @@ A useful query might be:
```
You can get a summary of errors in a prior run by invoking commands like
these:
those shown below. Note that the testrunner.tcl script can be run directly
on unix systems (including Macs) but you will need to add <tt>tclsh</tt>
to the front on Windows.
```
tclsh $(TESTDIR)/testrunner.tcl errors
tclsh $(TESTDIR)/testrunner.tcl errors -v
test/testrunner.tcl errors
test/testrunner.tcl errors -v
```
Running the command:
```
tclsh $(TESTDIR)/testrunner.tcl status
test/testrunner.tcl status
```
in the directory containing the testrunner.db database runs various queries
@@ -73,32 +104,43 @@ A good way to keep and eye on test progress is to run either of the two
following commands:
```
watch tclsh $(TESTDIR)/testrunner.tcl status
tclsh $(TESTDIR)/testrunner.tcl status -d 2
watch test/testrunner.tcl status
test/testrunner.tcl status -d 2
```
Both of the commands above accomplish about the same thing, but the second
one has the advantage of not requiring "watch" to be installed on your
system.
Sometimes testrunner.tcl uses the `testfixture` binary that it is run with
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
Sometimes testrunner.tcl uses the `testfixture` and `sqlite3` binaries that
are in the directory from which testrunner.tcl is run.
(see "Binary Tests" below). Sometimes it builds testfixture and
other binaries in specific configurations to test (see "Source Tests").
<a name=help></a>
## 1.4 Built-in help
Run this command:
```
test/testrunner.tcl help
```
To get a summary of all of the various command-line options available
with testrunner.tcl
<a name=binary_tests></a>
# 2. Binary Tests
The commands described in this section all run various combinations of the Tcl
test scripts using the `testfixture` binary used to run the testrunner.tcl
script (i.e. they do not invoke the compiler to build new binaries, or the
`make` command to run tests that are not Tcl scripts). The procedure to run
these tests is therefore:
1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
whatever method seems convenient.
2. Test the binary built in step 1 by running testrunner.tcl with it,
perhaps with various options.
test scripts using whatever `testfixture` binary (and maybe also the `sqlite3`
binary, depending on the test) that is found in the directory from which
testrunner.tcl is launched. So typically, one must first run something
like "`make testfixture sqlite3`" before launching binary tests. In other
words, testrunner.tcl does not automatically build the binaries under test
for binary tests. The testrunner.tcl expects the binaries to be available
already.
The following sub-sections describe the various options that can be
passed to testrunner.tcl to test binary testfixture builds.
@@ -140,22 +182,22 @@ are defined in file *testrunner_data.tcl*.
To run the "veryquick" test set, use either of the following:
```
./testfixture $TESTDIR/testrunner.tcl
./testfixture $TESTDIR/testrunner.tcl veryquick
test/testrunner.tcl
test/testrunner.tcl veryquick
```
To run the "full" test suite:
```
./testfixture $TESTDIR/testrunner.tcl full
test/testrunner.tcl full
```
To run the subset of the "full" test suite for which the test file name matches
a specified pattern (e.g. all tests that start with "fts5"), either of:
```
./testfixture $TESTDIR/testrunner.tcl fts5%
./testfixture $TESTDIR/testrunner.tcl 'fts5*'
test/testrunner.tcl fts5%
test/testrunner.tcl 'fts5*'
```
Strictly speaking, for a test to be run the pattern must match the script
@@ -167,7 +209,7 @@ characters specified as part of the pattern are transformed to "\*".
To run "all" tests (full + permutations):
```
./testfixture $TESTDIR/testrunner.tcl all
test/testrunner.tcl all
```
<a name=binary_test_failures></a>
@@ -186,10 +228,15 @@ If there is no permutation, the individual test script may be run with:
Or, if the failure occured as part of a permutation:
```
./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
test/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
```
TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?
One can also rerun all tests that failed or did not complete
in the previous invocation by typing:
```
test/testrunner.tcl retest
```
<a name=source_code_tests></a>
# 3. Source Code Tests
@@ -204,11 +251,9 @@ other tests. The advantages of this are that:
* it ensures that tests are always run using binaries created with the
same set of compiler options.
The testrunner.tcl commands described in this section may be run using
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".
TODO: ./configure + Makefile.msc build systems.
The testrunner.tcl commands described in this section do not require that
the testfixture and/or sqlite3 binaries be built ahead of time. Those
binaries will be constructed automatically.
<a name=commands_to_run_tests></a>
## 3.1. Commands to Run SQLite Tests
@@ -218,7 +263,7 @@ the `make fuzztest` target once for each of two --enable-all builds - one
with debugging enabled and one without:
```
tclsh $TESTDIR/testrunner.tcl mdevtest
test/testrunner.tcl mdevtest
```
In other words, it is equivalent to running:
@@ -227,13 +272,13 @@ In other words, it is equivalent to running:
$TOP/configure --enable-all --enable-debug
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
$TOP/test/testrunner.tcl veryquick
# Then, after removing files created by the tests above:
$TOP/configure --enable-all OPTS="-O0"
make fuzztest
make testfixture
./testfixture $TOP/test/testrunner.tcl veryquick
$TOP/test/testrunner.tcl veryquick
```
The **sdevtest** command is identical to the mdevtest command, except that the
@@ -241,7 +286,7 @@ second of the two builds is a sanitizer build. Specifically, this means that
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
```
tclsh $TESTDIR/testrunner.tcl sdevtest
test/testrunner.tcl sdevtest
```
The **release** command runs lots of tests under lots of builds. It runs
@@ -250,7 +295,7 @@ on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
of the specific tests run.
```
tclsh $TESTDIR/testrunner.tcl release
test/testrunner.tcl release
```
As with <a href=#source code tests>source code tests</a>, one or more patterns
@@ -258,7 +303,7 @@ may be appended to any of the above commands (mdevtest, sdevtest or release).
Pattern matching is used for both Tcl tests and fuzz tests.
```
tclsh $TESTDIR/testrunner.tcl release rtree%
test/testrunner.tcl release rtree%
```
<a name=zipvfs_tests></a>
@@ -268,14 +313,14 @@ testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
tests from the Zipvfs project with the following command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
```
This can be combined with any of "mdevtest", "sdevtest" or "release" to
test both SQLite and Zipvfs with a single command:
```
tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
```
<a name=source_code_test_failures></a>
@@ -295,10 +340,10 @@ a dos \*.bat file on windows. For example:
```
# Create a script that recreates build configuration "Device-One" on
# Linux or OSX:
tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh
test/testrunner.tcl script Device-One > make.sh
# Create a script that recreates build configuration "Have-Not" on Windows:
tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat
test/testrunner.tcl script Have-Not > make.bat
```
The generated bash or \*.bat file script accepts a single argument - a makefile
@@ -321,7 +366,7 @@ Thus, for example, to run a full releasetest including an external
dbsqlfuzz database, run a command like one of these:
```
tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
FUZZDB=../fuzz/20250415.db make releasetest
nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
```
@@ -331,7 +376,7 @@ databases. So if you want to run *only* tests involving the external
database, you can use a command something like this:
```
tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
```
<a name=testrunner_options></a>
@@ -345,7 +390,7 @@ required by a test, not to run any actual tests. For example:
```
# Build binaries required by release test.
tclsh $TESTDIR/testrunner.tcl --buildonly release"
test/testrunner.tcl --buildonly release"
```
The **--dryrun** option prevents testrunner.tcl from building any binaries
@@ -354,7 +399,7 @@ would normally execute into the testrunner.log file. Example:
```
# Log the shell commmands that make up the mdevtest test.
tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
test/testrunner.tcl --dryrun mdevtest"
```
The **--explain** option is similar to --dryrun in that it prevents
@@ -364,7 +409,7 @@ summary of all the builds and tests that would have been run.
```
# Show what builds and tests would have been run
tclsh $TESTDIR/testrunner.tcl --explain mdevtest
test/testrunner.tcl --explain mdevtest
```
The **--status** option uses VT100 escape sequences to display the test
@@ -380,7 +425,7 @@ When running either binary or source code tests, testrunner.tcl reports the
number of jobs it intends to use to stdout. e.g.
```
$ ./testfixture $TESTDIR/testrunner.tcl
$ test/testrunner.tcl
splitting work across 16 jobs
... more output ...
```
@@ -390,7 +435,7 @@ of real cores on the machine. This can be overridden using the "--jobs" (or -j)
switch:
```
$ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
$ test/testrunner.tcl --jobs 8
splitting work across 8 jobs
... more output ...
```
@@ -400,5 +445,5 @@ running by exucuting the following command from the directory containing the
testrunner.log and testrunner.db files:
```
$ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
$ test/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
```
+3 -6
View File
@@ -1816,9 +1816,7 @@ static int fts3CursorSeekStmt(Fts3Cursor *pCsr){
zSql = sqlite3_mprintf("SELECT %s WHERE rowid = ?", p->zReadExprlist);
if( !zSql ) return SQLITE_NOMEM;
p->bLock++;
rc = sqlite3_prepare_v3(
p->db, zSql,-1,SQLITE_PREPARE_PERSISTENT,&pCsr->pStmt,0
);
rc = sqlite3Fts3PrepareStmt(p, zSql, 1, 1, &pCsr->pStmt);
p->bLock--;
sqlite3_free(zSql);
}
@@ -3393,9 +3391,7 @@ static int fts3FilterMethod(
}
if( zSql ){
p->bLock++;
rc = sqlite3_prepare_v3(
p->db,zSql,-1,SQLITE_PREPARE_PERSISTENT,&pCsr->pStmt,0
);
rc = sqlite3Fts3PrepareStmt(p, zSql, 1, 1, &pCsr->pStmt);
p->bLock--;
sqlite3_free(zSql);
}else{
@@ -4018,6 +4014,7 @@ static int fts3IntegrityMethod(
UNUSED_PARAMETER(isQuick);
rc = sqlite3Fts3IntegrityCheck(p, &bOk);
assert( pVtab->zErrMsg==0 || rc!=SQLITE_OK );
assert( rc!=SQLITE_CORRUPT_VTAB );
if( rc==SQLITE_ERROR || (rc&0xFF)==SQLITE_CORRUPT ){
*pzErr = sqlite3_mprintf("unable to validate the inverted index for"
+19 -1
View File
@@ -203,7 +203,16 @@ typedef sqlite3_int64 i64; /* 8-byte signed integer */
#define LARGEST_INT64 (0xffffffff|(((i64)0x7fffffff)<<32))
#define SMALLEST_INT64 (((i64)-1) - LARGEST_INT64)
#define deliberate_fall_through
#if !defined(deliberate_fall_through)
# if defined(__has_attribute)
# if __has_attribute(fallthrough)
# define deliberate_fall_through __attribute__((fallthrough));
# endif
# endif
#endif
#if !defined(deliberate_fall_through)
# define deliberate_fall_through
#endif
/*
** Macros needed to provide flexible arrays in a portable way
@@ -601,6 +610,15 @@ int sqlite3Fts3Incrmerge(Fts3Table*,int,int);
(*(u8*)(p)&0x80) ? sqlite3Fts3GetVarint32(p, piVal) : (*piVal=*(u8*)(p), 1) \
)
int sqlite3Fts3PrepareStmt(
Fts3Table *p, /* Prepare for this connection */
const char *zSql, /* SQL to prepare */
int bPersist, /* True to set SQLITE_PREPARE_PERSISTENT */
int bAllowVtab, /* True to omit SQLITE_PREPARE_NO_VTAB */
sqlite3_stmt **pp /* OUT: Prepared statement */
);
/* fts3.c */
void sqlite3Fts3ErrMsg(char**,const char*,...);
int sqlite3Fts3PutVarint(char *, sqlite3_int64);
+1 -1
View File
@@ -325,7 +325,7 @@ static int fts3auxNextMethod(sqlite3_vtab_cursor *pCursor){
pCsr->aStat[1].nDoc++;
}
eState = 2;
/* fall through */
/* no break */ deliberate_fall_through
case 2:
if( v==0 ){ /* 0x00. Next integer will be a docid. */
+23 -5
View File
@@ -273,6 +273,24 @@ struct SegmentNode {
#define SQL_UPDATE_LEVEL_IDX 38
#define SQL_UPDATE_LEVEL 39
/*
** Wrapper around sqlite3_prepare_v3() to ensure that SQLITE_PREPARE_FROM_DDL
** is always set.
*/
int sqlite3Fts3PrepareStmt(
Fts3Table *p, /* Prepare for this connection */
const char *zSql, /* SQL to prepare */
int bPersist, /* True to set SQLITE_PREPARE_PERSISTENT */
int bAllowVtab, /* True to omit SQLITE_PREPARE_NO_VTAB */
sqlite3_stmt **pp /* OUT: Prepared statement */
){
int f = SQLITE_PREPARE_FROM_DDL
|((bAllowVtab==0) ? SQLITE_PREPARE_NO_VTAB : 0)
|(bPersist ? SQLITE_PREPARE_PERSISTENT : 0);
return sqlite3_prepare_v3(p->db, zSql, -1, f, pp, NULL);
}
/*
** This function is used to obtain an SQLite prepared statement handle
** for the statement identified by the second argument. If successful,
@@ -398,12 +416,12 @@ static int fts3SqlStmt(
pStmt = p->aStmt[eStmt];
if( !pStmt ){
int f = SQLITE_PREPARE_PERSISTENT|SQLITE_PREPARE_NO_VTAB;
int bAllowVtab = 0;
char *zSql;
if( eStmt==SQL_CONTENT_INSERT ){
zSql = sqlite3_mprintf(azSql[eStmt], p->zDb, p->zName, p->zWriteExprlist);
}else if( eStmt==SQL_SELECT_CONTENT_BY_ROWID ){
f &= ~SQLITE_PREPARE_NO_VTAB;
bAllowVtab = 1;
zSql = sqlite3_mprintf(azSql[eStmt], p->zReadExprlist);
}else{
zSql = sqlite3_mprintf(azSql[eStmt], p->zDb, p->zName);
@@ -411,7 +429,7 @@ static int fts3SqlStmt(
if( !zSql ){
rc = SQLITE_NOMEM;
}else{
rc = sqlite3_prepare_v3(p->db, zSql, -1, f, &pStmt, NULL);
rc = sqlite3Fts3PrepareStmt(p, zSql, 1, bAllowVtab, &pStmt);
sqlite3_free(zSql);
assert( rc==SQLITE_OK || pStmt==0 );
p->aStmt[eStmt] = pStmt;
@@ -3578,7 +3596,7 @@ static int fts3DoRebuild(Fts3Table *p){
if( !zSql ){
rc = SQLITE_NOMEM;
}else{
rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0);
rc = sqlite3Fts3PrepareStmt(p, zSql, 0, 1, &pStmt);
sqlite3_free(zSql);
}
@@ -5331,7 +5349,7 @@ int sqlite3Fts3IntegrityCheck(Fts3Table *p, int *pbOk){
if( !zSql ){
rc = SQLITE_NOMEM;
}else{
rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0);
rc = sqlite3Fts3PrepareStmt(p, zSql, 0, 1, &pStmt);
sqlite3_free(zSql);
}
+7 -1
View File
@@ -81,7 +81,13 @@ typedef sqlite3_uint64 u64;
# define FLEXARRAY 1
#endif
#endif
#endif /* SQLITE_AMALGAMATION */
/*
** Constants for the largest and smallest possible 32-bit signed integers.
*/
# define LARGEST_INT32 ((int)(0x7fffffff))
# define SMALLEST_INT32 ((int)((-1) - LARGEST_INT32))
/* Truncate very long tokens to this many bytes. Hard limit is
** (65536-1-1-4-9)==65521 bytes. The limiting factor is the 16-bit offset
+1 -1
View File
@@ -455,7 +455,7 @@ static void fts5SnippetFunction(
iBestCol = (iCol>=0 ? iCol : 0);
nPhrase = pApi->xPhraseCount(pFts);
aSeen = sqlite3_malloc(nPhrase);
aSeen = sqlite3_malloc64(nPhrase);
if( aSeen==0 ){
rc = SQLITE_NOMEM;
}
+1 -1
View File
@@ -288,7 +288,7 @@ char *sqlite3Fts5Strndup(int *pRc, const char *pIn, int nIn){
if( nIn<0 ){
nIn = (int)strlen(pIn);
}
zRet = (char*)sqlite3_malloc(nIn+1);
zRet = (char*)sqlite3_malloc64((i64)nIn+1);
if( zRet ){
memcpy(zRet, pIn, nIn);
zRet[nIn] = '\0';
+1 -3
View File
@@ -576,7 +576,7 @@ int sqlite3Fts5ConfigParse(
sqlite3_int64 nByte;
int bUnindexed = 0; /* True if there are one or more UNINDEXED */
*ppOut = pRet = (Fts5Config*)sqlite3_malloc(sizeof(Fts5Config));
*ppOut = pRet = (Fts5Config*)sqlite3_malloc64(sizeof(Fts5Config));
if( pRet==0 ) return SQLITE_NOMEM;
memset(pRet, 0, sizeof(Fts5Config));
pRet->pGlobal = pGlobal;
@@ -1123,5 +1123,3 @@ void sqlite3Fts5ConfigErrmsg(Fts5Config *pConfig, const char *zFmt, ...){
va_end(ap);
}
+2 -2
View File
@@ -314,7 +314,7 @@ int sqlite3Fts5ExprNew(
assert( sParse.rc!=SQLITE_OK || sParse.zErr==0 );
if( sParse.rc==SQLITE_OK ){
*ppNew = pNew = sqlite3_malloc(sizeof(Fts5Expr));
*ppNew = pNew = sqlite3_malloc64(sizeof(Fts5Expr));
if( pNew==0 ){
sParse.rc = SQLITE_NOMEM;
sqlite3Fts5ParseNodeFree(sParse.pExpr);
@@ -466,7 +466,7 @@ int sqlite3Fts5ExprAnd(Fts5Expr **pp1, Fts5Expr *p2){
p2->pRoot = 0;
if( sParse.rc==SQLITE_OK ){
Fts5ExprPhrase **ap = (Fts5ExprPhrase**)sqlite3_realloc(
Fts5ExprPhrase **ap = (Fts5ExprPhrase**)sqlite3_realloc64(
p1->apExprPhrase, nPhrase * sizeof(Fts5ExprPhrase*)
);
if( ap==0 ){
+1 -1
View File
@@ -91,7 +91,7 @@ int sqlite3Fts5HashNew(Fts5Config *pConfig, Fts5Hash **ppNew, int *pnByte){
int rc = SQLITE_OK;
Fts5Hash *pNew;
*ppNew = pNew = (Fts5Hash*)sqlite3_malloc(sizeof(Fts5Hash));
*ppNew = pNew = (Fts5Hash*)sqlite3_malloc64(sizeof(Fts5Hash));
if( pNew==0 ){
rc = SQLITE_NOMEM;
}else{
+41 -35
View File
@@ -2093,7 +2093,7 @@ static void fts5SegIterReverseInitPage(Fts5Index *p, Fts5SegIter *pIter){
/* If necessary, grow the pIter->aRowidOffset[] array. */
if( iRowidOffset>=pIter->nRowidOffset ){
int nNew = pIter->nRowidOffset + 8;
i64 nNew = pIter->nRowidOffset + 8;
int *aNew = (int*)sqlite3_realloc64(pIter->aRowidOffset,nNew*sizeof(int));
if( aNew==0 ){
p->rc = SQLITE_NOMEM;
@@ -5240,7 +5240,7 @@ static void fts5DoSecureDelete(
int iSegid = pSeg->pSeg->iSegid;
u8 *aPg = pSeg->pLeaf->p;
int nPg = pSeg->pLeaf->nn;
int iPgIdx = pSeg->pLeaf->szLeaf;
int iPgIdx = pSeg->pLeaf->szLeaf; /* Offset of page footer */
u64 iDelta = 0;
int iNextOff = 0;
@@ -5319,7 +5319,7 @@ static void fts5DoSecureDelete(
iSOP += fts5GetVarint32(&aPg[iSOP], nPos);
}
assert_nc( iSOP==pSeg->iLeafOffset );
iNextOff = pSeg->iLeafOffset + pSeg->nPos;
iNextOff = iSOP + pSeg->nPos;
}
}
@@ -5399,31 +5399,31 @@ static void fts5DoSecureDelete(
** is another term following it on this page. So the subsequent term
** needs to be moved to replace the term associated with the entry
** being removed. */
int nPrefix = 0;
int nSuffix = 0;
int nPrefix2 = 0;
int nSuffix2 = 0;
u64 nPrefix = 0;
u64 nSuffix = 0;
u64 nPrefix2 = 0;
u64 nSuffix2 = 0;
iDelKeyOff = iNextOff;
iNextOff += fts5GetVarint32(&aPg[iNextOff], nPrefix2);
iNextOff += fts5GetVarint32(&aPg[iNextOff], nSuffix2);
iNextOff += fts5GetVarint(&aPg[iNextOff], &nPrefix2);
iNextOff += fts5GetVarint(&aPg[iNextOff], &nSuffix2);
if( iKey!=1 ){
iKeyOff += fts5GetVarint32(&aPg[iKeyOff], nPrefix);
iKeyOff += fts5GetVarint(&aPg[iKeyOff], &nPrefix);
}
iKeyOff += fts5GetVarint32(&aPg[iKeyOff], nSuffix);
iKeyOff += fts5GetVarint(&aPg[iKeyOff], &nSuffix);
nPrefix = MIN(nPrefix, nPrefix2);
nSuffix = (nPrefix2 + nSuffix2) - nPrefix;
if( (iKeyOff+nSuffix)>iPgIdx || (iNextOff+nSuffix2)>iPgIdx ){
if( (iKeyOff+nSuffix)>(u64)iPgIdx || (iNextOff+nSuffix2)>(u64)iPgIdx ){
FTS5_CORRUPT_IDX(p);
}else{
if( iKey!=1 ){
iOff += sqlite3Fts5PutVarint(&aPg[iOff], nPrefix);
}
iOff += sqlite3Fts5PutVarint(&aPg[iOff], nSuffix);
if( nPrefix2>pSeg->term.n ){
if( nPrefix2>(u64)pSeg->term.n ){
FTS5_CORRUPT_IDX(p);
}else if( nPrefix2>nPrefix ){
memcpy(&aPg[iOff], &pSeg->term.p[nPrefix], nPrefix2-nPrefix);
@@ -5454,7 +5454,7 @@ static void fts5DoSecureDelete(
u8 *aTermIdx = &pTerm->p[pTerm->szLeaf];
int nTermIdx = pTerm->nn - pTerm->szLeaf;
int iTermIdx = 0;
int iTermOff = 0;
i64 iTermOff = 0;
while( 1 ){
u32 iVal = 0;
@@ -5465,12 +5465,15 @@ static void fts5DoSecureDelete(
}
nTermIdx = iTermIdx;
memmove(&pTerm->p[iTermOff], &pTerm->p[pTerm->szLeaf], nTermIdx);
fts5PutU16(&pTerm->p[2], iTermOff);
fts5DataWrite(p, iId, pTerm->p, iTermOff+nTermIdx);
if( nTermIdx==0 ){
fts5SecureDeleteIdxEntry(p, iSegid, pSeg->iTermLeafPgno);
if( iTermOff>pTerm->szLeaf ){
FTS5_CORRUPT_IDX(p);
}else{
memmove(&pTerm->p[iTermOff], &pTerm->p[pTerm->szLeaf], nTermIdx);
fts5PutU16(&pTerm->p[2], iTermOff);
fts5DataWrite(p, iId, pTerm->p, iTermOff+nTermIdx);
if( nTermIdx==0 ){
fts5SecureDeleteIdxEntry(p, iSegid, pSeg->iTermLeafPgno);
}
}
}
fts5DataRelease(pTerm);
@@ -5493,7 +5496,9 @@ static void fts5DoSecureDelete(
int iPrevKeyOut = 0;
int iKeyIn = 0;
memmove(&aPg[iOff], &aPg[iNextOff], nMove);
if( nMove>0 ){
memmove(&aPg[iOff], &aPg[iNextOff], nMove);
}
iPgIdx -= nShift;
nPg = iPgIdx;
fts5PutU16(&aPg[2], iPgIdx);
@@ -5931,7 +5936,7 @@ int sqlite3Fts5IndexMerge(Fts5Index *p, int nMerge){
fts5StructureRelease(pStruct);
pStruct = pNew;
nMin = 1;
nMerge = nMerge*-1;
nMerge = (nMerge==SMALLEST_INT32 ? LARGEST_INT32 : (nMerge*-1));
}
if( pStruct && pStruct->nLevel ){
if( fts5IndexMerge(p, &pStruct, nMerge, nMin) ){
@@ -6413,16 +6418,16 @@ struct Fts5TokenDataMap {
** aMap[] variables.
*/
struct Fts5TokenDataIter {
int nMapAlloc; /* Allocated size of aMap[] in entries */
int nMap; /* Number of valid entries in aMap[] */
i64 nMapAlloc; /* Allocated size of aMap[] in entries */
i64 nMap; /* Number of valid entries in aMap[] */
Fts5TokenDataMap *aMap; /* Array of (rowid+pos -> token) mappings */
/* The following are used for prefix-queries only. */
Fts5Buffer terms;
/* The following are used for other full-token tokendata queries only. */
int nIter;
int nIterAlloc;
i64 nIter;
i64 nIterAlloc;
Fts5PoslistReader *aPoslistReader;
int *aPoslistToIter;
Fts5Iter *apIter[FLEXARRAY];
@@ -6478,11 +6483,11 @@ static void fts5TokendataIterAppendMap(
){
if( p->rc==SQLITE_OK ){
if( pT->nMap==pT->nMapAlloc ){
int nNew = pT->nMapAlloc ? pT->nMapAlloc*2 : 64;
int nAlloc = nNew * sizeof(Fts5TokenDataMap);
i64 nNew = pT->nMapAlloc ? pT->nMapAlloc*2 : 64;
i64 nAlloc = nNew * sizeof(Fts5TokenDataMap);
Fts5TokenDataMap *aNew;
aNew = (Fts5TokenDataMap*)sqlite3_realloc(pT->aMap, nAlloc);
aNew = (Fts5TokenDataMap*)sqlite3_realloc64(pT->aMap, nAlloc);
if( aNew==0 ){
p->rc = SQLITE_NOMEM;
return;
@@ -6508,7 +6513,7 @@ static void fts5TokendataIterAppendMap(
*/
static void fts5TokendataIterSortMap(Fts5Index *p, Fts5TokenDataIter *pT){
Fts5TokenDataMap *aTmp = 0;
int nByte = pT->nMap * sizeof(Fts5TokenDataMap);
i64 nByte = pT->nMap * sizeof(Fts5TokenDataMap);
aTmp = (Fts5TokenDataMap*)sqlite3Fts5MallocZero(&p->rc, nByte);
if( aTmp ){
@@ -7042,9 +7047,10 @@ static Fts5TokenDataIter *fts5AppendTokendataIter(
if( p->rc==SQLITE_OK ){
if( pIn==0 || pIn->nIter==pIn->nIterAlloc ){
int nAlloc = pIn ? pIn->nIterAlloc*2 : 16;
int nByte = SZ_FTS5TOKENDATAITER(nAlloc+1);
Fts5TokenDataIter *pNew = (Fts5TokenDataIter*)sqlite3_realloc(pIn, nByte);
i64 nAlloc = pIn ? pIn->nIterAlloc*2 : 16;
i64 nByte = SZ_FTS5TOKENDATAITER(nAlloc+1);
Fts5TokenDataIter *pNew;
pNew = (Fts5TokenDataIter*)sqlite3_realloc64(pIn, nByte);
if( pNew==0 ){
p->rc = SQLITE_NOMEM;
@@ -7141,8 +7147,8 @@ static void fts5IterSetOutputsTokendata(Fts5Iter *pIter){
/* Ensure the token-mapping is large enough */
if( eDetail==FTS5_DETAIL_FULL && pT->nMapAlloc<(pT->nMap + nByte) ){
int nNew = (pT->nMapAlloc + nByte) * 2;
Fts5TokenDataMap *aNew = (Fts5TokenDataMap*)sqlite3_realloc(
i64 nNew = (pT->nMapAlloc + nByte) * 2;
Fts5TokenDataMap *aNew = (Fts5TokenDataMap*)sqlite3_realloc64(
pT->aMap, nNew*sizeof(Fts5TokenDataMap)
);
if( aNew==0 ){
+48 -22
View File
@@ -517,7 +517,7 @@ static void fts5SetEstimatedRows(sqlite3_index_info *pIdxInfo, i64 nRow){
if( sqlite3_libversion_number()>=3008002 )
#endif
{
pIdxInfo->estimatedRows = nRow;
pIdxInfo->estimatedRows = MAX(1, nRow);
}
#endif
}
@@ -586,19 +586,30 @@ static int fts5UsePatternMatch(
** a) If a MATCH operator is present, the cost depends on the other
** constraints also present. As follows:
**
** * No other constraints: cost=1000.0
** * One rowid range constraint: cost=750.0
** * Both rowid range constraints: cost=500.0
** * An == rowid constraint: cost=100.0
** * No other constraints: cost=50000.0
** * One rowid range constraint: cost=37500.0
** * Both rowid range constraints: cost=30000.0
** * An == rowid constraint: cost=25000.0
**
** b) Otherwise, if there is no MATCH:
**
** * No other constraints: cost=1000000.0
** * One rowid range constraint: cost=750000.0
** * Both rowid range constraints: cost=250000.0
** * An == rowid constraint: cost=10.0
** * No other constraints: cost=3000000.0
** * One rowid range constraints: cost=2250000.0
** * Both rowid range constraint: cost=750000.0
** * An == rowid constraint: cost=25.0
**
** Costs are not modified by the ORDER BY clause.
**
** The ratios used in case (a) are based on informal results obtained from
** the tool/fts5cost.tcl script. The "MATCH and ==" combination has the
** cost set quite high because the query may be a prefix query. Unless
** there is a prefix index, prefix queries with rowid constraints are much
** more expensive than non-prefix queries with rowid constraints.
**
** The estimated rows returned is set to the cost/40. For simple queries,
** experimental results show that cost/4 might be about right. But for
** more complex queries that use multiple terms the number of rows might
** be far fewer than this. So we compromise and use cost/40.
*/
static int fts5BestIndexMethod(sqlite3_vtab *pVTab, sqlite3_index_info *pInfo){
Fts5Table *pTab = (Fts5Table*)pVTab;
@@ -631,7 +642,7 @@ static int fts5BestIndexMethod(sqlite3_vtab *pVTab, sqlite3_index_info *pInfo){
return SQLITE_ERROR;
}
idxStr = (char*)sqlite3_malloc(pInfo->nConstraint * 8 + 1);
idxStr = (char*)sqlite3_malloc64((i64)pInfo->nConstraint * 8 + 1);
if( idxStr==0 ) return SQLITE_NOMEM;
pInfo->idxStr = idxStr;
pInfo->needToFreeIdxStr = 1;
@@ -724,21 +735,35 @@ static int fts5BestIndexMethod(sqlite3_vtab *pVTab, sqlite3_index_info *pInfo){
/* Calculate the estimated cost based on the flags set in idxFlags. */
if( bSeenEq ){
pInfo->estimatedCost = nSeenMatch ? 1000.0 : 25.0;
fts5SetUniqueFlag(pInfo);
pInfo->estimatedCost = nSeenMatch ? 25000.0 : 25.0;
fts5SetEstimatedRows(pInfo, 1);
fts5SetUniqueFlag(pInfo);
}else{
if( bSeenLt && bSeenGt ){
pInfo->estimatedCost = nSeenMatch ? 5000.0 : 750000.0;
}else if( bSeenLt || bSeenGt ){
pInfo->estimatedCost = nSeenMatch ? 7500.0 : 2250000.0;
i64 nEstRows;
if( nSeenMatch ){
if( bSeenLt && bSeenGt ){
pInfo->estimatedCost = 50000.0;
}else if( bSeenLt || bSeenGt ){
pInfo->estimatedCost = 37500.0;
}else{
pInfo->estimatedCost = 50000.0;
}
nEstRows = (i64)(pInfo->estimatedCost / 40.0);
for(i=1; i<nSeenMatch; i++){
pInfo->estimatedCost *= 2.5;
nEstRows = nEstRows / 2;
}
}else{
pInfo->estimatedCost = nSeenMatch ? 10000.0 : 3000000.0;
if( bSeenLt && bSeenGt ){
pInfo->estimatedCost = 750000.0;
}else if( bSeenLt || bSeenGt ){
pInfo->estimatedCost = 2250000.0;
}else{
pInfo->estimatedCost = 3000000.0;
}
nEstRows = (i64)(pInfo->estimatedCost / 4.0);
}
for(i=1; i<nSeenMatch; i++){
pInfo->estimatedCost *= 0.4;
}
fts5SetEstimatedRows(pInfo, (i64)(pInfo->estimatedCost / 4.0));
fts5SetEstimatedRows(pInfo, nEstRows);
}
pInfo->idxNum = idxFlags;
@@ -2081,6 +2106,7 @@ static int fts5UpdateMethod(
}
update_out:
sqlite3Fts5IndexCloseReader(pTab->p.pIndex);
pTab->p.pConfig->pzErrmsg = 0;
return rc;
}
@@ -3762,7 +3788,7 @@ static int fts5Init(sqlite3 *db){
int rc;
Fts5Global *pGlobal = 0;
pGlobal = (Fts5Global*)sqlite3_malloc(sizeof(Fts5Global));
pGlobal = (Fts5Global*)sqlite3_malloc64(sizeof(Fts5Global));
if( pGlobal==0 ){
rc = SQLITE_NOMEM;
}else{
+4 -4
View File
@@ -391,7 +391,7 @@ static int SQLITE_TCLAPI xF5tApi(
break;
}
CASE(12, "xSetAuxdata") {
F5tAuxData *pData = (F5tAuxData*)sqlite3_malloc(sizeof(F5tAuxData));
F5tAuxData *pData = (F5tAuxData*)sqlite3_malloc64(sizeof(F5tAuxData));
if( pData==0 ){
Tcl_AppendResult(interp, "out of memory", (char*)0);
return TCL_ERROR;
@@ -780,7 +780,7 @@ static int SQLITE_TCLAPI f5tTokenize(
}
if( nText>0 ){
pCopy = sqlite3_malloc(nText);
pCopy = sqlite3_malloc64(nText);
if( pCopy==0 ){
tokenizer.xDelete(pTok);
Tcl_AppendResult(interp, "error in sqlite3_malloc()", (char*)0);
@@ -1420,7 +1420,7 @@ static int f5tOrigintextCreate(
void *pTokCtx = 0;
int rc = SQLITE_OK;
pTok = (OriginTextTokenizer*)sqlite3_malloc(sizeof(OriginTextTokenizer));
pTok = (OriginTextTokenizer*)sqlite3_malloc64(sizeof(OriginTextTokenizer));
if( pTok==0 ){
rc = SQLITE_NOMEM;
}else if( nArg<1 ){
@@ -1480,7 +1480,7 @@ static int xOriginToken(
int nReq = nToken + 1 + (iEnd-iStart);
if( nReq>p->nBuf ){
sqlite3_free(p->aBuf);
p->aBuf = sqlite3_malloc(nReq*2);
p->aBuf = sqlite3_malloc64(nReq*2);
if( p->aBuf==0 ) return SQLITE_NOMEM;
p->nBuf = nReq*2;
}
+5 -5
View File
@@ -194,7 +194,7 @@ static int fts5tokConnectMethod(
}
if( rc==SQLITE_OK ){
pTab = (Fts5tokTable*)sqlite3_malloc(sizeof(Fts5tokTable));
pTab = (Fts5tokTable*)sqlite3_malloc64(sizeof(Fts5tokTable));
if( pTab==0 ){
rc = SQLITE_NOMEM;
}else{
@@ -275,7 +275,7 @@ static int fts5tokBestIndexMethod(
static int fts5tokOpenMethod(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCsr){
Fts5tokCursor *pCsr;
pCsr = (Fts5tokCursor *)sqlite3_malloc(sizeof(Fts5tokCursor));
pCsr = (Fts5tokCursor *)sqlite3_malloc64(sizeof(Fts5tokCursor));
if( pCsr==0 ){
return SQLITE_NOMEM;
}
@@ -347,7 +347,7 @@ static int fts5tokCb(
if( pCsr->nRow ){
pRow->iPos = pRow[-1].iPos + ((tflags & FTS5_TOKEN_COLOCATED) ? 0 : 1);
}
pRow->zToken = sqlite3_malloc(nToken+1);
pRow->zToken = sqlite3_malloc64((sqlite3_int64)nToken+1);
if( pRow->zToken==0 ) return SQLITE_NOMEM;
memcpy(pRow->zToken, pToken, nToken);
pRow->zToken[nToken] = 0;
@@ -373,8 +373,8 @@ static int fts5tokFilterMethod(
fts5tokResetCursor(pCsr);
if( idxNum==1 ){
const char *zByte = (const char *)sqlite3_value_text(apVal[0]);
int nByte = sqlite3_value_bytes(apVal[0]);
pCsr->zInput = sqlite3_malloc(nByte+1);
sqlite3_int64 nByte = sqlite3_value_bytes(apVal[0]);
pCsr->zInput = sqlite3_malloc64(nByte+1);
if( pCsr->zInput==0 ){
rc = SQLITE_NOMEM;
}else{
+4 -4
View File
@@ -72,7 +72,7 @@ static int fts5AsciiCreate(
if( nArg%2 ){
rc = SQLITE_ERROR;
}else{
p = sqlite3_malloc(sizeof(AsciiTokenizer));
p = sqlite3_malloc64(sizeof(AsciiTokenizer));
if( p==0 ){
rc = SQLITE_NOMEM;
}else{
@@ -367,7 +367,7 @@ static int fts5UnicodeCreate(
if( nArg%2 ){
rc = SQLITE_ERROR;
}else{
p = (Unicode61Tokenizer*)sqlite3_malloc(sizeof(Unicode61Tokenizer));
p = (Unicode61Tokenizer*)sqlite3_malloc64(sizeof(Unicode61Tokenizer));
if( p ){
const char *zCat = "L* N* Co";
int i;
@@ -590,7 +590,7 @@ static int fts5PorterCreate(
zBase = azArg[0];
}
pRet = (PorterTokenizer*)sqlite3_malloc(sizeof(PorterTokenizer));
pRet = (PorterTokenizer*)sqlite3_malloc64(sizeof(PorterTokenizer));
if( pRet ){
memset(pRet, 0, sizeof(PorterTokenizer));
rc = pApi->xFindTokenizer_v2(pApi, zBase, &pUserdata, &pV2);
@@ -1297,7 +1297,7 @@ static int fts5TriCreate(
rc = SQLITE_ERROR;
}else{
int i;
pNew = (TrigramTokenizer*)sqlite3_malloc(sizeof(*pNew));
pNew = (TrigramTokenizer*)sqlite3_malloc64(sizeof(*pNew));
if( pNew==0 ){
rc = SQLITE_NOMEM;
}else{
+1 -1
View File
@@ -666,7 +666,7 @@ static int fts5VocabFilterMethod(
const char *zCopy = (const char *)sqlite3_value_text(pLe);
if( zCopy==0 ) zCopy = "";
pCsr->nLeTerm = sqlite3_value_bytes(pLe);
pCsr->zLeTerm = sqlite3_malloc(pCsr->nLeTerm+1);
pCsr->zLeTerm = sqlite3_malloc64((i64)pCsr->nLeTerm+1);
if( pCsr->zLeTerm==0 ){
rc = SQLITE_NOMEM;
}else{
File diff suppressed because one or more lines are too long
-3
View File
@@ -379,9 +379,6 @@ do_execsql_test 12.2 {
db close
sqlite3 db test.db -readonly 1
explain_i {
PRAGMA integrity_check
}
do_execsql_test 12.3 {
PRAGMA integrity_check
} {ok}
+17
View File
@@ -64,4 +64,21 @@ foreach {tn sql} {
}
}
#-------------------------------------------------------------------------
# Verify that https://sqlite.org/forum/forumpost/95413eb410 has been
# fixed.
#
reset_db
do_execsql_test 2.0 {
CREATE VIRTUAL TABLE f1 USING fts5(x);
BEGIN TRANSACTION;
INSERT INTO f1(x) VALUES('abc def ghi');
}
do_test 2.1 {
sqlite3_interrupt db
} {}
do_execsql_test 2.2 {
ROLLBACK
}
finish_test
+9
View File
@@ -65,5 +65,14 @@ do_eqp_test 1.4 {
`--SCAN vt VIRTUAL TABLE INDEX 0:=
}
do_eqp_test 1.5 {
SELECT * FROM vt, t1
WHERE vt.rowid = t1.rowid AND vt MATCH ? AND b = ?
} {
QUERY PLAN
|--SCAN vt VIRTUAL TABLE INDEX 0:M1
`--SEARCH t1 USING INTEGER PRIMARY KEY (rowid=?)
}
finish_test
+16
View File
@@ -238,6 +238,22 @@ do_execsql_test 6.3 {
INSERT INTO g1(g1) VALUES('integrity-check');
}
#--------------------------------------------------------------------------
# Check that passing -2147483648 as the parameter to a merge command
# does not cause a signed integer overflow error.
#
reset_db
do_execsql_test 7.0 {
CREATE VIRTUAL TABLE f1 USING fts5(a);
}
do_execsql_test 7.1 {
INSERT INTO f1 VALUES('one two three');
INSERT INTO f1 VALUES('four five six');
INSERT INTO f1 VALUES('seven eight nine');
}
do_execsql_test 7.2 {
INSERT INTO f1(f1, rank) VALUES('merge', -2147483648);
}
finish_test
+153
View File
@@ -0,0 +1,153 @@
#
# 2026 March 20
#
# The author disclaims copyright to this source code. In place of
# a legal notice, here is a blessing:
#
# May you do good and not evil.
# May you find forgiveness for yourself and forgive others.
# May you share freely, never taking more than you give.
#
#--------------------------------------------------------------------------
#
# This script extracts the documentation for the API used by fts5 auxiliary
# functions from header file fts5.h. It outputs html text on stdout that
# is included in the documentation on the web.
#
sqlite3 db fts5cost.db
# Create an IPK table with 1,000,000 entries. Short records.
#
set res [list [catch { db eval {SELECT count(*) FROM t1} } msg] $msg]
if {$res!="0 1000000"} {
db eval {
PRAGMA mmap_size = 1000000000; -- 1GB
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT);
WITH s(i) AS (
SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<1_000_000
)
INSERT INTO t1 SELECT i, hex(randomblob(8)) FROM s;
}
}
# Create an FTS5 table with 1,000,000 entries. Each row contains a single
# column containing a document of 100 terms chosen pseudo-randomly from
# a vocabularly of 2000.
set res [list [catch { db eval {SELECT count(*) FROM f1} } msg] $msg]
if {$res!="0 1000000"} {
set nVocab 2000
set nTerm 100
db eval {
BEGIN;
DROP TABLE IF EXISTS vocab1;
CREATE TABLE vocab1(w);
}
for {set ii 0} {$ii<$nVocab} {incr ii} {
set word [format %06x [expr {int(abs(rand()) * 0xFFFFFF)}]]
db eval { INSERT INTO vocab1 VALUES($word) }
lappend lVocab $word
}
db func doc doc
proc doc {} {
for {set ii 0} {$ii<$::nTerm} {incr ii} {
lappend ret [lindex $::lVocab [expr int(abs(rand())*$::nVocab)]]
}
set ret
}
db eval {
DROP TABLE IF EXISTS f1;
CREATE VIRTUAL TABLE f1 USING fts5(x);
WITH s(i) AS (
SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<1_000_000
)
INSERT INTO f1(rowid, x) SELECT i, doc() FROM s;
COMMIT;
}
} else {
set lVocab [db eval { SELECT * FROM vocab1 }]
set nVocab [llength $lVocab]
}
proc rowid_query {n} {
set rowid 654
for {set ii 0} {$ii<$n} {incr ii} {
db eval { SELECT b FROM t1 WHERE a = $rowid }
set rowid [expr {($rowid + 7717) % 1000000}]
}
}
proc rowid_query_fts {n} {
set rowid 654
for {set ii 0} {$ii<$n} {incr ii} {
db eval { SELECT * FROM f1 WHERE rowid = $rowid }
set rowid [expr {($rowid + 7717) % 1000000}]
}
}
proc match_query_fts {n} {
set idx 654
for {set ii 0} {$ii<$n} {incr ii} {
set match [lrange $::lVocab $idx $idx+1]
db eval { SELECT * FROM f1($match) }
set idx [expr {($idx + 7717) % $::nVocab}]
}
}
proc prefix_query_fts {n} {
set idx 654
for {set ii 0} {$ii<$n} {incr ii} {
set match "[lindex $::lVocab $idx]*"
db eval { SELECT * FROM f1($match) }
set idx [expr {($idx + 7717) % $::nVocab}]
}
}
proc match_rowid_query_fts {n} {
set idx 654
for {set ii 0} {$ii<$n} {incr ii} {
set match "[lindex $::lVocab $idx]"
db eval { SELECT * FROM f1($match) WHERE rowid=500000 }
set idx [expr {($idx + 7717) % $::nVocab}]
}
}
proc prefix_rowid_query_fts {n} {
set idx 654
for {set ii 0} {$ii<$n} {incr ii} {
set match "[lindex $::lVocab $idx]*"
db eval { SELECT * FROM f1($match) WHERE rowid=500000 }
set idx [expr {($idx + 7717) % $::nVocab}]
}
}
proc mytime {cmd div} {
set tm [time $cmd]
expr {[lindex $tm 0] / $div}
}
#set us [mytime { match_rowid_query_fts 1000 } 1000]
#puts "1000 match/rowid queries on fts5 table: ${us} per query"
set us [mytime { prefix_rowid_query_fts 1000 } 1000]
puts "1000 prefix/rowid queries on fts5 table: ${us} per query"
set us [mytime { match_query_fts 10 } 10]
puts "10 match queries on fts5 table: ${us} per query"
set us [mytime { prefix_query_fts 10 } 10]
puts "10 prefix queries on fts5 table: ${us} per query"
set us [mytime { prefix_rowid_query_fts 1000 } 1000]
puts "1000 prefix/rowid queries on fts5 table: ${us} per query"
set us [mytime { rowid_query 10000 } 10000]
puts "10000 by-rowid queries on normal table: ${us} per query"
set us [mytime { rowid_query_fts 10000 } 10000]
puts "10000 by-rowid queries on fts5 table: ${us} per query"
+1 -1
View File
@@ -319,7 +319,7 @@ static int intckGetToken(const char *z){
char c = z[0];
int iRet = 1;
if( c=='\'' || c=='"' || c=='`' ){
while( 1 ){
while( z[iRet] ){
if( z[iRet]==c ){
iRet++;
if( z[iRet]!=c ) break;
+1 -1
View File
@@ -275,7 +275,7 @@ static void base64(sqlite3_context *context, int na, sqlite3_value *av[]){
#ifdef _WIN32
__declspec(dllexport)
#endif
int sqlite3_base_init
int sqlite3_base64_init
#else
static int sqlite3_base64_init
#endif
+7 -7
View File
@@ -262,7 +262,7 @@ static int allBase85( char *p, int len ){
#ifndef BASE85_STANDALONE
# ifndef OMIT_BASE85_CHECKER
#ifndef OMIT_BASE85_CHECKER
/* This function does the work for the SQLite is_base85(t) UDF. */
static void is_base85(sqlite3_context *context, int na, sqlite3_value *av[]){
assert(na==1);
@@ -282,7 +282,7 @@ static void is_base85(sqlite3_context *context, int na, sqlite3_value *av[]){
return;
}
}
# endif
#endif
/* This function does the work for the SQLite base85(x) UDF. */
static void base85(sqlite3_context *context, int na, sqlite3_value *av[]){
@@ -352,14 +352,14 @@ static void base85(sqlite3_context *context, int na, sqlite3_value *av[]){
#ifdef _WIN32
__declspec(dllexport)
#endif
int sqlite3_base_init
int sqlite3_base85_init
#else
static int sqlite3_base85_init
#endif
(sqlite3 *db, char **pzErr, const sqlite3_api_routines *pApi){
SQLITE_EXTENSION_INIT2(pApi);
(void)pzErr;
# ifndef OMIT_BASE85_CHECKER
#ifndef OMIT_BASE85_CHECKER
{
int rc = sqlite3_create_function
(db, "is_base85", 1,
@@ -367,7 +367,7 @@ static int sqlite3_base85_init
0, is_base85, 0, 0);
if( rc!=SQLITE_OK ) return rc;
}
# endif
#endif
return sqlite3_create_function
(db, "base85", 1,
SQLITE_DETERMINISTIC|SQLITE_INNOCUOUS|SQLITE_DIRECTONLY|SQLITE_UTF8,
@@ -432,9 +432,9 @@ int main(int na, char *av[]){
int nc = strlen(cBuf);
size_t nbo = fromBase85( cBuf, nc, bBuf ) - bBuf;
if( 1 != fwrite(bBuf, nbo, 1, fb) ) rc = 1;
# ifndef OMIT_BASE85_CHECKER
#ifndef OMIT_BASE85_CHECKER
b85Clean &= allBase85( cBuf, nc );
# endif
#endif
}
break;
default:
+5 -1
View File
@@ -306,6 +306,10 @@ static int binfoCompute(sqlite3 *db, int pgno, BinfoCursor *pCsr){
nEntry *= (nCell+1);
if( aData[0]==10 || aData[0]==13 ) break;
nPage *= (nCell+1);
if( 14+2*(nCell/2)>=pgsz ){
rc = SQLITE_CORRUPT;
break;
}
if( nCell<=1 ){
pgno = get_uint32(aData+8);
}else{
@@ -339,7 +343,7 @@ static int binfoColumn(
sqlite3 *db = sqlite3_context_db_handle(ctx);
int rc = binfoCompute(db, pgno, pCsr);
if( rc ){
pCursor->pVtab->zErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(db));
pCursor->pVtab->zErrMsg = sqlite3_mprintf("%s", sqlite3_errstr(rc));
return SQLITE_ERROR;
}
}
+25 -13
View File
@@ -199,6 +199,7 @@ static int completionNext(sqlite3_vtab_cursor *cur){
completion_cursor *pCur = (completion_cursor*)cur;
int eNextPhase = 0; /* Next phase to try if current phase reaches end */
int iCol = -1; /* If >=0, step pCur->pStmt and use the i-th column */
int rc;
pCur->iRowid++;
while( pCur->ePhase!=COMPLETION_EOF ){
switch( pCur->ePhase ){
@@ -224,22 +225,27 @@ static int completionNext(sqlite3_vtab_cursor *cur){
case COMPLETION_TABLES: {
if( pCur->pStmt==0 ){
sqlite3_stmt *pS2;
sqlite3_str* pStr = sqlite3_str_new(pCur->db);
char *zSql = 0;
const char *zSep = "";
sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, &pS2, 0);
while( sqlite3_step(pS2)==SQLITE_ROW ){
const char *zDb = (const char*)sqlite3_column_text(pS2, 1);
zSql = sqlite3_mprintf(
"%z%s"
sqlite3_str_appendf(pStr,
"%s"
"SELECT name FROM \"%w\".sqlite_schema",
zSql, zSep, zDb
zSep, zDb
);
if( zSql==0 ) return SQLITE_NOMEM;
zSep = " UNION ";
}
sqlite3_finalize(pS2);
sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0);
rc = sqlite3_finalize(pS2);
zSql = sqlite3_str_finish(pStr);
if( zSql==0 ) return SQLITE_NOMEM;
if( rc==SQLITE_OK ){
sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0);
}
sqlite3_free(zSql);
if( rc ) return rc;
}
iCol = 0;
eNextPhase = COMPLETION_COLUMNS;
@@ -248,24 +254,29 @@ static int completionNext(sqlite3_vtab_cursor *cur){
case COMPLETION_COLUMNS: {
if( pCur->pStmt==0 ){
sqlite3_stmt *pS2;
sqlite3_str *pStr = sqlite3_str_new(pCur->db);
char *zSql = 0;
const char *zSep = "";
sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, &pS2, 0);
while( sqlite3_step(pS2)==SQLITE_ROW ){
const char *zDb = (const char*)sqlite3_column_text(pS2, 1);
zSql = sqlite3_mprintf(
"%z%s"
sqlite3_str_appendf(pStr,
"%s"
"SELECT pti.name FROM \"%w\".sqlite_schema AS sm"
" JOIN pragma_table_xinfo(sm.name,%Q) AS pti"
" WHERE sm.type='table'",
zSql, zSep, zDb, zDb
zSep, zDb, zDb
);
if( zSql==0 ) return SQLITE_NOMEM;
zSep = " UNION ";
}
sqlite3_finalize(pS2);
sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0);
rc = sqlite3_finalize(pS2);
zSql = sqlite3_str_finish(pStr);
if( zSql==0 ) return SQLITE_NOMEM;
if( rc==SQLITE_OK ){
sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0);
}
sqlite3_free(zSql);
if( rc ) return rc;
}
iCol = 0;
eNextPhase = COMPLETION_EOF;
@@ -282,9 +293,10 @@ static int completionNext(sqlite3_vtab_cursor *cur){
pCur->szRow = sqlite3_column_bytes(pCur->pStmt, iCol);
}else{
/* When all rows are finished, advance to the next phase */
sqlite3_finalize(pCur->pStmt);
rc = sqlite3_finalize(pCur->pStmt);
pCur->pStmt = 0;
pCur->ePhase = eNextPhase;
if( rc ) return rc;
continue;
}
}
+2 -2
View File
@@ -59,7 +59,7 @@ static void compressFunc(
pIn = sqlite3_value_blob(argv[0]);
nIn = sqlite3_value_bytes(argv[0]);
nOut = 13 + nIn + (nIn+999)/1000;
pOut = sqlite3_malloc( nOut+5 );
pOut = sqlite3_malloc64( nOut+5 );
for(i=4; i>=0; i--){
x[i] = (nIn >> (7*(4-i)))&0x7f;
}
@@ -98,7 +98,7 @@ static void uncompressFunc(
nOut = (nOut<<7) | (pIn[i]&0x7f);
if( (pIn[i]&0x80)!=0 ){ i++; break; }
}
pOut = sqlite3_malloc( nOut+1 );
pOut = sqlite3_malloc64( nOut+1 );
rc = uncompress(pOut, &nOut, &pIn[i], nIn-i);
if( rc==Z_OK ){
sqlite3_result_blob(context, pOut, nOut, sqlite3_free);
+16 -13
View File
@@ -24,8 +24,8 @@
** schema= parameter, like this:
**
** CREATE VIRTUAL TABLE temp.csv2 USING csv(
** filename = "../http.log",
** schema = "CREATE TABLE x(date,ipaddr,url,referrer,userAgent)"
** filename = '../http.log',
** schema = 'CREATE TABLE x(date,ipaddr,url,referrer,userAgent)'
** );
**
** Instead of specifying a file, the text of the CSV can be loaded using
@@ -62,6 +62,10 @@ SQLITE_EXTENSION_INIT1
# define CSV_NOINLINE
#endif
#ifndef SQLITEINT_H
typedef sqlite3_int64 i64;
typedef sqlite3_uint64 u64;
#endif
/* Max size of the error message in a CsvReader */
#define CSV_MXERR 200
@@ -74,9 +78,9 @@ typedef struct CsvReader CsvReader;
struct CsvReader {
FILE *in; /* Read the CSV text from this input stream */
char *z; /* Accumulated text for a field */
int n; /* Number of bytes in z */
int nAlloc; /* Space allocated for z[] */
int nLine; /* Current line number */
i64 n; /* Number of bytes in z */
i64 nAlloc; /* Space allocated for z[] */
i64 nLine; /* Current line number */
int bNotFirst; /* True if prior text has been seen */
int cTerm; /* Character that terminated the most recent field */
size_t iIn; /* Next unread character in the input buffer */
@@ -174,7 +178,7 @@ static int csv_getc(CsvReader *p){
** Return 0 on success and non-zero if there is an OOM error */
static CSV_NOINLINE int csv_resize_and_append(CsvReader *p, char c){
char *zNew;
int nNew = p->nAlloc*2 + 100;
i64 nNew = p->nAlloc*2 + 100;
zNew = sqlite3_realloc64(p->z, nNew);
if( zNew ){
p->z = zNew;
@@ -218,7 +222,7 @@ static char *csv_read_one_field(CsvReader *p){
}
if( c=='"' ){
int pc, ppc;
int startLine = p->nLine;
i64 startLine = p->nLine;
pc = ppc = 0;
while( 1 ){
c = csv_getc(p);
@@ -322,7 +326,7 @@ typedef struct CsvCursor {
sqlite3_vtab_cursor base; /* Base class. Must be first */
CsvReader rdr; /* The CsvReader object */
char **azVal; /* Value of the current row */
int *aLen; /* Length of each entry */
i64 *aLen; /* Length of each entry */
sqlite3_int64 iRowid; /* The current rowid. Negative for EOF */
} CsvCursor;
@@ -494,7 +498,7 @@ static int csvtabConnect(
CsvTable *pNew = 0; /* The CsvTable object to construct */
int bHeader = -1; /* header= flags. -1 means not seen yet */
int rc = SQLITE_OK; /* Result code from this routine */
int i, j; /* Loop counters */
u64 i, j; /* Loop counters */
#ifdef SQLITE_TEST
int tstFlags = 0; /* Value for testflags=N parameter */
#endif
@@ -510,11 +514,10 @@ static int csvtabConnect(
# define CSV_DATA (azPValue[1])
# define CSV_SCHEMA (azPValue[2])
assert( sizeof(azPValue)==sizeof(azParam) );
memset(&sRdr, 0, sizeof(sRdr));
memset(azPValue, 0, sizeof(azPValue));
for(i=3; i<argc; i++){
for(i=3; i<(u64)argc; i++){
const char *z = argv[i];
const char *zValue;
for(j=0; j<sizeof(azParam)/sizeof(azParam[0]); j++){
@@ -707,12 +710,12 @@ static int csvtabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){
CsvTable *pTab = (CsvTable*)p;
CsvCursor *pCur;
size_t nByte;
nByte = sizeof(*pCur) + (sizeof(char*)+sizeof(int))*pTab->nCol;
nByte = sizeof(*pCur) + (sizeof(char*)+sizeof(i64))*pTab->nCol;
pCur = sqlite3_malloc64( nByte );
if( pCur==0 ) return SQLITE_NOMEM;
memset(pCur, 0, nByte);
pCur->azVal = (char**)&pCur[1];
pCur->aLen = (int*)&pCur->azVal[pTab->nCol];
pCur->aLen = (i64*)&pCur->azVal[pTab->nCol];
*ppCursor = &pCur->base;
if( csv_reader_open(&pCur->rdr, pTab->zFilename, pTab->zData) ){
csv_xfer_error(pTab, &pCur->rdr);
+63 -13
View File
@@ -31,6 +31,10 @@ SQLITE_EXTENSION_INIT1
#define IsSpace(X) isspace((unsigned char)X)
#endif
#ifndef SQLITE_DECIMAL_MAX_DIGIT
# define SQLITE_DECIMAL_MAX_DIGIT 10000000
#endif
/* A decimal object */
typedef struct Decimal Decimal;
struct Decimal {
@@ -69,6 +73,7 @@ static Decimal *decimalNewFromText(const char *zIn, int n){
int i;
int iExp = 0;
if( zIn==0 ) goto new_from_text_failed;
p = sqlite3_malloc( sizeof(*p) );
if( p==0 ) goto new_from_text_failed;
p->sign = 0;
@@ -128,9 +133,10 @@ static Decimal *decimalNewFromText(const char *zIn, int n){
}
}
if( iExp>0 ){
p->a = sqlite3_realloc64(p->a, (sqlite3_int64)p->nDigit
signed char *a = sqlite3_realloc64(p->a, (sqlite3_int64)p->nDigit
+ (sqlite3_int64)iExp + 1 );
if( p->a==0 ) goto new_from_text_failed;
if( a==0 ) goto new_from_text_failed;
p->a = a;
memset(p->a+p->nDigit, 0, iExp);
p->nDigit += iExp;
}
@@ -148,9 +154,10 @@ static Decimal *decimalNewFromText(const char *zIn, int n){
}
}
if( iExp>0 ){
p->a = sqlite3_realloc64(p->a, (sqlite3_int64)p->nDigit
signed char *a = sqlite3_realloc64(p->a, (sqlite3_int64)p->nDigit
+ (sqlite3_int64)iExp + 1 );
if( p->a==0 ) goto new_from_text_failed;
if( a==0 ) goto new_from_text_failed;
p->a = a;
memmove(p->a+iExp, p->a, p->nDigit);
memset(p->a, 0, iExp);
p->nDigit += iExp;
@@ -161,6 +168,7 @@ static Decimal *decimalNewFromText(const char *zIn, int n){
for(i=0; i<p->nDigit && p->a[i]==0; i++){}
if( i>=p->nDigit ) p->sign = 0;
}
if( p->nDigit>SQLITE_DECIMAL_MAX_DIGIT ) goto new_from_text_failed;
return p;
new_from_text_failed:
@@ -291,12 +299,38 @@ static void decimal_result(sqlite3_context *pCtx, Decimal *p){
sqlite3_result_text(pCtx, z, i, sqlite3_free);
}
/*
** Round a decimal value to N significant digits. N must be positive.
*/
static void decimal_round(Decimal *p, int N){
int i;
int nZero;
if( N<1 ) return;
if( p==0 ) return;
if( p->nDigit<=N ) return;
for(nZero=0; nZero<p->nDigit && p->a[nZero]==0; nZero++){}
N += nZero;
if( p->nDigit<=N ) return;
if( p->a[N]>4 ){
p->a[N-1]++;
for(i=N-1; i>0 && p->a[i]>9; i--){
p->a[i] = 0;
p->a[i-1]++;
}
if( p->a[0]>9 ){
p->a[0] = 1;
p->nFrac--;
}
}
memset(&p->a[N], 0, p->nDigit - N);
}
/*
** Make the given Decimal the result in an format similar to '%+#e'.
** In other words, show exponential notation with leading and trailing
** zeros omitted.
*/
static void decimal_result_sci(sqlite3_context *pCtx, Decimal *p){
static void decimal_result_sci(sqlite3_context *pCtx, Decimal *p, int N){
char *z; /* The output buffer */
int i; /* Loop counter */
int nZero; /* Number of leading zeros */
@@ -314,7 +348,8 @@ static void decimal_result_sci(sqlite3_context *pCtx, Decimal *p){
sqlite3_result_null(pCtx);
return;
}
for(nDigit=p->nDigit; nDigit>0 && p->a[nDigit-1]==0; nDigit--){}
if( N<1 ) N = 0;
for(nDigit=p->nDigit; nDigit>N && p->a[nDigit-1]==0; nDigit--){}
for(nZero=0; nZero<nDigit && p->a[nZero]==0; nZero++){}
nFrac = p->nFrac + (nDigit - p->nDigit);
nDigit -= nZero;
@@ -430,15 +465,18 @@ cmp_done:
static void decimal_expand(Decimal *p, int nDigit, int nFrac){
int nAddSig;
int nAddFrac;
signed char *a;
if( p==0 ) return;
nAddFrac = nFrac - p->nFrac;
nAddSig = (nDigit - p->nDigit) - nAddFrac;
if( nAddFrac==0 && nAddSig==0 ) return;
p->a = sqlite3_realloc64(p->a, nDigit+1);
if( p->a==0 ){
if( nDigit+1>SQLITE_DECIMAL_MAX_DIGIT ){ p->oom = 1; return; }
a = sqlite3_realloc64(p->a, nDigit+1);
if( a==0 ){
p->oom = 1;
return;
}
p->a = a;
if( nAddSig ){
memmove(p->a+nAddSig, p->a, p->nDigit);
memset(p->a, 0, nAddSig);
@@ -533,14 +571,18 @@ static void decimalMul(Decimal *pA, Decimal *pB){
signed char *acc = 0;
int i, j, k;
int minFrac;
sqlite3_int64 sumDigit;
if( pA==0 || pA->oom || pA->isNull
|| pB==0 || pB->oom || pB->isNull
){
goto mul_end;
}
acc = sqlite3_malloc64( (sqlite3_int64)pA->nDigit +
(sqlite3_int64)pB->nDigit + 2 );
sumDigit = pA->nDigit;
sumDigit += pB->nDigit;
sumDigit += 2;
if( sumDigit>SQLITE_DECIMAL_MAX_DIGIT ){ pA->oom = 1; return; }
acc = sqlite3_malloc64( sumDigit );
if( acc==0 ){
pA->oom = 1;
goto mul_end;
@@ -677,10 +719,16 @@ static void decimalFunc(
sqlite3_value **argv
){
Decimal *p = decimal_new(context, argv[0], 0);
UNUSED_PARAMETER(argc);
int N;
if( argc==2 ){
N = sqlite3_value_int(argv[1]);
if( N>0 ) decimal_round(p, N);
}else{
N = 0;
}
if( p ){
if( sqlite3_user_data(context)!=0 ){
decimal_result_sci(context, p);
decimal_result_sci(context, p, N);
}else{
decimal_result(context, p);
}
@@ -850,7 +898,7 @@ static void decimalPow2Func(
UNUSED_PARAMETER(argc);
if( sqlite3_value_type(argv[0])==SQLITE_INTEGER ){
Decimal *pA = decimalPow2(sqlite3_value_int(argv[0]));
decimal_result_sci(context, pA);
decimal_result_sci(context, pA, 0);
decimal_free(pA);
}
}
@@ -871,7 +919,9 @@ int sqlite3_decimal_init(
void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
} aFunc[] = {
{ "decimal", 1, 0, decimalFunc },
{ "decimal", 2, 0, decimalFunc },
{ "decimal_exp", 1, 1, decimalFunc },
{ "decimal_exp", 2, 1, decimalFunc },
{ "decimal_cmp", 2, 0, decimalCmpFunc },
{ "decimal_add", 2, 0, decimalAddFunc },
{ "decimal_sub", 2, 0, decimalSubFunc },
+173 -62
View File
@@ -73,11 +73,6 @@
** $path is a relative path, then $path is interpreted relative to $dir.
** And the paths returned in the "name" column of the table are also
** relative to directory $dir.
**
** Notes on building this extension for Windows:
** Unless linked statically with the SQLite library, a preprocessor
** symbol, FILEIO_WIN32_DLL, must be #define'd to create a stand-alone
** DLL form of this extension for WIN32. See its use below for details.
*/
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
@@ -94,12 +89,16 @@ SQLITE_EXTENSION_INIT1
# include <utime.h>
# include <sys/time.h>
# define STRUCT_STAT struct stat
# include <limits.h>
# include <stdlib.h>
#else
# include "windirent.h"
# include <direct.h>
# define STRUCT_STAT struct _stat
# define chmod(path,mode) fileio_chmod(path,mode)
# define mkdir(path,mode) fileio_mkdir(path)
extern LPWSTR sqlite3_win32_utf8_to_unicode(const char*);
extern char *sqlite3_win32_unicode_to_utf8(LPCWSTR);
#endif
#include <time.h>
#include <errno.h>
@@ -131,12 +130,9 @@ SQLITE_EXTENSION_INIT1
*/
#if defined(_WIN32) || defined(WIN32)
static int fileio_chmod(const char *zPath, int pmode){
sqlite3_int64 sz = strlen(zPath);
wchar_t *b1 = sqlite3_malloc64( (sz+1)*sizeof(b1[0]) );
int rc;
wchar_t *b1 = sqlite3_win32_utf8_to_unicode(zPath);
if( b1==0 ) return -1;
sz = MultiByteToWideChar(CP_UTF8, 0, zPath, sz, b1, sz);
b1[sz] = 0;
rc = _wchmod(b1, pmode);
sqlite3_free(b1);
return rc;
@@ -148,12 +144,9 @@ static int fileio_chmod(const char *zPath, int pmode){
*/
#if defined(_WIN32) || defined(WIN32)
static int fileio_mkdir(const char *zPath){
sqlite3_int64 sz = strlen(zPath);
wchar_t *b1 = sqlite3_malloc64( (sz+1)*sizeof(b1[0]) );
int rc;
wchar_t *b1 = sqlite3_win32_utf8_to_unicode(zPath);
if( b1==0 ) return -1;
sz = MultiByteToWideChar(CP_UTF8, 0, zPath, sz, b1, sz);
b1[sz] = 0;
rc = _wmkdir(b1);
sqlite3_free(b1);
return rc;
@@ -266,50 +259,7 @@ static sqlite3_uint64 fileTimeToUnixTime(
return (fileIntervals.QuadPart - epochIntervals.QuadPart) / 10000000;
}
#if defined(FILEIO_WIN32_DLL) && (defined(_WIN32) || defined(WIN32))
# /* To allow a standalone DLL, use this next replacement function: */
# undef sqlite3_win32_utf8_to_unicode
# define sqlite3_win32_utf8_to_unicode utf8_to_utf16
#
LPWSTR utf8_to_utf16(const char *z){
int nAllot = MultiByteToWideChar(CP_UTF8, 0, z, -1, NULL, 0);
LPWSTR rv = sqlite3_malloc(nAllot * sizeof(WCHAR));
if( rv!=0 && 0 < MultiByteToWideChar(CP_UTF8, 0, z, -1, rv, nAllot) )
return rv;
sqlite3_free(rv);
return 0;
}
#endif
/*
** This function attempts to normalize the time values found in the stat()
** buffer to UTC. This is necessary on Win32, where the runtime library
** appears to return these values as local times.
*/
static void statTimesToUtc(
const char *zPath,
STRUCT_STAT *pStatBuf
){
HANDLE hFindFile;
WIN32_FIND_DATAW fd;
LPWSTR zUnicodeName;
extern LPWSTR sqlite3_win32_utf8_to_unicode(const char*);
zUnicodeName = sqlite3_win32_utf8_to_unicode(zPath);
if( zUnicodeName ){
memset(&fd, 0, sizeof(WIN32_FIND_DATAW));
hFindFile = FindFirstFileW(zUnicodeName, &fd);
if( hFindFile!=NULL ){
pStatBuf->st_ctime = (time_t)fileTimeToUnixTime(&fd.ftCreationTime);
pStatBuf->st_atime = (time_t)fileTimeToUnixTime(&fd.ftLastAccessTime);
pStatBuf->st_mtime = (time_t)fileTimeToUnixTime(&fd.ftLastWriteTime);
FindClose(hFindFile);
}
sqlite3_free(zUnicodeName);
}
}
#endif
#endif /* _WIN32 */
/*
** This function is used in place of stat(). On Windows, special handling
@@ -321,14 +271,22 @@ static int fileStat(
STRUCT_STAT *pStatBuf
){
#if defined(_WIN32)
sqlite3_int64 sz = strlen(zPath);
wchar_t *b1 = sqlite3_malloc64( (sz+1)*sizeof(b1[0]) );
int rc;
wchar_t *b1 = sqlite3_win32_utf8_to_unicode(zPath);
if( b1==0 ) return 1;
sz = MultiByteToWideChar(CP_UTF8, 0, zPath, sz, b1, sz);
b1[sz] = 0;
rc = _wstat(b1, pStatBuf);
if( rc==0 ) statTimesToUtc(zPath, pStatBuf);
if( rc==0 ){
HANDLE hFindFile;
WIN32_FIND_DATAW fd;
memset(&fd, 0, sizeof(WIN32_FIND_DATAW));
hFindFile = FindFirstFileW(b1, &fd);
if( hFindFile!=NULL ){
pStatBuf->st_ctime = (time_t)fileTimeToUnixTime(&fd.ftCreationTime);
pStatBuf->st_atime = (time_t)fileTimeToUnixTime(&fd.ftLastAccessTime);
pStatBuf->st_mtime = (time_t)fileTimeToUnixTime(&fd.ftLastWriteTime);
FindClose(hFindFile);
}
}
sqlite3_free(b1);
return rc;
#else
@@ -1093,6 +1051,154 @@ static int fsdirRegister(sqlite3 *db){
# define fsdirRegister(x) SQLITE_OK
#endif
/*
** This version of realpath() works on any system. The string
** returned is held in memory allocated using sqlite3_malloc64().
** The caller is responsible for calling sqlite3_free().
*/
static char *portable_realpath(const char *zPath){
#if !defined(_WIN32) /* BEGIN unix */
char *zOut = 0; /* Result */
char *z; /* Temporary buffer */
#if defined(PATH_MAX)
char zBuf[PATH_MAX+1]; /* Space for the temporary buffer */
#endif
if( zPath==0 ) return 0;
#if defined(PATH_MAX)
z = realpath(zPath, zBuf);
if( z ){
zOut = sqlite3_mprintf("%s", zBuf);
}
#endif /* defined(PATH_MAX) */
if( zOut==0 ){
/* Try POSIX.1-2008 malloc behavior */
z = realpath(zPath, NULL);
if( z ){
zOut = sqlite3_mprintf("%s", z);
free(z);
}
}
return zOut;
#else /* End UNIX, Begin WINDOWS */
wchar_t *zPath16; /* UTF16 translation of zPath */
char *zOut = 0; /* Result */
wchar_t *z = 0; /* Temporary buffer */
if( zPath==0 ) return 0;
zPath16 = sqlite3_win32_utf8_to_unicode(zPath);
if( zPath16==0 ) return 0;
z = _wfullpath(NULL, zPath16, 0);
sqlite3_free(zPath16);
if( z ){
zOut = sqlite3_win32_unicode_to_utf8(z);
free(z);
}
return zOut;
#endif /* End WINDOWS, Begin common code */
}
/*
** SQL function: realpath(X)
**
** Try to convert file or pathname X into its real, absolute pathname.
** Return NULL if unable.
**
** The file or directory X is not required to exist. The answer is formed
** by calling system realpath() on the prefix of X that does exist and
** appending the tail of X that does not (yet) exist.
*/
static void realpathFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
const char *zPath; /* Original input path */
char *zCopy; /* An editable copy of zPath */
char *zOut; /* The result */
char cSep = 0; /* Separator turned into \000 */
size_t len; /* Prefix length before cSep */
#ifdef _WIN32
const int isWin = 1;
#else
const int isWin = 0;
#endif
(void)argc;
zPath = (const char*)sqlite3_value_text(argv[0]);
if( zPath==0 ) return;
if( zPath[0]==0 ) zPath = ".";
zCopy = sqlite3_mprintf("%s",zPath);
len = strlen(zCopy);
while( len>1 && (zCopy[len-1]=='/' || (isWin && zCopy[len-1]=='\\')) ){
len--;
}
zCopy[len] = 0;
while( 1 /*exit-by-break*/ ){
zOut = portable_realpath(zCopy);
zCopy[len] = cSep;
if( zOut ){
if( cSep ){
zOut = sqlite3_mprintf("%z%s",zOut,&zCopy[len]);
}
break;
}else{
size_t i = len-1;
while( i>0 ){
if( zCopy[i]=='/' || (isWin && zCopy[i]=='\\') ) break;
i--;
}
if( i<=0 ){
if( zCopy[0]=='/' ){
zOut = zCopy;
zCopy = 0;
}else if( (zOut = portable_realpath("."))!=0 ){
zOut = sqlite3_mprintf("%z/%s", zOut, zCopy);
}
break;
}
cSep = zCopy[i];
zCopy[i] = 0;
len = i;
}
}
sqlite3_free(zCopy);
if( zOut ){
/* Simplify any "/./" or "/../" that might have snuck into the
** pathname due to appending of zCopy. We only have to consider
** unix "/" separators, because the _wfilepath() system call on
** Windows will have already done this simplification for us. */
size_t i, j, n;
n = strlen(zOut);
for(i=j=0; i<n; i++){
if( zOut[i]=='/' ){
if( zOut[i+1]=='/' ) continue;
if( zOut[i+1]=='.' && i+2<n && zOut[i+2]=='/' ){
i += 1;
continue;
}
if( zOut[i+1]=='.' && i+3<n && zOut[i+2]=='.' && zOut[i+3]=='/' ){
while( j>0 && zOut[j-1]!='/' ){ j--; }
if( j>0 ){ j--; }
i += 2;
continue;
}
}
zOut[j++] = zOut[i];
}
zOut[j] = 0;
/* Return the result */
sqlite3_result_text(context, zOut, -1, sqlite3_free);
}
}
#ifdef _WIN32
__declspec(dllexport)
#endif
@@ -1119,5 +1225,10 @@ int sqlite3_fileio_init(
if( rc==SQLITE_OK ){
rc = fsdirRegister(db);
}
if( rc==SQLITE_OK ){
rc = sqlite3_create_function(db, "realpath", 1,
SQLITE_UTF8, 0,
realpathFunc, 0, 0);
}
return rc;
}
+6 -4
View File
@@ -38,9 +38,11 @@ SQLITE_EXTENSION_INIT1
#ifndef SQLITE_AMALGAMATION
/*
** The "u32" type must be an unsigned 32-bit integer. Adjust this
** The "u32" type must be an unsigned 32-bit integer. "u64" is
** an unsigned 64-bit integer.
*/
typedef unsigned int u32;
typedef sqlite3_uint64 u64;
/*
** Must be a 16-bit value
@@ -541,8 +543,8 @@ static int delta_apply(
int lenDelta, /* Length of the delta */
char *zOut /* Write the output into this preallocated buffer */
){
unsigned int limit;
unsigned int total = 0;
sqlite3_uint64 limit;
sqlite3_uint64 total = 0;
#ifdef FOSSIL_ENABLE_DELTA_CKSUM_TEST
char *zOrigOut = zOut;
#endif
@@ -570,7 +572,7 @@ static int delta_apply(
/* ERROR: copy exceeds output file size */
return -1;
}
if( ofst+cnt > lenSrc ){
if( (u64)ofst+(u64)cnt > (u64)lenSrc ){
/* ERROR: copy extends past end of input */
return -1;
}
+1 -1
View File
@@ -617,7 +617,7 @@ static int fuzzerRender(
int *pnBuf /* Size of the buffer */
){
const fuzzer_rule *pRule = pStem->pRule;
int n; /* Size of output term without nul-term */
sqlite3_int64 n; /* Size of output term without nul-term */
char *z; /* Buffer to assemble output term in */
n = pStem->nBasis + pRule->nTo - pRule->nFrom;
+35 -1
View File
@@ -179,9 +179,9 @@ static void ieee754func(
}
if( m<0 ){
if( m<(-9223372036854775807LL) ) return;
isNeg = 1;
m = -m;
if( m<0 ) return;
}else if( m==0 && e>-1000 && e<1000 ){
sqlite3_result_double(context, 0.0);
return;
@@ -259,6 +259,38 @@ static void ieee754func_to_blob(
}
}
/*
** Functions to convert between 64-bit integers and floats.
**
** The bit patterns are copied. The numeric values are different.
*/
static void ieee754func_from_int(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
UNUSED_PARAMETER(argc);
if( sqlite3_value_type(argv[0])==SQLITE_INTEGER ){
double r;
sqlite3_int64 v = sqlite3_value_int64(argv[0]);
memcpy(&r, &v, sizeof(r));
sqlite3_result_double(context, r);
}
}
static void ieee754func_to_int(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
UNUSED_PARAMETER(argc);
if( sqlite3_value_type(argv[0])==SQLITE_FLOAT ){
double r = sqlite3_value_double(argv[0]);
sqlite3_uint64 v;
memcpy(&v, &r, sizeof(v));
sqlite3_result_int64(context, v);
}
}
/*
** SQL Function: ieee754_inc(r,N)
**
@@ -311,6 +343,8 @@ int sqlite3_ieee_init(
{ "ieee754_exponent", 1, 2, ieee754func },
{ "ieee754_to_blob", 1, 0, ieee754func_to_blob },
{ "ieee754_from_blob", 1, 0, ieee754func_from_blob },
{ "ieee754_to_int", 1, 0, ieee754func_to_int },
{ "ieee754_from_int", 1, 0, ieee754func_from_int },
{ "ieee754_inc", 2, 0, ieee754inc },
};
unsigned int i;
+22 -8
View File
@@ -32,7 +32,7 @@
** ^X X occurring at the beginning of the string
** X$ X occurring at the end of the string
** . Match any single character
** \c Character c where c is one of \{}()[]|*+?.
** \c Character c where c is one of \{}()[]|*+?-.
** \c C-language escapes for c in afnrtv. ex: \t or \n
** \uXXXX Where XXXX is exactly 4 hex digits, unicode value XXXX
** \xXX Where XX is exactly 2 hex digits, unicode value XX
@@ -417,7 +417,7 @@ static int re_hex(int c, int *pV){
** return its interpretation.
*/
static unsigned re_esc_char(ReCompiled *p){
static const char zEsc[] = "afnrtv\\()*.+?[$^{|}]";
static const char zEsc[] = "afnrtv\\()*.+?[$^{|}]-";
static const char zTrans[] = "\a\f\n\r\t\v";
int i, v = 0;
char c;
@@ -740,11 +740,18 @@ static const char *re_compile(
}
/*
** Compute a reasonable limit on the length of the REGEXP NFA.
** The value of LIMIT_MAX_PATTERN_LENGTH.
*/
static int re_maxlen(sqlite3_context *context){
sqlite3 *db = sqlite3_context_db_handle(context);
return 75 + sqlite3_limit(db, SQLITE_LIMIT_LIKE_PATTERN_LENGTH,-1)/2;
return sqlite3_limit(db, SQLITE_LIMIT_LIKE_PATTERN_LENGTH,-1);
}
/*
** Maximum NFA size given a maximum pattern length.
*/
static int re_maxnfa(int mxlen){
return 75+mxlen/2;
}
/*
@@ -770,10 +777,17 @@ static void re_sql_func(
(void)argc; /* Unused */
pRe = sqlite3_get_auxdata(context, 0);
if( pRe==0 ){
int mxLen = re_maxlen(context);
int nPattern;
zPattern = (const char*)sqlite3_value_text(argv[0]);
if( zPattern==0 ) return;
zErr = re_compile(&pRe, zPattern, re_maxlen(context),
sqlite3_user_data(context)!=0);
nPattern = sqlite3_value_bytes(argv[0]);
if( nPattern>mxLen ){
zErr = "REGEXP pattern too big";
}else{
zErr = re_compile(&pRe, zPattern, re_maxnfa(mxLen),
sqlite3_user_data(context)!=0);
}
if( zErr ){
re_free(pRe);
sqlite3_result_error(context, zErr, -1);
@@ -814,7 +828,6 @@ static void re_bytecode_func(
int i;
int n;
char *z;
(void)argc;
static const char *ReOpName[] = {
"EOF",
"MATCH",
@@ -837,9 +850,10 @@ static void re_bytecode_func(
"ATSTART",
};
(void)argc;
zPattern = (const char*)sqlite3_value_text(argv[0]);
if( zPattern==0 ) return;
zErr = re_compile(&pRe, zPattern, re_maxlen(context),
zErr = re_compile(&pRe, zPattern, re_maxnfa(re_maxlen(context)),
sqlite3_user_data(context)!=0);
if( zErr ){
re_free(pRe);
+6 -2
View File
@@ -249,16 +249,19 @@ static void sha1Func(
SHA1Context cx;
int eType = sqlite3_value_type(argv[0]);
int nByte = sqlite3_value_bytes(argv[0]);
const unsigned char *pData;
char zOut[44];
assert( argc==1 );
if( eType==SQLITE_NULL ) return;
hash_init(&cx);
if( eType==SQLITE_BLOB ){
hash_step(&cx, sqlite3_value_blob(argv[0]), nByte);
pData = (const unsigned char*)sqlite3_value_blob(argv[0]);
}else{
hash_step(&cx, sqlite3_value_text(argv[0]), nByte);
pData = (const unsigned char*)sqlite3_value_text(argv[0]);
}
if( pData==0 ) return;
hash_step(&cx, pData, nByte);
if( sqlite3_user_data(context)!=0 ){
/* sha1b() - binary result */
hash_finish(&cx, zOut, 1);
@@ -320,6 +323,7 @@ static void sha1QueryFunc(
}
nCol = sqlite3_column_count(pStmt);
z = sqlite3_sql(pStmt);
if( z==0 ) z = "";
n = (int)strlen(z);
hash_step_vformat(&cx,"S%d:",n);
hash_step(&cx,(unsigned char*)z,n);
+1 -1
View File
@@ -84,7 +84,7 @@ static void sqlarUncompressFunc(
sqlite3_int64 sz;
assert( argc==2 );
sz = sqlite3_value_int(argv[1]);
sz = sqlite3_value_int64(argv[1]);
if( sz<=0 || sz==(nData = sqlite3_value_bytes(argv[0])) ){
sqlite3_result_value(context, argv[0]);
+1042
View File
File diff suppressed because it is too large Load Diff
+48 -6
View File
@@ -14,6 +14,8 @@
** on stdout when its key interfaces are called. This is intended for
** interactive analysis and debugging of virtual table interfaces.
**
** HOW TO COMPILE:
**
** To build this extension as a separately loaded shared library or
** DLL, use compiler command-lines similar to the following:
**
@@ -21,7 +23,7 @@
** (mac) clang -fPIC -dynamiclib vtablog.c -o vtablog.dylib
** (windows) cl vtablog.c -link -dll -out:vtablog.dll
**
** Usage example:
** USAGE EXAMPLE:
**
** .load ./vtablog
** CREATE VIRTUAL TABLE temp.log USING vtablog(
@@ -29,6 +31,23 @@
** rows=25
** );
** SELECT * FROM log;
**
** ARGUMENTS TO CREATE VIRTUAL TABLE:
**
** In "CREATE VIRTUAL TABLE temp.log AS vtablog(ARGS....)" statement, the
** ARGS argument is a list of key-value pairs that can be any of the
** following.
**
** schema=TEXT Text is a CREATE TABLE statement that defines
** the schema of the new virtual table.
**
** rows=N The table as N rows.
**
** consume_order_by=N If the left-most ORDER BY terms is ASC and
** against column N (where the leftmost column
** is #1) then set the orderByConsumed=1 flag in
** xBestIndex. Or if the left-most ORDER BY is
** DESC and against column -N, do likewise.
*/
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
@@ -49,6 +68,8 @@ struct vtablog_vtab {
char *zName; /* Table name. argv[2] of xConnect/xCreate */
int nRow; /* Number of rows in the table */
int nCursor; /* Number of cursors created */
int iConsumeOB; /* Consume the ORDER BY clause if on column N-th
** and consumeOB=N or consumeOB=(-N) and DESC */
};
/* vtablog_cursor is a subclass of sqlite3_vtab_cursor which will
@@ -180,6 +201,7 @@ static int vtablogConnectCreate(
int rc;
char *zSchema = 0;
char *zNRow = 0;
char *zConsumeOB = 0;
printf("%s.%s.%s():\n", argv[1], argv[2],
isCreate ? "xCreate" : "xConnect");
@@ -203,6 +225,10 @@ static int vtablogConnectCreate(
rc = SQLITE_ERROR;
goto vtablog_end_connect;
}
if( vtablog_string_parameter(pzErr, "consume_order_by", z, &zConsumeOB) ){
rc = SQLITE_ERROR;
goto vtablog_end_connect;
}
}
if( zSchema==0 ){
zSchema = sqlite3_mprintf("%s","CREATE TABLE x(a,b);");
@@ -221,6 +247,10 @@ static int vtablogConnectCreate(
pNew->nRow = 10;
if( zNRow ) pNew->nRow = atoi(zNRow);
printf(" nrow = %d\n", pNew->nRow);
if( zConsumeOB ) pNew->iConsumeOB = atoi(zConsumeOB);
if( pNew->iConsumeOB ){
printf(" consume_order_by = %d\n", pNew->iConsumeOB);
}
pNew->zDb = sqlite3_mprintf("%s", argv[1]);
pNew->zName = sqlite3_mprintf("%s", argv[2]);
}
@@ -228,6 +258,7 @@ static int vtablogConnectCreate(
vtablog_end_connect:
sqlite3_free(zSchema);
sqlite3_free(zNRow);
sqlite3_free(zConsumeOB);
return rc;
}
static int vtablogCreate(
@@ -514,16 +545,27 @@ static int vtablogBestIndex(
}
}
printf(" nOrderBy: %d\n", p->nOrderBy);
for(i=0; i<p->nOrderBy; i++){
printf(" orderby[%d]: col=%d desc=%d\n",
i,
p->aOrderBy[i].iColumn,
p->aOrderBy[i].desc);
if( p->nOrderBy ){
for(i=0; i<p->nOrderBy; i++){
printf(" orderby[%d]: col=%d desc=%d\n",
i,
p->aOrderBy[i].iColumn,
p->aOrderBy[i].desc);
}
if( pTab->iConsumeOB ){
int N = p->aOrderBy[0].iColumn+1;
if( (p->aOrderBy[0].desc && N==-pTab->iConsumeOB)
|| (!p->aOrderBy[0].desc && N==pTab->iConsumeOB)
){
p->orderByConsumed = 1;
}
}
}
p->estimatedCost = (double)500;
p->estimatedRows = 500;
printf(" idxNum=%d\n", p->idxNum);
printf(" idxStr=NULL\n");
printf(" sqlite3_vtab_distinct()=%d\n", sqlite3_vtab_distinct(p));
printf(" orderByConsumed=%d\n", p->orderByConsumed);
printf(" estimatedCost=%g\n", p->estimatedCost);
printf(" estimatedRows=%lld\n", p->estimatedRows);
+32 -27
View File
@@ -393,7 +393,7 @@ static int zipfileConnect(
rc = sqlite3_declare_vtab(db, ZIPFILE_SCHEMA);
if( rc==SQLITE_OK ){
pNew = (ZipfileTab*)sqlite3_malloc64((sqlite3_int64)nByte+nFile);
pNew = (ZipfileTab*)sqlite3_malloc64((i64)nByte+nFile);
if( pNew==0 ) return SQLITE_NOMEM;
memset(pNew, 0, nByte+nFile);
pNew->db = db;
@@ -539,14 +539,15 @@ static void zipfileCursorErr(ZipfileCsr *pCsr, const char *zFmt, ...){
static int zipfileReadData(
FILE *pFile, /* Read from this file */
u8 *aRead, /* Read into this buffer */
int nRead, /* Number of bytes to read */
i64 nRead, /* Number of bytes to read */
i64 iOff, /* Offset to read from */
char **pzErrmsg /* OUT: Error message (from sqlite3_malloc) */
){
size_t n;
fseek(pFile, (long)iOff, SEEK_SET);
n = fread(aRead, 1, nRead, pFile);
if( (int)n!=nRead ){
n = fread(aRead, 1, (long)nRead, pFile);
if( n!=(size_t)nRead ){
sqlite3_free(*pzErrmsg);
*pzErrmsg = sqlite3_mprintf("error in fread()");
return SQLITE_ERROR;
}
@@ -563,7 +564,7 @@ static int zipfileAppendData(
fseek(pTab->pWriteFd, (long)pTab->szCurrent, SEEK_SET);
n = fwrite(aWrite, 1, nWrite, pTab->pWriteFd);
if( (int)n!=nWrite ){
pTab->base.zErrMsg = sqlite3_mprintf("error in fwrite()");
zipfileTableErr(pTab,"error in fwrite()");
return SQLITE_ERROR;
}
pTab->szCurrent += nWrite;
@@ -704,7 +705,12 @@ static int zipfileScanExtra(u8 *aExtra, int nExtra, u32 *pmTime){
u8 *p = aExtra;
u8 *pEnd = &aExtra[nExtra];
while( p<pEnd ){
/* Stop when there are less than 9 bytes left to scan in the buffer. This
** is because the timestamp field requires exactly 9 bytes - 4 bytes of
** header fields and 5 bytes of data. If there are less than 9 bytes
** remaining, either it is some other field or else the extra data
** is corrupt. Either way, do not process it. */
while( p+(2*sizeof(u16) + 1 + sizeof(u32))<=pEnd ){
u16 id = zipfileRead16(p);
u16 nByte = zipfileRead16(p);
@@ -810,6 +816,7 @@ static void zipfileMtimeToDos(ZipfileCDS *pCds, u32 mUnixTime){
** generic corruption message and return SQLITE_CORRUPT;
*/
static int zipfileCorrupt(char **pzErr){
sqlite3_free(*pzErr);
*pzErr = sqlite3_mprintf("zip archive is corrupt");
return SQLITE_CORRUPT;
}
@@ -828,7 +835,7 @@ static int zipfileCorrupt(char **pzErr){
static int zipfileGetEntry(
ZipfileTab *pTab, /* Store any error message here */
const u8 *aBlob, /* Pointer to in-memory file image */
int nBlob, /* Size of aBlob[] in bytes */
i64 nBlob, /* Size of aBlob[] in bytes */
FILE *pFile, /* If aBlob==0, read from this file */
i64 iOff, /* Offset of CDS record */
ZipfileEntry **ppEntry /* OUT: Pointer to new object */
@@ -868,14 +875,14 @@ static int zipfileGetEntry(
memset(pNew, 0, sizeof(ZipfileEntry));
rc = zipfileReadCDS(aRead, &pNew->cds);
if( rc!=SQLITE_OK ){
*pzErr = sqlite3_mprintf("failed to read CDS at offset %lld", iOff);
zipfileTableErr(pTab, "failed to read CDS at offset %lld", iOff);
}else if( aBlob==0 ){
rc = zipfileReadData(
pFile, aRead, nExtra+nFile, iOff+ZIPFILE_CDS_FIXED_SZ, pzErr
);
}else{
aRead = (u8*)&aBlob[iOff + ZIPFILE_CDS_FIXED_SZ];
if( (iOff + ZIPFILE_LFH_FIXED_SZ + nFile + nExtra)>nBlob ){
if( (iOff + ZIPFILE_CDS_FIXED_SZ + nFile + nExtra)>nBlob ){
rc = zipfileCorrupt(pzErr);
}
}
@@ -900,7 +907,7 @@ static int zipfileGetEntry(
rc = zipfileReadData(pFile, aRead, szFix, pNew->cds.iOffset, pzErr);
}else{
aRead = (u8*)&aBlob[pNew->cds.iOffset];
if( (pNew->cds.iOffset + ZIPFILE_LFH_FIXED_SZ)>(unsigned)nBlob ){
if( ((i64)pNew->cds.iOffset + ZIPFILE_LFH_FIXED_SZ)>nBlob ){
rc = zipfileCorrupt(pzErr);
}
}
@@ -908,7 +915,7 @@ static int zipfileGetEntry(
memset(&lfh, 0, sizeof(lfh));
if( rc==SQLITE_OK ) rc = zipfileReadLFH(aRead, &lfh);
if( rc==SQLITE_OK ){
pNew->iDataOff = pNew->cds.iOffset + ZIPFILE_LFH_FIXED_SZ;
pNew->iDataOff = (i64)pNew->cds.iOffset + ZIPFILE_LFH_FIXED_SZ;
pNew->iDataOff += lfh.nFile + lfh.nExtra;
if( aBlob && pNew->cds.szCompressed ){
if( pNew->iDataOff + pNew->cds.szCompressed > nBlob ){
@@ -919,7 +926,7 @@ static int zipfileGetEntry(
}
}
}else{
*pzErr = sqlite3_mprintf("failed to read LFH at offset %d",
zipfileTableErr(pTab, "failed to read LFH at offset %d",
(int)pNew->cds.iOffset
);
}
@@ -943,7 +950,7 @@ static int zipfileNext(sqlite3_vtab_cursor *cur){
int rc = SQLITE_OK;
if( pCsr->pFile ){
i64 iEof = pCsr->eocd.iOffset + pCsr->eocd.nSize;
i64 iEof = (i64)pCsr->eocd.iOffset + (i64)pCsr->eocd.nSize;
zipfileEntryFree(pCsr->pCurrent);
pCsr->pCurrent = 0;
if( pCsr->iNextOff>=iEof ){
@@ -1181,12 +1188,12 @@ static int zipfileEof(sqlite3_vtab_cursor *cur){
static int zipfileReadEOCD(
ZipfileTab *pTab, /* Return errors here */
const u8 *aBlob, /* Pointer to in-memory file image */
int nBlob, /* Size of aBlob[] in bytes */
i64 nBlob, /* Size of aBlob[] in bytes */
FILE *pFile, /* Read from this file if aBlob==0 */
ZipfileEOCD *pEOCD /* Object to populate */
){
u8 *aRead = pTab->aBuffer; /* Temporary buffer */
int nRead; /* Bytes to read from file */
i64 nRead; /* Bytes to read from file */
int rc = SQLITE_OK;
memset(pEOCD, 0, sizeof(ZipfileEOCD));
@@ -1207,7 +1214,7 @@ static int zipfileReadEOCD(
}
if( rc==SQLITE_OK ){
int i;
i64 i;
/* Scan backwards looking for the signature bytes */
for(i=nRead-20; i>=0; i--){
@@ -1218,9 +1225,7 @@ static int zipfileReadEOCD(
}
}
if( i<0 ){
pTab->base.zErrMsg = sqlite3_mprintf(
"cannot find end of central directory record"
);
zipfileTableErr(pTab, "cannot find end of central directory record");
return SQLITE_ERROR;
}
@@ -1265,7 +1270,7 @@ static void zipfileAddEntry(
}
}
static int zipfileLoadDirectory(ZipfileTab *pTab, const u8 *aBlob, int nBlob){
static int zipfileLoadDirectory(ZipfileTab *pTab, const u8 *aBlob, i64 nBlob){
ZipfileEOCD eocd;
int rc;
int i;
@@ -1313,7 +1318,7 @@ static int zipfileFilter(
}else if( sqlite3_value_type(argv[0])==SQLITE_BLOB ){
static const u8 aEmptyBlob = 0;
const u8 *aBlob = (const u8*)sqlite3_value_blob(argv[0]);
int nBlob = sqlite3_value_bytes(argv[0]);
i64 nBlob = sqlite3_value_bytes(argv[0]);
assert( pTab->pFirstEntry==0 );
if( aBlob==0 ){
aBlob = &aEmptyBlob;
@@ -1511,7 +1516,7 @@ static int zipfileBegin(sqlite3_vtab *pVtab){
assert( pTab->pWriteFd==0 );
if( pTab->zFile==0 || pTab->zFile[0]==0 ){
pTab->base.zErrMsg = sqlite3_mprintf("zipfile: missing filename");
zipfileTableErr(pTab, "zipfile: missing filename");
return SQLITE_ERROR;
}
@@ -1521,9 +1526,9 @@ static int zipfileBegin(sqlite3_vtab *pVtab){
** in main-memory until the transaction is committed. */
pTab->pWriteFd = sqlite3_fopen(pTab->zFile, "ab+");
if( pTab->pWriteFd==0 ){
pTab->base.zErrMsg = sqlite3_mprintf(
"zipfile: failed to open file %s for writing", pTab->zFile
);
zipfileTableErr(pTab,
"zipfile: failed to open file %s for writing", pTab->zFile
);
rc = SQLITE_ERROR;
}else{
fseek(pTab->pWriteFd, 0, SEEK_END);
@@ -1988,7 +1993,7 @@ struct ZipfileCtx {
ZipfileBuffer cds;
};
static int zipfileBufferGrow(ZipfileBuffer *pBuf, int nByte){
static int zipfileBufferGrow(ZipfileBuffer *pBuf, i64 nByte){
if( pBuf->n+nByte>pBuf->nAlloc ){
u8 *aNew;
sqlite3_int64 nNew = pBuf->n ? pBuf->n*2 : 512;
@@ -2037,7 +2042,7 @@ static void zipfileStep(sqlite3_context *pCtx, int nVal, sqlite3_value **apVal){
char *zName = 0; /* Path (name) of new entry */
int nName = 0; /* Size of zName in bytes */
char *zFree = 0; /* Free this before returning */
int nByte;
i64 nByte;
memset(&e, 0, sizeof(e));
p = (ZipfileCtx*)sqlite3_aggregate_context(pCtx, sizeof(ZipfileCtx));
+56 -24
View File
@@ -90,16 +90,10 @@ while( zSql && zSql[0] ){
}
~~~
<a id="spec"></a>
## 2.0 The `sqlite3_qrf_spec` object
The most recent definition of `sqlite3_qrf_spec` is shown below.
Do not be alarmed by the complexity of this structure. You only need
to understand the properties you want to modify. Zero is always a good
default for all of the attributes (except iVersion and pzOutput/xWrite)
and so simply zeroing out the bulk of this structure is a good start.
You can then slowly make adjustments to individual fields to get the
results you desire.
The `sqlite3_qrf_spec` looks like this:
> ~~~
typedef struct sqlite3_qrf_spec sqlite3_qrf_spec;
@@ -120,6 +114,8 @@ struct sqlite3_qrf_spec {
short int nWrap; /* Wrap columns wider than this */
short int nScreenWidth; /* Maximum overall table width */
short int nLineLimit; /* Maximum number of lines for any row */
short int nTitleLimit; /* Maximum number of characters in a title */
unsigned int nMultiInsert; /* Add rows to one INSERT until size exceeds */
int nCharLimit; /* Maximum number of characters in a cell */
int nWidth; /* Number of entries in aWidth[] */
int nAlign; /* Number of entries in aAlignment[] */
@@ -138,16 +134,19 @@ struct sqlite3_qrf_spec {
};
~~~
Again, the only fields that must initialized are:
Do not be alarmed by the complexity of this structure. Everything can
be zeroed except for:
* `.iVersion`
* One of `.pzOutput` or `.xWrite`.
All other fields can be zeroed. Or they can contain other values to
alter the formatting of the query results.
You do not need to understand and configure every field of this object
in order to use QRF effectively. Start by zeroing out the whole structure,
then initializing iVersion and one of pzOutput or xWrite. Then maybe
tweak one or two other settings to get the output you want.
Further detail on the meanings of each of the fields in the
`sqlite3_qrf_spec` object are described below.
`sqlite3_qrf_spec` object is in the subsequent sections.
### 2.1 Structure Version Number
@@ -346,7 +345,7 @@ A value of QRF_BLOB_Size does not show any BLOB content at all.
Instead, it substitutes a text string that says how many bytes
the BLOB contains.
### 2.8 Maximum size of displayed content (nLineLimit, nCharLimit)
### 2.8 Maximum size of displayed content (nLineLimit, nCharLimit, nTitleLimit)
If the sqlite3_qrf_spec.nCharLimit setting is non-zero, then the formatter
will display only the first nCharLimit characters of each value.
@@ -362,14 +361,37 @@ If the sqlite3_qrf_spec.nLineLimit setting is non-zero, then the
formatter will only display the first nLineLimit lines of each value.
It does not matter if the value is split because it contains a newline
character, or if it split by wrapping. This setting merely limits
the number of displayed lines. This setting only works for
**Box**, **Column**, **Line**, **Markdown**, and **Table** styles.
the number of displayed lines. The nLineLimit setting currently only
works for **Box**, **Column**, **Line**, **Markdown**, and **Table**
styles, though that limitation might change in future releases.
The idea behind both of these settings is to prevent large renderings
when doing a query that (unexpectedly) contains very large text or
blob values: perhaps megabyes of text.
### 2.9 Word Wrapping In Columnar Styles (nWrap, bWordWrap)
If the sqlite3_qrf_spec.nTitleLimit is non-zero, then the formatter
attempts to limits the size of column titles to at most nTitleLimit
display characters in width and a single line of text. The nTitleLimit
is useful for queries that have result columns that are scalar
subqueries or complex expressions. If those columns lack an AS
clause, then the name of the column will be a copy of the expression
that defines the column, which in some queries can be hundreds of
characters and multiple lines in length, which can reduce the readability
of tabular displays. An nTitleLimit somewhere in the range of 10 to 20.
can improve readability. The nTitleLimit setting currently only
works for **Box**, **Column**, **Line**, **Markdown**, and **Table**
styles, though that limitation might change in future releases.
### 2.9 Multiple Tuples Per INSERT In QRF_STYLE_Insert (nMultiInsert)
If the sqlite3_qrf_spec.nMultiInsert value is positive, then the
QRF_STYLE_Insert output mode will generate multiple tuples in
each INSERT statement until the total number of bytes in the
statement exceeds nMultiInsert. A value of a few thousand is
recommended here, in order to generate SQL output that is parsed
and inserted at maximum speed by SQLite.
### 2.10 Word Wrapping In Columnar Styles (nWrap, bWordWrap)
When using columnar formatting modes (QRF_STYLE_Box, QRF_STYLE_Column,
QRF_STYLE_Markdown, or QRF_STYLE_Table), the formatter attempts to limit
@@ -388,7 +410,7 @@ anywhere, including in the middle of a word.
For narrow columns and wide words, it might sometimes be necessary to split
a column in the middle of a word, even when bWordWrap is QRF_Yes.
### 2.10 Helping The Output To Fit On The Terminal (nScreenWidth)
### 2.11 Helping The Output To Fit On The Terminal (nScreenWidth)
The sqlite3_qrf_spec.nScreenWidth field can be set the number of
characters that will fit on one line on the viewer output device.
@@ -408,7 +430,7 @@ The nScreenWidth field currently only makes a difference in
columnar styles (**Box**, **Column**, **Markdown**, and **Table**)
and in the **Line** style.
### 2.11 Individual Column Width (nWidth and aWidth)
### 2.12 Individual Column Width (nWidth and aWidth)
The sqlite3_qrf_spec.aWidth field is a pointer to an array of
signed 16-bit integers that control the width of individual columns
@@ -446,7 +468,7 @@ Again, negative values for aWidth\[\] entries are supported for
backwards compatibility only, and are not recommended for new
applications.
### 2.12 Alignment (nAlignment, aAlignment, eDfltAlign, eTitleAlign)
### 2.13 Alignment (nAlignment, aAlignment, eDfltAlign, eTitleAlign)
Some cells in a display table might contain a lot of text and thus
be wide, or they might contain newline characters or be wrapped by
@@ -525,7 +547,7 @@ specify a vertical alignment, then values are top-aligned
The vertical alignment settings are currently ignored and
the vertical alignment is always QRF_ALIGN_Top.*
### 2.13 Row and Column Separator Strings
### 2.14 Row and Column Separator Strings
The sqlite3_qrf_spec.zColumnSep and sqlite3_qrf_spec.zRowSep strings
are alternative column and row separator character sequences. If not
@@ -533,18 +555,18 @@ specified (if these pointers are left as NULL) then appropriate defaults
are used. Some output styles have hard-coded column and row separators
and these settings are ignored for those styles.
### 2.14 The Output Table Name
### 2.15 The Output Table Name
The sqlite3_qrf_spec.zTableName value is the name of the output table
when eStyle is QRF_STYLE_Insert.
### 2.15 The Rendering Of NULL (zNull)
### 2.16 The Rendering Of NULL (zNull)
If a value is NULL then show the NULL using the string
found in sqlite3_qrf_spec.zNull. If zNull is itself a NULL pointer
then NULL values are rendered as an empty string.
### 2.16 Optional Value Rendering Callback
### 2.17 Optional Value Rendering Callback
If the sqlite3_qrf_spec.xRender field is not NULL, then each
sqlite3_value coming out of the query is first passed to the
@@ -565,6 +587,13 @@ The eText, eBlob, and eEsc settings above become no-ops if the xRender
routine returns non-NULL. In other words, the application-supplied
xRender routine is expected to do all of its own quoting and formatting.
The xRender routine is expected to do character length limiting itself.
So the nCharLimit setting becomes a no-op if xRender is used. However
the nLineLimit setting is still applied. The nTitleLimit setting is
not applicable to xRender because title values come from the
sqlite3_column_name() interface not from sqlite3_column_value(),
and so that names of columns are never processed by xRender.
## 3.0 The `sqlite3_format_query_result()` Interface
Invoke the `sqlite3_format_query_result(P,S,E)` interface to run
@@ -690,7 +719,10 @@ the `<TABLE>..</TABLE>` around the outside.
The **Insert** style generates a series of SQL "INSERT" statements
that will inserts the data that is output into a table whose name is defined
by the zTableName field of `sqlite3_qrf_spec`. If zTableName is NULL,
then a substitute name is used.
then a substitute name is used. If nMultiInsert is positive, then the
output will add multiple rows to each INSERT statement until the size
of the INSERT statement exceeds nMultiInsert bytes before starting
a new INSERT statement.
The **Json** and **JObject** styles generates JSON text for the query result.
The **Json** style produces a JSON array of structures with one
+248 -59
View File
@@ -17,6 +17,7 @@
#endif
#include <string.h>
#include <assert.h>
#include <stdint.h>
typedef sqlite3_int64 i64;
@@ -34,7 +35,8 @@ typedef struct qrfEQPGraph qrfEQPGraph;
struct qrfEQPGraph {
qrfEQPGraphRow *pRow; /* Linked list of all rows of the EQP output */
qrfEQPGraphRow *pLast; /* Last element of the pRow list */
char zPrefix[100]; /* Graph prefix */
int nWidth; /* Width of the graph */
char zPrefix[400]; /* Graph prefix */
};
/*
@@ -56,7 +58,7 @@ struct Qrf {
union {
struct { /* Content for QRF_STYLE_Line */
int mxColWth; /* Maximum display width of any column */
const char **azCol; /* Names of output columns (MODE_Line) */
char **azCol; /* Names of output columns (MODE_Line) */
} sLine;
qrfEQPGraph *pGraph; /* EQP graph (Eqp, Stats, and StatsEst) */
struct { /* Content for QRF_STYLE_Explain */
@@ -64,6 +66,7 @@ struct Qrf {
int iIndent; /* Current slot */
int *aiIndent; /* Indentation for each opcode */
} sExpln;
unsigned int nIns; /* Bytes used for current INSERT stmt */
} u;
sqlite3_int64 nRow; /* Number of rows handled so far */
int *actualWidth; /* Actual width of each column */
@@ -140,6 +143,15 @@ static void qrfOom(Qrf *p){
qrfError(p, SQLITE_NOMEM, "out of memory");
}
/*
** Transfer any error in pStr over into p.
*/
static void qrfStrErr(Qrf *p, sqlite3_str *pStr){
int rc = pStr ? sqlite3_str_errcode(pStr) : 0;
if( rc ){
qrfError(p, rc, sqlite3_errstr(rc));
}
}
/*
@@ -220,6 +232,45 @@ static void qrfEqpRenderLevel(Qrf *p, int iEqpId){
}
}
/*
** Render the 64-bit value N in a more human-readable format into
** pOut.
**
** + Only show the first three significant digits.
** + Append suffixes K, M, G, T, P, and E for 1e3, 1e6, ... 1e18
*/
static void qrfApproxInt64(sqlite3_str *pOut, i64 N){
static const char aSuffix[] = { 'K', 'M', 'G', 'T', 'P', 'E' };
int i;
if( N<0 ){
N = N==INT64_MIN ? INT64_MAX : -N;
sqlite3_str_append(pOut, "-", 1);
}
if( N<10000 ){
sqlite3_str_appendf(pOut, "%4lld ", N);
return;
}
for(i=1; i<=18; i++){
N = (N+5)/10;
if( N<10000 ){
int n = (int)N;
switch( i%3 ){
case 0:
sqlite3_str_appendf(pOut, "%d.%02d", n/1000, (n%1000)/10);
break;
case 1:
sqlite3_str_appendf(pOut, "%2d.%d", n/100, (n%100)/10);
break;
case 2:
sqlite3_str_appendf(pOut, "%4d", n/10);
break;
}
sqlite3_str_append(pOut, &aSuffix[i/3], 1);
break;
}
}
}
/*
** Display and reset the EXPLAIN QUERY PLAN data
*/
@@ -235,7 +286,26 @@ static void qrfEqpRender(Qrf *p, i64 nCycle){
p->u.pGraph->pRow = pRow->pNext;
sqlite3_free(pRow);
}else if( nCycle>0 ){
sqlite3_str_appendf(p->pOut, "QUERY PLAN (cycles=%lld [100%%])\n",nCycle);
int nSp = p->u.pGraph->nWidth - 2;
if( p->spec.eStyle==QRF_STYLE_StatsEst ){
sqlite3_str_appendchar(p->pOut, nSp, ' ');
sqlite3_str_appendall(p->pOut,
"Cycles Loops (est) Rows (est)\n");
sqlite3_str_appendchar(p->pOut, nSp, ' ');
sqlite3_str_appendall(p->pOut,
"---------- ------------ ------------\n");
}else{
sqlite3_str_appendchar(p->pOut, nSp, ' ');
sqlite3_str_appendall(p->pOut,
"Cycles Loops Rows \n");
sqlite3_str_appendchar(p->pOut, nSp, ' ');
sqlite3_str_appendall(p->pOut,
"---------- ----- -----\n");
}
sqlite3_str_appendall(p->pOut, "QUERY PLAN");
sqlite3_str_appendchar(p->pOut, nSp - 10, ' ');
qrfApproxInt64(p->pOut, nCycle);
sqlite3_str_appendall(p->pOut, " 100%\n");
}else{
sqlite3_str_appendall(p->pOut, "QUERY PLAN\n");
}
@@ -290,6 +360,8 @@ static void qrfEqpStats(Qrf *p){
int i = 0;
i64 nTotal = 0;
int nWidth = 0;
int prevPid = -1; /* Previous iPid */
double rEstCum = 1.0; /* Cumulative row estimate */
sqlite3_str *pLine = sqlite3_str_new(p->db);
sqlite3_str *pStats = sqlite3_str_new(p->db);
qrfEqpReset(p);
@@ -303,7 +375,7 @@ static void qrfEqpStats(Qrf *p){
n = (int)strlen(z) + qrfStatsHeight(pS,i)*3;
if( n>nWidth ) nWidth = n;
}
nWidth += 4;
nWidth += 2;
sqlite3_stmt_scanstatus_v2(pS,-1, SQLITE_SCANSTAT_NCYCLE, f, (void*)&nTotal);
for(i=0; 1; i++){
@@ -319,40 +391,49 @@ static void qrfEqpStats(Qrf *p){
if( sqlite3_stmt_scanstatus_v2(pS,i,SQLITE_SCANSTAT_EXPLAIN,f,(void*)&zo) ){
break;
}
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_PARENTID,f,(void*)&iPid);
if( iPid!=prevPid ){
prevPid = iPid;
rEstCum = 1.0;
}
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_EST,f,(void*)&rEst);
rEstCum *= rEst;
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_NLOOP,f,(void*)&nLoop);
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_NVISIT,f,(void*)&nRow);
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_NCYCLE,f,(void*)&nCycle);
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_SELECTID,f,(void*)&iId);
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_PARENTID,f,(void*)&iPid);
sqlite3_stmt_scanstatus_v2(pS,i, SQLITE_SCANSTAT_NAME,f,(void*)&zName);
if( nCycle>=0 || nLoop>=0 || nRow>=0 ){
const char *zSp = "";
double rpl;
int nSp = 0;
sqlite3_str_reset(pStats);
if( nCycle>=0 && nTotal>0 ){
sqlite3_str_appendf(pStats, "cycles=%lld [%d%%]",
nCycle, ((nCycle*100)+nTotal/2) / nTotal
qrfApproxInt64(pStats, nCycle);
sqlite3_str_appendf(pStats, " %3d%%",
((nCycle*100)+nTotal/2) / nTotal
);
zSp = " ";
nSp = 2;
}
if( nLoop>=0 ){
sqlite3_str_appendf(pStats, "%sloops=%lld", zSp, nLoop);
zSp = " ";
if( nSp ) sqlite3_str_appendchar(pStats, nSp, ' ');
qrfApproxInt64(pStats, nLoop);
nSp = 2;
if( p->spec.eStyle==QRF_STYLE_StatsEst ){
sqlite3_str_appendf(pStats, " ");
qrfApproxInt64(pStats, (i64)(rEstCum/rEst));
}
}
if( nRow>=0 ){
sqlite3_str_appendf(pStats, "%srows=%lld", zSp, nRow);
zSp = " ";
if( nSp ) sqlite3_str_appendchar(pStats, nSp, ' ');
qrfApproxInt64(pStats, nRow);
nSp = 2;
if( p->spec.eStyle==QRF_STYLE_StatsEst ){
sqlite3_str_appendf(pStats, " ");
qrfApproxInt64(pStats, (i64)rEstCum);
}
}
if( p->spec.eStyle==QRF_STYLE_StatsEst ){
rpl = (double)nRow / (double)nLoop;
sqlite3_str_appendf(pStats, "%srpl=%.1f est=%.1f", zSp, rpl, rEst);
}
sqlite3_str_appendf(pLine,
"% *s (%s)", -1*(nWidth-qrfStatsHeight(pS,i)*3), zo,
"% *s %s", -1*(nWidth-qrfStatsHeight(pS,i)*3), zo,
sqlite3_str_value(pStats)
);
sqlite3_str_reset(pStats);
@@ -362,7 +443,10 @@ static void qrfEqpStats(Qrf *p){
qrfEqpAppend(p, iId, iPid, zo);
}
}
if( p->u.pGraph ) p->u.pGraph->nWidth = nWidth;
qrfStrErr(p, pLine);
sqlite3_free(sqlite3_str_finish(pLine));
qrfStrErr(p, pStats);
sqlite3_free(sqlite3_str_finish(pStats));
#endif
}
@@ -726,9 +810,11 @@ static void qrfEscape(
static int qrfRelaxable(Qrf *p, const char *z){
size_t i, n;
if( z[0]=='\'' || qrfSpace(z[0]) ) return 0;
if( z[0]==0 && (p->spec.zNull==0 || p->spec.zNull[0]==0) ) return 0;
if( z[0]==0 ){
return (p->spec.zNull!=0 && p->spec.zNull[0]!=0);
}
n = strlen(z);
if( z[n-1]=='\'' || qrfSpace(z[n-1]) ) return 0;
if( n==0 || z[n-1]=='\'' || qrfSpace(z[n-1]) ) return 0;
if( p->spec.zNull && strcmp(p->spec.zNull,z)==0 ) return 0;
i = (z[0]=='-' || z[0]=='+');
if( strcmp(z+i,"Inf")==0 ) return 0;
@@ -945,6 +1031,53 @@ static const char *qrfJsonbToJson(Qrf *p, int iCol){
}
}
/*
** Adjust the input string zIn[] such that it is no more than N display
** characters wide. If it is wider than that, then truncate and add
** ellipsis. Or if zIn[] contains a \r or \n, truncate at that point,
** adding ellipsis. Embedded tabs in zIn[] are converted into ordinary
** spaces.
**
** Return this display width of the modified title string.
*/
static int qrfTitleLimit(char *zIn, int N){
unsigned char *z = (unsigned char*)zIn;
int n = 0;
unsigned char *zEllipsis = 0;
while( 1 /*exit-by-break*/ ){
if( z[0]<' ' ){
int k;
if( z[0]==0 ){
zEllipsis = 0;
break;
}else if( z[0]=='\033' && (k = qrfIsVt100(z))>0 ){
z += k;
}else if( z[0]=='\t' ){
z[0] = ' ';
}else if( z[0]=='\n' || z[0]=='\r' ){
z[0] = ' ';
}else{
z++;
}
}else if( (0x80&z[0])==0 ){
if( n>=(N-3) && zEllipsis==0 ) zEllipsis = z;
if( n==N ){ z[0] = 0; break; }
n++;
z++;
}else{
int u = 0;
int len = sqlite3_qrf_decode_utf8(z, &u);
if( n+len>(N-3) && zEllipsis==0 ) zEllipsis = z;
if( n+len>N ){ z[0] = 0; break; }
z += len;
n += sqlite3_qrf_wcwidth(u);
}
}
if( zEllipsis && N>=3 ) memcpy(zEllipsis,"...",4);
return n;
}
/*
** Render value pVal into pOut
*/
@@ -1827,8 +1960,15 @@ static void qrfColumnar(Qrf *p){
pStr = sqlite3_str_new(p->db);
qrfEncodeText(p, pStr, z ? z : "");
n = sqlite3_str_length(pStr);
qrfStrErr(p, pStr);
z = data.az[data.n] = sqlite3_str_finish(pStr);
data.aiWth[data.n] = w = qrfDisplayWidth(z, n, &nNL);
if( p->spec.nTitleLimit ){
nNL = 0;
data.aiWth[data.n] = w = qrfTitleLimit(data.az[data.n],
p->spec.nTitleLimit );
}else{
data.aiWth[data.n] = w = qrfDisplayWidth(z, n, &nNL);
}
data.n++;
if( w>data.a[i].mxW ) data.a[i].mxW = w;
if( nNL ) data.bMultiRow = 1;
@@ -1848,6 +1988,7 @@ static void qrfColumnar(Qrf *p){
pStr = sqlite3_str_new(p->db);
qrfRenderValue(p, pStr, i);
n = sqlite3_str_length(pStr);
qrfStrErr(p, pStr);
z = data.az[data.n] = sqlite3_str_finish(pStr);
data.abNum[data.n] = eType==SQLITE_INTEGER || eType==SQLITE_FLOAT;
data.aiWth[data.n] = w = qrfDisplayWidth(z, n, &nNL);
@@ -2002,7 +2143,7 @@ static void qrfColumnar(Qrf *p){
}else{
bRTrim = 0;
}
for(i=0; i<data.n; i+=nColumn){
for(i=0; i<data.n && sqlite3_str_errcode(p->pOut)==SQLITE_OK; i+=nColumn){
int bMore;
int nRow = 0;
@@ -2189,7 +2330,7 @@ static void qrfExplain(Qrf *p){
assert( 0==sqlite3_stricmp( sqlite3_column_name(p->pStmt, 2), "p1" ) );
assert( 0==sqlite3_stricmp( sqlite3_column_name(p->pStmt, 3), "p2" ) );
for(iOp=0; SQLITE_ROW==sqlite3_step(p->pStmt); iOp++){
for(iOp=0; SQLITE_ROW==sqlite3_step(p->pStmt) && !p->iErr; iOp++){
int iAddr = sqlite3_column_int(p->pStmt, 0);
const char *zOp = (const char*)sqlite3_column_text(p->pStmt, 1);
int p1 = sqlite3_column_int(p->pStmt, 2);
@@ -2246,7 +2387,7 @@ static void qrfExplain(Qrf *p){
}
if( nArg>nWidth ) nArg = nWidth;
for(iOp=0; sqlite3_step(p->pStmt)==SQLITE_ROW; iOp++){
for(iOp=0; sqlite3_step(p->pStmt)==SQLITE_ROW && !p->iErr; iOp++){
/* If this is the first row seen, print out the headers */
if( iOp==0 ){
for(i=0; i<nArg; i++){
@@ -2423,30 +2564,46 @@ static void qrfOneSimpleRow(Qrf *p){
break;
}
case QRF_STYLE_Insert: {
if( qrf_need_quote(p->spec.zTableName) ){
sqlite3_str_appendf(p->pOut,"INSERT INTO \"%w\"",p->spec.zTableName);
}else{
sqlite3_str_appendf(p->pOut,"INSERT INTO %s",p->spec.zTableName);
}
if( p->spec.bTitles==QRF_Yes ){
for(i=0; i<p->nCol; i++){
const char *zCName = sqlite3_column_name(p->pStmt, i);
if( qrf_need_quote(zCName) ){
sqlite3_str_appendf(p->pOut, "%c\"%w\"",
i==0 ? '(' : ',', zCName);
}else{
sqlite3_str_appendf(p->pOut, "%c%s",
i==0 ? '(' : ',', zCName);
}
unsigned int mxIns = p->spec.nMultiInsert;
int szStart = sqlite3_str_length(p->pOut);
if( p->u.nIns==0 || p->u.nIns>=mxIns ){
if( p->u.nIns ){
sqlite3_str_append(p->pOut, ";\n", 2);
p->u.nIns = 0;
}
sqlite3_str_append(p->pOut, ")", 1);
if( qrf_need_quote(p->spec.zTableName) ){
sqlite3_str_appendf(p->pOut,"INSERT INTO \"%w\"",p->spec.zTableName);
}else{
sqlite3_str_appendf(p->pOut,"INSERT INTO %s",p->spec.zTableName);
}
if( p->spec.bTitles==QRF_Yes ){
for(i=0; i<p->nCol; i++){
const char *zCName = sqlite3_column_name(p->pStmt, i);
if( qrf_need_quote(zCName) ){
sqlite3_str_appendf(p->pOut, "%c\"%w\"",
i==0 ? '(' : ',', zCName);
}else{
sqlite3_str_appendf(p->pOut, "%c%s",
i==0 ? '(' : ',', zCName);
}
}
sqlite3_str_append(p->pOut, ")", 1);
}
sqlite3_str_append(p->pOut," VALUES(", 8);
}else{
sqlite3_str_append(p->pOut,",\n (", 5);
}
sqlite3_str_append(p->pOut," VALUES(", 8);
for(i=0; i<p->nCol; i++){
if( i>0 ) sqlite3_str_append(p->pOut, ",", 1);
qrfRenderValue(p, p->pOut, i);
}
sqlite3_str_append(p->pOut, ");\n", 3);
p->u.nIns += sqlite3_str_length(p->pOut) + 2 - szStart;
if( p->u.nIns>=mxIns ){
sqlite3_str_append(p->pOut, ");\n", 3);
p->u.nIns = 0;
}else{
sqlite3_str_append(p->pOut, ")", 1);
}
qrfWrite(p);
break;
}
@@ -2454,6 +2611,7 @@ static void qrfOneSimpleRow(Qrf *p){
sqlite3_str *pVal;
int mxW;
int bWW;
int nSep;
if( p->u.sLine.azCol==0 ){
p->u.sLine.azCol = sqlite3_malloc64( p->nCol*sizeof(char*) );
if( p->u.sLine.azCol==0 ){
@@ -2463,28 +2621,35 @@ static void qrfOneSimpleRow(Qrf *p){
p->u.sLine.mxColWth = 0;
for(i=0; i<p->nCol; i++){
int sz;
p->u.sLine.azCol[i] = sqlite3_column_name(p->pStmt, i);
if( p->u.sLine.azCol[i]==0 ) p->u.sLine.azCol[i] = "unknown";
const char *zCName = sqlite3_column_name(p->pStmt, i);
if( zCName==0 ) zCName = "unknown";
p->u.sLine.azCol[i] = sqlite3_mprintf("%s", zCName);
if( p->spec.nTitleLimit>0 ){
(void)qrfTitleLimit(p->u.sLine.azCol[i], p->spec.nTitleLimit);
}
sz = (int)sqlite3_qrf_wcswidth(p->u.sLine.azCol[i]);
if( sz > p->u.sLine.mxColWth ) p->u.sLine.mxColWth = sz;
}
}
if( p->nRow ) sqlite3_str_append(p->pOut, "\n", 1);
pVal = sqlite3_str_new(p->db);
mxW = p->mxWidth - (3 + p->u.sLine.mxColWth);
nSep = (int)strlen(p->spec.zColumnSep);
mxW = p->mxWidth - (nSep + p->u.sLine.mxColWth);
bWW = p->spec.bWordWrap==QRF_Yes;
for(i=0; i<p->nCol; i++){
const char *zVal;
int cnt = 0;
qrfWidthPrint(p, p->pOut, -p->u.sLine.mxColWth, p->u.sLine.azCol[i]);
sqlite3_str_append(p->pOut, " = ", 3);
sqlite3_str_append(p->pOut, p->spec.zColumnSep, nSep);
qrfRenderValue(p, pVal, i);
zVal = sqlite3_str_value(pVal);
if( zVal==0 ) zVal = "";
do{
int nThis, nWide, iNext;
qrfWrapLine(zVal, mxW, bWW, &nThis, &nWide, &iNext);
if( cnt ) sqlite3_str_appendchar(p->pOut,p->u.sLine.mxColWth+3,' ');
if( cnt ){
sqlite3_str_appendchar(p->pOut,p->u.sLine.mxColWth+nSep,' ');
}
cnt++;
if( cnt>p->mxHeight ){
zVal = "...";
@@ -2496,6 +2661,7 @@ static void qrfOneSimpleRow(Qrf *p){
}while( zVal[0] );
sqlite3_str_reset(pVal);
}
qrfStrErr(p, pVal);
sqlite3_free(sqlite3_str_finish(pVal));
qrfWrite(p);
break;
@@ -2546,7 +2712,7 @@ static void qrfInitialize(
size_t sz; /* Size of pSpec[], based on pSpec->iVersion */
memset(p, 0, sizeof(*p));
p->pzErr = pzErr;
if( pSpec->iVersion!=1 ){
if( pSpec->iVersion>1 ){
qrfError(p, SQLITE_ERROR,
"unusable sqlite3_qrf_spec.iVersion (%d)",
pSpec->iVersion);
@@ -2559,7 +2725,7 @@ static void qrfInitialize(
qrfOom(p);
return;
}
p->iErr = 0;
p->iErr = SQLITE_OK;
p->nCol = sqlite3_column_count(p->pStmt);
p->nRow = 0;
sz = sizeof(sqlite3_qrf_spec);
@@ -2606,6 +2772,13 @@ qrf_reinit:
if( p->spec.zTableName==0 || p->spec.zTableName[0]==0 ){
p->spec.zTableName = "tab";
}
p->u.nIns = 0;
break;
}
case QRF_STYLE_Line: {
if( p->spec.zColumnSep==0 ){
p->spec.zColumnSep = ": ";
}
break;
}
case QRF_STYLE_Csv: {
@@ -2698,42 +2871,58 @@ static void qrfFinalize(Qrf *p){
switch( p->spec.eStyle ){
case QRF_STYLE_Count: {
sqlite3_str_appendf(p->pOut, "%lld\n", p->nRow);
qrfWrite(p);
break;
}
case QRF_STYLE_Json: {
if( p->nRow>0 ){
sqlite3_str_append(p->pOut, "}]\n", 3);
qrfWrite(p);
}
break;
}
case QRF_STYLE_JObject: {
if( p->nRow>0 ){
sqlite3_str_append(p->pOut, "}\n", 2);
qrfWrite(p);
}
break;
}
case QRF_STYLE_Insert: {
if( p->u.nIns ){
sqlite3_str_append(p->pOut, ";\n", 2);
}
break;
}
case QRF_STYLE_Line: {
if( p->u.sLine.azCol ) sqlite3_free(p->u.sLine.azCol);
if( p->u.sLine.azCol ){
int i;
for(i=0; i<p->nCol; i++) sqlite3_free(p->u.sLine.azCol[i]);
sqlite3_free(p->u.sLine.azCol);
}
break;
}
case QRF_STYLE_Stats:
case QRF_STYLE_StatsEst:
case QRF_STYLE_StatsEst: {
i64 nCycle = 0;
#ifdef SQLITE_ENABLE_STMT_SCANSTATUS
sqlite3_stmt_scanstatus_v2(p->pStmt, -1, SQLITE_SCANSTAT_NCYCLE,
SQLITE_SCANSTAT_COMPLEX, (void*)&nCycle);
#endif
qrfEqpRender(p, nCycle);
break;
}
case QRF_STYLE_Eqp: {
qrfEqpRender(p, 0);
qrfWrite(p);
break;
}
}
qrfWrite(p);
qrfStrErr(p, p->pOut);
if( p->spec.pzOutput ){
if( p->spec.pzOutput[0] ){
sqlite3_int64 n, sz;
char *zCombined;
sz = strlen(p->spec.pzOutput[0]);
n = sqlite3_str_length(p->pOut);
zCombined = sqlite3_realloc(p->spec.pzOutput[0], sz+n+1);
zCombined = sqlite3_realloc64(p->spec.pzOutput[0], sz+n+1);
if( zCombined==0 ){
sqlite3_free(p->spec.pzOutput[0]);
p->spec.pzOutput[0] = 0;
+2
View File
@@ -41,6 +41,8 @@ struct sqlite3_qrf_spec {
short int nWrap; /* Wrap columns wider than this */
short int nScreenWidth; /* Maximum overall table width */
short int nLineLimit; /* Maximum number of lines for any row */
short int nTitleLimit; /* Maximum number of characters in a title */
unsigned int nMultiInsert; /* Add rows to one INSERT until size exceeds */
int nCharLimit; /* Maximum number of characters in a cell */
int nWidth; /* Number of entries in aWidth[] */
int nAlign; /* Number of entries in aAlignment[] */
+28
View File
@@ -192,4 +192,32 @@ do_test 4.7.2 {
list [catch {rbu close} msg] $msg
} {1 {SQLITE_ERROR - rbu_state mismatch error}}
#-------------------------------------------------------------------------
# https://sqlite.org/forum/info/6d0a31e22a435877
#
reset_db
forcedelete rbu.db
do_execsql_test 5.0 {
CREATE TABLE t1(a BLOB);
INSERT INTO t1 VALUES(x'41');
}
sqlite3 dbRbu rbu.db
dbRbu eval {
CREATE TABLE data_t1(a, rbu_control, rbu_rowid);
INSERT INTO data_t1 VALUES(X'310a313a5a337e7e7e7e7e40302c303b','f',1);
}
dbRbu close
do_test 5.1 {
sqlite3rbu rbu test.db rbu.db
rbu step
} {SQLITE_ERROR}
do_test 5.2 {
list [catch {rbu close} msg] $msg
} {1 {SQLITE_ERROR - corrupt fossil delta}}
finish_test
+3 -3
View File
@@ -623,7 +623,7 @@ static int rbuDeltaApply(
/* ERROR: copy exceeds output file size */
return -1;
}
if( (int)(ofst+cnt) > lenSrc ){
if( (u64)ofst+(u64)cnt > (u64)lenSrc ){
/* ERROR: copy extends past end of input */
return -1;
}
@@ -2269,8 +2269,8 @@ static char *rbuObjIterGetIndexWhere(sqlite3rbu *p, RbuObjIter *pIter){
/* If necessary, grow the pIter->aIdxCol[] array */
if( iIdxCol==nIdxAlloc ){
RbuSpan *aIdxCol = (RbuSpan*)sqlite3_realloc(
pIter->aIdxCol, (nIdxAlloc+16)*sizeof(RbuSpan)
RbuSpan *aIdxCol = (RbuSpan*)sqlite3_realloc64(
pIter->aIdxCol, nIdxAlloc*sizeof(RbuSpan) + 16*sizeof(RbuSpan)
);
if( aIdxCol==0 ){
rc = SQLITE_NOMEM;
-16
View File
@@ -1,16 +0,0 @@
This folder contains extensions and utility programs intended to analyze
live database files, detect problems, and possibly fix them.
As SQLite is being used on larger and larger databases, database sizes
are growing into the terabyte range. At that size, hardware malfunctions
and/or cosmic rays will occasionally corrupt a database file. Detecting
problems and fixing errors a terabyte-sized databases can take hours or days,
and it is undesirable to take applications that depend on the databases
off-line for such a long time.
The utilities in the folder are intended to provide mechanisms for
detecting and fixing problems in large databases while those databases
are in active use.
The utilities and extensions in this folder are experimental and under
active development at the time of this writing (2017-10-12). If and when
they stabilize, this README will be updated to reflect that fact.
-310
View File
@@ -1,310 +0,0 @@
/*
** 2017 October 11
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
**
** This module exports a single C function:
**
** int sqlite3_check_freelist(sqlite3 *db, const char *zDb);
**
** This function checks the free-list in database zDb (one of "main",
** "temp", etc.) and reports any errors by invoking the sqlite3_log()
** function. It returns SQLITE_OK if successful, or an SQLite error
** code otherwise. It is not an error if the free-list is corrupted but
** no IO or OOM errors occur.
**
** If this file is compiled and loaded as an SQLite loadable extension,
** it adds an SQL function "checkfreelist" to the database handle, to
** be invoked as follows:
**
** SELECT checkfreelist(<database-name>);
**
** This function performs the same checks as sqlite3_check_freelist(),
** except that it returns all error messages as a single text value,
** separated by newline characters. If the freelist is not corrupted
** in any way, an empty string is returned.
**
** To compile this module for use as an SQLite loadable extension:
**
** gcc -Os -fPIC -shared checkfreelist.c -o checkfreelist.so
*/
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
#ifndef SQLITE_AMALGAMATION
# include <string.h>
# include <stdio.h>
# include <stdlib.h>
# include <assert.h>
# if defined(SQLITE_COVERAGE_TEST) || defined(SQLITE_MUTATION_TEST)
# define SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS 1
# endif
# if defined(SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS)
# define ALWAYS(X) (1)
# define NEVER(X) (0)
# elif !defined(NDEBUG)
# define ALWAYS(X) ((X)?1:(assert(0),0))
# define NEVER(X) ((X)?(assert(0),1):0)
# else
# define ALWAYS(X) (X)
# define NEVER(X) (X)
# endif
typedef unsigned char u8;
typedef unsigned short u16;
typedef unsigned int u32;
#define get4byte(x) ( \
((u32)((x)[0])<<24) + \
((u32)((x)[1])<<16) + \
((u32)((x)[2])<<8) + \
((u32)((x)[3])) \
)
#endif
/*
** Execute a single PRAGMA statement and return the integer value returned
** via output parameter (*pnOut).
**
** The SQL statement passed as the third argument should be a printf-style
** format string containing a single "%s" which will be replace by the
** value passed as the second argument. e.g.
**
** sqlGetInteger(db, "main", "PRAGMA %s.page_count", pnOut)
**
** executes "PRAGMA main.page_count" and stores the results in (*pnOut).
*/
static int sqlGetInteger(
sqlite3 *db, /* Database handle */
const char *zDb, /* Database name ("main", "temp" etc.) */
const char *zFmt, /* SQL statement format */
u32 *pnOut /* OUT: Integer value */
){
int rc, rc2;
char *zSql;
sqlite3_stmt *pStmt = 0;
int bOk = 0;
zSql = sqlite3_mprintf(zFmt, zDb);
if( zSql==0 ){
rc = SQLITE_NOMEM;
}else{
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
sqlite3_free(zSql);
}
if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){
*pnOut = (u32)sqlite3_column_int(pStmt, 0);
bOk = 1;
}
rc2 = sqlite3_finalize(pStmt);
if( rc==SQLITE_OK ) rc = rc2;
if( rc==SQLITE_OK && bOk==0 ) rc = SQLITE_ERROR;
return rc;
}
/*
** Argument zFmt must be a printf-style format string and must be
** followed by its required arguments. If argument pzOut is NULL,
** then the results of printf()ing the format string are passed to
** sqlite3_log(). Otherwise, they are appended to the string
** at (*pzOut).
*/
static int checkFreelistError(char **pzOut, const char *zFmt, ...){
int rc = SQLITE_OK;
char *zErr = 0;
va_list ap;
va_start(ap, zFmt);
zErr = sqlite3_vmprintf(zFmt, ap);
if( zErr==0 ){
rc = SQLITE_NOMEM;
}else{
if( pzOut ){
*pzOut = sqlite3_mprintf("%s%z%s", *pzOut?"\n":"", *pzOut, zErr);
if( *pzOut==0 ) rc = SQLITE_NOMEM;
}else{
sqlite3_log(SQLITE_ERROR, "checkfreelist: %s", zErr);
}
sqlite3_free(zErr);
}
va_end(ap);
return rc;
}
static int checkFreelist(
sqlite3 *db,
const char *zDb,
char **pzOut
){
/* This query returns one row for each page on the free list. Each row has
** two columns - the page number and page content. */
const char *zTrunk =
"WITH freelist_trunk(i, d, n) AS ("
"SELECT 1, NULL, sqlite_readint32(data, 32) "
"FROM sqlite_dbpage(:1) WHERE pgno=1 "
"UNION ALL "
"SELECT n, data, sqlite_readint32(data) "
"FROM freelist_trunk, sqlite_dbpage(:1) WHERE pgno=n "
")"
"SELECT i, d FROM freelist_trunk WHERE i!=1;";
int rc, rc2; /* Return code */
sqlite3_stmt *pTrunk = 0; /* Compilation of zTrunk */
u32 nPage = 0; /* Number of pages in db */
u32 nExpected = 0; /* Expected number of free pages */
u32 nFree = 0; /* Number of pages on free list */
if( zDb==0 ) zDb = "main";
if( (rc = sqlGetInteger(db, zDb, "PRAGMA %s.page_count", &nPage))
|| (rc = sqlGetInteger(db, zDb, "PRAGMA %s.freelist_count", &nExpected))
){
return rc;
}
rc = sqlite3_prepare_v2(db, zTrunk, -1, &pTrunk, 0);
if( rc!=SQLITE_OK ) return rc;
sqlite3_bind_text(pTrunk, 1, zDb, -1, SQLITE_STATIC);
while( rc==SQLITE_OK && sqlite3_step(pTrunk)==SQLITE_ROW ){
u32 i;
u32 iTrunk = (u32)sqlite3_column_int(pTrunk, 0);
const u8 *aData = (const u8*)sqlite3_column_blob(pTrunk, 1);
u32 nData = (u32)sqlite3_column_bytes(pTrunk, 1);
u32 iNext = get4byte(&aData[0]);
u32 nLeaf = get4byte(&aData[4]);
if( nLeaf>((nData/4)-2-6) ){
rc = checkFreelistError(pzOut,
"leaf count out of range (%d) on trunk page %d",
(int)nLeaf, (int)iTrunk
);
nLeaf = (nData/4) - 2 - 6;
}
nFree += 1+nLeaf;
if( iNext>nPage ){
rc = checkFreelistError(pzOut,
"trunk page %d is out of range", (int)iNext
);
}
for(i=0; rc==SQLITE_OK && i<nLeaf; i++){
u32 iLeaf = get4byte(&aData[8 + 4*i]);
if( iLeaf==0 || iLeaf>nPage ){
rc = checkFreelistError(pzOut,
"leaf page %d is out of range (child %d of trunk page %d)",
(int)iLeaf, (int)i, (int)iTrunk
);
}
}
}
if( rc==SQLITE_OK && nFree!=nExpected ){
rc = checkFreelistError(pzOut,
"free-list count mismatch: actual=%d header=%d",
(int)nFree, (int)nExpected
);
}
rc2 = sqlite3_finalize(pTrunk);
if( rc==SQLITE_OK ) rc = rc2;
return rc;
}
int sqlite3_check_freelist(sqlite3 *db, const char *zDb){
return checkFreelist(db, zDb, 0);
}
static void checkfreelist_function(
sqlite3_context *pCtx,
int nArg,
sqlite3_value **apArg
){
const char *zDb;
int rc;
char *zOut = 0;
sqlite3 *db = sqlite3_context_db_handle(pCtx);
assert( nArg==1 );
zDb = (const char*)sqlite3_value_text(apArg[0]);
rc = checkFreelist(db, zDb, &zOut);
if( rc==SQLITE_OK ){
sqlite3_result_text(pCtx, zOut?zOut:"ok", -1, SQLITE_TRANSIENT);
}else{
sqlite3_result_error_code(pCtx, rc);
}
sqlite3_free(zOut);
}
/*
** An SQL function invoked as follows:
**
** sqlite_readint32(BLOB) -- Decode 32-bit integer from start of blob
*/
static void readint_function(
sqlite3_context *pCtx,
int nArg,
sqlite3_value **apArg
){
const u8 *zBlob;
int nBlob;
int iOff = 0;
u32 iRet = 0;
if( nArg!=1 && nArg!=2 ){
sqlite3_result_error(
pCtx, "wrong number of arguments to function sqlite_readint32()", -1
);
return;
}
if( nArg==2 ){
iOff = sqlite3_value_int(apArg[1]);
}
zBlob = sqlite3_value_blob(apArg[0]);
nBlob = sqlite3_value_bytes(apArg[0]);
if( nBlob>=(iOff+4) ){
iRet = get4byte(&zBlob[iOff]);
}
sqlite3_result_int64(pCtx, (sqlite3_int64)iRet);
}
/*
** Register the SQL functions.
*/
static int cflRegister(sqlite3 *db){
int rc = sqlite3_create_function(
db, "sqlite_readint32", -1, SQLITE_UTF8, 0, readint_function, 0, 0
);
if( rc!=SQLITE_OK ) return rc;
rc = sqlite3_create_function(
db, "checkfreelist", 1, SQLITE_UTF8, 0, checkfreelist_function, 0, 0
);
return rc;
}
/*
** Extension load function.
*/
#ifdef _WIN32
__declspec(dllexport)
#endif
int sqlite3_checkfreelist_init(
sqlite3 *db,
char **pzErrMsg,
const sqlite3_api_routines *pApi
){
SQLITE_EXTENSION_INIT2(pApi);
return cflRegister(db);
}
-929
View File
@@ -1,929 +0,0 @@
/*
** 2017 October 27
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
*/
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
/*
** Stuff that is available inside the amalgamation, but which we need to
** declare ourselves if this module is compiled separately.
*/
#ifndef SQLITE_AMALGAMATION
# include <string.h>
# include <stdio.h>
# include <stdlib.h>
# include <assert.h>
typedef unsigned char u8;
typedef unsigned short u16;
typedef unsigned int u32;
#define get4byte(x) ( \
((u32)((x)[0])<<24) + \
((u32)((x)[1])<<16) + \
((u32)((x)[2])<<8) + \
((u32)((x)[3])) \
)
#endif
typedef struct CidxTable CidxTable;
typedef struct CidxCursor CidxCursor;
struct CidxTable {
sqlite3_vtab base; /* Base class. Must be first */
sqlite3 *db;
};
struct CidxCursor {
sqlite3_vtab_cursor base; /* Base class. Must be first */
sqlite3_int64 iRowid; /* Row number of the output */
char *zIdxName; /* Copy of the index_name parameter */
char *zAfterKey; /* Copy of the after_key parameter */
sqlite3_stmt *pStmt; /* SQL statement that generates the output */
};
typedef struct CidxColumn CidxColumn;
struct CidxColumn {
char *zExpr; /* Text for indexed expression */
int bDesc; /* True for DESC columns, otherwise false */
int bKey; /* Part of index, not PK */
};
typedef struct CidxIndex CidxIndex;
struct CidxIndex {
char *zWhere; /* WHERE clause, if any */
int nCol; /* Elements in aCol[] array */
CidxColumn aCol[1]; /* Array of indexed columns */
};
static void *cidxMalloc(int *pRc, int n){
void *pRet = 0;
assert( n!=0 );
if( *pRc==SQLITE_OK ){
pRet = sqlite3_malloc(n);
if( pRet ){
memset(pRet, 0, n);
}else{
*pRc = SQLITE_NOMEM;
}
}
return pRet;
}
static void cidxCursorError(CidxCursor *pCsr, const char *zFmt, ...){
va_list ap;
va_start(ap, zFmt);
assert( pCsr->base.pVtab->zErrMsg==0 );
pCsr->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap);
va_end(ap);
}
/*
** Connect to the incremental_index_check virtual table.
*/
static int cidxConnect(
sqlite3 *db,
void *pAux,
int argc, const char *const*argv,
sqlite3_vtab **ppVtab,
char **pzErr
){
int rc = SQLITE_OK;
CidxTable *pRet;
#define IIC_ERRMSG 0
#define IIC_CURRENT_KEY 1
#define IIC_INDEX_NAME 2
#define IIC_AFTER_KEY 3
#define IIC_SCANNER_SQL 4
rc = sqlite3_declare_vtab(db,
"CREATE TABLE xyz("
" errmsg TEXT," /* Error message or NULL if everything is ok */
" current_key TEXT," /* SQLite quote() text of key values */
" index_name HIDDEN," /* IN: name of the index being scanned */
" after_key HIDDEN," /* IN: Start scanning after this key */
" scanner_sql HIDDEN" /* debugging info: SQL used for scanner */
")"
);
pRet = cidxMalloc(&rc, sizeof(CidxTable));
if( pRet ){
pRet->db = db;
}
*ppVtab = (sqlite3_vtab*)pRet;
return rc;
}
/*
** Disconnect from or destroy an incremental_index_check virtual table.
*/
static int cidxDisconnect(sqlite3_vtab *pVtab){
CidxTable *pTab = (CidxTable*)pVtab;
sqlite3_free(pTab);
return SQLITE_OK;
}
/*
** idxNum and idxStr are not used. There are only three possible plans,
** which are all distinguished by the number of parameters.
**
** No parameters: A degenerate plan. The result is zero rows.
** 1 Parameter: Scan all of the index starting with first entry
** 2 parameters: Scan the index starting after the "after_key".
**
** Provide successively smaller costs for each of these plans to encourage
** the query planner to select the one with the most parameters.
*/
static int cidxBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pInfo){
int iIdxName = -1;
int iAfterKey = -1;
int i;
for(i=0; i<pInfo->nConstraint; i++){
struct sqlite3_index_constraint *p = &pInfo->aConstraint[i];
if( p->usable==0 ) continue;
if( p->op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue;
if( p->iColumn==IIC_INDEX_NAME ){
iIdxName = i;
}
if( p->iColumn==IIC_AFTER_KEY ){
iAfterKey = i;
}
}
if( iIdxName<0 ){
pInfo->estimatedCost = 1000000000.0;
}else{
pInfo->aConstraintUsage[iIdxName].argvIndex = 1;
pInfo->aConstraintUsage[iIdxName].omit = 1;
if( iAfterKey<0 ){
pInfo->estimatedCost = 1000000.0;
}else{
pInfo->aConstraintUsage[iAfterKey].argvIndex = 2;
pInfo->aConstraintUsage[iAfterKey].omit = 1;
pInfo->estimatedCost = 1000.0;
}
}
return SQLITE_OK;
}
/*
** Open a new btreeinfo cursor.
*/
static int cidxOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){
CidxCursor *pRet;
int rc = SQLITE_OK;
pRet = cidxMalloc(&rc, sizeof(CidxCursor));
*ppCursor = (sqlite3_vtab_cursor*)pRet;
return rc;
}
/*
** Close a btreeinfo cursor.
*/
static int cidxClose(sqlite3_vtab_cursor *pCursor){
CidxCursor *pCsr = (CidxCursor*)pCursor;
sqlite3_finalize(pCsr->pStmt);
sqlite3_free(pCsr->zIdxName);
sqlite3_free(pCsr->zAfterKey);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** Move a btreeinfo cursor to the next entry in the file.
*/
static int cidxNext(sqlite3_vtab_cursor *pCursor){
CidxCursor *pCsr = (CidxCursor*)pCursor;
int rc = sqlite3_step(pCsr->pStmt);
if( rc!=SQLITE_ROW ){
rc = sqlite3_finalize(pCsr->pStmt);
pCsr->pStmt = 0;
if( rc!=SQLITE_OK ){
sqlite3 *db = ((CidxTable*)pCsr->base.pVtab)->db;
cidxCursorError(pCsr, "Cursor error: %s", sqlite3_errmsg(db));
}
}else{
pCsr->iRowid++;
rc = SQLITE_OK;
}
return rc;
}
/* We have reached EOF if previous sqlite3_step() returned
** anything other than SQLITE_ROW;
*/
static int cidxEof(sqlite3_vtab_cursor *pCursor){
CidxCursor *pCsr = (CidxCursor*)pCursor;
return pCsr->pStmt==0;
}
static char *cidxMprintf(int *pRc, const char *zFmt, ...){
char *zRet = 0;
va_list ap;
va_start(ap, zFmt);
zRet = sqlite3_vmprintf(zFmt, ap);
if( *pRc==SQLITE_OK ){
if( zRet==0 ){
*pRc = SQLITE_NOMEM;
}
}else{
sqlite3_free(zRet);
zRet = 0;
}
va_end(ap);
return zRet;
}
static sqlite3_stmt *cidxPrepare(
int *pRc, CidxCursor *pCsr, const char *zFmt, ...
){
sqlite3_stmt *pRet = 0;
char *zSql;
va_list ap; /* ... printf arguments */
va_start(ap, zFmt);
zSql = sqlite3_vmprintf(zFmt, ap);
if( *pRc==SQLITE_OK ){
if( zSql==0 ){
*pRc = SQLITE_NOMEM;
}else{
sqlite3 *db = ((CidxTable*)pCsr->base.pVtab)->db;
*pRc = sqlite3_prepare_v2(db, zSql, -1, &pRet, 0);
if( *pRc!=SQLITE_OK ){
cidxCursorError(pCsr, "SQL error: %s", sqlite3_errmsg(db));
}
}
}
sqlite3_free(zSql);
va_end(ap);
return pRet;
}
static void cidxFinalize(int *pRc, sqlite3_stmt *pStmt){
int rc = sqlite3_finalize(pStmt);
if( *pRc==SQLITE_OK ) *pRc = rc;
}
char *cidxStrdup(int *pRc, const char *zStr){
char *zRet = 0;
if( *pRc==SQLITE_OK ){
int n = (int)strlen(zStr);
zRet = cidxMalloc(pRc, n+1);
if( zRet ) memcpy(zRet, zStr, n+1);
}
return zRet;
}
static void cidxFreeIndex(CidxIndex *pIdx){
if( pIdx ){
int i;
for(i=0; i<pIdx->nCol; i++){
sqlite3_free(pIdx->aCol[i].zExpr);
}
sqlite3_free(pIdx->zWhere);
sqlite3_free(pIdx);
}
}
static int cidx_isspace(char c){
return c==' ' || c=='\t' || c=='\r' || c=='\n';
}
static int cidx_isident(char c){
return c<0
|| (c>='0' && c<='9') || (c>='a' && c<='z')
|| (c>='A' && c<='Z') || c=='_';
}
#define CIDX_PARSE_EOF 0
#define CIDX_PARSE_COMMA 1 /* "," */
#define CIDX_PARSE_OPEN 2 /* "(" */
#define CIDX_PARSE_CLOSE 3 /* ")" */
/*
** Argument zIn points into the start, middle or end of a CREATE INDEX
** statement. If argument pbDoNotTrim is non-NULL, then this function
** scans the input until it finds EOF, a comma (",") or an open or
** close parenthesis character. It then sets (*pzOut) to point to said
** character and returns a CIDX_PARSE_XXX constant as appropriate. The
** parser is smart enough that special characters inside SQL strings
** or comments are not returned for.
**
** Or, if argument pbDoNotTrim is NULL, then this function sets *pzOut
** to point to the first character of the string that is not whitespace
** or part of an SQL comment and returns CIDX_PARSE_EOF.
**
** Additionally, if pbDoNotTrim is not NULL and the element immediately
** before (*pzOut) is an SQL comment of the form "-- comment", then
** (*pbDoNotTrim) is set before returning. In all other cases it is
** cleared.
*/
static int cidxFindNext(
const char *zIn,
const char **pzOut,
int *pbDoNotTrim /* OUT: True if prev is -- comment */
){
const char *z = zIn;
while( 1 ){
while( cidx_isspace(*z) ) z++;
if( z[0]=='-' && z[1]=='-' ){
z += 2;
while( z[0]!='\n' ){
if( z[0]=='\0' ) return CIDX_PARSE_EOF;
z++;
}
while( cidx_isspace(*z) ) z++;
if( pbDoNotTrim ) *pbDoNotTrim = 1;
}else
if( z[0]=='/' && z[1]=='*' ){
z += 2;
while( z[0]!='*' || z[1]!='/' ){
if( z[1]=='\0' ) return CIDX_PARSE_EOF;
z++;
}
z += 2;
}else{
*pzOut = z;
if( pbDoNotTrim==0 ) return CIDX_PARSE_EOF;
switch( *z ){
case '\0':
return CIDX_PARSE_EOF;
case '(':
return CIDX_PARSE_OPEN;
case ')':
return CIDX_PARSE_CLOSE;
case ',':
return CIDX_PARSE_COMMA;
case '"':
case '\'':
case '`': {
char q = *z;
z++;
while( *z ){
if( *z==q ){
z++;
if( *z!=q ) break;
}
z++;
}
break;
}
case '[':
while( *z++!=']' );
break;
default:
z++;
break;
}
*pbDoNotTrim = 0;
}
}
assert( 0 );
return -1;
}
static int cidxParseSQL(CidxCursor *pCsr, CidxIndex *pIdx, const char *zSql){
const char *z = zSql;
const char *z1;
int e;
int rc = SQLITE_OK;
int nParen = 1;
int bDoNotTrim = 0;
CidxColumn *pCol = pIdx->aCol;
e = cidxFindNext(z, &z, &bDoNotTrim);
if( e!=CIDX_PARSE_OPEN ) goto parse_error;
z1 = z+1;
z++;
while( nParen>0 ){
e = cidxFindNext(z, &z, &bDoNotTrim);
if( e==CIDX_PARSE_EOF ) goto parse_error;
if( (e==CIDX_PARSE_COMMA || e==CIDX_PARSE_CLOSE) && nParen==1 ){
const char *z2 = z;
if( pCol->zExpr ) goto parse_error;
if( bDoNotTrim==0 ){
while( cidx_isspace(z[-1]) ) z--;
if( !sqlite3_strnicmp(&z[-3], "asc", 3) && 0==cidx_isident(z[-4]) ){
z -= 3;
while( cidx_isspace(z[-1]) ) z--;
}else
if( !sqlite3_strnicmp(&z[-4], "desc", 4) && 0==cidx_isident(z[-5]) ){
z -= 4;
while( cidx_isspace(z[-1]) ) z--;
}
while( cidx_isspace(z1[0]) ) z1++;
}
pCol->zExpr = cidxMprintf(&rc, "%.*s", z-z1, z1);
pCol++;
z = z1 = z2+1;
}
if( e==CIDX_PARSE_OPEN ) nParen++;
if( e==CIDX_PARSE_CLOSE ) nParen--;
z++;
}
/* Search for a WHERE clause */
cidxFindNext(z, &z, 0);
if( 0==sqlite3_strnicmp(z, "where", 5) ){
pIdx->zWhere = cidxMprintf(&rc, "%s\n", &z[5]);
}else if( z[0]!='\0' ){
goto parse_error;
}
return rc;
parse_error:
cidxCursorError(pCsr, "Parse error in: %s", zSql);
return SQLITE_ERROR;
}
static int cidxLookupIndex(
CidxCursor *pCsr, /* Cursor object */
const char *zIdx, /* Name of index to look up */
CidxIndex **ppIdx, /* OUT: Description of columns */
char **pzTab /* OUT: Table name */
){
int rc = SQLITE_OK;
char *zTab = 0;
CidxIndex *pIdx = 0;
sqlite3_stmt *pFindTab = 0;
sqlite3_stmt *pInfo = 0;
/* Find the table for this index. */
pFindTab = cidxPrepare(&rc, pCsr,
"SELECT tbl_name, sql FROM sqlite_schema WHERE name=%Q AND type='index'",
zIdx
);
if( rc==SQLITE_OK && sqlite3_step(pFindTab)==SQLITE_ROW ){
const char *zSql = (const char*)sqlite3_column_text(pFindTab, 1);
zTab = cidxStrdup(&rc, (const char*)sqlite3_column_text(pFindTab, 0));
pInfo = cidxPrepare(&rc, pCsr, "PRAGMA index_xinfo(%Q)", zIdx);
if( rc==SQLITE_OK ){
int nAlloc = 0;
int iCol = 0;
while( sqlite3_step(pInfo)==SQLITE_ROW ){
const char *zName = (const char*)sqlite3_column_text(pInfo, 2);
const char *zColl = (const char*)sqlite3_column_text(pInfo, 4);
CidxColumn *p;
if( zName==0 ) zName = "rowid";
if( iCol==nAlloc ){
int nByte = sizeof(CidxIndex) + sizeof(CidxColumn)*(nAlloc+8);
pIdx = (CidxIndex*)sqlite3_realloc(pIdx, nByte);
nAlloc += 8;
}
p = &pIdx->aCol[iCol++];
p->bDesc = sqlite3_column_int(pInfo, 3);
p->bKey = sqlite3_column_int(pInfo, 5);
if( zSql==0 || p->bKey==0 ){
p->zExpr = cidxMprintf(&rc, "\"%w\" COLLATE %s",zName,zColl);
}else{
p->zExpr = 0;
}
pIdx->nCol = iCol;
pIdx->zWhere = 0;
}
cidxFinalize(&rc, pInfo);
}
if( rc==SQLITE_OK && zSql ){
rc = cidxParseSQL(pCsr, pIdx, zSql);
}
}
cidxFinalize(&rc, pFindTab);
if( rc==SQLITE_OK && zTab==0 ){
rc = SQLITE_ERROR;
}
if( rc!=SQLITE_OK ){
sqlite3_free(zTab);
cidxFreeIndex(pIdx);
}else{
*pzTab = zTab;
*ppIdx = pIdx;
}
return rc;
}
static int cidxDecodeAfter(
CidxCursor *pCsr,
int nCol,
const char *zAfterKey,
char ***pazAfter
){
char **azAfter;
int rc = SQLITE_OK;
int nAfterKey = (int)strlen(zAfterKey);
azAfter = cidxMalloc(&rc, sizeof(char*)*nCol + nAfterKey+1);
if( rc==SQLITE_OK ){
int i;
char *zCopy = (char*)&azAfter[nCol];
char *p = zCopy;
memcpy(zCopy, zAfterKey, nAfterKey+1);
for(i=0; i<nCol; i++){
while( *p==' ' ) p++;
/* Check NULL values */
if( *p=='N' ){
if( memcmp(p, "NULL", 4) ) goto parse_error;
p += 4;
}
/* Check strings and blob literals */
else if( *p=='X' || *p=='\'' ){
azAfter[i] = p;
if( *p=='X' ) p++;
if( *p!='\'' ) goto parse_error;
p++;
while( 1 ){
if( *p=='\0' ) goto parse_error;
if( *p=='\'' ){
p++;
if( *p!='\'' ) break;
}
p++;
}
}
/* Check numbers */
else{
azAfter[i] = p;
while( (*p>='0' && *p<='9')
|| *p=='.' || *p=='+' || *p=='-' || *p=='e' || *p=='E'
){
p++;
}
}
while( *p==' ' ) p++;
if( *p!=(i==(nCol-1) ? '\0' : ',') ){
goto parse_error;
}
*p++ = '\0';
}
}
*pazAfter = azAfter;
return rc;
parse_error:
sqlite3_free(azAfter);
*pazAfter = 0;
cidxCursorError(pCsr, "%s", "error parsing after value");
return SQLITE_ERROR;
}
static char *cidxWhere(
int *pRc, CidxColumn *aCol, char **azAfter, int iGt, int bLastIsNull
){
char *zRet = 0;
const char *zSep = "";
int i;
for(i=0; i<iGt; i++){
zRet = cidxMprintf(pRc, "%z%s(%s) IS %s", zRet,
zSep, aCol[i].zExpr, (azAfter[i] ? azAfter[i] : "NULL")
);
zSep = " AND ";
}
if( bLastIsNull ){
zRet = cidxMprintf(pRc, "%z%s(%s) IS NULL", zRet, zSep, aCol[iGt].zExpr);
}
else if( azAfter[iGt] ){
zRet = cidxMprintf(pRc, "%z%s(%s) %s %s", zRet,
zSep, aCol[iGt].zExpr, (aCol[iGt].bDesc ? "<" : ">"),
azAfter[iGt]
);
}else{
zRet = cidxMprintf(pRc, "%z%s(%s) IS NOT NULL", zRet, zSep,aCol[iGt].zExpr);
}
return zRet;
}
#define CIDX_CLIST_ALL 0
#define CIDX_CLIST_ORDERBY 1
#define CIDX_CLIST_CURRENT_KEY 2
#define CIDX_CLIST_SUBWHERE 3
#define CIDX_CLIST_SUBEXPR 4
/*
** This function returns various strings based on the contents of the
** CidxIndex structure and the eType parameter.
*/
static char *cidxColumnList(
int *pRc, /* IN/OUT: Error code */
const char *zIdx,
CidxIndex *pIdx, /* Indexed columns */
int eType /* True to include ASC/DESC */
){
char *zRet = 0;
if( *pRc==SQLITE_OK ){
const char *aDir[2] = {"", " DESC"};
int i;
const char *zSep = "";
for(i=0; i<pIdx->nCol; i++){
CidxColumn *p = &pIdx->aCol[i];
assert( pIdx->aCol[i].bDesc==0 || pIdx->aCol[i].bDesc==1 );
switch( eType ){
case CIDX_CLIST_ORDERBY:
zRet = cidxMprintf(pRc, "%z%s%d%s", zRet, zSep, i+1, aDir[p->bDesc]);
zSep = ",";
break;
case CIDX_CLIST_CURRENT_KEY:
zRet = cidxMprintf(pRc, "%z%squote(i%d)", zRet, zSep, i);
zSep = "||','||";
break;
case CIDX_CLIST_SUBWHERE:
if( p->bKey==0 ){
zRet = cidxMprintf(pRc, "%z%s%s IS i.i%d", zRet,
zSep, p->zExpr, i
);
zSep = " AND ";
}
break;
case CIDX_CLIST_SUBEXPR:
if( p->bKey==1 ){
zRet = cidxMprintf(pRc, "%z%s%s IS i.i%d", zRet,
zSep, p->zExpr, i
);
zSep = " AND ";
}
break;
default:
assert( eType==CIDX_CLIST_ALL );
zRet = cidxMprintf(pRc, "%z%s(%s) AS i%d", zRet, zSep, p->zExpr, i);
zSep = ", ";
break;
}
}
}
return zRet;
}
/*
** Generate SQL (in memory obtained from sqlite3_malloc()) that will
** continue the index scan for zIdxName starting after zAfterKey.
*/
int cidxGenerateScanSql(
CidxCursor *pCsr, /* The cursor which needs the new statement */
const char *zIdxName, /* index to be scanned */
const char *zAfterKey, /* start after this key, if not NULL */
char **pzSqlOut /* OUT: Write the generated SQL here */
){
int rc;
char *zTab = 0;
char *zCurrentKey = 0;
char *zOrderBy = 0;
char *zSubWhere = 0;
char *zSubExpr = 0;
char *zSrcList = 0;
char **azAfter = 0;
CidxIndex *pIdx = 0;
*pzSqlOut = 0;
rc = cidxLookupIndex(pCsr, zIdxName, &pIdx, &zTab);
zOrderBy = cidxColumnList(&rc, zIdxName, pIdx, CIDX_CLIST_ORDERBY);
zCurrentKey = cidxColumnList(&rc, zIdxName, pIdx, CIDX_CLIST_CURRENT_KEY);
zSubWhere = cidxColumnList(&rc, zIdxName, pIdx, CIDX_CLIST_SUBWHERE);
zSubExpr = cidxColumnList(&rc, zIdxName, pIdx, CIDX_CLIST_SUBEXPR);
zSrcList = cidxColumnList(&rc, zIdxName, pIdx, CIDX_CLIST_ALL);
if( rc==SQLITE_OK && zAfterKey ){
rc = cidxDecodeAfter(pCsr, pIdx->nCol, zAfterKey, &azAfter);
}
if( rc==SQLITE_OK ){
if( zAfterKey==0 ){
*pzSqlOut = cidxMprintf(&rc,
"SELECT (SELECT %s FROM %Q AS t WHERE %s), %s "
"FROM (SELECT %s FROM %Q INDEXED BY %Q %s%sORDER BY %s) AS i",
zSubExpr, zTab, zSubWhere, zCurrentKey,
zSrcList, zTab, zIdxName,
(pIdx->zWhere ? "WHERE " : ""), (pIdx->zWhere ? pIdx->zWhere : ""),
zOrderBy
);
}else{
const char *zSep = "";
char *zSql;
int i;
zSql = cidxMprintf(&rc,
"SELECT (SELECT %s FROM %Q WHERE %s), %s FROM (",
zSubExpr, zTab, zSubWhere, zCurrentKey
);
for(i=pIdx->nCol-1; i>=0; i--){
int j;
if( pIdx->aCol[i].bDesc && azAfter[i]==0 ) continue;
for(j=0; j<2; j++){
char *zWhere = cidxWhere(&rc, pIdx->aCol, azAfter, i, j);
zSql = cidxMprintf(&rc, "%z"
"%sSELECT * FROM ("
"SELECT %s FROM %Q INDEXED BY %Q WHERE %s%s%z ORDER BY %s"
")",
zSql, zSep, zSrcList, zTab, zIdxName,
pIdx->zWhere ? pIdx->zWhere : "",
pIdx->zWhere ? " AND " : "",
zWhere, zOrderBy
);
zSep = " UNION ALL ";
if( pIdx->aCol[i].bDesc==0 ) break;
}
}
*pzSqlOut = cidxMprintf(&rc, "%z) AS i", zSql);
}
}
sqlite3_free(zTab);
sqlite3_free(zCurrentKey);
sqlite3_free(zOrderBy);
sqlite3_free(zSubWhere);
sqlite3_free(zSubExpr);
sqlite3_free(zSrcList);
cidxFreeIndex(pIdx);
sqlite3_free(azAfter);
return rc;
}
/*
** Position a cursor back to the beginning.
*/
static int cidxFilter(
sqlite3_vtab_cursor *pCursor,
int idxNum, const char *idxStr,
int argc, sqlite3_value **argv
){
int rc = SQLITE_OK;
CidxCursor *pCsr = (CidxCursor*)pCursor;
const char *zIdxName = 0;
const char *zAfterKey = 0;
sqlite3_free(pCsr->zIdxName);
pCsr->zIdxName = 0;
sqlite3_free(pCsr->zAfterKey);
pCsr->zAfterKey = 0;
sqlite3_finalize(pCsr->pStmt);
pCsr->pStmt = 0;
if( argc>0 ){
zIdxName = (const char*)sqlite3_value_text(argv[0]);
if( argc>1 ){
zAfterKey = (const char*)sqlite3_value_text(argv[1]);
}
}
if( zIdxName ){
char *zSql = 0;
pCsr->zIdxName = sqlite3_mprintf("%s", zIdxName);
pCsr->zAfterKey = zAfterKey ? sqlite3_mprintf("%s", zAfterKey) : 0;
rc = cidxGenerateScanSql(pCsr, zIdxName, zAfterKey, &zSql);
if( zSql ){
pCsr->pStmt = cidxPrepare(&rc, pCsr, "%z", zSql);
}
}
if( pCsr->pStmt ){
assert( rc==SQLITE_OK );
rc = cidxNext(pCursor);
}
pCsr->iRowid = 1;
return rc;
}
/*
** Return a column value.
*/
static int cidxColumn(
sqlite3_vtab_cursor *pCursor,
sqlite3_context *ctx,
int iCol
){
CidxCursor *pCsr = (CidxCursor*)pCursor;
assert( iCol>=IIC_ERRMSG && iCol<=IIC_SCANNER_SQL );
switch( iCol ){
case IIC_ERRMSG: {
const char *zVal = 0;
if( sqlite3_column_type(pCsr->pStmt, 0)==SQLITE_INTEGER ){
if( sqlite3_column_int(pCsr->pStmt, 0)==0 ){
zVal = "row data mismatch";
}
}else{
zVal = "row missing";
}
sqlite3_result_text(ctx, zVal, -1, SQLITE_STATIC);
break;
}
case IIC_CURRENT_KEY: {
sqlite3_result_value(ctx, sqlite3_column_value(pCsr->pStmt, 1));
break;
}
case IIC_INDEX_NAME: {
sqlite3_result_text(ctx, pCsr->zIdxName, -1, SQLITE_TRANSIENT);
break;
}
case IIC_AFTER_KEY: {
sqlite3_result_text(ctx, pCsr->zAfterKey, -1, SQLITE_TRANSIENT);
break;
}
case IIC_SCANNER_SQL: {
char *zSql = 0;
cidxGenerateScanSql(pCsr, pCsr->zIdxName, pCsr->zAfterKey, &zSql);
sqlite3_result_text(ctx, zSql, -1, sqlite3_free);
break;
}
}
return SQLITE_OK;
}
/* Return the ROWID for the sqlite_btreeinfo table */
static int cidxRowid(sqlite3_vtab_cursor *pCursor, sqlite_int64 *pRowid){
CidxCursor *pCsr = (CidxCursor*)pCursor;
*pRowid = pCsr->iRowid;
return SQLITE_OK;
}
/*
** Register the virtual table modules with the database handle passed
** as the only argument.
*/
static int ciInit(sqlite3 *db){
static sqlite3_module cidx_module = {
0, /* iVersion */
0, /* xCreate */
cidxConnect, /* xConnect */
cidxBestIndex, /* xBestIndex */
cidxDisconnect, /* xDisconnect */
0, /* xDestroy */
cidxOpen, /* xOpen - open a cursor */
cidxClose, /* xClose - close a cursor */
cidxFilter, /* xFilter - configure scan constraints */
cidxNext, /* xNext - advance a cursor */
cidxEof, /* xEof - check for end of scan */
cidxColumn, /* xColumn - read data */
cidxRowid, /* xRowid - read data */
0, /* xUpdate */
0, /* xBegin */
0, /* xSync */
0, /* xCommit */
0, /* xRollback */
0, /* xFindMethod */
0, /* xRename */
0, /* xSavepoint */
0, /* xRelease */
0, /* xRollbackTo */
0, /* xShadowName */
0 /* xIntegrity */
};
return sqlite3_create_module(db, "incremental_index_check", &cidx_module, 0);
}
/*
** Extension load function.
*/
#ifdef _WIN32
__declspec(dllexport)
#endif
int sqlite3_checkindex_init(
sqlite3 *db,
char **pzErrMsg,
const sqlite3_api_routines *pApi
){
SQLITE_EXTENSION_INIT2(pApi);
return ciInit(db);
}
-85
View File
@@ -1,85 +0,0 @@
/*
** Read an SQLite database file and analyze its space utilization. Generate
** text on standard output.
*/
#define TCLSH_INIT_PROC sqlite3_checker_init_proc
#define SQLITE_ENABLE_DBPAGE_VTAB 1
#undef SQLITE_THREADSAFE
#define SQLITE_THREADSAFE 0
#undef SQLITE_ENABLE_COLUMN_METADATA
#define SQLITE_OMIT_DECLTYPE 1
#define SQLITE_OMIT_DEPRECATED 1
#define SQLITE_OMIT_PROGRESS_CALLBACK 1
#define SQLITE_OMIT_SHARED_CACHE 1
#define SQLITE_DEFAULT_MEMSTATUS 0
#define SQLITE_MAX_EXPR_DEPTH 0
INCLUDE sqlite3.c
INCLUDE $ROOT/src/tclsqlite.c
INCLUDE $ROOT/ext/misc/btreeinfo.c
INCLUDE $ROOT/ext/repair/checkindex.c
INCLUDE $ROOT/ext/repair/checkfreelist.c
/*
** Decode a pointer to an sqlite3 object.
*/
int getDbPointer(Tcl_Interp *interp, const char *zA, sqlite3 **ppDb){
struct SqliteDb *p;
Tcl_CmdInfo cmdInfo;
if( Tcl_GetCommandInfo(interp, zA, &cmdInfo) ){
p = (struct SqliteDb*)cmdInfo.objClientData;
*ppDb = p->db;
return TCL_OK;
}else{
*ppDb = 0;
return TCL_ERROR;
}
return TCL_OK;
}
/*
** sqlite3_imposter db main rootpage {CREATE TABLE...} ;# setup an imposter
** sqlite3_imposter db main ;# rm all imposters
*/
static int sqlite3_imposter(
void *clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[]
){
sqlite3 *db;
const char *zSchema;
int iRoot;
const char *zSql;
if( objc!=3 && objc!=5 ){
Tcl_WrongNumArgs(interp, 1, objv, "DB SCHEMA [ROOTPAGE SQL]");
return TCL_ERROR;
}
if( getDbPointer(interp, Tcl_GetString(objv[1]), &db) ) return TCL_ERROR;
zSchema = Tcl_GetString(objv[2]);
if( objc==3 ){
sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, zSchema, 0, 1);
}else{
if( Tcl_GetIntFromObj(interp, objv[3], &iRoot) ) return TCL_ERROR;
zSql = Tcl_GetString(objv[4]);
sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, zSchema, 1, iRoot);
sqlite3_exec(db, zSql, 0, 0, 0);
sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, zSchema, 0, 0);
}
return TCL_OK;
}
#include <stdio.h>
const char *sqlite3_checker_init_proc(Tcl_Interp *interp){
Tcl_CreateObjCommand(interp, "sqlite3_imposter",
(Tcl_ObjCmdProc*)sqlite3_imposter, 0, 0);
sqlite3_auto_extension((void(*)(void))sqlite3_btreeinfo_init);
sqlite3_auto_extension((void(*)(void))sqlite3_checkindex_init);
sqlite3_auto_extension((void(*)(void))sqlite3_checkfreelist_init);
return
BEGIN_STRING
INCLUDE $ROOT/ext/repair/sqlite3_checker.tcl
END_STRING
;
}
-264
View File
@@ -1,264 +0,0 @@
# This TCL script is the main driver script for the sqlite3_checker utility
# program.
#
# Special case:
#
# sqlite3_checker --test FILENAME ARGS
#
# uses FILENAME in place of this script.
#
if {[lindex $argv 0]=="--test" && [llength $argv]>1} {
set ::argv0 [lindex $argv 1]
set argv [lrange $argv 2 end]
source $argv0
exit 0
}
# Emulate a TCL shell
#
proc tclsh {} {
set line {}
while {![eof stdin]} {
if {$line!=""} {
puts -nonewline "> "
} else {
puts -nonewline "% "
}
flush stdout
append line [gets stdin]
if {[info complete $line]} {
if {[catch {uplevel #0 $line} result]} {
puts stderr "Error: $result"
} elseif {$result!=""} {
puts $result
}
set line {}
} else {
append line \n
}
}
}
# Do an incremental integrity check of a single index
#
proc check_index {idxname batchsize bTrace} {
set i 0
set more 1
set nerr 0
set pct 00.0
set max [db one {SELECT nEntry FROM sqlite_btreeinfo('main')
WHERE name=$idxname}]
puts -nonewline "$idxname: $i of $max rows ($pct%)\r"
flush stdout
if {$bTrace} {
set sql {SELECT errmsg, current_key AS key,
CASE WHEN rowid=1 THEN scanner_sql END AS traceOut
FROM incremental_index_check($idxname)
WHERE after_key=$key
LIMIT $batchsize}
} else {
set sql {SELECT errmsg, current_key AS key, NULL AS traceOut
FROM incremental_index_check($idxname)
WHERE after_key=$key
LIMIT $batchsize}
}
while {$more} {
set more 0
db eval $sql {
set more 1
if {$errmsg!=""} {
incr nerr
puts "$idxname: key($key): $errmsg"
} elseif {$traceOut!=""} {
puts "$idxname: $traceOut"
}
incr i
}
set x [format {%.1f} [expr {($i*100.0)/$max}]]
if {$x!=$pct} {
puts -nonewline "$idxname: $i of $max rows ($pct%)\r"
flush stdout
set pct $x
}
}
puts "$idxname: $nerr errors out of $i entries"
}
# Print a usage message on standard error, then quit.
#
proc usage {} {
set argv0 [file rootname [file tail [info nameofexecutable]]]
puts stderr "Usage: $argv0 OPTIONS database-filename"
puts stderr {
Do sanity checking on a live SQLite3 database file specified by the
"database-filename" argument.
Options:
--batchsize N Number of rows to check per transaction
--freelist Perform a freelist check
--index NAME Run a check of the index NAME
--summary Print summary information about the database
--table NAME Run a check of all indexes for table NAME
--tclsh Run the built-in TCL interpreter (for debugging)
--trace (Debugging only:) Output trace information on the scan
--version Show the version number of SQLite
}
exit 1
}
set file_to_analyze {}
append argv {}
set bFreelistCheck 0
set bSummary 0
set zIndex {}
set zTable {}
set batchsize 1000
set bAll 1
set bTrace 0
set argc [llength $argv]
for {set i 0} {$i<$argc} {incr i} {
set arg [lindex $argv $i]
if {[regexp {^-+tclsh$} $arg]} {
tclsh
exit 0
}
if {[regexp {^-+version$} $arg]} {
sqlite3 mem :memory:
puts [mem one {SELECT sqlite_version()||' '||sqlite_source_id()}]
mem close
exit 0
}
if {[regexp {^-+freelist$} $arg]} {
set bFreelistCheck 1
set bAll 0
continue
}
if {[regexp {^-+summary$} $arg]} {
set bSummary 1
set bAll 0
continue
}
if {[regexp {^-+trace$} $arg]} {
set bTrace 1
continue
}
if {[regexp {^-+batchsize$} $arg]} {
incr i
if {$i>=$argc} {
puts stderr "missing argument on $arg"
exit 1
}
set batchsize [lindex $argv $i]
continue
}
if {[regexp {^-+index$} $arg]} {
incr i
if {$i>=$argc} {
puts stderr "missing argument on $arg"
exit 1
}
set zIndex [lindex $argv $i]
set bAll 0
continue
}
if {[regexp {^-+table$} $arg]} {
incr i
if {$i>=$argc} {
puts stderr "missing argument on $arg"
exit 1
}
set zTable [lindex $argv $i]
set bAll 0
continue
}
if {[regexp {^-} $arg]} {
puts stderr "Unknown option: $arg"
usage
}
if {$file_to_analyze!=""} {
usage
} else {
set file_to_analyze $arg
}
}
if {$file_to_analyze==""} usage
# If a TCL script is specified on the command-line, then run that
# script.
#
if {[file extension $file_to_analyze]==".tcl"} {
source $file_to_analyze
exit 0
}
set root_filename $file_to_analyze
regexp {^file:(//)?([^?]*)} $file_to_analyze all x1 root_filename
if {![file exists $root_filename]} {
puts stderr "No such file: $root_filename"
exit 1
}
if {![file readable $root_filename]} {
puts stderr "File is not readable: $root_filename"
exit 1
}
if {[catch {sqlite3 db $file_to_analyze} res]} {
puts stderr "Cannot open datababase $root_filename: $res"
exit 1
}
if {$bFreelistCheck || $bAll} {
puts -nonewline "freelist-check: "
flush stdout
db eval BEGIN
puts [db one {SELECT checkfreelist('main')}]
db eval END
}
if {$bSummary} {
set scale 0
set pgsz [db one {PRAGMA page_size}]
db eval {SELECT nPage*$pgsz AS sz, name, tbl_name
FROM sqlite_btreeinfo
WHERE type='index'
ORDER BY 1 DESC, name} {
if {$scale==0} {
if {$sz>10000000} {
set scale 1000000.0
set unit MB
} else {
set scale 1000.0
set unit KB
}
}
puts [format {%7.1f %s index %s of table %s} \
[expr {$sz/$scale}] $unit $name $tbl_name]
}
}
if {$zIndex!=""} {
check_index $zIndex $batchsize $bTrace
}
if {$zTable!=""} {
foreach idx [db eval {SELECT name FROM sqlite_master
WHERE type='index' AND rootpage>0
AND tbl_name=$zTable}] {
check_index $idx $batchsize $bTrace
}
}
if {$bAll} {
set allidx [db eval {SELECT name FROM sqlite_btreeinfo('main')
WHERE type='index' AND rootpage>0
ORDER BY nEntry}]
foreach idx $allidx {
check_index $idx $batchsize $bTrace
}
}
-13
View File
@@ -1,13 +0,0 @@
To run these tests, first build sqlite3_checker:
> make sqlite3_checker
Then run the "test.tcl" script using:
> ./sqlite3_checker --test $path/test.tcl
Optionally add the full pathnames of individual *.test modules
-92
View File
@@ -1,92 +0,0 @@
# 2017-10-11
set testprefix checkfreelist
do_execsql_test 1.0 {
PRAGMA page_size=1024;
CREATE TABLE t1(a, b);
}
do_execsql_test 1.2 { SELECT checkfreelist('main') } {ok}
do_execsql_test 1.3 {
WITH s(i) AS (
SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<10000
)
INSERT INTO t1 SELECT randomblob(400), randomblob(400) FROM s;
DELETE FROM t1 WHERE rowid%3;
PRAGMA freelist_count;
} {6726}
do_execsql_test 1.4 { SELECT checkfreelist('main') } {ok}
do_execsql_test 1.5 {
WITH freelist_trunk(i, d, n) AS (
SELECT 1, NULL, sqlite_readint32(data, 32) FROM sqlite_dbpage WHERE pgno=1
UNION ALL
SELECT n, data, sqlite_readint32(data)
FROM freelist_trunk, sqlite_dbpage WHERE pgno=n
)
SELECT i FROM freelist_trunk WHERE i!=1;
} {
10009 9715 9343 8969 8595 8222 7847 7474 7102 6727 6354 5982 5608 5234
4860 4487 4112 3740 3367 2992 2619 2247 1872 1499 1125 752 377 5
}
do_execsql_test 1.6 { SELECT checkfreelist('main') } {ok}
proc set_int {blob idx newval} {
binary scan $blob I* ints
lset ints $idx $newval
binary format I* $ints
}
db func set_int set_int
proc get_int {blob idx} {
binary scan $blob I* ints
lindex $ints $idx
}
db func get_int get_int
do_execsql_test 1.7 {
BEGIN;
UPDATE sqlite_dbpage
SET data = set_int(data, 1, get_int(data, 1)-1)
WHERE pgno=4860;
SELECT checkfreelist('main');
ROLLBACK;
} {{free-list count mismatch: actual=6725 header=6726}}
do_execsql_test 1.8 {
BEGIN;
UPDATE sqlite_dbpage
SET data = set_int(data, 5, (SELECT * FROM pragma_page_count)+1)
WHERE pgno=4860;
SELECT checkfreelist('main');
ROLLBACK;
} {{leaf page 10092 is out of range (child 3 of trunk page 4860)}}
do_execsql_test 1.9 {
BEGIN;
UPDATE sqlite_dbpage
SET data = set_int(data, 5, 0)
WHERE pgno=4860;
SELECT checkfreelist('main');
ROLLBACK;
} {{leaf page 0 is out of range (child 3 of trunk page 4860)}}
do_execsql_test 1.10 {
BEGIN;
UPDATE sqlite_dbpage
SET data = set_int(data, get_int(data, 1)+1, 0)
WHERE pgno=5;
SELECT checkfreelist('main');
ROLLBACK;
} {{leaf page 0 is out of range (child 247 of trunk page 5)}}
do_execsql_test 1.11 {
BEGIN;
UPDATE sqlite_dbpage
SET data = set_int(data, 1, 249)
WHERE pgno=5;
SELECT checkfreelist('main');
ROLLBACK;
} {{leaf count out of range (249) on trunk page 5}}
-349
View File
@@ -1,349 +0,0 @@
# 2017-10-11
#
set testprefix checkindex
do_execsql_test 1.0 {
CREATE TABLE t1(a, b);
CREATE INDEX i1 ON t1(a);
INSERT INTO t1 VALUES('one', 2);
INSERT INTO t1 VALUES('two', 4);
INSERT INTO t1 VALUES('three', 6);
INSERT INTO t1 VALUES('four', 8);
INSERT INTO t1 VALUES('five', 10);
CREATE INDEX i2 ON t1(a DESC);
} {}
proc incr_index_check {idx nStep} {
set Q {
SELECT errmsg, current_key FROM incremental_index_check($idx, $after)
LIMIT $nStep
}
set res [list]
while {1} {
unset -nocomplain current_key
set res1 [db eval $Q]
if {[llength $res1]==0} break
set res [concat $res $res1]
set after [lindex $res end]
}
return $res
}
proc do_index_check_test {tn idx res} {
uplevel [list do_execsql_test $tn.1 "
SELECT errmsg, current_key FROM incremental_index_check('$idx');
" $res]
uplevel [list do_test $tn.2 "incr_index_check $idx 1" [list {*}$res]]
uplevel [list do_test $tn.3 "incr_index_check $idx 2" [list {*}$res]]
uplevel [list do_test $tn.4 "incr_index_check $idx 5" [list {*}$res]]
}
do_execsql_test 1.2.1 {
SELECT rowid, errmsg IS NULL, current_key FROM incremental_index_check('i1');
} {
1 1 'five',5
2 1 'four',4
3 1 'one',1
4 1 'three',3
5 1 'two',2
}
do_execsql_test 1.2.2 {
SELECT errmsg IS NULL, current_key, index_name, after_key, scanner_sql
FROM incremental_index_check('i1') LIMIT 1;
} {
1
'five',5
i1
{}
{SELECT (SELECT a IS i.i0 FROM 't1' AS t WHERE "rowid" COLLATE BINARY IS i.i1), quote(i0)||','||quote(i1) FROM (SELECT (a) AS i0, ("rowid" COLLATE BINARY) AS i1 FROM 't1' INDEXED BY 'i1' ORDER BY 1,2) AS i}
}
do_index_check_test 1.3 i1 {
{} 'five',5
{} 'four',4
{} 'one',1
{} 'three',3
{} 'two',2
}
do_index_check_test 1.4 i2 {
{} 'two',2
{} 'three',3
{} 'one',1
{} 'four',4
{} 'five',5
}
do_test 1.5 {
set tblroot [db one { SELECT rootpage FROM sqlite_master WHERE name='t1' }]
sqlite3_imposter db main $tblroot {CREATE TABLE xt1(a,b)}
db eval {
UPDATE xt1 SET a='six' WHERE rowid=3;
DELETE FROM xt1 WHERE rowid = 5;
}
sqlite3_imposter db main
} {}
do_index_check_test 1.6 i1 {
{row missing} 'five',5
{} 'four',4
{} 'one',1
{row data mismatch} 'three',3
{} 'two',2
}
do_index_check_test 1.7 i2 {
{} 'two',2
{row data mismatch} 'three',3
{} 'one',1
{} 'four',4
{row missing} 'five',5
}
#--------------------------------------------------------------------------
do_execsql_test 2.0 {
CREATE TABLE t2(a INTEGER PRIMARY KEY, b, c, d);
INSERT INTO t2 VALUES(1, NULL, 1, 1);
INSERT INTO t2 VALUES(2, 1, NULL, 1);
INSERT INTO t2 VALUES(3, 1, 1, NULL);
INSERT INTO t2 VALUES(4, 2, 2, 1);
INSERT INTO t2 VALUES(5, 2, 2, 2);
INSERT INTO t2 VALUES(6, 2, 2, 3);
INSERT INTO t2 VALUES(7, 2, 2, 1);
INSERT INTO t2 VALUES(8, 2, 2, 2);
INSERT INTO t2 VALUES(9, 2, 2, 3);
CREATE INDEX i3 ON t2(b, c, d);
CREATE INDEX i4 ON t2(b DESC, c DESC, d DESC);
CREATE INDEX i5 ON t2(d, c DESC, b);
} {}
do_index_check_test 2.1 i3 {
{} NULL,1,1,1
{} 1,NULL,1,2
{} 1,1,NULL,3
{} 2,2,1,4
{} 2,2,1,7
{} 2,2,2,5
{} 2,2,2,8
{} 2,2,3,6
{} 2,2,3,9
}
do_index_check_test 2.2 i4 {
{} 2,2,3,6
{} 2,2,3,9
{} 2,2,2,5
{} 2,2,2,8
{} 2,2,1,4
{} 2,2,1,7
{} 1,1,NULL,3
{} 1,NULL,1,2
{} NULL,1,1,1
}
do_index_check_test 2.3 i5 {
{} NULL,1,1,3
{} 1,2,2,4
{} 1,2,2,7
{} 1,1,NULL,1
{} 1,NULL,1,2
{} 2,2,2,5
{} 2,2,2,8
{} 3,2,2,6
{} 3,2,2,9
}
#--------------------------------------------------------------------------
do_execsql_test 3.0 {
CREATE TABLE t3(w, x, y, z PRIMARY KEY) WITHOUT ROWID;
CREATE INDEX t3wxy ON t3(w, x, y);
CREATE INDEX t3wxy2 ON t3(w DESC, x DESC, y DESC);
INSERT INTO t3 VALUES(NULL, NULL, NULL, 1);
INSERT INTO t3 VALUES(NULL, NULL, NULL, 2);
INSERT INTO t3 VALUES(NULL, NULL, NULL, 3);
INSERT INTO t3 VALUES('a', NULL, NULL, 4);
INSERT INTO t3 VALUES('a', NULL, NULL, 5);
INSERT INTO t3 VALUES('a', NULL, NULL, 6);
INSERT INTO t3 VALUES('a', 'b', NULL, 7);
INSERT INTO t3 VALUES('a', 'b', NULL, 8);
INSERT INTO t3 VALUES('a', 'b', NULL, 9);
} {}
do_index_check_test 3.1 t3wxy {
{} NULL,NULL,NULL,1 {} NULL,NULL,NULL,2 {} NULL,NULL,NULL,3
{} 'a',NULL,NULL,4 {} 'a',NULL,NULL,5 {} 'a',NULL,NULL,6
{} 'a','b',NULL,7 {} 'a','b',NULL,8 {} 'a','b',NULL,9
}
do_index_check_test 3.2 t3wxy2 {
{} 'a','b',NULL,7 {} 'a','b',NULL,8 {} 'a','b',NULL,9
{} 'a',NULL,NULL,4 {} 'a',NULL,NULL,5 {} 'a',NULL,NULL,6
{} NULL,NULL,NULL,1 {} NULL,NULL,NULL,2 {} NULL,NULL,NULL,3
}
#--------------------------------------------------------------------------
# Test with an index that uses non-default collation sequences.
#
do_execsql_test 4.0 {
CREATE TABLE t4(a INTEGER PRIMARY KEY, c1 TEXT, c2 TEXT);
INSERT INTO t4 VALUES(1, 'aaa', 'bbb');
INSERT INTO t4 VALUES(2, 'AAA', 'CCC');
INSERT INTO t4 VALUES(3, 'aab', 'ddd');
INSERT INTO t4 VALUES(4, 'AAB', 'EEE');
CREATE INDEX t4cc ON t4(c1 COLLATE nocase, c2 COLLATE nocase);
}
do_index_check_test 4.1 t4cc {
{} 'aaa','bbb',1
{} 'AAA','CCC',2
{} 'aab','ddd',3
{} 'AAB','EEE',4
}
do_test 4.2 {
set tblroot [db one { SELECT rootpage FROM sqlite_master WHERE name='t4' }]
sqlite3_imposter db main $tblroot \
{CREATE TABLE xt4(a INTEGER PRIMARY KEY, c1 TEXT, c2 TEXT)}
db eval {
UPDATE xt4 SET c1='hello' WHERE rowid=2;
DELETE FROM xt4 WHERE rowid = 3;
}
sqlite3_imposter db main
} {}
do_index_check_test 4.3 t4cc {
{} 'aaa','bbb',1
{row data mismatch} 'AAA','CCC',2
{row missing} 'aab','ddd',3
{} 'AAB','EEE',4
}
#--------------------------------------------------------------------------
# Test an index on an expression.
#
do_execsql_test 5.0 {
CREATE TABLE t5(x INTEGER PRIMARY KEY, y TEXT, UNIQUE(y));
INSERT INTO t5 VALUES(1, '{"x":1, "y":1}');
INSERT INTO t5 VALUES(2, '{"x":2, "y":2}');
INSERT INTO t5 VALUES(3, '{"x":3, "y":3}');
INSERT INTO t5 VALUES(4, '{"w":4, "z":4}');
INSERT INTO t5 VALUES(5, '{"x":5, "y":5}');
CREATE INDEX t5x ON t5( json_extract(y, '$.x') );
CREATE INDEX t5y ON t5( json_extract(y, '$.y') DESC );
}
do_index_check_test 5.1.1 t5x {
{} NULL,4 {} 1,1 {} 2,2 {} 3,3 {} 5,5
}
do_index_check_test 5.1.2 t5y {
{} 5,5 {} 3,3 {} 2,2 {} 1,1 {} NULL,4
}
do_index_check_test 5.1.3 sqlite_autoindex_t5_1 {
{} {'{"w":4, "z":4}',4}
{} {'{"x":1, "y":1}',1}
{} {'{"x":2, "y":2}',2}
{} {'{"x":3, "y":3}',3}
{} {'{"x":5, "y":5}',5}
}
do_test 5.2 {
set tblroot [db one { SELECT rootpage FROM sqlite_master WHERE name='t5' }]
sqlite3_imposter db main $tblroot \
{CREATE TABLE xt5(a INTEGER PRIMARY KEY, c1 TEXT);}
db eval {
UPDATE xt5 SET c1='{"x":22, "y":11}' WHERE rowid=1;
DELETE FROM xt5 WHERE rowid = 4;
}
sqlite3_imposter db main
} {}
do_index_check_test 5.3.1 t5x {
{row missing} NULL,4
{row data mismatch} 1,1
{} 2,2
{} 3,3
{} 5,5
}
do_index_check_test 5.3.2 sqlite_autoindex_t5_1 {
{row missing} {'{"w":4, "z":4}',4}
{row data mismatch} {'{"x":1, "y":1}',1}
{} {'{"x":2, "y":2}',2}
{} {'{"x":3, "y":3}',3}
{} {'{"x":5, "y":5}',5}
}
#-------------------------------------------------------------------------
#
do_execsql_test 6.0 {
CREATE TABLE t6(x INTEGER PRIMARY KEY, y, z);
CREATE INDEX t6x1 ON t6(y, /* one,two,three */ z);
CREATE INDEX t6x2 ON t6(z, -- hello,world,
y);
CREATE INDEX t6x3 ON t6(z -- hello,world
, y);
INSERT INTO t6 VALUES(1, 2, 3);
INSERT INTO t6 VALUES(4, 5, 6);
}
do_index_check_test 6.1 t6x1 {
{} 2,3,1
{} 5,6,4
}
do_index_check_test 6.2 t6x2 {
{} 3,2,1
{} 6,5,4
}
do_index_check_test 6.2 t6x3 {
{} 3,2,1
{} 6,5,4
}
#-------------------------------------------------------------------------
#
do_execsql_test 7.0 {
CREATE TABLE t7(x INTEGER PRIMARY KEY, y, z);
INSERT INTO t7 VALUES(1, 1, 1);
INSERT INTO t7 VALUES(2, 2, 0);
INSERT INTO t7 VALUES(3, 3, 1);
INSERT INTO t7 VALUES(4, 4, 0);
CREATE INDEX t7i1 ON t7(y) WHERE z=1;
CREATE INDEX t7i2 ON t7(y) /* hello,world */ WHERE z=1;
CREATE INDEX t7i3 ON t7(y) WHERE -- yep
z=1;
CREATE INDEX t7i4 ON t7(y) WHERE z=1 -- yep;
}
do_index_check_test 7.1 t7i1 {
{} 1,1 {} 3,3
}
do_index_check_test 7.2 t7i2 {
{} 1,1 {} 3,3
}
do_index_check_test 7.3 t7i3 {
{} 1,1 {} 3,3
}
do_index_check_test 7.4 t7i4 {
{} 1,1 {} 3,3
}
-67
View File
@@ -1,67 +0,0 @@
# Run this script using
#
# sqlite3_checker --test $thisscript $testscripts
#
# The $testscripts argument is optional. If omitted, all *.test files
# in the same directory as $thisscript are run.
#
set NTEST 0
set NERR 0
# Invoke the do_test procedure to run a single test
#
# The $expected parameter is the expected result. The result is the return
# value from the last TCL command in $cmd.
#
# Normally, $expected must match exactly. But if $expected is of the form
# "/regexp/" then regular expression matching is used. If $expected is
# "~/regexp/" then the regular expression must NOT match. If $expected is
# of the form "#/value-list/" then each term in value-list must be numeric
# and must approximately match the corresponding numeric term in $result.
# Values must match within 10%. Or if the $expected term is A..B then the
# $result term must be in between A and B.
#
proc do_test {name cmd expected} {
if {[info exists ::testprefix]} {
set name "$::testprefix$name"
}
incr ::NTEST
puts -nonewline $name...
flush stdout
if {[catch {uplevel #0 "$cmd;\n"} result]} {
puts -nonewline $name...
puts "\nError: $result"
incr ::NERR
} else {
set ok [expr {[string compare $result $expected]==0}]
if {!$ok} {
puts "\n! $name expected: \[$expected\]\n! $name got: \[$result\]"
incr ::NERR
} else {
puts " Ok"
}
}
flush stdout
}
#
# do_execsql_test TESTNAME SQL RES
#
proc do_execsql_test {testname sql {result {}}} {
uplevel [list do_test $testname [list db eval $sql] [list {*}$result]]
}
if {[llength $argv]==0} {
set dir [file dirname $argv0]
set argv [glob -nocomplain $dir/*.test]
}
foreach testfile $argv {
file delete -force test.db
sqlite3 db test.db
source $testfile
catch {db close}
}
puts "$NERR errors out of $NTEST tests"
+1 -1
View File
@@ -200,7 +200,7 @@ static int geopolyParseNumber(GeoParse *p, GeoCoord *pVal){
/* The sqlite3AtoF() routine is much much faster than atof(), if it
** is available */
double r;
(void)sqlite3AtoF((const char*)p->z, &r, j, SQLITE_UTF8);
(void)sqlite3AtoF((const char*)p->z, &r);
*pVal = r;
#else
*pVal = (GeoCoord)atof((const char*)p->z);
+14 -13
View File
@@ -1037,7 +1037,17 @@ static void rtreeRelease(Rtree *pRtree){
pRtree->inWrTrans = 0;
assert( pRtree->nCursor==0 );
nodeBlobReset(pRtree);
assert( pRtree->nNodeRef==0 || pRtree->bCorrupt );
if( pRtree->nNodeRef ){
int i;
assert( pRtree->bCorrupt );
for(i=0; i<HASHSIZE; i++){
while( pRtree->aHash[i] ){
RtreeNode *pNext = pRtree->aHash[i]->pNext;
sqlite3_free(pRtree->aHash[i]);
pRtree->aHash[i] = pNext;
}
}
}
sqlite3_finalize(pRtree->pWriteNode);
sqlite3_finalize(pRtree->pDeleteNode);
sqlite3_finalize(pRtree->pReadRowid);
@@ -2329,7 +2339,7 @@ static int AdjustTree(
int iCell;
cnt++;
if( NEVER(cnt>100) ){
if( cnt>100 ){
RTREE_IS_CORRUPT(pRtree);
return SQLITE_CORRUPT_VTAB;
}
@@ -2687,15 +2697,6 @@ static int SplitNode(
rc = updateMapping(pRtree, pCell->iRowid, pLeft, iHeight);
}
if( rc==SQLITE_OK ){
rc = nodeRelease(pRtree, pRight);
pRight = 0;
}
if( rc==SQLITE_OK ){
rc = nodeRelease(pRtree, pLeft);
pLeft = 0;
}
splitnode_out:
nodeRelease(pRtree, pRight);
nodeRelease(pRtree, pLeft);
@@ -2880,7 +2881,7 @@ static int rtreeInsertCell(
rc = SplitNode(pRtree, pNode, pCell, iHeight);
}else{
rc = AdjustTree(pRtree, pNode, pCell);
if( ALWAYS(rc==SQLITE_OK) ){
if( rc==SQLITE_OK ){
if( iHeight==0 ){
rc = rowidWrite(pRtree, pCell->iRowid, pNode->iNode);
}else{
@@ -3775,7 +3776,7 @@ static void rtreenode(sqlite3_context *ctx, int nArg, sqlite3_value **apArg){
if( node.zData==0 ) return;
nData = sqlite3_value_bytes(apArg[1]);
if( nData<4 ) return;
if( nData<NCELL(&node)*tree.nBytesPerCell ) return;
if( nData<4+NCELL(&node)*tree.nBytesPerCell ) return;
pOut = sqlite3_str_new(0);
for(ii=0; ii<NCELL(&node); ii++){
+10
View File
@@ -47,4 +47,14 @@ ifcapable rtree_int_only {
do_rtree_integrity_test rtreeB-1.2 t1
# https://sqlite.org/forum/forumpost/2026-01-08T23:32:19Z
#
db null NULL
do_execsql_test rtreeB-2.1 {
SELECT rtreenode(1,x'00000001'||randomblob(15)) IS NULL;
} {1}
do_execsql_test rtreeB-2.2 {
SELECT rtreenode(1,x'00000001'||randomblob(16)) IS NOT NULL;
} {1}
finish_test
+1
View File
@@ -135,6 +135,7 @@ foreach {tn blob} {
54 540101743400120003001200010000000000000002120002400C000000000002120002400C00000000000050040100000074310017FF0050040100000074310017FF7F00000000000000050100000000000000030100000003001700010000666F7572
55 540101743400120003001200010000000000000002120002400C00000000000050040100000074310017000100010080000001000000020003010100000300170100000003001700010000666F7572
56 5487ffffff7f
57 54015052494e5446110017120004
} {
do_test 2.$tn {
set changeset [binary decode hex $blob]
+10
View File
@@ -192,6 +192,16 @@ do_test 3.3 {
}
} {1 1 3 3}
#-------------------------------------------------------------------------
#
reset_db
set C [binary format c* 0x54 0x01 0x01 0x00 0x12 0x00 0x05]
do_test 4.0 {
sqlite3changegroup grp
list [catch { grp add $C } msg] $msg
} {1 SQLITE_CORRUPT}
grp delete
finish_test
+44 -16
View File
@@ -349,6 +349,20 @@ static int sessionVarintGet(const u8 *aBuf, int *piVal){
return getVarint32(aBuf, *piVal);
}
/*
** Read a varint value from buffer aBuf[], size nBuf bytes, into *piVal.
** Return the number of bytes read.
*/
static int sessionVarintGetSafe(const u8 *aBuf, int nBuf, int *piVal){
u8 aCopy[5];
const u8 *aRead = aBuf;
if( nBuf<5 ){
memcpy(aCopy, aBuf, nBuf);
aRead = aCopy;
}
return getVarint32(aRead, *piVal);
}
/* Load an unaligned and unsigned 32-bit integer */
#define SESSION_UINT32(x) (((u32)(x)[0]<<24)|((x)[1]<<16)|((x)[2]<<8)|(x)[3])
@@ -643,14 +657,10 @@ static unsigned int sessionChangeHash(
int isPK = pTab->abPK[i];
if( bPkOnly && isPK==0 ) continue;
/* It is not possible for eType to be SQLITE_NULL here. The session
** module does not record changes for rows with NULL values stored in
** primary key columns. */
assert( eType==SQLITE_INTEGER || eType==SQLITE_FLOAT
|| eType==SQLITE_TEXT || eType==SQLITE_BLOB
|| eType==SQLITE_NULL || eType==0
);
assert( !isPK || (eType!=0 && eType!=SQLITE_NULL) );
if( isPK ){
a++;
@@ -658,12 +668,16 @@ static unsigned int sessionChangeHash(
if( eType==SQLITE_INTEGER || eType==SQLITE_FLOAT ){
h = sessionHashAppendI64(h, sessionGetI64(a));
a += 8;
}else{
}else if( eType==SQLITE_TEXT || eType==SQLITE_BLOB ){
int n;
a += sessionVarintGet(a, &n);
h = sessionHashAppendBlob(h, n, a);
a += n;
}
/* It should not be possible for eType to be SQLITE_NULL or 0x00 here,
** as the session module does not record changes for rows with NULL
** values stored in primary key columns. But a corrupt changesets
** may contain such a value. */
}else{
a += sessionSerialLen(a);
}
@@ -3072,10 +3086,13 @@ static int sessionGenerateChangeset(
}
if( pSession->rc ) return pSession->rc;
rc = sqlite3_exec(pSession->db, "SAVEPOINT changeset", 0, 0, 0);
if( rc!=SQLITE_OK ) return rc;
sqlite3_mutex_enter(sqlite3_db_mutex(db));
rc = sqlite3_exec(pSession->db, "SAVEPOINT changeset", 0, 0, 0);
if( rc!=SQLITE_OK ){
sqlite3_mutex_leave(sqlite3_db_mutex(db));
return rc;
}
for(pTab=pSession->pTable; rc==SQLITE_OK && pTab; pTab=pTab->pNext){
if( pTab->nEntry ){
@@ -3558,7 +3575,8 @@ static int sessionReadRecord(
u8 *aVal = &pIn->aData[pIn->iNext];
if( eType==SQLITE_TEXT || eType==SQLITE_BLOB ){
int nByte;
pIn->iNext += sessionVarintGet(aVal, &nByte);
int nRem = pIn->nData - pIn->iNext;
pIn->iNext += sessionVarintGetSafe(aVal, nRem, &nByte);
rc = sessionInputBuffer(pIn, nByte);
if( rc==SQLITE_OK ){
if( nByte<0 || nByte>pIn->nData-pIn->iNext ){
@@ -3611,7 +3629,8 @@ static int sessionChangesetBufferTblhdr(SessionInput *pIn, int *pnByte){
rc = sessionInputBuffer(pIn, 9);
if( rc==SQLITE_OK ){
nRead += sessionVarintGet(&pIn->aData[pIn->iNext + nRead], &nCol);
int nBuf = pIn->nData - pIn->iNext;
nRead += sessionVarintGetSafe(&pIn->aData[pIn->iNext], nBuf, &nCol);
/* The hard upper limit for the number of columns in an SQLite
** database table is, according to sqliteLimit.h, 32676. So
** consider any table-header that purports to have more than 65536
@@ -3631,8 +3650,15 @@ static int sessionChangesetBufferTblhdr(SessionInput *pIn, int *pnByte){
while( (pIn->iNext + nRead)<pIn->nData && pIn->aData[pIn->iNext + nRead] ){
nRead++;
}
/* Break out of the loop if if the nul-terminator byte has been found.
** Otherwise, read some more input data and keep seeking. If there is
** no more input data, consider the changeset corrupt. */
if( (pIn->iNext + nRead)<pIn->nData ) break;
rc = sessionInputBuffer(pIn, nRead + 100);
if( rc==SQLITE_OK && (pIn->iNext + nRead)>=pIn->nData ){
rc = SQLITE_CORRUPT_BKPT;
}
}
*pnByte = nRead+1;
return rc;
@@ -3764,10 +3790,10 @@ static int sessionChangesetNextOne(
memset(p->apValue, 0, sizeof(sqlite3_value*)*p->nCol*2);
}
/* Make sure the buffer contains at least 10 bytes of input data, or all
** remaining data if there are less than 10 bytes available. This is
** sufficient either for the 'T' or 'P' byte and the varint that follows
** it, or for the two single byte values otherwise. */
/* Make sure the buffer contains at least 2 bytes of input data, or all
** remaining data if there are less than 2 bytes available. This is
** sufficient either for the 'T' or 'P' byte that begins a new table,
** or for the "op" and "bIndirect" single bytes otherwise. */
p->rc = sessionInputBuffer(&p->in, 2);
if( p->rc!=SQLITE_OK ) return p->rc;
@@ -3797,11 +3823,13 @@ static int sessionChangesetNextOne(
return (p->rc = SQLITE_CORRUPT_BKPT);
}
p->op = op;
p->bIndirect = p->in.aData[p->in.iNext++];
if( p->op!=SQLITE_UPDATE && p->op!=SQLITE_DELETE && p->op!=SQLITE_INSERT ){
if( (op!=SQLITE_UPDATE && op!=SQLITE_DELETE && op!=SQLITE_INSERT)
|| (p->in.iNext>=p->in.nData)
){
return (p->rc = SQLITE_CORRUPT_BKPT);
}
p->op = op;
p->bIndirect = p->in.aData[p->in.iNext++];
if( paRec ){
int nVal; /* Number of values to buffer */
+45 -21
View File
@@ -7,6 +7,8 @@
#include <string.h>
#include "tclsqlite.h"
#include <stdlib.h>
#ifndef SQLITE_AMALGAMATION
typedef unsigned char u8;
#endif
@@ -856,6 +858,21 @@ static int testStreamInput(
return SQLITE_OK;
}
/*
** This works like Tcl_GetByteArrayFromObj(), except that it returns a buffer
** allocated using malloc() that must be freed by the caller. This is done
** because Tcl's buffers are often padded by a few bytes, which prevents
** small overreads from being detected when tests are run under asan.
*/
static void *testGetByteArrayFromObj(Tcl_Obj *p, Tcl_Size *pnByte){
Tcl_Size nByte = 0;
void *aByte = Tcl_GetByteArrayFromObj(p, &nByte);
void *aCopy = malloc(nByte ? (size_t)nByte : 1);
memcpy(aCopy, aByte, (size_t)nByte);
*pnByte = nByte;
return aCopy;
}
static int SQLITE_TCLAPI testSqlite3changesetApply(
int iVersion,
@@ -920,7 +937,7 @@ static int SQLITE_TCLAPI testSqlite3changesetApply(
return TCL_ERROR;
}
db = *(sqlite3 **)info.objClientData;
pChangeset = (void *)Tcl_GetByteArrayFromObj(objv[2], &nChangeset);
pChangeset = (void *)testGetByteArrayFromObj(objv[2], &nChangeset);
ctx.pConflictScript = objv[3];
ctx.pFilterScript = objc==5 ? objv[4] : 0;
ctx.interp = interp;
@@ -972,6 +989,7 @@ static int SQLITE_TCLAPI testSqlite3changesetApply(
}
}
free(pChangeset);
if( rc!=SQLITE_OK ){
return test_session_error(interp, rc, 0);
}else{
@@ -1195,7 +1213,12 @@ static int SQLITE_TCLAPI test_sqlite3session_foreach(
pCS = objv[2];
pScript = objv[3];
pChangeset = (void *)Tcl_GetByteArrayFromObj(pCS, &nChangeset);
/* Take a copy of the changeset into an exact sized buffer allocated
** using malloc(). The Tcl buffer will be padded by a few bytes, which
** prevents small overreads from being detected by ASAN when the tests
** are run. */
pChangeset = (void*)testGetByteArrayFromObj(pCS, &nChangeset);
sStr.nStream = test_tcl_integer(interp, SESSION_STREAM_TCL_VAR);
if( isInvert ){
int f = SQLITE_CHANGESETSTART_INVERT;
@@ -1216,32 +1239,33 @@ static int SQLITE_TCLAPI test_sqlite3session_foreach(
rc = sqlite3changeset_start_strm(&pIter, testStreamInput, (void*)&sStr);
}
}
if( rc!=SQLITE_OK ){
return test_session_error(interp, rc, 0);
}
while( SQLITE_ROW==sqlite3changeset_next(pIter) ){
Tcl_Obj *pVar = 0; /* Tcl value to set $VARNAME to */
pVar = testIterData(pIter);
Tcl_ObjSetVar2(interp, pVarname, 0, pVar, 0);
rc = Tcl_EvalObjEx(interp, pScript, 0);
if( rc!=TCL_OK && rc!=TCL_CONTINUE ){
sqlite3changeset_finalize(pIter);
return rc==TCL_BREAK ? TCL_OK : rc;
if( rc==SQLITE_OK ){
while( SQLITE_ROW==sqlite3changeset_next(pIter) ){
Tcl_Obj *pVar = 0; /* Tcl value to set $VARNAME to */
pVar = testIterData(pIter);
Tcl_ObjSetVar2(interp, pVarname, 0, pVar, 0);
rc = Tcl_EvalObjEx(interp, pScript, 0);
if( rc!=TCL_OK && rc!=TCL_CONTINUE ){
sqlite3changeset_finalize(pIter);
free(pChangeset);
return rc==TCL_BREAK ? TCL_OK : rc;
}
}
if( isCheckNext ){
int rc2 = sqlite3changeset_next(pIter);
rc = sqlite3changeset_finalize(pIter);
assert( (rc2==SQLITE_DONE && rc==SQLITE_OK) || rc2==rc );
}else{
rc = sqlite3changeset_finalize(pIter);
}
}
if( isCheckNext ){
int rc2 = sqlite3changeset_next(pIter);
rc = sqlite3changeset_finalize(pIter);
assert( (rc2==SQLITE_DONE && rc==SQLITE_OK) || rc2==rc );
}else{
rc = sqlite3changeset_finalize(pIter);
}
free(pChangeset);
if( rc!=SQLITE_OK ){
return test_session_error(interp, rc, 0);
}
return TCL_OK;
}
+69 -21
View File
@@ -86,6 +86,7 @@ emo.strip = 💈
emo.test = 🧪
emo.tool = 🔨
emo.wasm-opt = 🧼
emo.cleanup = 🧼
# 👷🪄🧮🧫🧽🍿⛽🚧🎱🪚🏆🧼
#
@@ -205,7 +206,7 @@ b.mkdir@ = if [ ! -d $(dir $@) ]; then \
# $1 = logtag, $2 = src file(s). $3 = dest dir
b.cp = $(call b.mkdir@); \
echo '$(logtag.$(1)) $(emo.disk) $(2) ==> $3'; \
cp -p $(2) $(3) || exit
cp -f -p $(2) $(3) || exit
#
# $(call b.c-pp.shcmd,LOGTAG,src,dest,-Dx=y...)
@@ -317,7 +318,7 @@ endif
#
sqlite3.canonical.c = $(dir.top)/sqlite3.c
sqlite3.c ?= $(firstword $(wildcard $(dir.top)/sqlite3-see.c) $(sqlite3.canonical.c))
sqlite3.h = $(dir $(sqlite3.c))/sqlite3.h
sqlite3.h = $(dir $(sqlite3.c))sqlite3.h
#
# bin.version-info = binary to output various sqlite3 version info for
@@ -565,12 +566,14 @@ WASM_CUSTOM_INSTANTIATE = 1
# -D... flags which should be included in all invocations should be
# appended to $(b.c-pp.target.flags).
#
bin.c-pp = ./c-pp-lite
$(bin.c-pp): c-pp-lite.c $(sqlite3.c) $(MAKEFILE)
$(CC) -O0 -o $@ c-pp-lite.c $(sqlite3.c) '-DCMPP_DEFAULT_DELIM="//#"' -I$(dir.top) \
bin.c-pp = ./c-pp
$(bin.c-pp): libcmpp.c $(sqlite3.c) $(MAKEFILE)
$(CC) -O0 -o $@ libcmpp.c $(dir.top)/sqlite3.c -I$(dir.top) \
-DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_OMIT_DEPRECATED -DSQLITE_OMIT_UTF16 \
-DSQLITE_OMIT_SHARED_CACHE -DSQLITE_OMIT_WAL -DSQLITE_THREADSAFE=0 \
-DSQLITE_TEMP_STORE=3
-DSQLITE_TEMP_STORE=3 \
'-DCMPP_DEFAULT_DELIM="//#"' -DCMPP_MAIN -DCMPP_OMIT_D_MODULE \
-DCMPP_OMIT_D_PIPE
DISTCLEAN_FILES += $(bin.c-pp)
b.c-pp.target.flags ?=
ifeq (1,$(SQLITE_C_IS_SEE))
@@ -868,7 +871,7 @@ $(sqlite3-license-version.js): $(bin.version-info) \
$(dir.api)/sqlite3-license-version-header.js $(MAKEFILE)
@$(call b.mkdir@); echo '$(logtag.@) $(emo.disk)'; { \
cat $(dir.api)/sqlite3-license-version-header.js || exit $$?; \
echo '/*'; \
echo '/* @preserve'; \
echo '** This code was built from sqlite3 version...'; \
echo '**'; \
awk '/define SQLITE_VERSION/{$$1=""; print "**" $$0}' $(sqlite3.h); \
@@ -910,8 +913,10 @@ ifeq (0,$(wasm-bare-bones))
sqlite3-api.jses += $(dir.api)/sqlite3-vtab-helper.c-pp.js
endif
sqlite3-api.jses += $(dir.api)/sqlite3-vfs-kvvfs.c-pp.js
sqlite3-api.jses += $(dir.api)/opfs-common-shared.c-pp.js
sqlite3-api.jses += $(dir.api)/sqlite3-vfs-opfs.c-pp.js
sqlite3-api.jses += $(dir.api)/sqlite3-vfs-opfs-sahpool.c-pp.js
sqlite3-api.jses += $(dir.api)/sqlite3-vfs-opfs-wl.c-pp.js
# Parallel builds can fail if $(sqlite3-license-version.js) is not
# created early enough, so make all files in $(sqlite-api.jses) except
@@ -1181,13 +1186,13 @@ all: demos
#######################################################################
#
# "SOAP" is a static file which is not part of the amalgamation but
# gets copied into the build output folder and into each of the fiddle
# builds.
# "SOAP" is not part of the amalgamation but gets copied into the
# build output folder and into each of the fiddle builds.
#
sqlite3.ext.js += $(dir.dout)/sqlite3-opfs-async-proxy.js
$(dir.dout)/sqlite3-opfs-async-proxy.js: $(dir.api)/sqlite3-opfs-async-proxy.js
@$(call b.cp,@,$<,$@)
$(eval $(call b.c-pp.target,soap,\
$(dir.api)/sqlite3-opfs-async-proxy.c-pp.js,\
$(dir.dout)/sqlite3-opfs-async-proxy.js))
#
# Add a dep of $(sqlite3.ext.js) on every individual build's JS file.
@@ -1220,7 +1225,7 @@ $(foreach B,$(b.names),$(eval post: $(post-js.$(B).js)))
# These flags get applied via $(bin.mkwb).
emcc.speedtest1.common = $(emcc_opt_full)
emcc.speedtest1 = -I. -I$(dir $(sqlite3.canonical.c))
emcc.speedtest1 += -sENVIRONMENT=web
emcc.speedtest1 += -sENVIRONMENT=web,worker
emcc.speedtest1 += -sALLOW_MEMORY_GROWTH
emcc.speedtest1 += -sINITIAL_MEMORY=$(emcc.INITIAL_MEMORY.32)
emcc.speedtest1.common += -sINVOKE_RUN=0
@@ -1263,23 +1268,33 @@ $(EXPORTED_FUNCTIONS.speedtest1): $(EXPORTED_FUNCTIONS.api)
@$(call b.mkdir@); $(call b.echo,@,$(emo.disk)); \
{ echo _wasm_main; cat $(EXPORTED_FUNCTIONS.api); } > $@ || exit
speedtest1: b-speedtest1
st: speedtest1
#
# Generate 64-bit variants of speedtest1*.{js,html}
#
# $1 = input file
# $2 = output file
#
# TODO: preprocess these like we do the rest.
#
define gen-st64
$(2): $(1)
@$$(call b.echo,speedtest164,$$(emo.disk)$(emo.lock) Creating from $$<)
@rm -f $$@; \
sed -e 's/$(3)\.js/$(3)-64bit\.js/' < $$< > $$@; \
rm -f $$@; \
sed -e 's/speedtest1\.js/speedtest1-64bit\.js/' \
-e 's/speedtest1-worker\.js/speedtest1-worker-64bit\.js/' \
< $$< > $$@; \
chmod -w $$@
b-speedtest164: $(2)
$(2): b-speedtest164
speedtest1: $(1) $(2)
CLEAN_FILES += $(2)
endef
speedtest1: b-speedtest164
$(eval $(call gen-st64,speedtest1.html,speedtest1-64bit.html,speedtest1))
$(eval $(call gen-st64,speedtest1-worker.html,speedtest1-worker-64bit.html,speedtest1-worker))
$(eval $(call gen-st64,speedtest1-worker.js,speedtest1-worker-64bit.js,speedtest1-worker))
$(eval $(call gen-st64,speedtest1.html,speedtest1-64bit.html))
$(eval $(call gen-st64,speedtest1-worker.html,speedtest1-worker-64bit.html))
$(eval $(call gen-st64,speedtest1-worker.js,speedtest1-worker-64bit.js))
# end speedtest1.js
########################################################################
@@ -1517,9 +1532,9 @@ update-docs:
exit 127
else
wasm.docs.jswasm = $(wasm.docs.home)/jswasm
update-docs: $(bin.stripccomments) $(out.sqlite3.js) $(out.sqlite3.wasm)
update-docs: $(bin.stripccomments) $(out.vanilla.js) $(out.vanilla.wasm)
@echo "Copying files to the /wasm docs. Be sure to use an -Oz build for this!";
cp -p $(sqlite3.wasm) $(wasm.docs.jswasm)/.
cp -p $(out.vanilla.wasm) $(wasm.docs.jswasm)/.
$(bin.stripccomments) -k -k < $(out.vanilla.js) \
| sed -e '/^[ \t]*$$/d' > $(wasm.docs.jswasm)/sqlite3.js
cp -p demo-123.js demo-123.html demo-123-worker.html $(wasm.docs.home)/.
@@ -1582,6 +1597,39 @@ endif
# ^^^ making dist/snapshot
CLEAN_FILES += $(wildcard sqlite-wasm-*.zip)
########################################################################
# The npm target is specifically for preparing files for the downstream
# https://github.com/sqlite/sqlite-wasm (npm) distribution.
#
# Per agreement with that project's maintainers, these filenames need
# to remain stable. To avoid breakage in their deployment process, any
# changes (like renaming files) which potentially break their deployment
# needs to be communicated to that project via opening a new ticket or
# direct coordination with its maintainers.
#
# This target does a full clean/rebuild so that we can ensure that the
# optimization level is set consistently across all files.
#
npm.bundle.zip = npm-bundle.zip
CLEAN_FILES += $(npm.bundle.zip)
# Distributables which need to be built for npm:
npm_files = $(addprefix $(dir.dout)/, \
sqlite3-bundler-friendly.mjs \
sqlite3-opfs-async-proxy.js \
sqlite3-worker1-bundler-friendly.mjs \
sqlite3-worker1-promiser.mjs \
sqlite3-worker1.mjs \
sqlite3.mjs \
sqlite3.wasm \
sqlite3-node.mjs \
)
npm: $(sqlite3.canonical.c)
@echo "$(emo.cleanup) Forcing a clean rebuild to ensure consistent optimization flags."
$(MAKE) clean
$(MAKE) -e "emcc_opt=-Oz $(emcc-opt-extra)" $(npm_files)
rm -f $(npm.bundle.zip); zip -r $(npm.bundle.zip) $(npm_files)
unzip -l $(npm.bundle.zip)
########################################################################
# Explanation of, and some commentary on, various emcc build flags
# follows. Full docs for these can be found at:
+4 -3
View File
@@ -13,6 +13,7 @@ _sqlite3_bind_parameter_index
_sqlite3_bind_parameter_name
_sqlite3_bind_pointer
_sqlite3_bind_text
_sqlite3_bind_zeroblob
_sqlite3_busy_handler
_sqlite3_busy_timeout
_sqlite3_cancel_auto_extension
@@ -225,14 +226,14 @@ _sqlite3session_object_config
_sqlite3session_patchset
_sqlite3session_patchset_strm
_sqlite3session_table_filter
//#endif not bare-bones
//#/if not bare-bones
//#if enable-see
_sqlite3_key
_sqlite3_key_v2
_sqlite3_rekey
_sqlite3_rekey_v2
_sqlite3_activate_see
//#endif enable-see
//#/if enable-see
//#if fiddle
_fiddle_db_arg
_fiddle_db_filename
@@ -244,4 +245,4 @@ _fiddle_reset_db
_fiddle_db_handle
_fiddle_db_vfs
_fiddle_export_db
//#endif fiddle
//#/if fiddle
+6 -4
View File
@@ -14,7 +14,7 @@
*/
//#if target:es6-module
const toExportForESM =
//#endif
//#/if
(function(){
//console.warn("this is extern-post-js");
/**
@@ -73,6 +73,8 @@ const toExportForESM =
const sIM = globalThis.sqlite3InitModule = function ff(...args){
//console.warn("Using replaced sqlite3InitModule()",globalThis.location);
sIMS.emscriptenLocateFile = args[0]?.locateFile /* see pre-js.c-pp.js [tag:locateFile] */;
sIMS.emscriptenInstantiateWasm = args[0]?.instantiateWasm /* see pre-js.c-pp.js [tag:locateFile] */;
return originalInit(...args).then((EmscriptenModule)=>{
sIMS.debugModule("sqlite3InitModule() sIMS =",sIMS);
sIMS.debugModule("sqlite3InitModule() EmscriptenModule =",EmscriptenModule);
@@ -95,7 +97,7 @@ const toExportForESM =
//console.warn("sqlite3InitModule() returning E-module.",EmscriptenModule);
return EmscriptenModule;
}
//#endif
//#/if
return s;
}).catch((e)=>{
console.error("Exception loading sqlite3 module:",e);
@@ -126,10 +128,10 @@ const toExportForESM =
}
/* AMD modules get injected in a way we cannot override,
so we can't handle those here. */
//#endif // !target:es6-module
//#/if // !target:es6-module
return sIM;
})();
//#if target:es6-module
sqlite3InitModule = toExportForESM;
export default sqlite3InitModule;
//#endif
//#/if
+191
View File
@@ -0,0 +1,191 @@
//#if 0
/**
This file is for preprocessor #include into the "opfs" and
"opfs-wl" impls, as well as their async-proxy part. It must be
inlined in those files, as opposed to being a shared copy in the
library, because (A) the async proxy does not load the library and
(B) it references an object which is local to each of those files
but which has a 99% identical structure for each.
*/
//#/if
//#// vfs.metrics.enable is a refactoring crutch.
//#define vfs.metrics.enable 0
const initS11n = function(){
/**
This proxy de/serializes cross-thread function arguments and
output-pointer values via the state.sabIO SharedArrayBuffer,
using the region defined by (state.sabS11nOffset,
state.sabS11nOffset + state.sabS11nSize]. Only one dataset is
recorded at a time.
This is not a general-purpose format. It only supports the
range of operations, and data sizes, needed by the
sqlite3_vfs and sqlite3_io_methods operations. Serialized
data are transient and this serialization algorithm may
change at any time.
The data format can be succinctly summarized as:
Nt...Td...D
Where:
- N = number of entries (1 byte)
- t = type ID of first argument (1 byte)
- ...T = type IDs of the 2nd and subsequent arguments (1 byte
each).
- d = raw bytes of first argument (per-type size).
- ...D = raw bytes of the 2nd and subsequent arguments (per-type
size).
All types except strings have fixed sizes. Strings are stored
using their TextEncoder/TextDecoder representations. It would
arguably make more sense to store them as Int16Arrays of
their JS character values, but how best/fastest to get that
in and out of string form is an open point. Initial
experimentation with that approach did not gain us any speed.
Historical note: this impl was initially about 1% this size by
using using JSON.stringify/parse(), but using fit-to-purpose
serialization saves considerable runtime.
*/
if(state.s11n) return state.s11n;
const textDecoder = new TextDecoder(),
textEncoder = new TextEncoder('utf-8'),
viewU8 = new Uint8Array(state.sabIO, state.sabS11nOffset, state.sabS11nSize),
viewDV = new DataView(state.sabIO, state.sabS11nOffset, state.sabS11nSize);
state.s11n = Object.create(null);
/* Only arguments and return values of these types may be
serialized. This covers the whole range of types needed by the
sqlite3_vfs API. */
const TypeIds = Object.create(null);
TypeIds.number = { id: 1, size: 8, getter: 'getFloat64', setter: 'setFloat64' };
TypeIds.bigint = { id: 2, size: 8, getter: 'getBigInt64', setter: 'setBigInt64' };
TypeIds.boolean = { id: 3, size: 4, getter: 'getInt32', setter: 'setInt32' };
TypeIds.string = { id: 4 };
const getTypeId = (v)=>(
TypeIds[typeof v]
|| toss("Maintenance required: this value type cannot be serialized.",v)
);
const getTypeIdById = (tid)=>{
switch(tid){
case TypeIds.number.id: return TypeIds.number;
case TypeIds.bigint.id: return TypeIds.bigint;
case TypeIds.boolean.id: return TypeIds.boolean;
case TypeIds.string.id: return TypeIds.string;
default: toss("Invalid type ID:",tid);
}
};
/**
Returns an array of the deserialized state stored by the most
recent serialize() operation (from this thread or the
counterpart thread), or null if the serialization buffer is
empty. If passed a truthy argument, the serialization buffer
is cleared after deserialization.
*/
state.s11n.deserialize = function(clear=false){
//#if vfs.metrics.enable
++metrics.s11n.deserialize.count;
//#/if
const t = performance.now();
const argc = viewU8[0];
const rc = argc ? [] : null;
if(argc){
const typeIds = [];
let offset = 1, i, n, v;
for(i = 0; i < argc; ++i, ++offset){
typeIds.push(getTypeIdById(viewU8[offset]));
}
for(i = 0; i < argc; ++i){
const t = typeIds[i];
if(t.getter){
v = viewDV[t.getter](offset, state.littleEndian);
offset += t.size;
}else{/*String*/
n = viewDV.getInt32(offset, state.littleEndian);
offset += 4;
v = textDecoder.decode(viewU8.slice(offset, offset+n));
offset += n;
}
rc.push(v);
}
}
if(clear) viewU8[0] = 0;
//log("deserialize:",argc, rc);
//#if vfs.metrics.enable
metrics.s11n.deserialize.time += performance.now() - t;
//#/if
return rc;
};
/**
Serializes all arguments to the shared buffer for consumption
by the counterpart thread.
This routine is only intended for serializing OPFS VFS
arguments and (in at least one special case) result values,
and the buffer is sized to be able to comfortably handle
those.
If passed no arguments then it zeroes out the serialization
state.
*/
state.s11n.serialize = function(...args){
const t = performance.now();
//#if vfs.metrics.enable
++metrics.s11n.serialize.count;
//#/if
if(args.length){
//log("serialize():",args);
const typeIds = [];
let i = 0, offset = 1;
viewU8[0] = args.length & 0xff /* header = # of args */;
for(; i < args.length; ++i, ++offset){
/* Write the TypeIds.id value into the next args.length
bytes. */
typeIds.push(getTypeId(args[i]));
viewU8[offset] = typeIds[i].id;
}
for(i = 0; i < args.length; ++i) {
/* Deserialize the following bytes based on their
corresponding TypeIds.id from the header. */
const t = typeIds[i];
if(t.setter){
viewDV[t.setter](offset, args[i], state.littleEndian);
offset += t.size;
}else{/*String*/
const s = textEncoder.encode(args[i]);
viewDV.setInt32(offset, s.byteLength, state.littleEndian);
offset += 4;
viewU8.set(s, offset);
offset += s.byteLength;
}
}
//log("serialize() result:",viewU8.slice(0,offset));
}else{
viewU8[0] = 0;
}
//#if vfs.metrics.enable
metrics.s11n.serialize.time += performance.now() - t;
//#/if
};
//#if defined opfs-async-proxy
state.s11n.storeException = state.asyncS11nExceptions
? ((priority,e)=>{
if(priority<=state.asyncS11nExceptions){
state.s11n.serialize([e.name,': ',e.message].join(""));
}
})
: ()=>{};
//#/if
return state.s11n;
//#undef vfs.metrics.enable
}/*initS11n()*/;
File diff suppressed because it is too large Load Diff
+1
View File
@@ -34,6 +34,7 @@ Module.runSQLite3PostLoadInit = async function(
- sqlite3-vtab-helper.c-pp.js => Utilities for virtual table impls
- sqlite3-vfs-opfs.c-pp.js => OPFS VFS
- sqlite3-vfs-opfs-sahpool.c-pp.js => OPFS SAHPool VFS
- sqlite3-vfs-opfs-wl.c-pp.js => WebLock-using OPFS VFS
- post-js-footer.js => this file's epilogue
And all of that gets sandwiched between extern-pre-js.js and
+28 -8
View File
@@ -17,9 +17,9 @@
/**
This file was preprocessed using:
//#@policy error
//#@ policy error
@c-pp::argv@
//#@policy off
//#@ policy off
*/
//#if unsupported-build
/**
@@ -27,10 +27,14 @@
This SQLite JS build configuration is entirely unsupported! It has
not been tested beyond the ability to compile it. It may not
load. It may not work properly. Only builds targeting browser
environments are supported and tested.
load. It may not work properly. Only builds _directly_ targeting
browser environments ("vanilla" JS and ESM modules) are supported
and tested. Builds which _indirectly_ target browsers (namely
bundler-friendly builds and any node builds) are not supported
deliverables.
*/
//#endif
//#/if
//#if not target:es6-bundler-friendly
(function(Module){
const sIMS =
globalThis.sqlite3InitModuleState/*from extern-post-js.c-pp.js*/
@@ -68,6 +72,14 @@
approach.
*/
Module['locateFile'] = function(path, prefix) {
if( this.emscriptenLocateFile instanceof Function ){
/* [tag:locateFile] Client-overridden impl. We do not support
this but offer it as a back-door which will go away the
moment either Emscripten changes that interface or we manage
to get non-Emscripten builds working.
https://sqlite.org/forum/forumpost/1eec339854c935bd */
return this.emscriptenLocateFile(path, prefix);
}
//#if target:es6-module
return new URL(path, import.meta.url).href;
//#else
@@ -90,10 +102,10 @@
"result =", theFile
);
return theFile;
//#endif target:es6-module
//#/if target:es6-module
}.bind(sIMS);
//#if Module.instantiateWasm and not wasmfs
//#if Module.instantiateWasm and not wasmfs and not target:node
/**
Override Module.instantiateWasm().
@@ -102,8 +114,15 @@
https://github.com/emscripten-core/emscripten/issues/17951
In such builds we must disable this.
It's disabled in the (unsupported/untested) node builds because
node does not do fetch().
*/
Module['instantiateWasm'] = function callee(imports,onSuccess){
if( this.emscriptenInstantiateWasm instanceof Function ){
/* See [tag:locateFile]. Same story here */
return this.emscriptenInstantiateWasm(imports, onSuccess);
}
const sims = this;
const uri = Module.locateFile(
sims.wasmFilename, (
@@ -129,6 +148,7 @@
.then(finalThen)
return loadWasm();
}.bind(sIMS);
//#endif Module.instantiateWasm and not wasmfs
//#/if Module.instantiateWasm and not wasmfs
})(Module);
//#/if not target:es6-bundler-friendly
/* END FILE: api/pre-js.js. */
+7 -5
View File
@@ -98,6 +98,8 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
["sqlite3_bind_parameter_name", "string", "sqlite3_stmt*", "int"],
["sqlite3_bind_pointer", "int",
"sqlite3_stmt*", "int", "*", "string:static", "*"],
/* sqlite_bind_text() is hand-written */
["sqlite3_bind_zeroblob", "int", "sqlite3_stmt*", "int", "int"],
["sqlite3_busy_handler","int", [
"sqlite3*",
new wasm.xWrap.FuncPtrAdapter({
@@ -120,11 +122,11 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
["sqlite3_column_double","f64", "sqlite3_stmt*", "int"],
["sqlite3_column_int","int", "sqlite3_stmt*", "int"],
["sqlite3_column_name","string", "sqlite3_stmt*", "int"],
//#define proxy-text-apis=1
//#define proxy-text-apis 1
//#if not proxy-text-apis
/* Search this file for tag:proxy-text-apis to see what this is about. */
["sqlite3_column_text","string", "sqlite3_stmt*", "int"],
//#endif
//#/if
["sqlite3_column_type","int", "sqlite3_stmt*", "int"],
["sqlite3_column_value","sqlite3_value*", "sqlite3_stmt*", "int"],
["sqlite3_commit_hook", "void*", [
@@ -324,7 +326,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
["sqlite3_value_subtype", "int", "sqlite3_value*"],
//#if not proxy-text-apis
["sqlite3_value_text", "string", "sqlite3_value*"],
//#endif
//#/if
["sqlite3_value_type", "int", "sqlite3_value*"],
["sqlite3_vfs_find", "*", "string"],
["sqlite3_vfs_register", "int", "sqlite3_vfs*", "int"],
@@ -500,7 +502,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
["sqlite3_activate_see", undefined, "string"]
);
}
//#endif enable-see
//#/if enable-see
if( wasm.bigIntEnabled && !!wasm.exports.sqlite3_declare_vtab ){
bindingSignatures.int64.push(
@@ -1694,7 +1696,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
: null;
};
}/*text-return-related bindings*/
//#endif proxy-text-apis
//#/if proxy-text-apis
{/* sqlite3_config() */
/**
+174 -23
View File
@@ -26,6 +26,20 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
the sqlite3 binding if, e.g., the wrapper is in the main thread
and the sqlite3 API is in a worker. */
const outWrapper = function(f){
return (...args)=>f("sqlite3.oo1:",...args);
};
const debug = sqlite3.__isUnderTest
? outWrapper(console.debug.bind(console))
: outWrapper(sqlite3.config.debug);
const warn = sqlite3.__isUnderTest
? outWrapper(console.warn.bind(console))
: outWrapper(sqlite3.config.warn);
const error = sqlite3.__isUnderTest
? outWrapper(console.error.bind(console))
: outWrapper(sqlite3.config.error);
/**
In order to keep clients from manipulating, perhaps
inadvertently, the underlying pointer values of DB and Stmt
@@ -201,7 +215,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
if(stmt) stmt.finalize();
}
};
//#endif enable-see
//#/if enable-see
/**
A proxy for DB class constructors. It must be called with the
@@ -214,13 +228,16 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
properties:
- `.filename`: the db filename. It may be a special name like ":memory:"
or "".
or "". It may also be a URI-style name.
- `.flags`: as documented in the DB constructor.
- `.vfs`: as documented in the DB constructor.
It also accepts those as the first 3 arguments.
In non-default builds it may accept additional configuration
options.
*/
const dbCtorHelper = function ctor(...args){
const opt = ctor.normalizeArgs(...args);
@@ -284,7 +301,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
try{
//#if enable-see
dbCtorApplySEEKey(this,opt);
//#endif
//#/if
// Check for per-VFS post-open SQL/callback...
const pVfs = capi.sqlite3_js_db_vfs(pDb)
|| toss3("Internal error: cannot get VFS for new db handle.");
@@ -436,7 +453,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
is supplied and the database is encrypted, execution of the
post-initialization SQL will fail, causing the constructor to
throw.
//#endif enable-see
//#/if enable-see
The `filename` and `vfs` arguments may be either JS strings or
C-strings allocated via WASM. `flags` is required to be a JS
@@ -785,6 +802,9 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
sqlite3_db_filename() value for the given database name,
defaulting to "main". The argument may be either a JS string
or a pointer to a WASM-allocated C-string.
this.filename may be in the form of a URI-style string, whereas
the returned string contains only the filename part.
*/
dbFilename: function(dbName='main'){
return capi.sqlite3_db_filename(affirmDbOpen(this).pointer, dbName);
@@ -906,15 +926,15 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
result set, but only if that statement has any result rows. The
callback's "this" is the options object, noting that this
function synthesizes one if the caller does not pass one to
exec(). The second argument passed to the callback is always
the current Stmt object, as it's needed if the caller wants to
fetch the column names or some such (noting that they could
also be fetched via `this.columnNames`, if the client provides
the `columnNames` option). If the callback returns a literal
`false` (as opposed to any other falsy value, e.g. an implicit
`undefined` return), any ongoing statement-`step()` iteration
stops without an error. The return value of the callback is
otherwise ignored.
exec(). The first argument passed to the callback is described
below. The second argument is always the current Stmt object,
as it's needed if the caller wants to fetch the column names or
some such (noting that they could also be fetched via
`this.columnNames`, if the client provides the `columnNames`
option). If the callback returns a literal `false` (as opposed
to any other falsy value, e.g. an implicit `undefined` return),
any ongoing statement-`step()` iteration stops without an
error. The return value of the callback is otherwise ignored.
ACHTUNG: The callback MUST NOT modify the Stmt object. Calling
any of the Stmt.get() variants, Stmt.getColumnName(), or
@@ -947,20 +967,23 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
A.3) `'stmt'` causes the current Stmt to be passed to the
callback, but this mode will trigger an exception if
`resultRows` is an array because appending the transient
statement to the array would be downright unhelpful.
statement to the array would be downright unhelpful. This
option is a legacy feature, retained for backwards
compatibility. The statement object is passed as the second
argument to the callback, as described above.
B) An integer, indicating a zero-based column in the result
row. Only that one single value will be passed on.
row. Only that one single value, in JS form, will be passed on.
C) A string with a minimum length of 2 and leading character of
'$' will fetch the row as an object, extract that one field,
and pass that field's value to the callback. Note that these
keys are case-sensitive so must match the case used in the
and pass that field's value to the callback. These keys are
case-sensitive so must match the case used in the
SQL. e.g. `"select a A from t"` with a `rowMode` of `'$A'`
would work but `'$a'` would not. A reference to a column not in
the result set will trigger an exception on the first row (as
the check is not performed until rows are fetched). Note also
that `$` is a legal identifier character in JS so need not be
the check is not performed until rows are fetched). Note that
`$` is a legal identifier character in JS so need not be
quoted.
Any other `rowMode` value triggers an exception.
@@ -1000,6 +1023,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
- `callback` and `resultRows`: permit an array entries with
semantics similar to those described for `bind` above.
OTOH, this function already does too much.
*/
exec: function(/*(sql [,obj]) || (obj)*/){
affirmDbOpen(this);
@@ -1023,7 +1047,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
/* Optimization: if the SQL is a TypedArray we can save some string
conversion costs. */;
/* Allocate the two output pointers (ppStmt, pzTail) and heap
space for the SQL (pSql). When prepare_v2() returns, pzTail
space for the SQL (pSql). When prepare_v3() returns, pzTail
will point to somewhere in pSql. */
let sqlByteLen = isTA ? arg.sql.byteLength : wasm.jstrlen(arg.sql);
const ppStmt = wasm.scopedAlloc(
@@ -1035,8 +1059,8 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
const pSqlEnd = wasm.ptr.add(pSql, sqlByteLen);
if(isTA) wasm.heap8().set(arg.sql, pSql);
else wasm.jstrcpy(arg.sql, wasm.heap8(), pSql, sqlByteLen, false);
wasm.poke(wasm.ptr.add(pSql, sqlByteLen), 0/*NUL terminator*/);
while(pSql && wasm.peek(pSql, 'i8')
wasm.poke8(wasm.ptr.add(pSql, sqlByteLen), 0/*NUL terminator*/);
while(pSql && wasm.peek8(pSql)
/* Maintenance reminder:^^^ _must_ be 'i8' or else we
will very likely cause an endless loop. What that's
doing is checking for a terminating NUL byte. If we
@@ -1100,6 +1124,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
/* In order to trigger an exception in the
INSERT...RETURNING locking scenario:
https://sqlite.org/forum/forumpost/36f7a2e7494897df
[tag:insert-returning-reset]
*/).finalize();
stmt = null;
}/*prepare() loop*/
@@ -1116,6 +1141,132 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
return arg.returnVal();
}/*exec()*/,
//#if 0
/**
Experimental and untested - do not use.
Prepares one or more SQL statements, passing each to a callback
for processing.
It requires an options object with the following properties:
- "sql": SQL in any format accepted by exec().
- "callback" (function): gets passed each prepared statement,
as described below.
- "asPointer" (bool=false): if true, the callback is passed the
WASM (sqlite3*) pointer instead of a Stmt object.
- "saveSql" (array): if set, the SQL of each prepared statement
is appended to this array. This can be used without a callback
to split SQL into its component statements. Purely empty
statements (for for which sqlite3_prepare() returns a NULL
sqlite3_stmt, i.e. spaces and comments) are not added to this
list unless...
- "saveEmpty" (bool=false): If true, empty statements are
retained in opt.saveSql, but their leading/trailing whitespace
is trimmed (as for queries) so they may be empty.
For each statement in the input SQL:
1) If opt.saveSql is set, the SQL is appended to it.
2) If callback is set, callback(S) is called, where S is either
a Stmt object (by default) or an (sqlite3*) WASM pointer (if
opt.asPointer is true). If the callback returns a literal true
(as opposed to any other truthy value), ownership of S is
transferred to the callback, otherwise S is reset and finalized
as soon as the callback returns. If the callback throws, S is
unconditionally finalized.
If neither of opt.saveSql nor opt.callback are set, this
function does nothing more than prepare and finalize each
statement, which will trigger an exception if any of them
contain invalid SQL.
*/
forEachStmt: function(opt){
affirmDbOpen(this);
opt ??= Object.create(null);
if(!opt.sql){
return toss3("exec() requires an SQL string.");
}
const sql = util.flexibleString(opt.sql);
const callback = opt.callback;
let stmt, pStmt;
const stack = wasm.scopedAllocPush();
const saveSql = Array.isArray(opt.saveSql) ? opt.saveSql : undefined;
try{
const isTA = util.isSQLableTypedArray(opt.sql)
/* Optimization: if the SQL is a TypedArray we can save some string
conversion costs. */;
/* Allocate the two output pointers (ppStmt, pzTail) and heap
space for the SQL (pSql). When prepare_v3() returns, pzTail
will point to somewhere in pSql. */
let sqlByteLen = isTA ? opt.sql.byteLength : wasm.jstrlen(sql);
const ppStmt = wasm.scopedAlloc(
/* output (sqlite3_stmt**) arg and pzTail */
(2 * wasm.ptr.size) + (sqlByteLen + 1/* SQL + NUL */)
);
const pzTail = wasm.ptr.add(ppStmt, wasm.ptr.size) /* final arg to sqlite3_prepare_v2() */;
let pSql = wasm.ptr.add(pzTail, wasm.ptr.size) /* start of the SQL string */;
const pSqlEnd = wasm.ptr.add(pSql, sqlByteLen);
if(isTA) wasm.heap8().set(sql, pSql);
else wasm.jstrcpy(sql, wasm.heap8(), pSql, sqlByteLen, false);
wasm.poke8(wasm.ptr.add(pSql, sqlByteLen), 0/*NUL terminator*/);
while( pSql && wasm.peek8(pSql) ){
pStmt = stmt = null;
wasm.pokePtr([ppStmt, pzTail], 0);
const zHead = pSql;
DB.checkRc(this, capi.sqlite3_prepare_v3(
this.pointer, pSql, sqlByteLen, 0, ppStmt, pzTail
));
[pStmt, pSql] = wasm.peekPtr([ppStmt, pzTail]);
sqlByteLen = wasm.ptr.addn(pSqlEnd,-pSql);
if(opt.saveSql){
if( pStmt ) opt.saveSql.push(capi.sqlite3_sql(pStmt).trim());
else if( opt.saveEmpty ){
saveSql.push(wasm.typedArrayToString(
wasm.heap8u(), Number(zHead),
wasm.ptr.addn(zHead, sqlByteLen)
).trim(/*arguable*/));
}
}
if(!pStmt) continue;
//sqlite3.config.debug("forEachStmt() pSql =",capi.sqlite3_sql(pStmt));
if( !opt.callback ){
capi.sqlite3_finalize(pStmt);
pStmt = null;
continue;
}
stmt = opt.asPointer ? null : new Stmt(this, pStmt, BindTypes);
if( true===callaback(stmt || pStmt) ){
stmt = pStmt = null /*callback took ownership */;
}else if(stmt){
pStmt = null;
stmt.reset(
/* See [tag:insert-returning-reset]. The thinking here is
that if the callback didn't throw for this, it
probably should have.
*/).finalize();
stmt = null;
}else{
const rx = capi.sqlite3_reset(pStmt/*[tag:insert-returning-reset]*/);
capi.sqlite3_finalize(pStmt);
pStmt = null;
DB.checkRc(this, rx);
}
}/*prepare() loop*/
}finally{
if(stmt) stmt.finalize();
else if(pStmt) capi.sqlite3_finalize(pStmt);
wasm.scopedAllocPop(stack);
}
return this;
}/*forEachStmt()*/,
//#/if nope
/**
Creates a new UDF (User-Defined Function) which is accessible
via SQL code. This function may be called in any of the
@@ -2275,4 +2426,4 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
});
//#else
/* Built with the omit-oo1 flag. */
//#endif if not omit-oo1
//#/if if not omit-oo1
+101 -27
View File
@@ -99,6 +99,13 @@
used in WASMFS-capable builds of the library (which the canonical
builds do not include).
- `disable` (as of 3.53.0) may be an object with the following
properties:
- `vfs`, an object, may contain a map of VFS names to booleans.
Any mapping to falsy are disabled. The supported names
are: "kvvfs", "opfs", "opfs-sahpool", "opfs-wl".
- Other disabling options may be added in the future.
[^1] = This property may optionally be a function, in which case
this function calls that function to fetch the value,
enabling delayed evaluation.
@@ -145,7 +152,8 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
);
return sqlite3ApiBootstrap.sqlite3;
}
const config = Object.assign(Object.create(null),{
const nu = (...obj)=>Object.assign(Object.create(null),...obj);
const config = nu({
exports: undefined,
memory: undefined,
bigIntEnabled: !!globalThis.BigInt64Array,
@@ -162,7 +170,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
certain wasm.xWrap.resultAdapter()s.
*/
useStdAlloc: false
}, apiConfig || {});
}, apiConfig);
Object.assign(config, {
allocExportName: config.useStdAlloc ? 'malloc' : 'sqlite3_malloc',
@@ -195,7 +203,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
not documented are installed as 1-to-1 proxies for their
C-side counterparts.
*/
const capi = Object.create(null);
const capi = nu();
/**
Holds state which are specific to the WASM-related
infrastructure and glue code.
@@ -204,7 +212,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
dynamically after the api object is fully constructed, so
not all are documented in this file.
*/
const wasm = Object.create(null);
const wasm = nu();
/** Internal helper for SQLite3Error ctor. */
const __rcStr = (rc)=>{
@@ -752,6 +760,12 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
toss: function(...args){throw new Error(args.join(' '))},
toss3,
typedArrayPart: wasm.typedArrayPart,
nu,
assert: function(arg,msg){
if( !arg ){
util.toss("Assertion failed:",msg);
}
},
/**
Given a byte array or ArrayBuffer, this function throws if the
lead bytes of that buffer do not hold a SQLite3 database header,
@@ -887,7 +901,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
Like this.alloc.impl(), this.realloc.impl() is a direct binding
to the underlying realloc() implementation which does not throw
exceptions, instead returning 0 on allocation error.
exceptions, instead returning 0 (or 0n) on allocation error.
*/
realloc: undefined/*installed later*/,
@@ -1003,7 +1017,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
rv[1] = m ? (f._rxInt.test(m[2]) ? +m[2] : m[2]) : true;
};
}
const rc = Object.create(null), ov = [0,0];
const rc = nu(), ov = [0,0];
let i = 0, k;
while((k = capi.sqlite3_compileoption_get(i++))){
f._opt(k,ov);
@@ -1011,7 +1025,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
}
return f._result = rc;
}else if(Array.isArray(optName)){
const rc = Object.create(null);
const rc = nu();
optName.forEach((v)=>{
rc[v] = capi.sqlite3_compileoption_used(v);
});
@@ -1062,7 +1076,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
The memory lives in the WASM heap and can be used with routines
such as wasm.poke() and wasm.heap8u().slice().
*/
wasm.pstack = Object.assign(Object.create(null),{
wasm.pstack = nu({
/**
Sets the current pstack position to the given pointer. Results
are undefined if the passed-in value did not come from
@@ -1284,7 +1298,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
// sqlite3__wasm_init_wasmfs() is not available
return this.dir = "";
}
}.bind(Object.create(null));
}.bind(nu());
/**
Returns true if sqlite3.capi.sqlite3_wasmfs_opfs_dir() is a
@@ -1408,7 +1422,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
or not provided, then "main" is assumed.
*/
capi.sqlite3_js_db_vfs =
(dbPointer, dbName=0)=>util.sqlite3__wasm_db_vfs(dbPointer, dbName);
(dbPointer, dbName=wasm.ptr.null)=>util.sqlite3__wasm_db_vfs(dbPointer, dbName);
/**
A thin wrapper around capi.sqlite3_aggregate_context() which
@@ -1638,6 +1652,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
case capi.SQLITE_DBCONFIG_ENABLE_ATTACH_CREATE:
case capi.SQLITE_DBCONFIG_ENABLE_ATTACH_WRITE:
case capi.SQLITE_DBCONFIG_ENABLE_COMMENTS:
case capi.SQLITE_DBCONFIG_FP_DIGITS:
if( !this.ip ){
this.ip = wasm.xWrap('sqlite3__wasm_db_config_ip','int',
['sqlite3*', 'int', 'int', '*']);
@@ -1659,7 +1674,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
default:
return capi.SQLITE_MISUSE;
}
}.bind(Object.create(null));
}.bind(nu());
/**
Given a (sqlite3_value*), this function attempts to convert it
@@ -1893,7 +1908,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
if(rc) return SQLite3Error.toss(rc,arguments[2]+"() failed with code "+rc);
const pv = wasm.peekPtr(this.ptr);
return pv ? capi.sqlite3_value_to_js( pv, true ) : undefined;
}.bind(Object.create(null));
}.bind(nu());
/**
A wrapper around sqlite3_preupdate_new() which fetches the
@@ -1933,6 +1948,62 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
'sqlite3changeset_old');
}/*changeset/preupdate additions*/
/**
EXPERIMENTAL. For tentative addition in 3.53.0.
sqlite3_js_retry_busy(maxTimes,callback[,beforeRetry])
Calls the given _synchronous_ callback function. If that function
returns sqlite3.capi.SQLITE_BUSY _or_ throws an SQLite3Error
with a resultCode property of that value then it will suppress
that error and try again, up to the given maximum number of
times. If the callback returns any other value than that,
it is returned. If the maximum number of retries has been
reached, an SQLite3Error with a resultCode value of
sqlite3.capi.SQLITE_BUSY is thrown. If the callback throws any
exception other than the aforementioned BUSY exception, it is
propagated. If it throws a BUSY exception on its final attempt,
that is propagated as well.
If the beforeRetry argument is given, it must be a _synchronous_
function. It is called immediately before each retry of the
callback (not for the initial call), passed the attempt number
(so it starts with 2, not 1). If it throws, the exception is
handled as described above. Its result value is ignored.
To effectively retry "forever", pass a negative maxTimes value,
with the caveat that there is no recovery from that unless the
beforeRetry() can figure out when to throw.
TODO: an async variant of this.
*/
capi.sqlite3_js_retry_busy = function(maxTimes, callback, beforeRetry){
for(let n = 1; n <= maxTimes; ++n){
try{
if( beforeRetry && n>1 ) beforeRetry(n);
const rc = callback();
if( capi.SQLITE_BUSY===rc ){
if( n===maxTimes ){
throw new SQLite3Error(rc, [
"sqlite3_js_retry_busy() max retry attempts (",
maxTimes,
") reached."
].join(''));
}
continue;
}
return rc;
}catch(e){
if( n<maxTimes
&& (e instanceof SQLite3Error)
&& e.resultCode===capi.SQLITE_BUSY ){
continue;
}
throw e;
}
}
};
/* The remainder of the API will be set up in later steps. */
const sqlite3 = {
WasmAllocError: WasmAllocError,
@@ -1951,7 +2022,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
This object is initially a placeholder which gets replaced by a
build-generated object.
*/
version: Object.create(null),
version: nu(),
/**
The library reserves the 'client' property for client-side use
@@ -1997,6 +2068,7 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
so that we can add tests for them. */
delete sqlite3.util;
delete sqlite3.StructBinder;
delete sqlite3.opfs;
}
return sqlite3;
};
@@ -2016,19 +2088,24 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
return ff.isReady = p.catch(catcher);
}.bind(sqlite3ApiBootstrap),
/**
scriptInfo ideally gets injected into this object by the
infrastructure which assembles the JS/WASM module. It contains
state which must be collected before sqlite3ApiBootstrap() can
be declared. It is not necessarily available to any
sqlite3ApiBootstrap.initializers but "should" be in place (if
it's added at all) by the time that
sqlite3ApiBootstrap.initializersAsync is processed.
scriptInfo holds information about the currenty-loading script
so that we can locate the WASM file if it's somewhere other
than the build-time-defined directory. It ideally gets injected
into this object by the infrastructure which assembles the
JS/WASM module. It contains state which must be collected
before sqlite3ApiBootstrap() can be declared. It is not
necessarily available to any sqlite3ApiBootstrap.initializers
but "should" be in place (if it's added at all) by the time
that sqlite3ApiBootstrap.initializersAsync is processed.
This state is not part of the public API, only intended for use
with the sqlite3 API bootstrapping and wasm-loading process.
*/
scriptInfo: undefined
};
if( 'undefined'!==typeof sqlite3IsUnderTest/* from post-js-header.js */ ){
sqlite3.__isUnderTest = !!sqlite3IsUnderTest;
}
try{
sqlite3ApiBootstrap.initializers.forEach((f)=>{
f(sqlite3);
@@ -2051,9 +2128,6 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
builds, this would need to be injected in somewhere to get
that VFS loading. */ = sqlite3InitScriptInfo;
}
sqlite3.__isUnderTest =
('undefined'===typeof sqlite3IsUnderTest /* from post-js-header.js */)
? false : !!sqlite3IsUnderTest;
if( sqlite3.__isUnderTest ){
if( 'undefined'!==typeof EmscriptenModule ){
sqlite3.config.emscripten = EmscriptenModule;
@@ -2066,10 +2140,10 @@ globalThis.sqlite3ApiBootstrap = async function sqlite3ApiBootstrap(
loader, and if we lose that capability for some reason then
we'll lose access to this metadata.
These data are interesting for exploring how the wasm/JS
pieces connect, e.g. for exploring exactly what Emscripten
imports into WASM from its JS glue, but it's not
SQLite-related.
These data are interesting for exploring how the wasm/JS pieces
connect, e.g. for exploring exactly what Emscripten imports into
WASM from its JS glue, but it's not SQLite-related and is not
required for the library to work.
*/
const iw = sqlite3.scriptInfo?.instantiateWasm;
if( iw ){
+1 -1
View File
@@ -677,4 +677,4 @@ sqlite3.initWorker1API = function(){
});
//#else
/* Built with the omit-oo1 flag. */
//#endif if not omit-oo1
//#/if if not omit-oo1
@@ -1,4 +1,5 @@
/*
/* @preserve
**
** LICENSE for the sqlite3 WebAssembly/JavaScript APIs.
**
** This bundle (typically released as sqlite3.js or sqlite3.mjs)
@@ -1,4 +1,4 @@
/*
/* @preserve
2022-09-16
The author disclaims copyright to this source code. In place of a
@@ -46,10 +46,22 @@
theFunc().then(...) is not compatible with the change to
synchronous, but we do do not use those APIs that way. i.e. we don't
_need_ to change anything for this, but at some point (after Chrome
versions (approximately) 104-107 are extinct) should change our
versions (approximately) 104-107 are extinct) we should change our
usage of those methods to remove the "await".
*/
"use strict";
const urlParams = new URL(globalThis.location.href).searchParams;
const vfsName = urlParams.get('vfs');
if( !vfsName ){
throw new Error("Expecting vfs=opfs|opfs-wl URL argument for this worker");
}
/**
We use this to allow us to differentiate debug output from
multiple instances, e.g. multiple Workers to the "opfs"
VFS or both the "opfs" and "opfs-wl" VFSes.
*/
const workerId = (Math.random() * 10000000) | 0;
const isWebLocker = 'opfs-wl'===urlParams.get('vfs');
const wPost = (type,...args)=>postMessage({type, payload:args});
const installAsyncProxy = function(){
const toss = function(...args){throw new Error(args.join(' '))};
@@ -66,6 +78,13 @@ const installAsyncProxy = function(){
*/
const state = Object.create(null);
/* initS11n() is preprocessor-injected so that we have identical
copies in the synchronous and async halves. This side does not
load the SQLite library, so does not have access to that copy. */
//#define opfs-async-proxy
//#include api/opfs-common-inline.c-pp.js
//#undef opfs-async-proxy
/**
verbose:
@@ -82,7 +101,7 @@ const installAsyncProxy = function(){
2:console.log.bind(console)
};
const logImpl = (level,...args)=>{
if(state.verbose>level) loggers[level]("OPFS asyncer:",...args);
if(state.verbose>level) loggers[level](vfsName+' async-proxy',workerId+":",...args);
};
const log = (...args)=>logImpl(2, ...args);
const warn = (...args)=>logImpl(1, ...args);
@@ -97,12 +116,13 @@ const installAsyncProxy = function(){
*/
const __openFiles = Object.create(null);
/**
__implicitLocks is a Set of sqlite3_file pointers (integers) which were
"auto-locked". i.e. those for which we obtained a sync access
handle without an explicit xLock() call. Such locks will be
released during db connection idle time, whereas a sync access
handle obtained via xLock(), or subsequently xLock()'d after
auto-acquisition, will not be released until xUnlock() is called.
__implicitLocks is a Set of sqlite3_file pointers (integers)
which were "auto-locked". i.e. those for which we necessarily
obtain a sync access handle without an explicit xLock() call
guarding access. Such locks will be released during
`waitLoop()`'s idle time, whereas a sync access handle obtained
via xLock(), or subsequently xLock()'d after auto-acquisition,
will not be released until xUnlock() is called.
Maintenance reminder: if we relinquish auto-locks at the end of the
operation which acquires them, we pay a massive performance
@@ -271,10 +291,11 @@ const installAsyncProxy = function(){
In order to help alleviate cross-tab contention for a dabase, if
an exception is thrown while acquiring the handle, this routine
will wait briefly and try again, up to some fixed number of
times. If acquisition still fails at that point it will give up
and propagate the exception. Client-level code will see that as
an I/O error.
will wait briefly and try again, up to `maxTries` of times. If
acquisition still fails at that point it will give up and
propagate the exception. Client-level code will see that either
as an I/O error or SQLITE_BUSY, depending on the exception and
the context.
2024-06-12: there is a rare race condition here which has been
reported a single time:
@@ -289,13 +310,31 @@ const installAsyncProxy = function(){
there's another race condition there). That's easy to say but
creating a viable test for that condition has proven challenging
so far.
Interface quirk: if fh.xLock is falsy and the handle is acquired
then fh.fid is added to __implicitLocks(). If fh.xLock is truthy,
it is not added as an implicit lock. i.e. xLock() impls must set
fh.xLock immediately _before_ calling this and must arrange to
restore it to its previous value if this function throws.
2026-03-06:
- baseWaitTime is the number of milliseconds to wait for the
first retry, increasing by one factor for each retry. It defaults
to (state.asyncIdleWaitTime*2).
- maxTries is the number of attempt to make, each one spaced out
by one additional factor of the baseWaitTime (e.g. 300, then 600,
then 900, the 1200...). This MUST be an integer >0.
Only the Web Locks impl should use the 3rd and 4th parameters.
*/
const getSyncHandle = async (fh,opName)=>{
const getSyncHandle = async (fh, opName, baseWaitTime, maxTries = 6)=>{
if(!fh.syncHandle){
const t = performance.now();
log("Acquiring sync handle for",fh.filenameAbs);
const maxTries = 6,
msBase = state.asyncIdleWaitTime * 2;
const msBase = baseWaitTime ?? (state.asyncIdleWaitTime * 2);
maxTries ??= 6;
let i = 1, ms = msBase;
for(; true; ms = msBase * ++i){
try {
@@ -329,6 +368,9 @@ const installAsyncProxy = function(){
/**
Stores the given value at state.sabOPView[state.opIds.rc] and then
Atomics.notify()'s it.
The opName is only used for logging and debugging - all result
codes are expected on the same state.sabOPView slot.
*/
const storeAndNotify = (opName, value)=>{
log(opName+"() => notify(",value,")");
@@ -458,24 +500,12 @@ const installAsyncProxy = function(){
await releaseImplicitLock(fh);
storeAndNotify('xFileSize', rc);
},
xLock: async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/){
const fh = __openFiles[fid];
let rc = 0;
const oldLockType = fh.xLock;
fh.xLock = lockType;
if( !fh.syncHandle ){
try {
await getSyncHandle(fh,'xLock');
__implicitLocks.delete(fid);
}catch(e){
state.s11n.storeException(1,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_LOCK);
fh.xLock = oldLockType;
}
}
storeAndNotify('xLock',rc);
},
/**
The first argument is semantically invalid here - it's an
address in the synchronous side's heap. We can do nothing with
it here except use it as a unique-per-file identifier.
i.e. a lookup key.
*/
xOpen: async function(fid/*sqlite3_file pointer*/, filename,
flags/*SQLITE_OPEN_...*/,
opfsFlags/*OPFS_...*/){
@@ -533,7 +563,7 @@ const installAsyncProxy = function(){
rc = state.sq3Codes.SQLITE_IOERR_SHORT_READ;
}
}catch(e){
error("xRead() failed",e,fh);
//error("xRead() failed",e,fh);
state.s11n.storeException(1,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_READ);
}
@@ -560,15 +590,192 @@ const installAsyncProxy = function(){
affirmNotRO('xTruncate', fh);
await (await getSyncHandle(fh,'xTruncate')).truncate(size);
}catch(e){
error("xTruncate():",e,fh);
//error("xTruncate():",e,fh);
state.s11n.storeException(2,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_TRUNCATE);
}
await releaseImplicitLock(fh);
storeAndNotify('xTruncate',rc);
},
xUnlock: async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/){
xWrite: async function(fid/*sqlite3_file pointer*/,n,offset64){
let rc;
const fh = __openFiles[fid];
try{
affirmNotRO('xWrite', fh);
rc = (
n === (await getSyncHandle(fh,'xWrite'))
.write(fh.sabView.subarray(0, n),
{at: Number(offset64)})
) ? 0 : state.sq3Codes.SQLITE_IOERR_WRITE;
}catch(e){
//error("xWrite():",e,fh);
state.s11n.storeException(1,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_WRITE);
}
await releaseImplicitLock(fh);
storeAndNotify('xWrite',rc);
}
}/*vfsAsyncImpls*/;
if( isWebLocker ){
/* We require separate xLock() and xUnlock() implementations for the
original and Web Lock implementations. The ones in this block
are for the WebLock impl.
The Golden Rule for this impl is: if we have a web lock, we
must also hold the SAH. When "upgrading" an implicit lock to a
requested (explicit) lock, we must remove the SAH from the
__implicitLocks set. When we unlock, we release both the web
lock and the SAH. That invariant must be kept intact or race
conditions on SAHs will ensue.
*/
/** Registry of active Web Locks: fid -> { mode, resolveRelease } */
const __activeWebLocks = Object.create(null);
vfsAsyncImpls.xLock = async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/,
isFromUnlock/*only if called from this.xUnlock()*/){
const whichOp = isFromUnlock ? 'xUnlock' : 'xLock';
const fh = __openFiles[fid];
//error("xLock()",fid, lockType, isFromUnlock, fh);
const requestedMode = (lockType >= state.sq3Codes.SQLITE_LOCK_RESERVED)
? 'exclusive' : 'shared';
const existing = __activeWebLocks[fid];
if( existing ){
if( existing.mode === requestedMode
|| (existing.mode === 'exclusive'
&& requestedMode === 'shared') ) {
fh.xLock = lockType;
storeAndNotify(whichOp, 0);
/* Don't do this: existing.mode = requestedMode;
Paraphrased from advice given by a consulting developer:
If you hold an exclusive lock and SQLite requests shared,
you should keep exiting.mode as exclusive in because the
underlying Web Lock is still exclusive. Changing it to
shared would trick xLock into thinking it needs to
perform a release/re-acquire dance if an exclusive is
later requested.
*/
return 0 /* Already held at required or higher level */;
}
/*
Upgrade path: we must release shared and acquire exclusive.
This transition is NOT atomic in Web Locks API.
It _effectively_ is atomic if we don't call
closeSyncHandle(fh), as no other worker can lock that until
we let it go. But we can't do that without eventually
leading to deadly embrace situations, so we don't do that.
(That's not a hypothetical, it has happened.)
*/
await closeSyncHandle(fh);
existing.resolveRelease();
delete __activeWebLocks[fid];
}
const lockName = "sqlite3-vfs-opfs:" + fh.filenameAbs;
const oldLockType = fh.xLock;
return new Promise((resolveWaitLoop) => {
//error("xLock() initial promise entered...");
navigator.locks.request(lockName, { mode: requestedMode }, async (lock) => {
//error("xLock() Web Lock entered.", fh);
__implicitLocks.delete(fid);
let rc = 0;
try{
fh.xLock = lockType/*must be set before getSyncHandle() is called!*/;
await getSyncHandle(fh, 'xLock', state.asyncIdleWaitTime, 5);
}catch(e){
fh.xLock = oldLockType;
state.s11n.storeException(1, e);
rc = GetSyncHandleError.convertRc(e, state.sq3Codes.SQLITE_BUSY);
}
const releasePromise = rc
? undefined
: new Promise((resolveRelease) => {
__activeWebLocks[fid] = { mode: requestedMode, resolveRelease };
});
storeAndNotify(whichOp, rc) /* unblock the C side */;
resolveWaitLoop(0) /* unblock waitLoop() */;
await releasePromise /* hold the lock until xUnlock */;
});
});
};
/** Internal helper for the opfs-wl xUnlock() */
const wlCloseHandle = async(fh)=>{
let rc = 0;
try{
/* For the record, we've never once seen closeSyncHandle()
throw, nor should it because destructors do not throw. */
await closeSyncHandle(fh);
}catch(e){
state.s11n.storeException(1,e);
rc = state.sq3Codes.SQLITE_IOERR_UNLOCK;
}
return rc;
};
vfsAsyncImpls.xUnlock = async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/){
const fh = __openFiles[fid];
const existing = __activeWebLocks[fid];
if( !existing ){
const rc = await wlCloseHandle(fh);
storeAndNotify('xUnlock', rc);
return rc;
}
//error("xUnlock()",fid, lockType, fh);
let rc = 0;
if( lockType === state.sq3Codes.SQLITE_LOCK_NONE ){
/* SQLite usually unlocks all the way to NONE */
rc = await wlCloseHandle(fh);
existing.resolveRelease();
delete __activeWebLocks[fid];
fh.xLock = lockType;
}else if( lockType === state.sq3Codes.SQLITE_LOCK_SHARED
&& existing.mode === 'exclusive' ){
/* downgrade Exclusive -> Shared */
rc = await wlCloseHandle(fh);
if( 0===rc ){
fh.xLock = lockType;
existing.resolveRelease();
delete __activeWebLocks[fid];
return vfsAsyncImpls.xLock(fid, lockType, true);
}
}else{
/* ??? */
error("xUnlock() unhandled condition", fh);
}
storeAndNotify('xUnlock', rc);
return 0;
}
}else{
/* Original/"legacy" xLock() and xUnlock() */
vfsAsyncImpls.xLock = async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/){
const fh = __openFiles[fid];
let rc = 0;
const oldLockType = fh.xLock;
fh.xLock = lockType;
if( !fh.syncHandle ){
try {
await getSyncHandle(fh,'xLock');
__implicitLocks.delete(fid);
}catch(e){
state.s11n.storeException(1,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_LOCK);
fh.xLock = oldLockType;
}
}
storeAndNotify('xLock',rc);
};
vfsAsyncImpls.xUnlock = async function(fid/*sqlite3_file pointer*/,
lockType/*SQLITE_LOCK_...*/){
let rc = 0;
const fh = __openFiles[fid];
if( fh.syncHandle
@@ -582,173 +789,112 @@ const installAsyncProxy = function(){
}
}
storeAndNotify('xUnlock',rc);
},
xWrite: async function(fid/*sqlite3_file pointer*/,n,offset64){
let rc;
const fh = __openFiles[fid];
try{
affirmNotRO('xWrite', fh);
rc = (
n === (await getSyncHandle(fh,'xWrite'))
.write(fh.sabView.subarray(0, n),
{at: Number(offset64)})
) ? 0 : state.sq3Codes.SQLITE_IOERR_WRITE;
}catch(e){
error("xWrite():",e,fh);
state.s11n.storeException(1,e);
rc = GetSyncHandleError.convertRc(e,state.sq3Codes.SQLITE_IOERR_WRITE);
}
await releaseImplicitLock(fh);
storeAndNotify('xWrite',rc);
}
}/*vfsAsyncImpls*/;
const initS11n = ()=>{
/**
ACHTUNG: this code is 100% duplicated in the other half of this
proxy! The documentation is maintained in the "synchronous half".
*/
if(state.s11n) return state.s11n;
const textDecoder = new TextDecoder(),
textEncoder = new TextEncoder('utf-8'),
viewU8 = new Uint8Array(state.sabIO, state.sabS11nOffset, state.sabS11nSize),
viewDV = new DataView(state.sabIO, state.sabS11nOffset, state.sabS11nSize);
state.s11n = Object.create(null);
const TypeIds = Object.create(null);
TypeIds.number = { id: 1, size: 8, getter: 'getFloat64', setter: 'setFloat64' };
TypeIds.bigint = { id: 2, size: 8, getter: 'getBigInt64', setter: 'setBigInt64' };
TypeIds.boolean = { id: 3, size: 4, getter: 'getInt32', setter: 'setInt32' };
TypeIds.string = { id: 4 };
const getTypeId = (v)=>(
TypeIds[typeof v]
|| toss("Maintenance required: this value type cannot be serialized.",v)
);
const getTypeIdById = (tid)=>{
switch(tid){
case TypeIds.number.id: return TypeIds.number;
case TypeIds.bigint.id: return TypeIds.bigint;
case TypeIds.boolean.id: return TypeIds.boolean;
case TypeIds.string.id: return TypeIds.string;
default: toss("Invalid type ID:",tid);
}
};
state.s11n.deserialize = function(clear=false){
const argc = viewU8[0];
const rc = argc ? [] : null;
if(argc){
const typeIds = [];
let offset = 1, i, n, v;
for(i = 0; i < argc; ++i, ++offset){
typeIds.push(getTypeIdById(viewU8[offset]));
}
for(i = 0; i < argc; ++i){
const t = typeIds[i];
if(t.getter){
v = viewDV[t.getter](offset, state.littleEndian);
offset += t.size;
}else{/*String*/
n = viewDV.getInt32(offset, state.littleEndian);
offset += 4;
v = textDecoder.decode(viewU8.slice(offset, offset+n));
offset += n;
}
rc.push(v);
}
}
if(clear) viewU8[0] = 0;
//log("deserialize:",argc, rc);
return rc;
};
state.s11n.serialize = function(...args){
if(args.length){
//log("serialize():",args);
const typeIds = [];
let i = 0, offset = 1;
viewU8[0] = args.length & 0xff /* header = # of args */;
for(; i < args.length; ++i, ++offset){
/* Write the TypeIds.id value into the next args.length
bytes. */
typeIds.push(getTypeId(args[i]));
viewU8[offset] = typeIds[i].id;
}
for(i = 0; i < args.length; ++i) {
/* Deserialize the following bytes based on their
corresponding TypeIds.id from the header. */
const t = typeIds[i];
if(t.setter){
viewDV[t.setter](offset, args[i], state.littleEndian);
offset += t.size;
}else{/*String*/
const s = textEncoder.encode(args[i]);
viewDV.setInt32(offset, s.byteLength, state.littleEndian);
offset += 4;
viewU8.set(s, offset);
offset += s.byteLength;
}
}
//log("serialize() result:",viewU8.slice(0,offset));
}else{
viewU8[0] = 0;
}
};
state.s11n.storeException = state.asyncS11nExceptions
? ((priority,e)=>{
if(priority<=state.asyncS11nExceptions){
state.s11n.serialize([e.name,': ',e.message].join(""));
}
})
: ()=>{};
return state.s11n;
}/*initS11n()*/;
}/*xLock() and xUnlock() impls*/
const waitLoop = async function f(){
const opHandlers = Object.create(null);
for(let k of Object.keys(state.opIds)){
const vi = vfsAsyncImpls[k];
if(!vi) continue;
const o = Object.create(null);
opHandlers[state.opIds[k]] = o;
o.key = k;
o.f = vi;
if( !f.inited ){
f.inited = true;
f.opHandlers = Object.create(null);
for(let k of Object.keys(state.opIds)){
const vi = vfsAsyncImpls[k];
if(!vi) continue;
const o = Object.create(null);
f.opHandlers[state.opIds[k]] = o;
o.key = k;
o.f = vi;
}
}
const opIds = state.opIds;
const opView = state.sabOPView;
const slotWhichOp = opIds.whichOp;
const idleWaitTime = state.asyncIdleWaitTime;
const hasWaitAsync = !!Atomics.waitAsync;
//#if 0
error("waitLoop init: isWebLocker",isWebLocker,
"idleWaitTime",idleWaitTime,
"hasWaitAsync",hasWaitAsync);
//#/if
while(!flagAsyncShutdown){
try {
if('not-equal'!==Atomics.wait(
state.sabOPView, state.opIds.whichOp, 0, state.asyncIdleWaitTime
)){
/* Maintenance note: we compare against 'not-equal' because
let opId;
if( hasWaitAsync ){
opId = Atomics.load(opView, slotWhichOp);
if( 0===opId ){
const rv = Atomics.waitAsync(opView, slotWhichOp, 0,
idleWaitTime);
if( rv.async ) await rv.value;
await releaseImplicitLocks();
continue;
}
}else{
/**
For browsers without Atomics.waitAsync(), we require
the legacy implementation. Browser versions where
waitAsync() arrived:
https://github.com/tomayac/sqlite-wasm/issues/12
Chrome: 90 (2021-04-13)
Firefox: 145 (2025-11-11)
Safari: 16.4 (2023-03-27)
is reporting that this occasionally, under high loads,
returns 'ok', which leads to the whichOp being 0 (which
isn't a valid operation ID and leads to an exception,
along with a corresponding ugly console log
message). Unfortunately, the conditions for that cannot
be reliably reproduced. The only place in our code which
writes a 0 to the state.opIds.whichOp SharedArrayBuffer
index is a few lines down from here, and that instance
is required in order for clear communication between
the sync half of this proxy and this half.
The "opfs" VFS was not born until Chrome was somewhere in
the v104-108 range (Summer/Autumn 2022) and did not work
with Safari < v17 (2023-09-18) due to a WebKit bug which
restricted OPFS access from sub-Workers.
The waitAsync() counterpart of this block can be used by
both "opfs" and "opfs-wl", whereas this block can only be
used by "opfs". Performance comparisons between the two
in high-contention tests have been indecisive.
*/
await releaseImplicitLocks();
continue;
if('not-equal'!==Atomics.wait(
state.sabOPView, slotWhichOp, 0, state.asyncIdleWaitTime
)){
/* Maintenance note: we compare against 'not-equal' because
https://github.com/tomayac/sqlite-wasm/issues/12
is reporting that this occasionally, under high loads,
returns 'ok', which leads to the whichOp being 0 (which
isn't a valid operation ID and leads to an exception,
along with a corresponding ugly console log
message). Unfortunately, the conditions for that cannot
be reliably reproduced. The only place in our code which
writes a 0 to the state.opIds.whichOp SharedArrayBuffer
index is a few lines down from here, and that instance
is required in order for clear communication between
the sync half of this proxy and this half.
Much later (2026-03-07): that phenomenon is apparently
called a spurious wakeup.
*/
await releaseImplicitLocks();
continue;
}
opId = Atomics.load(state.sabOPView, slotWhichOp);
}
const opId = Atomics.load(state.sabOPView, state.opIds.whichOp);
Atomics.store(state.sabOPView, state.opIds.whichOp, 0);
const hnd = opHandlers[opId] ?? toss("No waitLoop handler for whichOp #",opId);
Atomics.store(opView, slotWhichOp, 0);
const hnd = f.opHandlers[opId]?.f ?? toss("No waitLoop handler for whichOp #",opId);
const args = state.s11n.deserialize(
true /* clear s11n to keep the caller from confusing this with
an exception string written by the upcoming
operation */
) || [];
//warn("waitLoop() whichOp =",opId, hnd, args);
if(hnd.f) await hnd.f(...args);
else error("Missing callback for opId",opId);
//error("waitLoop() whichOp =",opId, f.opHandlers[opId].key, args);
//#if 0
if( isWebLocker && (opId==opIds.xLock || opIds==opIds.xUnlock) ){
/* An expert suggests that this introduces a race condition,
but my eyes aren't seeing it. The hope was that this
would improve the lock speed a tick, but it does not
appear to. */
hnd(...args);
continue;
}
//#/if
await hnd(...args);
}catch(e){
error('in waitLoop():',e);
error('in waitLoop():', e);
}
}
};
@@ -756,6 +902,7 @@ const installAsyncProxy = function(){
navigator.storage.getDirectory().then(function(d){
state.rootDir = d;
globalThis.onmessage = function({data}){
//log(globalThis.location.href,"onmessage()",data);
switch(data.type){
case 'opfs-async-init':{
/* Receive shared state from synchronous partner */
@@ -771,6 +918,7 @@ const installAsyncProxy = function(){
}
});
initS11n();
//warn("verbosity =",opt.verbose, state.verbose);
log("init state",state);
wPost('opfs-async-inited');
waitLoop();
@@ -782,22 +930,27 @@ const installAsyncProxy = function(){
flagAsyncShutdown = false;
waitLoop();
}
break;
break;
}
};
wPost('opfs-async-loaded');
}).catch((e)=>error("error initializing OPFS asyncer:",e));
}/*installAsyncProxy()*/;
if(!globalThis.SharedArrayBuffer){
if(globalThis.window === globalThis){
wPost('opfs-unavailable',
"This code cannot run from the main thread.",
"Load it as a Worker from a separate Worker.");
}else if(!globalThis.SharedArrayBuffer){
wPost('opfs-unavailable', "Missing SharedArrayBuffer API.",
"The server must emit the COOP/COEP response headers to enable that.");
}else if(!globalThis.Atomics){
wPost('opfs-unavailable', "Missing Atomics API.",
"The server must emit the COOP/COEP response headers to enable that.");
}else if(isWebLocker && !globalThis.Atomics.waitAsync){
wPost('opfs-unavailable',"Missing required Atomics.waitSync() for "+vfsName);
}else if(!globalThis.FileSystemHandle ||
!globalThis.FileSystemDirectoryHandle ||
!globalThis.FileSystemFileHandle ||
!globalThis.FileSystemFileHandle.prototype.createSyncAccessHandle ||
!globalThis.FileSystemFileHandle?.prototype?.createSyncAccessHandle ||
!navigator?.storage?.getDirectory){
wPost('opfs-unavailable',"Missing required OPFS APIs.");
}else{
File diff suppressed because it is too large Load Diff
+17 -5
View File
@@ -55,6 +55,10 @@
*/
globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
'use strict';
if( sqlite3.config.disable?.vfs?.['opfs-sahpool'] ){
return;
}
const toss = sqlite3.util.toss;
const toss3 = sqlite3.util.toss3;
const initPromises = Object.create(null) /* cache of (name:result) of VFS init results */;
@@ -410,7 +414,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
}/*vfsMethods*/;
/**
Creates and initializes an sqlite3_vfs instance for an
Creates, initializes, and returns an sqlite3_vfs instance for an
OpfsSAHPool. The argument is the VFS's name (JS string).
Throws if the VFS name is already registered or if something
@@ -1157,8 +1161,9 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
described at the end of these docs.
This function accepts an options object to configure certain
parts but it is only acknowledged for the very first call and
ignored for all subsequent calls.
parts but it is only acknowledged for the very first call for
each distinct name and ignored for all subsequent calls with that
same name.
The options, in alphabetical order:
@@ -1224,7 +1229,14 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
- Paths given to it _must_ be absolute. Relative paths will not
be properly recognized. This is arguably a bug but correcting it
requires some hoop-jumping in routines which have no business
doing such tricks.
doing such tricks. (2026-01-19 (2.5 years later): the specifics
are lost to history, but this was a side effect of xOpen()
receiving an immutable C-string filename, to which no implicit
"/" can be prefixed without causing a discrepancy between what
the user provided and what the VFS stores. Its conceivable that
that quirk could be glossed over in xFullPathname(), but
regressions when doing so cannot be ruled out, so there are no
current plans to change this behavior.)
- It is possible to install multiple instances under different
names, each sandboxed from one another inside their own private
@@ -1459,4 +1471,4 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
The OPFS SAH Pool VFS parts are elided from builds targeting
node.js.
*/
//#endif target:node
//#/if target:node
+133
View File
@@ -0,0 +1,133 @@
//#if not target:node
/*
2026-02-20
The author disclaims copyright to this source code. In place of a
legal notice, here is a blessing:
* May you do good and not evil.
* May you find forgiveness for yourself and forgive others.
* May you share freely, never taking more than you give.
***********************************************************************
This file is a reimplementation of the "opfs" VFS (as distinct from
"opfs-sahpool") which uses WebLocks for locking instead of a bespoke
custom Atomics.wait()/notify() protocol. This file holds the
"synchronous half" of the VFS, whereas it shares the "asynchronous
half" of the "opfs" VFS.
This file is intended to be appended to the main sqlite3 JS
deliverable somewhere after sqlite3-api-oo1.js.
TODOs (2026-03-03):
- For purposes of tester1.js we need to figure out which of these
VFSes will install the (internal-use-only) sqlite3.opfs utility code
namespace. We need that in order to clean up OPFS files during test
runs. Alternately, move those into their own
sqlite3ApiBootstrap.initializers entry which precedes both of the
VFSes, so they'll have access to it during bootstrapping. The
sqlite3.opfs namespace is removed at the end of bootstrapping unless
the library is told to run in testing mode (which is not a
documented feature).
*/
'use strict';
globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
if( !sqlite3.opfs || sqlite3.config.disable?.vfs?.['opfs-wl'] ){
return;
}
const util = sqlite3.util,
toss = sqlite3.util.toss;
const opfsUtil = sqlite3.opfs;
const vfsName = 'opfs-wl';
/**
installOpfsWlVfs() returns a Promise which, on success, installs an
sqlite3_vfs named "opfs-wl", suitable for use with all sqlite3 APIs
which accept a VFS. It is intended to be called via
sqlite3ApiBootstrap.initializers or an equivalent mechanism.
This VFS is essentially identical to the "opfs" VFS but uses
WebLocks for its xLock() and xUnlock() implementations.
Quirks specific to this VFS:
- The (officially undocumented) 'opfs-wl-disable' URL
argument will disable OPFS, making this function a no-op.
Aside from locking differences in the VFSes, this function
otherwise behaves the same as
sqlite3-vfs-opfs.c-pp.js:installOpfsVfs().
*/
const installOpfsWlVfs = async function(options){
options = opfsUtil.initOptions(vfsName,options);
if( !options ) return sqlite3;
const capi = sqlite3.capi,
state = opfsUtil.createVfsState(),
opfsVfs = state.vfs,
metrics = opfsVfs.metrics.counters,
mTimeStart = opfsVfs.mTimeStart,
mTimeEnd = opfsVfs.mTimeEnd,
opRun = opfsVfs.opRun,
debug = (...args)=>sqlite3.config.debug(vfsName+":",...args),
warn = (...args)=>sqlite3.config.warn(vfsName+":",...args),
__openFiles = opfsVfs.__openFiles;
//debug("state",JSON.stringify(options));
/*
At this point, createVfsState() has populated:
- state: the configuration object we share with the async proxy.
- opfsVfs: an sqlite3_vfs instance with lots of JS state attached
to it.
with any code common to both the "opfs" and "opfs-wl" VFSes. Now
comes the VFS-dependent work...
*/
return opfsVfs.bindVfs(util.nu({
xLock: function(pFile,lockType){
mTimeStart('xLock');
//debug("xLock()...");
const f = __openFiles[pFile];
const rc = opRun('xLock', pFile, lockType);
if( !rc ) f.lockType = lockType;
mTimeEnd();
return rc;
},
xUnlock: function(pFile,lockType){
mTimeStart('xUnlock');
const f = __openFiles[pFile];
const rc = opRun('xUnlock', pFile, lockType);
if( !rc ) f.lockType = lockType;
mTimeEnd();
return rc;
}
}), function(sqlite3, vfs){
/* Post-VFS-registration initialization... */
if(sqlite3.oo1){
const OpfsWlDb = function(...args){
const opt = sqlite3.oo1.DB.dbCtorHelper.normalizeArgs(...args);
opt.vfs = vfs.$zName;
sqlite3.oo1.DB.dbCtorHelper.call(this, opt);
};
OpfsWlDb.prototype = Object.create(sqlite3.oo1.DB.prototype);
sqlite3.oo1.OpfsWlDb = OpfsWlDb;
OpfsWlDb.importDb = opfsUtil.importDb;
/* The "opfs" VFS variant adds a
oo1.DB.dbCtorHelper.setVfsPostOpenCallback() callback to set
a high busy_timeout. That was a design mis-decision and is
inconsistent with sqlite3_open() and friends, but is retained
against the risk of introducing regressions if it's removed.
This variant does not repeat that mistake.
*/
}
})/*bindVfs()*/;
}/*installOpfsWlVfs()*/;
globalThis.sqlite3ApiBootstrap.initializersAsync.push(async (sqlite3)=>{
return installOpfsWlVfs().catch((e)=>{
sqlite3.config.warn("Ignoring inability to install the",vfsName,"sqlite3_vfs:",e);
});
});
}/*sqlite3ApiBootstrap.initializers.push()*/);
//#/if target:node
File diff suppressed because it is too large Load Diff
+1 -4
View File
@@ -172,10 +172,7 @@ globalThis.sqlite3ApiBootstrap.initializers.push(function(sqlite3){
Works like unget() plus it calls dispose() on the
StructType object.
*/
dispose: (pCObj)=>{
const o = __xWrap(pCObj,true);
if(o) o.dispose();
}
dispose: (pCObj)=>__xWrap(pCObj,true)?.dispose?.()
});
};
+140 -141
View File
@@ -229,15 +229,18 @@
** not by client code, so an argument can be made for reducing their
** visibility by not including them in any build-time export lists.
**
** 2022-09-11: it's not yet _proven_ that this approach works in
** non-Emscripten builds. If not, such builds will need to export
** those using the --export=... wasm-ld flag (or equivalent). As of
** this writing we are tied to Emscripten for various reasons
** and cannot test the library with other build environments.
** 2025-12-01: for use in non-Emscripten builds, we need a more
** invasive macro which explicitly names the export:
** SQLITE_WASM_EXPORT2.
*/
#define SQLITE_WASM_EXPORT __attribute__((used,visibility("default")))
// See also:
//__attribute__((export_name("theExportedName"), used, visibility("default")))
#define SQLITE_WASM_EXPORT_NAMED(X) __attribute__((export_name(#X),used,visibility("default")))
#define SQLITE_WASM_EXPORT2(RETTYPE,NAME,SIG) SQLITE_WASM_EXPORT_NAMED(NAME) RETTYPE NAME SIG
#if 1
/** Increase the kvvfs key size limit from 32. */
#define KVRECORD_KEY_SZ 128
#endif
/*
** Which sqlite3.c we're using needs to be configurable to enable
@@ -261,45 +264,6 @@
#undef INC__STRINGIFY
#undef SQLITE_C
#if 0
/*
** An EXPERIMENT in implementing a stack-based allocator analog to
** Emscripten's stackSave(), stackAlloc(), stackRestore().
** Unfortunately, this cannot work together with Emscripten because
** Emscripten defines its own native one and we'd stomp on each
** other's memory. Other than that complication, basic tests show it
** to work just fine.
**
** Another option is to malloc() a chunk of our own and call that our
** "stack".
*/
SQLITE_WASM_EXPORT void * sqlite3__wasm_stack_end(void){
extern void __heap_base
/* see https://stackoverflow.com/questions/10038964 */;
return &__heap_base;
}
SQLITE_WASM_EXPORT void * sqlite3__wasm_stack_begin(void){
extern void __data_end;
return &__data_end;
}
static void * pWasmStackPtr = 0;
SQLITE_WASM_EXPORT void * sqlite3__wasm_stack_ptr(void){
if(!pWasmStackPtr) pWasmStackPtr = sqlite3__wasm_stack_end();
return pWasmStackPtr;
}
SQLITE_WASM_EXPORT void sqlite3__wasm_stack_restore(void * p){
pWasmStackPtr = p;
}
SQLITE_WASM_EXPORT void * sqlite3__wasm_stack_alloc(int n){
if(n<=0) return 0;
n = (n + 7) & ~7 /* align to 8-byte boundary */;
unsigned char * const p = (unsigned char *)sqlite3__wasm_stack_ptr();
unsigned const char * const b = (unsigned const char *)sqlite3__wasm_stack_begin();
if(b + n >= p || b + n < b/*overflow*/) return 0;
return pWasmStackPtr = p - n;
}
#endif /* stack allocator experiment */
/*
** State for the "pseudo-stack" allocator implemented in
** sqlite3__wasm_pstack_xyz(). In order to avoid colliding with
@@ -348,10 +312,10 @@ SQLITE_WASM_EXPORT void sqlite3__wasm_pstack_restore(unsigned char * p){
** the memory on success, 0 on error (including a negative n value). n
** is always adjusted to be a multiple of 8 and returned memory is
** always zeroed out before returning (because this keeps the client
** JS code from having to do so, and most uses of the pstack will
** call for doing so).
** JS code from having to do so, and most uses of the pstack call for
** doing so).
*/
SQLITE_WASM_EXPORT void * sqlite3__wasm_pstack_alloc(int n){
SQLITE_WASM_EXPORT2(void *,sqlite3__wasm_pstack_alloc,(int n)){
if( n<=0 ) return 0;
n = (n + 7) & ~7 /* align to 8-byte boundary */;
if( PStack.pBegin + n > PStack.pPos /*not enough space left*/
@@ -363,7 +327,7 @@ SQLITE_WASM_EXPORT void * sqlite3__wasm_pstack_alloc(int n){
** Return the number of bytes left which can be
** sqlite3__wasm_pstack_alloc()'d.
*/
SQLITE_WASM_EXPORT int sqlite3__wasm_pstack_remaining(void){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_pstack_remaining,(void)){
assert(PStack.pPos >= PStack.pBegin);
assert(PStack.pPos <= PStack.pEnd);
return (int)(PStack.pPos - PStack.pBegin);
@@ -374,7 +338,7 @@ SQLITE_WASM_EXPORT int sqlite3__wasm_pstack_remaining(void){
** any space which is currently allocated. This value is a
** compile-time constant.
*/
SQLITE_WASM_EXPORT int sqlite3__wasm_pstack_quota(void){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_pstack_quota,(void)){
return (int)(PStack.pEnd - PStack.pBegin);
}
@@ -387,8 +351,7 @@ struct WasmTestStruct {
void (*xFunc)(void*);
};
typedef struct WasmTestStruct WasmTestStruct;
SQLITE_WASM_EXPORT
void sqlite3__wasm_test_struct(WasmTestStruct * s){
SQLITE_WASM_EXPORT2(void,sqlite3__wasm_test_struct,(WasmTestStruct * s)){
if(s){
if( 0 ){
/* Do not be alarmed by the small (and odd) pointer values.
@@ -428,8 +391,7 @@ void sqlite3__wasm_test_struct(WasmTestStruct * s){
** fails to compile with "tables may not be 64-bit" but does not tell
** us where it's happening.
*/
SQLITE_WASM_EXPORT
const char * sqlite3__wasm_enum_json(void){
SQLITE_WASM_EXPORT2(const char *,sqlite3__wasm_enum_json,(void)){
static char aBuffer[1024 * 20] =
{0} /* where the JSON goes. 2025-09-19: output size=19295, but
that can vary slightly from build to build, so a little
@@ -609,6 +571,7 @@ const char * sqlite3__wasm_enum_json(void){
DefInt(SQLITE_DBCONFIG_ENABLE_ATTACH_WRITE);
DefInt(SQLITE_DBCONFIG_ENABLE_COMMENTS);
DefInt(SQLITE_DBCONFIG_MAX);
DefInt(SQLITE_DBCONFIG_FP_DIGITS);
} _DefGroup;
DefGroup(dbStatus){
@@ -632,6 +595,7 @@ const char * sqlite3__wasm_enum_json(void){
DefGroup(encodings) {
/* Noting that the wasm binding only aims to support UTF-8. */
DefInt(SQLITE_UTF8);
DefInt(SQLITE_UTF8_ZT);
DefInt(SQLITE_UTF16LE);
DefInt(SQLITE_UTF16BE);
DefInt(SQLITE_UTF16);
@@ -735,6 +699,8 @@ const char * sqlite3__wasm_enum_json(void){
DefInt(SQLITE_MAX_TRIGGER_DEPTH);
DefInt(SQLITE_LIMIT_WORKER_THREADS);
DefInt(SQLITE_MAX_WORKER_THREADS);
DefInt(SQLITE_LIMIT_PARSER_DEPTH);
DefInt(SQLITE_MAX_PARSER_DEPTH);
} _DefGroup;
DefGroup(openFlags) {
@@ -768,6 +734,7 @@ const char * sqlite3__wasm_enum_json(void){
DefInt(SQLITE_PREPARE_PERSISTENT);
DefInt(SQLITE_PREPARE_NORMALIZE);
DefInt(SQLITE_PREPARE_NO_VTAB);
DefInt(SQLITE_PREPARE_FROM_DDL);
} _DefGroup;
DefGroup(resultCodes) {
@@ -1019,13 +986,15 @@ const char * sqlite3__wasm_enum_json(void){
/** ^^^ indirection needed to expand CurrentStruct */
#define StructBinder StructBinder_(CurrentStruct)
#define _StructBinder CloseBrace(2)
#define M(MEMBER,SIG) \
outf("%s\"%s\": " \
"{\"offset\":%d,\"sizeof\": %d,\"signature\":\"%s\"}", \
(n++ ? ", " : ""), #MEMBER, \
(int)offsetof(CurrentStruct,MEMBER), \
(int)sizeof(((CurrentStruct*)0)->MEMBER), \
SIG)
#define M3(MEMBER,SIG,READONLY) \
outf("%s\"%s\": " \
"{\"offset\":%d,\"sizeof\":%d,\"signature\":\"%s\"%s}", \
(n++ ? ", " : ""), #MEMBER, \
(int)offsetof(CurrentStruct,MEMBER), \
(int)sizeof(((CurrentStruct*)0)->MEMBER), \
SIG, (READONLY ? ",\"readOnly\":true" : ""))
#define M(MEMBER,SIG) M3(MEMBER,SIG,0)
#define MRO(MEMBER,SIG) M3(MEMBER,SIG,1)
nStruct = 0;
out(", \"structs\": ["); {
@@ -1088,11 +1057,30 @@ const char * sqlite3__wasm_enum_json(void){
#undef CurrentStruct
#define CurrentStruct sqlite3_kvvfs_methods
/* From os_kv.c */
StructBinder {
M(xRead, "i(sspi)");
M(xWrite, "i(sss)");
M(xDelete, "i(ss)");
M(nKeySize, "i");
M(xRcrdRead, "i(sspi)");
M(xRcrdWrite, "i(sss)");
M(xRcrdDelete, "i(ss)");
MRO(nKeySize, "i");
MRO(nBufferSize, "i");
M(pVfs, "p");
M(pIoDb, "p");
M(pIoJrnl, "p");
} _StructBinder;
#undef CurrentStruct
#define CurrentStruct KVVfsFile
/* From os_kv.c */
StructBinder {
M(base, "p")/*sqlite3_file base*/;
M(zClass, "s");
M(isJournal, "i");
M(nJrnl, "i")/*actually unsigned!*/;
M(aJrnl, "p");
M(szPage, "i");
M(szDb, "j");
M(aData, "p");
} _StructBinder;
#undef CurrentStruct
@@ -1149,7 +1137,13 @@ const char * sqlite3__wasm_enum_json(void){
** sqlite3_index_info, we have to uplift those into constructs we
** can access by type name. These structs _must_ match their
** in-sqlite3_index_info counterparts byte for byte.
*/
**
** 2025-11-21: this uplifing is no longer necessary, as Jaccwabyt
** can now handle nested structs, but "it ain't broke" so there's
** no pressing need to rewire this. Also, it's conceivable that
** rewiring it might break downstream vtab impls, so it shouldn't
** be rewired.
*/
typedef struct {
int iColumn;
unsigned char op;
@@ -1244,6 +1238,8 @@ const char * sqlite3__wasm_enum_json(void){
#undef StructBinder_
#undef StructBinder__
#undef M
#undef MRO
#undef M3
#undef _StructBinder
#undef CloseBrace
#undef out
@@ -1261,8 +1257,7 @@ const char * sqlite3__wasm_enum_json(void){
** method, SQLITE_MISUSE is returned, else the result of the xDelete()
** call is returned.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_vfs_unlink(sqlite3_vfs *pVfs, const char *zName){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_vfs_unlink,(sqlite3_vfs *pVfs, const char *zName)){
int rc = SQLITE_MISUSE /* ??? */;
if( 0==pVfs && 0!=zName ) pVfs = sqlite3_vfs_find(0);
if( zName && pVfs && pVfs->xDelete ){
@@ -1279,8 +1274,7 @@ int sqlite3__wasm_vfs_unlink(sqlite3_vfs *pVfs, const char *zName){
** defaulting to "main" if zDbName is 0. Returns 0 if no db with the
** given name is open.
*/
SQLITE_WASM_EXPORT
sqlite3_vfs * sqlite3__wasm_db_vfs(sqlite3 *pDb, const char *zDbName){
SQLITE_WASM_EXPORT2(sqlite3_vfs *,sqlite3__wasm_db_vfs,(sqlite3 *pDb, const char *zDbName)){
sqlite3_vfs * pVfs = 0;
sqlite3_file_control(pDb, zDbName ? zDbName : "main",
SQLITE_FCNTL_VFS_POINTER, &pVfs);
@@ -1302,8 +1296,7 @@ sqlite3_vfs * sqlite3__wasm_db_vfs(sqlite3 *pDb, const char *zDbName){
** Returns 0 on success, an SQLITE_xxx code on error. Returns
** SQLITE_MISUSE if pDb is NULL.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_db_reset(sqlite3 *pDb){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_db_reset,(sqlite3 *pDb)){
int rc = SQLITE_MISUSE;
if( pDb ){
sqlite3_table_column_metadata(pDb, "main", 0, 0, 0, 0, 0, 0, 0);
@@ -1384,10 +1377,10 @@ int sqlite3__wasm_db_export_chunked( sqlite3* pDb,
** If `*pOut` is not NULL, the caller is responsible for passing it to
** sqlite3_free() to free it.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_db_serialize( sqlite3 *pDb, const char *zSchema,
unsigned char **pOut,
sqlite3_int64 *nOut, unsigned int mFlags ){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_db_serialize,
(sqlite3 *pDb, const char *zSchema,
unsigned char **pOut,
sqlite3_int64 *nOut, unsigned int mFlags)){
unsigned char * z;
if( !pDb || !pOut ) return SQLITE_MISUSE;
if( nOut ) *nOut = 0;
@@ -1448,11 +1441,9 @@ int sqlite3__wasm_db_serialize( sqlite3 *pDb, const char *zSchema,
** portability, so that the API can still work in builds where BigInt
** support is disabled or unavailable.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_vfs_create_file( sqlite3_vfs *pVfs,
const char *zFilename,
const unsigned char * pData,
int nData ){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_vfs_create_file,
(sqlite3_vfs *pVfs, const char *zFilename,
const unsigned char * pData, int nData)){
int rc;
sqlite3_file *pFile = 0;
sqlite3_io_methods const *pIo;
@@ -1538,10 +1529,9 @@ int sqlite3__wasm_vfs_create_file( sqlite3_vfs *pVfs,
** zFilename, appends pData bytes to it, and returns 0 on success or
** SQLITE_IOERR on error.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_posix_create_file( const char *zFilename,
const unsigned char * pData,
int nData ){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_posix_create_file,
(const char *zFilename, const unsigned char * pData,
int nData)){
int rc;
FILE * pFile = 0;
int fileExisted = 0;
@@ -1561,22 +1551,30 @@ int sqlite3__wasm_posix_create_file( const char *zFilename,
** This function is NOT part of the sqlite3 public API. It is strictly
** for use by the sqlite project's own JS/WASM bindings.
**
** Allocates sqlite3KvvfsMethods.nKeySize bytes from
** sqlite3__wasm_pstack_alloc() and returns 0 if that allocation fails,
** else it passes that string to kvstorageMakeKey() and returns a
** NUL-terminated pointer to that string. It is up to the caller to
** use sqlite3__wasm_pstack_restore() to free the returned pointer.
** This returns either a pointer to a static buffer or zKeyIn directly
** (if zClass is NULL or empty).
*/
SQLITE_WASM_EXPORT
char * sqlite3__wasm_kvvfsMakeKeyOnPstack(const char *zClass,
const char *zKeyIn){
SQLITE_WASM_EXPORT2(const char *,sqlite3__wasm_kvvfsMakeKey,
(const char *zClass, const char *zKeyIn)){
static char buf[SQLITE_KVOS_SZ+1] = {0};
assert(sqlite3KvvfsMethods.nKeySize>24);
char *zKeyOut =
(char *)sqlite3__wasm_pstack_alloc(sqlite3KvvfsMethods.nKeySize);
if(zKeyOut){
kvstorageMakeKey(zClass, zKeyIn, zKeyOut);
if( zClass && *zClass ){
kvrecordMakeKey(zClass, zKeyIn, buf);
return buf;
}else{
#if 1
/* We can return zKeyIn here only because the JS API takes special
** care with its lifetime.*/
return zKeyIn;
#else
/* It would be nice to be able to return zKeyIn directly here, but
** it may have been allocated as part of the automated JS-to-WASM
** conversions, in which case it will be freed before reaching the
** caller. */
sqlite3_snprintf(KVRECORD_KEY_SZ, buf, "%s", zKeyIn);
return buf;
#endif
}
return zKeyOut;
}
/*
@@ -1586,8 +1584,7 @@ char * sqlite3__wasm_kvvfsMakeKeyOnPstack(const char *zClass,
** Returns the pointer to the singleton object which holds the kvvfs
** I/O methods and associated state.
*/
SQLITE_WASM_EXPORT
sqlite3_kvvfs_methods * sqlite3__wasm_kvvfs_methods(void){
SQLITE_WASM_EXPORT2(sqlite3_kvvfs_methods *,sqlite3__wasm_kvvfs_methods,(void)){
return &sqlite3KvvfsMethods;
}
@@ -1602,8 +1599,8 @@ sqlite3_kvvfs_methods * sqlite3__wasm_kvvfs_methods(void){
** sqlite3_vtab_config(), or SQLITE_MISUSE if the 2nd arg is not a
** valid value.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_vtab_config(sqlite3 *pDb, int op, int arg){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_vtab_config,
(sqlite3 *pDb, int op, int arg)){
switch(op){
case SQLITE_VTAB_DIRECTONLY:
case SQLITE_VTAB_INNOCUOUS:
@@ -1623,8 +1620,8 @@ int sqlite3__wasm_vtab_config(sqlite3 *pDb, int op, int arg){
** Wrapper for the variants of sqlite3_db_config() which take
** (int,int*) variadic args.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_db_config_ip(sqlite3 *pDb, int op, int arg1, int* pArg2){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_db_config_ip,
(sqlite3 *pDb, int op, int arg1, int* pArg2)){
switch(op){
case SQLITE_DBCONFIG_ENABLE_FKEY:
case SQLITE_DBCONFIG_ENABLE_TRIGGER:
@@ -1647,6 +1644,7 @@ int sqlite3__wasm_db_config_ip(sqlite3 *pDb, int op, int arg1, int* pArg2){
case SQLITE_DBCONFIG_ENABLE_ATTACH_CREATE:
case SQLITE_DBCONFIG_ENABLE_ATTACH_WRITE:
case SQLITE_DBCONFIG_ENABLE_COMMENTS:
case SQLITE_DBCONFIG_FP_DIGITS:
return sqlite3_db_config(pDb, op, arg1, pArg2);
default: return SQLITE_MISUSE;
}
@@ -1659,8 +1657,9 @@ int sqlite3__wasm_db_config_ip(sqlite3 *pDb, int op, int arg1, int* pArg2){
** Wrapper for the variants of sqlite3_db_config() which take
** (void*,int,int) variadic args.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_db_config_pii(sqlite3 *pDb, int op, void * pArg1, int arg2, int arg3){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_db_config_pii,
(sqlite3 *pDb, int op, void * pArg1, int arg2,
int arg3)){
switch(op){
case SQLITE_DBCONFIG_LOOKASIDE:
return sqlite3_db_config(pDb, op, pArg1, arg2, arg3);
@@ -1675,8 +1674,8 @@ int sqlite3__wasm_db_config_pii(sqlite3 *pDb, int op, void * pArg1, int arg2, in
** Wrapper for the variants of sqlite3_db_config() which take
** (const char *) variadic args.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_db_config_s(sqlite3 *pDb, int op, const char *zArg){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_db_config_s,(sqlite3 *pDb, int op,
const char *zArg)){
switch(op){
case SQLITE_DBCONFIG_MAINDBNAME:
return sqlite3_db_config(pDb, op, zArg);
@@ -1692,8 +1691,7 @@ int sqlite3__wasm_db_config_s(sqlite3 *pDb, int op, const char *zArg){
** Binding for combinations of sqlite3_config() arguments which take
** a single integer argument.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_config_i(int op, int arg){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_config_i,(int op, int arg)){
return sqlite3_config(op, arg);
}
@@ -1704,8 +1702,7 @@ int sqlite3__wasm_config_i(int op, int arg){
** Binding for combinations of sqlite3_config() arguments which take
** two int arguments.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_config_ii(int op, int arg1, int arg2){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_config_ii,(int op, int arg1, int arg2)){
return sqlite3_config(op, arg1, arg2);
}
@@ -1716,8 +1713,7 @@ int sqlite3__wasm_config_ii(int op, int arg1, int arg2){
** Binding for combinations of sqlite3_config() arguments which take
** a single i64 argument.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_config_j(int op, sqlite3_int64 arg){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_config_j,(int op, sqlite3_int64 arg)){
return sqlite3_config(op, arg);
}
@@ -1729,8 +1725,7 @@ int sqlite3__wasm_config_j(int op, sqlite3_int64 arg){
** sqlite3_mprintf()'s %Q modifier (if addQuotes is true) or %q (if
** addQuotes is 0). Returns NULL if z is NULL or on OOM.
*/
SQLITE_WASM_EXPORT
char * sqlite3__wasm_qfmt_token(char *z, int addQuotes){
SQLITE_WASM_EXPORT2(char *,sqlite3__wasm_qfmt_token,(char *z, int addQuotes)){
char * rc = 0;
if( z ){
rc = addQuotes
@@ -1740,6 +1735,21 @@ char * sqlite3__wasm_qfmt_token(char *z, int addQuotes){
return rc;
}
/*
** This function is NOT part of the sqlite3 public API. It is strictly
** for use by the sqlite project's own JS/WASM bindings.
**
** A WASM wrapper for the interal os_kv.c:kvvfsDecode() for internal
** use by the kvvfs v2 API.
*/
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_kvvfs_decode,(const char *a, char *aOut, int nOut)){
return kvvfsDecode(a, aOut, nOut);
}
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_kvvfs_encode,(const char *a, int nA, char *aOut)){
return kvvfsEncode(a, nA, aOut);
}
#if defined(__EMSCRIPTEN__) && defined(SQLITE_ENABLE_WASMFS)
#include <emscripten/console.h>
#include <emscripten/wasmfs.h>
@@ -1765,8 +1775,7 @@ char * sqlite3__wasm_qfmt_token(char *z, int addQuotes){
** the virtual FS fails. In builds compiled without SQLITE_ENABLE_WASMFS
** defined, SQLITE_NOTFOUND is returned without side effects.
*/
SQLITE_WASM_EXPORT
int sqlite3__wasm_init_wasmfs(const char *zMountPoint){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_init_wasmfs,(const char *zMountPoint)){
static backend_t pOpfs = 0;
if( !zMountPoint || !*zMountPoint ) zMountPoint = "/opfs";
if( !pOpfs ){
@@ -1785,8 +1794,7 @@ int sqlite3__wasm_init_wasmfs(const char *zMountPoint){
return pOpfs ? 0 : SQLITE_NOMEM;
}
#else
SQLITE_WASM_EXPORT
int sqlite3__wasm_init_wasmfs(const char *zUnused){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_init_wasmfs,(const char *zUnused)){
//emscripten_console_warn("WASMFS OPFS is not compiled in.");
(void)zUnused;
return SQLITE_NOTFOUND;
@@ -1795,52 +1803,43 @@ int sqlite3__wasm_init_wasmfs(const char *zUnused){
#if SQLITE_WASM_ENABLE_C_TESTS
SQLITE_WASM_EXPORT
int sqlite3__wasm_test_intptr(int * p){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_test_intptr,(int * p)){
return *p = *p * 2;
}
SQLITE_WASM_EXPORT
void * sqlite3__wasm_test_voidptr(void * p){
SQLITE_WASM_EXPORT2(void *,sqlite3__wasm_test_voidptr,(void * p)){
return p;
}
SQLITE_WASM_EXPORT
int64_t sqlite3__wasm_test_int64_max(void){
SQLITE_WASM_EXPORT2(int64_t,sqlite3__wasm_test_int64_max,(void)){
return (int64_t)0x7fffffffffffffff;
}
SQLITE_WASM_EXPORT
int64_t sqlite3__wasm_test_int64_min(void){
SQLITE_WASM_EXPORT2(int64_t,sqlite3__wasm_test_int64_min,(void)){
return ~sqlite3__wasm_test_int64_max();
}
SQLITE_WASM_EXPORT
int64_t sqlite3__wasm_test_int64_times2(int64_t x){
SQLITE_WASM_EXPORT2(int64_t,sqlite3__wasm_test_int64_times2,(int64_t x)){
return x * 2;
}
SQLITE_WASM_EXPORT
void sqlite3__wasm_test_int64_minmax(int64_t * min, int64_t *max){
SQLITE_WASM_EXPORT2(void,sqlite3__wasm_test_int64_minmax,(int64_t * min, int64_t *max)){
*max = sqlite3__wasm_test_int64_max();
*min = sqlite3__wasm_test_int64_min();
/*printf("minmax: min=%lld, max=%lld\n", *min, *max);*/
}
SQLITE_WASM_EXPORT
int64_t sqlite3__wasm_test_int64ptr(int64_t * p){
SQLITE_WASM_EXPORT2(int64_t,sqlite3__wasm_test_int64ptr,(int64_t * p)){
/*printf("sqlite3__wasm_test_int64ptr( @%lld = 0x%llx )\n", (int64_t)p, *p);*/
return *p = *p * 2;
}
SQLITE_WASM_EXPORT
void sqlite3__wasm_test_stack_overflow(int recurse){
SQLITE_WASM_EXPORT2(void,sqlite3__wasm_test_stack_overflow,(int recurse)){
if(recurse) sqlite3__wasm_test_stack_overflow(recurse);
}
/* For testing the 'string:dealloc' whwasmutil.xWrap() conversion. */
SQLITE_WASM_EXPORT
char * sqlite3__wasm_test_str_hello(int fail){
SQLITE_WASM_EXPORT2(char *,sqlite3__wasm_test_str_hello,(int fail)){
char * s = fail ? 0 : (char *)sqlite3_malloc(6);
if(s){
memcpy(s, "hello", 5);
@@ -1954,8 +1953,8 @@ static int sqlite3__wasm_SQLTester_strnotglob(const char *zGlob, const char *z){
return *z==0;
}
SQLITE_WASM_EXPORT
int sqlite3__wasm_SQLTester_strglob(const char *zGlob, const char *z){
SQLITE_WASM_EXPORT2(int,sqlite3__wasm_SQLTester_strglob,
(const char *zGlob, const char *z)){
return !sqlite3__wasm_SQLTester_strnotglob(zGlob, z);
}
+20 -14
View File
@@ -19,10 +19,12 @@
slightly simpler client-side interface than the slightly-lower-level
Worker API does.
This script necessarily exposes one global symbol, but clients may
freely `delete` that symbol after calling it.
In non-ESM builds this file necessarily exposes one global symbol,
but clients may freely `delete` that symbol after calling it.
*/
//#if not defined target:es6-module
'use strict';
//#/if
/**
Configures an sqlite3 Worker API #1 Worker such that it can be
manipulated via a Promise-based interface and returns a factory
@@ -109,10 +111,12 @@
the callback is called one time for each row of the result set,
passed the same worker message format as the worker API emits:
{type:typeString,
{
type:typeString,
row:VALUE,
rowNumber:1-based-#,
columnNames: array}
columnNames: array
}
Where `typeString` is an internally-synthesized message type string
used temporarily for worker message dispatching. It can be ignored
@@ -123,10 +127,9 @@
callback.
At the end of the result set, the same event is fired with
(row=undefined, rowNumber=null) to indicate that
the end of the result set has been reached. Note that the rows
arrive via worker-posted messages, with all the implications
of that.
(row=undefined, rowNumber=null) to indicate that the end of the
result set has been reached. The rows arrive via worker-posted
messages, with all the implications of that.
Notable shortcomings:
@@ -257,7 +260,9 @@ globalThis.sqlite3Worker1Promiser.defaultConfig = {
type: 'module'
});
//#elif target:es6-module
return new Worker(new URL("sqlite3-worker1.js", import.meta.url));
return new Worker(new URL("sqlite3-worker1.mjs", import.meta.url),{
type: 'module'
});
//#else
let theJs = "sqlite3-worker1.js";
if(this.currentScript){
@@ -273,15 +278,15 @@ globalThis.sqlite3Worker1Promiser.defaultConfig = {
}
}
return new Worker(theJs + globalThis.location.search);
//#endif
//#/if
}
//#if not target:es6-module
.bind({
currentScript: globalThis?.document?.currentScript
})
//#endif
//#/if
,
onerror: (...args)=>console.error('worker1 promiser error',...args)
onerror: (...args)=>console.error('sqlite3Worker1Promiser():',...args)
}/*defaultConfig*/;
/**
@@ -343,7 +348,8 @@ globalThis.sqlite3Worker1Promiser.v2.defaultConfig =
incompatibility.
*/
export default sqlite3Worker1Promiser.v2;
//#endif /* target:es6-module */
delete globalThis.sqlite3Worker1Promiser;
//#/if /* target:es6-module */
//#else
/* Built with the omit-oo1 flag. */
//#endif if not omit-oo1
//#/if if not omit-oo1
+4 -4
View File
@@ -33,9 +33,9 @@
directory from which `sqlite3.js` will be loaded.
*/
//#if target:es6-bundler-friendly
import {default as sqlite3InitModule} from './sqlite3-bundler-friendly.mjs';
import sqlite3InitModule from './sqlite3-bundler-friendly.mjs';
//#elif target:es6-module
return new Worker(new URL("sqlite3.js", import.meta.url));
import sqlite3InitModule from './sqlite3.mjs';
//#else
"use strict";
{
@@ -49,8 +49,8 @@ import {default as sqlite3InitModule} from './sqlite3-bundler-friendly.mjs';
//console.warn("worker1 theJs =",theJs);
importScripts(theJs);
}
//#endif
//#/if
sqlite3InitModule().then(sqlite3 => sqlite3.initWorker1API());
//#else
/* Built with the omit-oo1 flag. */
//#endif if not omit-oo1
//#/if if not omit-oo1
-2804
View File
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -44,7 +44,8 @@
/** abort() if expr is false. If expr is a function, it
is called and its result is evaluated.
*/
assert: function f(expr, msg){
assert: function f(expr, ...msg){
msg = msg?.join?.(' ');
if(!f._){
f._ = ('undefined'===typeof abort
? (msg)=>{throw new Error(msg)}
+4 -4
View File
@@ -16,7 +16,7 @@
*/
'use strict';
(function(){
const T = self.SqliteTestUtil;
const T = globalThis.SqliteTestUtil;
const toss = function(...args){throw new Error(args.join(' '))};
const debug = console.debug.bind(console);
const eOutput = document.querySelector('#test-output');
@@ -40,7 +40,7 @@
const error = function(...args){
logHtml('error',...args);
};
const runTests = function(sqlite3){
const capi = sqlite3.capi,
oo = sqlite3.oo1,
@@ -51,7 +51,7 @@
error("This build is not kvvfs-capable.");
return;
}
const dbStorage = 0 ? 'session' : 'local';
const theStore = 's'===dbStorage[0] ? sessionStorage : localStorage;
const db = new oo.JsStorageDb( dbStorage );
@@ -108,7 +108,7 @@
}
};
sqlite3InitModule(self.sqlite3TestModule).then((sqlite3)=>{
sqlite3InitModule(globalThis.sqlite3TestModule).then((sqlite3)=>{
runTests(sqlite3);
});
})();
+6 -6
View File
@@ -6,11 +6,11 @@
<link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon">
<link rel="stylesheet" href="common/emscripten.css"/>
<link rel="stylesheet" href="common/testing.css"/>
//#if target=es6-module
<title>worker-promise (via ESM) tests</title>
//#if target:es6-module
<title>Worker1-promiser (ESM) tests</title>
//#else
<title>worker-promise tests</title>
//#endif
<title>Worker1-promiser tests</title>
//#/if
</head>
<body>
<header id='titlebar'><span>worker-promise tests</span></header>
@@ -32,11 +32,11 @@
<hr>
<div id='test-output'></div>
<script src="common/SqliteTestUtil.js"></script>
//#if target=es6-module
//#if target:es6-module
<script src="demo-worker1-promiser.mjs" type="module"></script>
//#else
<script src="jswasm/sqlite3-worker1-promiser.js"></script>
<script src="demo-worker1-promiser.js"></script>
//#endif
//#/if
</body>
</html>
+2 -2
View File
@@ -19,7 +19,7 @@ import {default as promiserFactory} from "./jswasm/sqlite3-worker1-promiser.mjs"
"use strict";
const promiserFactory = globalThis.sqlite3Worker1Promiser.v2;
delete globalThis.sqlite3Worker1Promiser;
//#endif
//#/if
(async function(){
const T = globalThis.SqliteTestUtil;
const eOutput = document.querySelector('#test-output');
@@ -53,7 +53,7 @@ delete globalThis.sqlite3Worker1Promiser;
before workerPromise is set. */
console.warn("This is the v2 interface - you don't need an onready() function.");
},
//#endif
//#/if
debug: 1 ? undefined : (...args)=>console.debug('worker debug',...args),
onunhandled: function(ev){
error("Unhandled worker message:",ev.data);
+3 -3
View File
@@ -18,7 +18,7 @@
*/
'use strict';
(function(){
const T = self.SqliteTestUtil;
const T = globalThis.SqliteTestUtil;
const SW = new Worker("jswasm/sqlite3-worker1.js");
const DbState = {
id: undefined
@@ -323,7 +323,7 @@
switch(ev.result){
case 'worker1-ready':
log("Message:",ev);
self.sqlite3TestModule.setStatus(null);
globalThis.sqlite3TestModule.setStatus(null);
runTests();
return;
default:
@@ -344,5 +344,5 @@
};
log("Init complete, but async init bits may still be running.");
log("Installing Worker into global scope SW for dev purposes.");
self.SW = SW;
globalThis.SW = SW;
})();
-4
View File
@@ -175,10 +175,6 @@
"features (e.g. upload) do not yet work with OPFS.");
}
stdout('\nEnter ".help" for usage hints.');
this.exec([ // initialization commands...
'.nullvalue NULL',
'.headers on'
].join('\n'));
return true;
},
/**
+1 -1
View File
@@ -27,7 +27,7 @@
-->
<script src="jqterm/jquery.terminal.bundle.min.js"></script>
<link rel="stylesheet" href="jqterm/jquery.terminal.min.css">
//#endif
//#/if
<style>
/* The following styles are for app-level use. */
:root {

Some files were not shown because too many files have changed in this diff Show More