Add file doc/vdbesort-memory.md, containing notes on the way vdbesort.c uses heap memory.

FossilOrigin-Name: 07f53899a929cce93f17c0332819610c9517c7e54e00092af20efd746cf84438
This commit is contained in:
dan
2020-11-20 14:16:41 +00:00
parent 6fc74a4454
commit 6a5e21149b
3 changed files with 57 additions and 8 deletions
+49
View File
@@ -0,0 +1,49 @@
20-11-2020
# Memory Allocation In vdbesort.c
Memory allocation is slightly different depending on:
* whether or not SQLITE_CONFIG_SMALL_MALLOC is set, and
* whether or not worker threads are enabled.
## SQLITE_CONFIG_SMALL_MALLOC=0
Assuming SQLITE_CONFIG_SMALL_MALLOC is not set, keys passed to the sorter are
added to an in-memory buffer. This buffer is grown using sqlite3Realloc() as
required it reaches the size configured for the main pager cache using "PRAGMA
cache_size". i.e. if the user has executed "PRAGMA main.cache_size = -2048",
then this buffer is allowed to grow up to 2MB in size.
Once the buffer has grown to its threshold, keys are sorted and written to
a temp file. If worker threads are not enabled, this is the only significant
allocation the sorter module makes. After keys are sorted and flushed out to
the temp file, the buffer is reused to accumulate the next batch of keys.
If worker threads are available, then the buffer is passed to a worker thread
to sort and flush once it is full, and a new buffer allocated to allow the
main thread to continue to accumulate keys. Buffers are reused once they
have been flushed, so in this case at most (nWorker+1) buffers are allocated
and used, where nWorker is the number of configured worker threads.
There are no other significant users of heap memory in the sorter module.
Once sorted buffers of keys have been flushed to disk, they are read back
either by mapping the file (via sqlite3_file.xFetch()) or else read back
in one page at a time.
All buffers are allocated by the main thread. A sorter object is associated
with a single database connection, to which it holds a pointer.
## SQLITE_CONFIG_SMALL_MALLOC=1
This case is similar to the above, except that instead of accumulating
multiple keys in a single large buffer, sqlite3VdbeSorterWrite() stores
keys in a regular heap-memory linked list (one allocation per element).
List elements are freed as they are flushed to disk, either by the main
thread or by a worker thread.
Each time a key is added the sorter (and an allocation made),
sqlite3HeapNearlyFull() is called. If it returns true, the current
list of keys is flushed to a temporary file, even if it has not yet
reached the size threshold.
+7 -7
View File
@@ -1,5 +1,5 @@
C The\sxFullPathname\smethod\sof\sthe\sunix\sVFS\snow\snormalizes\sthe\sfilenames\sby\nomitting\ssurplus\s"/",\s"/./"\sand\s"/../"\sstrings.
D 2020-11-19T21:12:08.560
C Add\sfile\sdoc/vdbesort-memory.md,\scontaining\snotes\son\sthe\sway\svdbesort.c\suses\sheap\smemory.
D 2020-11-20T14:16:41.814
F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724
@@ -41,6 +41,7 @@ F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd
F doc/lemon.html c5d8ba85ac1daef7be8c2d389899480eb62451ff5c09b0c28ff8157bb8770746
F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a
F doc/wal-lock.md 781726aaba20bafeceb7ba9f91d5c98c6731691b30c954e37cf0b49a053d461d
F ext/README.md fd5f78013b0a2bc6f0067afb19e6ad040e89a10179b4f6f03eee58fac5f169bd
@@ -1885,8 +1886,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
P 842c432772e6cd8464cdb7bfdb38789adeea9aa9e0486d4034cc9841f085f517 6f0d5f0949130a4688319adf05d37c5ccd683ca9ad2dacb7ce433a5fe18ac3c3
R 9bca7f34bcf3572651c390350ebafdef
T +closed 6f0d5f0949130a4688319adf05d37c5ccd683ca9ad2dacb7ce433a5fe18ac3c3
U drh
Z 8c74c867e3ffdcd79c4ff4d10ad118b3
P c38dec6f52c01614c1bee8356daf0fcd9f708d029116e9bff51e06719a730dde
R b9b1a27eff3b385f6d301a9808892950
U dan
Z d17ac9b00e741eb4707d9deb9c3699e7
+1 -1
View File
@@ -1 +1 @@
c38dec6f52c01614c1bee8356daf0fcd9f708d029116e9bff51e06719a730dde
07f53899a929cce93f17c0332819610c9517c7e54e00092af20efd746cf84438