The command line shell for SQLite can be customized to modify or add certain kinds of features, without altering its source. This document details this extensibility and how such extension may be accomplished by shell users.
Shell extensibility serves to reduce the tension between keeping the shell simple with broadly useful features and allowing the shell to become an ever-growing tool meeting diverse needs for those not ready to satisfy them with a custom program using the SQLite libary. Extensions contributed by the SQLite developers or others will become (or are) available for use in situations that may not justify permanently adding the same features to the core shell published by the SQLite project in binary form.
Only certain categories of features may be added or modified via extension, namely:
The "dot" commands implemented by the shell, along with help text for them, may be augmented or overridden. A meta-command effected with an extension may be used in the same ways as one available in the core, non-extended shell.
The ways in which data may be imported to a DB table may be augmented or overriden. Once that is done, the .import meta-command will either have a new option to specify a new import handler or an existing .import option can be overridden for this.
The display formatting or other disposition of query results may be augmented or overriden. Once that is done, the .mode meta-command will either have a new option to specify a new result handler or an existing .mode option can be overridden for this.
New handlers added via extension are not restricted to importing data from a file or result display formatting. They may be considered more generally to be either a data source or a data sink, producing or accepting data row sets. The origin or destination of the data is up to the handler, as may be affected by arguments to the .import or most recent .mode command.
Shell extension may be effected at different times in different ways according to convenience and need.
Extension at runtime (when the shell is running) is effected via the .load command with a --shell flag. In this way, a dynamically loaded library (DLL) is loaded with provision made for its sqlite3_X_init() function to obtain the shell extension API entry points and thereby register with the shell core any new meta-commands, or import or query result handlers that it implements.
Extension when the shell is built is effected by specifying certain option values to either the "make" invocation or to a utility, (tool/mkshellc.tcl), which assembles and transforms sources to produce the shell's unitary source file (shell.c).
With certain coding conventions and methods followed, the same source code can be used either to produce most of a runtime shell extension DLL or to be incorporated into shell.c as a built-in shell extension. (See the extension code samples for details.)
Just as new meta-commands can be readily incorporated into the shell when it is built, many of the core meta-commands can be readily omitted from the shell build. This is done with a variation of the option values that may be given to "make" or to tool/mkshellc.tcl as the shell is built. (See tool/mkshellc.tcl --help output for details.) Such omission of meta-commands might be done when building a customized shell which need not have the various meta-commands which exist for the purpose of testing the SQLite library.
The following details relate to src/shext_linkage.h, a header in which declarations appear for objects and functions that facilitate runtime interaction between the shell core and shell extensions written in C/C++. Comments in that header tersely summarize these explanations:
The ShellStateX object, (maintained by the shell and known as shellState), consists of a public portion, which is available and stable for use in shell/extension interactions, and a private portion which may not be stable. Shell extension code used only for build-time extension might use the private part, (to which it has access because such code is compiled within the same translation unit as the core shell code), but such usage generally precludes (or makes hazardous) use of runtime loadable extensions built from such code.
An object of this type serves to uniquely identify an extension loaded at runtime so that it may be unloaded later. It must be passed back to the shell (in the ShellExtensionLink eid member) by the sqlite3_X_init() function if the extension DLL is ever to be unloaded during that shell session.
An object of this type is passed between the shell core and its import or query result handlers to: (1) convey or keep parameters and data related to formatting or parsing data rows in an external form; (2) keep state associated with the progression of an import or result handling operation from initiation to completion; (3) to facilitate access to exposed shell state generally useful to such handlers or meta-commands; or (4) to provide for abnormal shell exits.
The shell core .mode and .import implementations also use the same instance of this type to affect result output and import operations. That instance resides in a ShellStateX object kept by the shell so that extension meta-commands can access it, possibly to change it. Meta-commands or handlers which alter this instance for their own purposes (rather than for intended effect) should take care to restore its prior value as the operation completes.
The function addressed by this member will be called prior to the extension being unloaded (if the pointer is non-zero.) This is an out parameter from the sqlite3_X_init() function. It may perform any cleanup or deallocations necessitated by successful initialization generally (and will never be called after failed initialization.)
The objects registered with the shell core to provided extension functionality may be implemented in C or C++ (or anything else producing the same ABI.) In the below descriptions of their interfaces, it should be understood that: C++ implementations need not explicitly deal with anything like a Whatsit_Vtable struct and will refer to the object pointer, passed implicitly, as "this"; and C implementations will need to populate a static Whatsit_Vtable and refer to the initial object pointer as "pThis".
All shell extension interfaces have a method, destruct(), which is (or may be) called by the shell core prior to deactivating any registered meta-command, output result or import handler. This call will be made in addition to any automatic (or implicit) takedown that may occur due to atexit() or C++ destructor calls, so destruct()'s responsibility should be limited to reversing the per-registered-object effects of sqlite3_X_init().
A registered object is deactivated when either: the extension is immanently going to be unloaded; or the registered object is being overridden by some like-named object (such that it can no longer be reached by the core shell.)
These objects represent an extension meta-command, including a dispatch table for the public interface and any accompanying data (which is opaque to the core shell.) Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerMetaCommand() calls.
These objects represent the dispatch table of a MetaCommand object.
All methods are given the same leading (or lone) argument:
(1) the address of the registered MetaCommand object.
This method is called prior to unloading a runtime extension for any registered MetaCommand object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the meta-command (sans leading '.'.) The returned pointer must remain valid throughout the lifetime of the registered MetaCommand object.
This method returns help text for the meta-command. This text should be formatted and aligned with the built-in meta-command help text so that it can be displayed seamlessly.
There is one additional argument:
(2) an integer directing what help text to return, with
0 indicating primary, single-line help, or
1 indicating any more detailed help beyond the primary level.
The return is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the MetaCommand object.
This method returns a pair of unsigned integers indicating the range of valid argument counts for the meta-command.
The returned .minArgs value indicates the minimum number of arguments required for a successful execute call. The returned .maxArgs value indicates the maximum number of arguments allowed for a successful execute call, with ~0 indicating no (practical) upper limit. The execute method will never be called with an argument count not within this range. (This significantly simplifies argument checking.)
This method performs whatever work the meta-command is supposed
to do when invoked. It has 4 additional arguments:
(2) present ShellStateX, passed by reference (an in/out parameter);
(3) an error message pointer, passed by reference, set upon error but
otherwise not modified, to be freed by the shell core;
(4) the number of invocation arguments;
(5) an array of C strings constituting the invocation arguments;
The return is 0 for success, or anything else to indicate error. The special value SHELL_INVALID_ARGS (aka SQLITE_MISUSE) should be returned when argument checking shows an invalid call. This will cause the command dispatcher to issue usage help (in interactive shell sessions) without further fuss by the execute() method itself. (And if an extension causes this error to be returned from the SQLite library, which is a serious error by itself, it must either translate it for return or trouble its users with a misleading error message from the dispatcher.) If the output error message is set, it will be issued in lieu of the standard error message for invalid calls.
These objects represent an extension query result handler, including a dispatch table for the public interface and any accompanying data which is opaque to the core shell. Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerOutMode() calls.
These objects represent the dispatch table of an OutModeHandler object.
All methods in the dispatch table are given at
least this leading argument:
(1) The OutModeHandler address registered via registerOutMode();
This method is called prior to unloading a runtime extension for any registered OutModeHandler object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the OutModeHandler, which users specify to the .mode command (as the mode's name) to designate use of the registered OutModeHandler for subsequent query results. The returned pointer must remain valid throughout the lifetime of the registered MetaCommand object.
This method returns help text for the OutModeHandler.
There is one additional argument:
(2) an integer directing what help text to return, with
0 indicating primary, single-line help, or
1 indicating any more detailed help beyond the primary level.
The primary help is included in the .mode command's own
detailed help text, so it should be aligned accordingly.
The detailed help is shown by the .mode command's --help option.
The return is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the MetaCommand object.
The following methods are given these 2 additional arguments:
(2) A ShellExState object passed by reference; and
(3) An error message pointer, passed by reference, to receive errors.
This method is called when a query output is setup, (via the .mode command with the handler's name given as a --flag.)
It is given 3 additional arguments:
(4) the number of arguments in the said .mode command;
(5) an array of C strings with the said argument values; and
(6) the name given as a --flag which caused the handler to be used.
When the extension handler is activated via a .mode command, parsing that command and behaving accordingly is the responsibility of this method alone. (The command is parsed and acted upon by the default .mode implementation only when no OutModeHandler is used.)
Once this method is called and succeeds, it is guaranteed that the closeResultsOutStream method will be called.
This method should return SQLITE_OK only upon success. Any other return will abort remaining calls in the handling sequence.
This method is called when a query succeeds for which this handler will be given the results. This is purely preparatory; zero or more result rows may follow. It is up to this method to determine if any results can even be had (by considering the return from sqlite3_column_count()) and acting accordingly.
It is given 1 additional argument:
(4) the query prepared statement (pointer), not yet stepped.
This method should return SQLITE_OK only upon success. Any other return will abort the next 2 calls in the handling sequence. (TBD: Such aborts without error should be supported for DDL and DML.)
This method is called when a query's prepared statement is stepped, for each result row available.
It is given 1 additional argument:
(4) the query prepared statement (pointer), stepped with
a result row available. It is permitted for this method to
perform the remaining stepping, as indicated by the return.
This method should return SQLITE_OK only upon success without having completed stepping. If this method completes stepping, (by making its own calls to sqlite3_step() until it returns SQLITE_DONE), it must return SQLITE_DONE. Any other return will abort the next call in the handling sequence.
This method is called when result set stepping has been completed.
It is given additional argument:
(4) the query prepared statement (pointer), completely stepped.
This method should return SQLITE_OK upon success, or may return something else to indicate error with no effect upon succeeding calls.
This method is called when a new .mode command is invoked specifying some output mode not using this OutModeHandler or when the extension is about to be unloaded with this OutModeHandler selected for output. It should free resources allocated or held as a result of the previous openResultsOutStream call.
These objects represent an extension data import handler, including a dispatch table for the public interface and any accompanying data which is opaque to the core shell. Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerImporter() calls.
These objects represent the dispatch table of an ImportHandler object.
All methods in the dispatch table are given this 1 leading argument:
(1) The ImportHandler address registered via registerImporter().
This method is called prior to unloading a runtime extension for any registered OutModeHandler object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the importer, which is to be passed with leading '--' (or '-') to the .import command to specify use of the registered importer for that invocation. The returned pointer must remain valid throughout the lifetime of the registered MetaCommand object.
This method returns help text for the importer.
There is one additional argument:
(2) an integer directing what help text to return, with
0 indicating primary, single-line help, or
1 indicating any more detailed help beyond the primary level.
The primary help is included in the .import command's own
detailed help text, so it should be aligned accordingly.
The detailed help is shown by the .import command's --help option.
The return is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the MetaCommand object.
The following methods are given these 2 additional arguments:
(2) A ShellExState object passed by reference; and
(3) An error message pointer, passed by reference, to receive errors.
This method is called when a .import command is invoked with the '--' (or '-') prefixed name of the ImportHandler as an argument.
These 3 additional arguments are passed:
(4) the number of arguments to said .import command;
(5) an array of C strings with the said argument values; and
(6) the name of the ImportHandler (as its name method would return.)
When the extension handler is activated via a .import command, responsibility for parsing that command and behaving accordingly is shared between the shell core .import implementation and this method. The shell core normally takes the last argument as the name of a table (to be created if necessary) which will receive the imported data. (But see return values for exceptions to this treatment.) The shell core also detects any --flag argument selecting a registered ImportHandler (giving effect to the last one) and calls this and following methods to perform the input part of the import operation. This method may interpret any or all of the arguments as needed and (supposedly) documented by the associated help(...) method returns.
Once this method is called and succeeds, it is guaranteed that the closeDataInStream method will be called.
This method should return SQLITE_OK upon success where disposition of the imported data (into a table named by the last argument) is to be handled normally, by the shell core. An alternative success return is SQLITE_DONE, indicating that disposition of the imported data has been done by the ImportHandler. In that case, the next 3 methods (*DataInput(...)) will not be called and this method should have completed the whole import operation. After a success return, closeDataInStream is guaranteed to be called. Other returns will abort all remaining calls in the handling sequence.
This method prepares for the particular import operation commenced with a .import invocation. It may take into account the shape of the input data (and possibly its type) as discovered during the call.
This 1 additional argument is passed:
(4) a to-be-prepared statement (pointer), passed by reference.
This is an out parameter conveying a prepared statement created
by this method specifically for the single import operation.
The prepared statement should, in some manner wholly determined by the extension handler, incorporate compiled SQL (possibly with as-yet unbound parameters) which will (or may) produce a result set. (This may be no more than a query such as "SELECT @1 as one, ...", or could be a SELECT from a temporary table used for buffering.)
The return should be SQLITE_OK upon success, in which case the following 2 methods will also be called. Any other return will abort the {prepare,row,finish}DataInput() call sequence. In that case, no prepared statement should be returned either.
This method is called to collect imported data, making it available through the prepared statement passed as the last argument with as many steps as are needed to get SQLITE_DONE from sqlite3_step(). It will be called repeatedly until its return indicates no more data.
It is given 1 additional argument:
(4) the prepared statement (pointer) returned by prepareDataInput().
This prepared statement can have values bound to it or be reset as
necessary to return some or more data. However, it must remain the
same sqlite3_statement instance returned by prepareDataInput().
The return should be SQLITE_DONE when no more data is available, or SQLITE_ROW to indicate that more data is or might be available. Any other return (including SQLITE_OK) indicates a condition which will abort the data collection phase of the import operation. Any of those 3 returns is treated as success by the shell core.
This method is called to complete the data input phase of the import operation commenced by prepareDataInput().
It is given 1 additional argument:
(4) the prepared statement (pointer) returned by prepareDataInput().
This prepared statement should be finalized by this method. Other
cleanup or import wrap-up related to the transfer may also be performed.
This method is called as a .import command which specified this ImportHandler completes. It should free resources allocated or held as a result of the previous openDataInStream call.
An object of this type is passed (somewhat indirectly) to the sqlite3_X_init(...) function which is called when a runtime extension is loaded via a .load command with a --shell flag. It is used to establish linkage between the loaded extension and a shell core API exposed specifically for extensibility.
At present, the extension API is limited to registration of meta-commands, query result handlers, and import handlers implemented by extensions. This API may be extended in future versions of the core shell, in a backwards-compatible manner. (See the pExtra sentinel, whose offset may increase.)
The 1st or 2nd parameter passed to the sqlite3_X_init() function as an extension is dynamically loaded is used to obtain a pointer to the ShellExtensionLink object (provided that .load was invoked with the --shell flag.) To verify that the correct object type was passed and reference it if so, the extension's sqlite3_X_init() function can use one of these two means:
The least code-intensive means is implemented by a macro, EXTENSION_LINKAGE_PTR(pzem), parameter of which is the char** passed as the 2nd sqlite3_X_init() argument. This is reliable except in the case of a maliciously written shell core (which portends worse problems than the undefined behavior which could arise from invoking .load from such a shell.)
A more code-intensive means, (which is no safer in the case of a maliciously written shell core), is to use a C macro, DEFINE_SHDB_TO_SHEXT_API(function_name), in the extension source to define a function named per the macro argument. Then that function may be called with its lone argument being the sqlite3 * (db) pointer which was passed as the 1st argument to sqlite3_X_init().
Either means will return (or yield) either a null pointer or a verified ShellExtensionLink pointer. In the former case, sqlite3_X_init() may return SHELL_INVALID_ARGS to induce the emission of help on using the .load command. (Said help will mention the need to use the --shell flag for extensions.) In the latter case, the obtained pointer may be called to register the extension's feature implementor(s) as described.
There is critical tension between competing uses of the ShellState object which is kept and used by the core shell for state which must persist between meta-command invocations. For unhindered implementation and feature expansion flexibility, the structure of the ShellState data should be unconstrained across versions of the shell. However, unless extension meta-commands and data transferers are to act entirely independently of built-in meta-commands, (excepting interaction through the above-described extension methods), some portion of the ShellState data needs to be shared in a stable manner between extensions and the core shell code.
With respect to interface stability concerns, it is nearly immaterial whether the sharing occurs through exposed data structures or an extension API devised to convey similar data. (Only data layout and possible change notification are at stake in that choice.)
For simplicity, and because it can work in a way consistent with how built-in shell features are implemented now, the chosen sharing method is to simply directly expose a subset of the ShellState data. That subset will be extremely limited to minimize hinderance of what has previously been unfettered change to that data structure/meaning. It will initially be: the present output stream, as affected by .output and .once commands; the currently open user DB (if any); the dedicated shell DB; and the data related to formatting and transfer of data in external forms. This data resides in the ShellStateX object as a ShellExState struct.
As shell extensibility evolves, additional data items may need to move into the publicly exposed portion of the ShellStateX object, either directly (via exposed data members) or by means of additional extension APIs defined in the ShellExtensionLink object (which has been defined to accommodate growth of its function pointer list in a backwards-compatible manner.)