The command line shell for SQLite can be customized to modify or add certain kinds of features, without altering its source. This document details this extensibility and how such extension may be accomplished by shell users.
Shell extensibility serves to reduce the tension between keeping the shell simple with broadly useful features and allowing the shell to become an ever-growing tool meeting diverse needs for those not ready to satisfy them with a custom program using the SQLite libary. Extensions contributed by the SQLite developers or others will become (or are) available for use in situations that may not justify permanently adding the same features to the core shell published by the SQLite project in binary form.
Only certain categories of features may be added or modified via extension, namely:
The "dot" commands implemented by the shell, along with help text for them, may be augmented or overridden. A meta-command effected with an extension may be used in the same ways as one available in the core, non-extended shell.
The ways in which data may be imported to a DB table may be augmented or overriden. Once that is done, the .import meta-command will either have a new option to specify a new import handler or an existing .import option can be overridden for this.
The display formatting or other disposition of query results may be augmented or overriden. Once that is done, the .mode meta-command will either have a new option to specify a new result handler or an existing .mode option can be overridden for this.
New handlers added via extension are not restricted to importing data from a file or result display formatting. They may be considered more generally to be either a data source or a data sink, producing or accepting data row sets. The origin or destination of the data is up to the handler, as may be affected by arguments to the .import or most recent .mode command.
An shell extension may provide for integration of a scripting subsystem with the shell. Such a subsystem may participate in classification of input, extending the recognized classes from { meta-command SQL-statement(s) comment } to add { script }, and then handle input in the "script" input class. The shell exposes some its internal data and code to script implementations so that they may act in a manner providing adaptation or enhancement of shell features rather than entirely independent new features.
Shell extension may be effected at different times in different ways according to convenience and need.
Extension at runtime (when the shell is running) is effected via the .shxload command. In this way, a dynamically loaded library (DLL) is loaded with provision made for its sqlite3_X_init() function to obtain the shell extension API entry points and thereby register with the shell core any new meta-commands, or import or query result handlers or scripting support that it implements.
Extension when the shell is built is effected by specifying certain option values to either the "make" invocation or to a utility, (tool/mkshellc.tcl), which assembles and transforms sources to produce the shell's unitary source file (shell.c).
With certain coding conventions and methods followed, the same source code can be used either to produce most of a runtime shell extension DLL or to be incorporated into shell.c as a built-in shell extension. (See the extension code samples for details. (Not fully implemented.))
Just as new meta-commands can be readily incorporated into the shell when it is built, many of the core meta-commands can be readily omitted from the shell build. This is done with a variation of the option values that may be given to "make" or to tool/mkshellc.tcl as the shell is built. (See tool/mkshellc.tcl --help output for details.) Such omission of meta-commands might be done when building a customized shell which need not have the various meta-commands which exist for the purpose of testing the SQLite library.
The following details relate to src/shext_linkage.h, a header in which declarations appear for objects and functions that facilitate runtime interaction between the shell core and shell extensions written in C/C++. Comments in that header tersely summarize these explanations:
The Shell{In,Ex}State objects, are maintained by the shell and known as shell state. The shell state consists of a public portion, which is available and stable for use in shell/extension interactions, and a private portion which may not be stable. Shell extension code used only for build-time extension might use the private part, (to which it has access because such code is compiled within the same translation unit as the core shell code), but such usage generally precludes (or makes hazardous) use of runtime loadable extensions built from such code.
An object of this type serves to uniquely identify an extension loaded at runtime so that it may be unloaded later. It must be passed back to the shell (in the ShellExtensionLink eid member) by the sqlite3_X_init() function if the extension DLL is ever to be unloaded during that shell session.
An object of this type is passed between the shell core and its import or query result handlers to: (1) convey or keep parameters and data related to formatting or parsing data rows in an external form; (2) keep state associated with the progression of an import or result handling operation from initiation to completion; (3) to facilitate access to exposed shell state generally useful to such handlers or meta-commands; or (4) to provide for abnormal shell exits.
The shell core .mode and .import implementations also use the same instance of this type to affect result output and import operations. That instance resides in a ShellStateX object kept by the shell so that extension meta-commands can access it, possibly to change it. Meta-commands or handlers which alter this instance for their own purposes (rather than for intended effect) should take care to restore its prior value as the operation completes.
The function addressed by this member will be called prior to exit or the extension being unloaded (if the pointer is non-zero.) This is an out parameter from the sqlite3_X_init() function. It may perform any cleanup or deallocations necessitated by successful initialization generally (and will never be called after failed initialization.)
The objects registered with the shell core to provided extension functionality may be implemented in C or C++ (or anything else presenting the same ABI.) In the below descriptions of their interfaces, it should be understood that: C++ implementations need not explicitly deal with anything like a Whatsit_Vtable struct and will refer to the object pointer, passed implicitly, as "this"; and C implementations will need to populate a static Whatsit_Vtable and refer to the initial object pointer as "pThis".
All shell extension interfaces have a method, destruct(), which is (or may be) called by the shell core prior to deactivating any registered meta-command, output result or import handler. This call will be made in addition to any automatic (or implicit) takedown that may occur due to atexit() or C++ destructor calls, so destruct()'s responsibility should be limited to reversing the per-registered-object effects of sqlite3_X_init().
A registered object is deactivated when either: the extension is immanently going to be unloaded; the registered object is being overridden by some like-named object (such that it can no longer be reached by the core shell); or the shell is about to return or exit.
These objects represent an extension meta-command, including a dispatch table for the public interface and any accompanying data (which is opaque to the core shell.) Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerDotCommand() calls.
These objects represent the dispatch table of a DotCommand object.
All methods are given the same leading (or lone) argument:
(1) the address of the registered DotCommand object.
This method is called prior to unloading a runtime extension for any registered DotCommand object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the meta-command (sans leading '.'.) The returned pointer must remain valid throughout the lifetime of the registered DotCommand object.
This method returns help text for the meta-command. This text should be formatted and aligned with the built-in meta-command help text so that it can be displayed seamlessly.
There is one additional argument:
(2) a const char * directing what help text to return, with
0 indicating primary, single-line help, or
&"" indicating more detailed help beyond the primary level, or
&"(other)" reserved for expansion.
The return for the first 2 forms is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the DotCommand object.
This method returns a DotCmdRC value indicating the validity (or invalidity) of arguments for the meta-command, using the coding scheme defined for the DotCmdRC type.
There are three additional arguments:
(2) a char** to possibly receive an error message, which the
caller must eventually pass to sqlite3_free().
(3) a count of all provided arguments; and
(4) a char* array of given arguments, led by meta-command's name.
This method performs whatever work the meta-command is supposed
to do when invoked. It has 4 additional arguments:
(2) present ShellExState, passed by reference (an in/out parameter);
(3) an error message pointer, passed by reference, set upon error but
otherwise not modified, to be freed by the shell core;
(4) the number of invocation arguments; and
(5) an array of C strings constituting the invocation arguments;
The return is a DotCmdRC value, indicating success, error, or other disposition as documented for the DotCmdRC type.
These objects represent an extension query result handler, including a dispatch table for the public interface and any accompanying data which is opaque to the core shell. Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerExporter() calls.
These objects represent the dispatch table of an ExportHandler object.
All methods in the dispatch table are given at
least this leading argument:
(1) The ExportHandler address registered via registerExporter();
This method is called prior to the shell's return or exit or unloading the runtime extension for a registered ExportHandler object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the ExportHandler, which users specify to the .mode command (as the mode's name) to designate use of the registered ExportHandler for subsequent query results. The returned pointer must remain valid throughout the lifetime of the registered ExportHandler object.
This method returns help text for the ExportHandler.
There is one additional argument:
(2) a const char * directing what help text to return, with
0 indicating primary, single-line help, or
&"" indicating more detailed help beyond the primary level, or
&"(other)" reserved for expansion.
The primary help is included in the .mode command's own detailed help text, so it should be aligned accordingly. The detailed help is shown by the .mode command's --help option.
The return is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the ExportHandler object.
The following methods are given these 2 additional arguments:
(2) A ShellExState object passed by reference; and
(3) An error message pointer, passed by reference, to receive errors.
This method is called when a query output is setup, (via the .mode command with the handler's name given as a --flag.)
It is given 3 additional arguments:
(4) the number of arguments in the said .mode command;
(5) an array of C strings with the said argument values; and
(6) the name given as a --flag which caused the handler to be used.
When the extension handler is activated via a .mode command, parsing that command and behaving accordingly is the responsibility of this method alone. (The command is parsed and acted upon by the default .mode implementation only when no ExportHandler is used.)
Once this method is called and succeeds, it is guaranteed that the closeResultsOutStream method will be called.
This method should return SQLITE_OK only upon success. Any other return will abort remaining calls in the handling sequence.
This method is called when a query succeeds for which this handler will be given the results. This is purely preparatory; zero or more result rows may follow. It is up to this method to determine if any results can even be had (by considering the return from sqlite3_column_count()) and acting accordingly.
It is given 1 additional argument:
(4) the query prepared statement (pointer), not yet stepped.
This method should return SQLITE_OK only upon success. Any other return will abort the next 2 calls in the handling sequence. (TBD: Such aborts without error should be supported for DDL and DML.)
This method is called when a query's prepared statement is stepped, for each result row available.
It is given 1 additional argument:
(4) the query prepared statement (pointer), stepped with
a result row available. It is permitted for this method to
perform the remaining stepping, as indicated by the return.
This method should return SQLITE_OK only upon success without having completed stepping. If this method completes stepping, (by making its own calls to sqlite3_step() until it returns SQLITE_DONE), it must return SQLITE_DONE. Any other return will avoid the next call in the handling sequence.
This method is called when result set stepping has been completed.
It is given additional argument:
(4) the query prepared statement (pointer), completely stepped.
This method should return SQLITE_OK upon success, or may return something else to indicate error with no effect upon succeeding calls.
This method is called when a new .mode command is invoked specifying some output mode not using this ExportHandler or when the extension is about to be unloaded with this ExportHandler selected for output. It should free resources allocated or held as a result of the previous openResultsOutStream call.
These objects represent an extension data import handler, including a dispatch table for the public interface and any accompanying data which is opaque to the core shell. Such objects are created by extensions and passed to the core shell only by reference. They are made known to the shell core via registerImporter() calls.
These objects represent the dispatch table of an ImportHandler object.
All methods in the dispatch table are given this 1 leading argument:
(1) The ImportHandler address registered via registerImporter().
This method is called prior to the shell's return or exit or unloading the runtime extension for a registered ImportHandler object, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the importer, which is to be passed with leading '--' (or '-') to the .import command to specify use of the registered importer for that invocation. The returned pointer must remain valid throughout the lifetime of the registered ImportHandler object.
This method returns help text for the ImportHandler.
There is one additional argument:
(2) a const char * directing what help text to return, with
0 indicating primary, single-line help, or
&"" indicating more detailed help beyond the primary level, or
&"(other)" reserved for expansion.
The primary help is included in the .import command's own detailed help text, so it should be aligned accordingly. The detailed help is shown by the .import command's --help option.
The return is either a C string or null pointer. The C string will not be freed by the core shell, and must remain valid during the lifetime of the ImportHandler object.
The following methods are given these 2 additional arguments:
(2) A ShellExState object passed by reference; and
(3) An error message pointer, passed by reference, to receive errors.
This method is called when a .import command is invoked with the '--' (or '-') prefixed name of the ImportHandler as an argument.
These 3 additional arguments are passed:
(4) the number of arguments to said .import command;
(5) an array of C strings with the said argument values; and
(6) the name of the ImportHandler (as its name method would return.)
When the extension handler is activated via a .import command, responsibility for parsing that command and behaving accordingly is shared between the shell core .import implementation and this method. The shell core normally takes the last argument as the name of a table (to be created if necessary) which will receive the imported data. (But see return values for exceptions to this treatment.) The shell core also detects any --flag argument selecting a registered ImportHandler (giving effect to the last one) and calls this and following methods to perform the input part of the import operation. This method may interpret any or all of the arguments as needed and (supposedly) documented by the associated help(...) method returns.
Once this method is called and succeeds, it is guaranteed that the closeDataInStream method will be called.
This method should return SQLITE_OK upon success where disposition of the imported data (into a table named by the last argument) is to be handled normally, by the shell core. An alternative success return is SQLITE_DONE, indicating that disposition of the imported data has been done by the ImportHandler. In that case, the next 3 methods (*DataInput(...)) will not be called and this method should have completed the whole import operation. After a success return, closeDataInStream is guaranteed to be called. Other returns will abort all remaining calls in the handling sequence.
This method prepares for the particular import operation commenced with a .import invocation. It may take into account the shape of the input data (and possibly its type) as discovered during the call.
This 1 additional argument is passed:
(4) a to-be-prepared statement (pointer), passed by reference.
This is an out parameter conveying a prepared statement created
by this method specifically for the single import operation.
The prepared statement should, in some manner wholly determined by the extension handler, incorporate compiled SQL (possibly with as-yet unbound parameters) which will (or may) produce a result set. (This may be no more than a query such as "SELECT @1 as one, ...", or could be a SELECT from a temporary table used for buffering.)
The return should be SQLITE_OK upon success, in which case the following 2 methods will also be called. Any other return will abort the {prepare,row,finish}DataInput() call sequence. In that case, no prepared statement should be returned either.
This method is called to collect imported data, making it available through the prepared statement passed as the last argument with as many steps as are needed to get SQLITE_DONE from sqlite3_step(). It will be called repeatedly until its return indicates no more data.
It is given 1 additional argument:
(4) the prepared statement (pointer) returned by prepareDataInput().
This prepared statement can have values bound to it or be reset as
necessary to return some or more data. However, it must remain the
same sqlite3_statement instance returned by prepareDataInput().
The return should be SQLITE_DONE when no more data is available, or SQLITE_ROW to indicate that more data is or might be available. Any other return (including SQLITE_OK) indicates a condition which will abort the data collection phase of the import operation. Any of those 3 returns is treated as success by the shell core.
This method is called to complete the data input phase of the import operation commenced by prepareDataInput().
It is given 1 additional argument:
(4) the prepared statement (pointer) returned by prepareDataInput().
This prepared statement should be finalized by this method. Other
cleanup or import wrap-up related to the transfer may also be performed.
This method is called as a .import command which specified this ImportHandler completes. It should free resources allocated or held as a result of the previous openDataInStream call.
An object of this type is passed by an extension to registerScripting() to enable the shell core to gain scripting functionality provided by the extension.
All methods in the dispatch table have this 1 leading argument:
(1) The object address registered via registerScripting().
The object's v-table contains these methods:
This method is called prior to the shell's return or exit or unloading the runtime extension, provided its dispatch table entry is non-zero. It should free resources allocated during the sqlite3_X_init() call associated with creation or preparation of the object.
This method returns the name of the ScriptSupport object. There is no presently planned scheme for using this to switch among scripting providers or even to identify one that is loaded.
This method returns help text for the ExportHandler. There is no presently implemented use for this method's return.
There is one additional argument:
(2) a const char * directing what help text to return.
This method allows load-time configuration of the extension.
It returns a DotCmdRC code indicating status of the extension
load and subsequent configuration.
It has these 4 additional arguments:
(2) ShellExState reference, for general use by the extension;
(3) A char ** for an error message associated with configuration;
(4) Count of arguments passed in next parameter; and
(5) A char * array of arguments passed in the .shxload command tail.
This method returns TRUE if the provided line text should be
considered part of an input line group to be handled by the extension
(rather than being treated as a dot-command or SQL), otherwise FALSE.
There is one additional argument:
(2) char * zLineLead, the initial line of a shell input group.
This method returns TRUE if the provided line group text should be
considered ready to execute, otherwise FALSE.
There are two additional arguments:
(2) char * zScript, the possibly ready-to-exectue text; and
(3) char ** pzWhyNot, for possibly saying why it is not ready.
This method is called by the shell, with no additional arguments, to (possibly) restore the extension's input line scanning state. It has no return.
This method executes the provided script text, returning a
DotCmdRC value to indicate success, error, or other dispositions.
It has three additional arguments:
(2) char * zScript, the to-be-executed text;
(3) ShellExState *, (same struct as is passed to configure); and
(4) A char ** for an error message associated with the execution.
An object of this type is passed (somewhat indirectly) to the sqlite3_X_init(...) function which is called when a runtime extension is loaded via a .load command with a --shell flag. It is used to establish linkage between the loaded extension and a shell core API exposed specifically for extensibility.
At present, the extension API is limited to registration of meta-commands, query result handlers, import handlers, and scripting support implemented by extensions. This API may be extended in future versions of the core shell, in a backwards- compatible manner. (See the pExtra sentinel, whose offset may increase.)
An extension's sqlite3_X_init() function is called soon after the extension is dynamically loaded. It may obtain a pointer to the ShellExtensionLink object by the following means. (Or, without reference to that object, it can do nothing useful.)
During extension build, a macro provided by shext_linkage.h, SHDB_TO_SHEXTLINK(link_function_name), is used to define a function named per its argument. This function accepts a sqlite3 *, as passed into sqlite3_X_init(db, ...), and reads that DB to retrieve the address of a ShellExtensionLink object, which it returns. (Or, it returns NULL if somebody has loaded the extension erroneously using the .load meta-command.)
The sqlite3_X_init() function may call this macro-written function to obtain a reference to a ShellExtensionLink object which will remain valid while the extension is loaded.
There is critical tension between competing uses of the ShellState object which is kept and used by the core shell for state which must persist between meta-command invocations. For unhindered implementation and feature expansion flexibility, the structure of the ShellState data should be unconstrained across versions of the shell. However, unless extension meta-commands and data transferers are to act entirely independently of built-in meta-commands, (excepting interaction through the above-described extension methods), some portion of the ShellState data needs to be shared in a stable manner between extensions and the core shell code.
With respect to interface stability concerns, it is nearly immaterial whether the sharing occurs through exposed data structures or an extension API devised to convey similar data. (Only data layout and possible change notification are at stake in that choice.)
For simplicity, and because it can work in a way consistent with how built-in shell features are implemented now, the chosen sharing method is to simply directly expose a subset of the ShellState data. That subset will be extremely limited to minimize hinderance of what has previously been unfettered change to that data structure/meaning. It will initially be: the present output stream, as affected by .output and .once commands; the currently open user DB (if any); the dedicated shell DB; and the data related to formatting and transfer of data in external forms. This data resides in the ShellStateX object as a ShellExState struct.
As shell extensibility evolves, additional data items may need to move into the publicly exposed portion of the ShellStateX object, either directly (via exposed data members) or by means of additional extension APIs defined in the ShellExtensionLink object (which has been defined to accommodate growth of its function pointer list in a backwards-compatible manner.)