# ticketchng

login: drh
mimetype: text/x-fossil-wiki
icomment:
  Documentation updated to explain that an setting the 3rd parameter of
  sqlite3_bind_text(), sqlite3_bind_text16() or sqlite3_bind_blob() to
  a NULL pointer is the same as calling sqlite3_bind_null().


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This is a known issue with the parser.  It is documented.  We won't change it
  because making the change might break legacy.


login: nobody
username: Roger Binns
mimetype: text/x-fossil-plain
icomment:
  Additional examples from the mailing list:

  * Doing "natrual" join
  * Nonsensical aggregate function use
  * Using double quotes for strings
  * Adding an ORDER BY random() if no order is present


login: nobody
mimetype: text/x-fossil-plain
icomment:
  more than one row returned by a subquery used as an expression

  http://thread.gmane.org/gmane.comp.db.sqlite.general/79670


login: anonymous
mimetype: text/x-fossil-plain
icomment:
  I'm really sorry, but I have to reopen the ticket.

  The following commit 597333f1024092b94bcd8772541e19a0f707bd40 (http://www.sqlite.org/src/info/597333f102) breaks the compilation on systems with no posix_fallocate available (e.g. systems using uClibc as libc-implementation, i.e. a lot of embedded systems).

  Inverting the result of configure test is incorrect, i.e. the value of HAVE_POSIX_FALLOCATE should be left as is.

  I didn't test it myself, but from reading the code I would say the problem reported in this ticket is only solved for systems with posix_fallocate available. posix_fallocate is optional (s. http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_fallocate.html), therefore the ticket is only partially solved. Thus reopened.

  Regards,
  Gene


login: drh
mimetype: text/x-fossil-wiki
icomment:
  We no longer support the async extension.  The features provided
  by async is now provided in a much better way by WAL mode.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script, run in "testfixture", fails, possibly with a segfault,
  when db2 tries to access a collating sequence function for db1 after db1 has
  been closed:

  <verbatim>
  sqlite3_enable_shared_cache 1
  file delete -force test.db
  sqlite3 db1 test.db
  sqlite3 db2 test.db
  set counter 0
  proc mycollate_db1 {a b} {incr ::counter; string compare $a $b}
  proc mycollate_db2 {a b} {string compare $a $b}
  db1 collate mycollate mycollate_db1
  db2 collate mycollate mycollate_db2
  db1 eval {
    CREATE TABLE t1(a, CHECK (a COLLATE mycollate IN ('one', 'two', 'three')));
    INSERT INTO t1 VALUES('one');
  }
  db1 close
  puts counter=$counter ;# expect to see mycollate_db1 called several times
  db2 eval {INSERT INTO t1 VALUES('two');}
  db2 close
  puts counter=$counter ;# No additional calls to mycollate_db1
  </verbatim>


login: nobody
mimetype: text/x-fossil-plain
icomment:
  When using sub queries I can no longer make use of the rowid. With version 
  3.7.13 which is shipped in Debian testing (and probably 3.7.14, too) it works as expected.

  A small example:

  sqlite3 test.db
  sqlite> create table test (line varchar);
  sqlite> insert into test (line) values ("a");
  sqlite> insert into test (line) values ("b");
  sqlite> insert into test (line) values ("c");
  sqlite> insert into test (line) values ("d");
  sqlite> select rowid from (select * from test limit 0, -1) where line="c";

  With 3.7.15 this returns an empty line while 3.7.13 returns '3' as 
  expected.

  I already reported this against Debian in [0] but since people using macports and Gentoo reported the same problem I decided to better tell you directly.

  [0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=696022


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Subqueries do not (necessarily) have rowids.  This is not a bug.

  You should not use rowids.  If you want an integer primary key, then declare
  one as part of your schema and use that instead.  Rowids are an example of the
  implementation showing through into the interface and should be avoided
  in all applications.


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  Try the following:
  <pre>
   create table name (
     type integer,
     path text primary key);

   select (name.type = 0) as container from Name name where name.type = 0 or    container order by name.path
  </pre>
  ---

  The select statement crashes during query preparation.


  [7b96115e81deab0aa0472839cf787ba3f2fc64a7]

  <pre>
    char sqlite3ExprAffinity(Expr *pExpr){
      int op;
      pExpr = sqlite3ExprSkipCollate(pExpr);
      int op = pExpr->op;                               
      op = pExpr->op;
      ...
  </pre>
  last line crashes because sqlite3ExprSkipCollate returns null.


login: nobody
mimetype: text/x-fossil-plain
icomment:
  pExpr is already NULL from caller 'exprAnalyzeOrTerm' -> affRight = sqlite3ExprAffinity(pOrTerm->pExpr->pRight);


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Bisecting shows that the bug was introduced by the refactoring of COLLATE at
  check-in [8542e6180d4321d45b], just a few days prior to the release of
  version 3.7.15, and needed to address bug [71e333e7d2e642afc96d2].  

  So, we fixed one bug but in the process introduced another....


login: nobody
mimetype: text/x-fossil-plain
icomment:
  /usr/bin/sqlite2 reports "file is encrypted or is not a database" when given an sqlite3 database file. This is not exactly correct. A more user friendly diagnostics could include the hint about version 3.

  With e.g. openSUSE it is likely to stumble into this issue, as /usr/bin/sqlite (version 2.x) is installed per default (but but not sqlite3); while most applications write sqlite3 files these days.

  Sugested patch for better "usability":

  https://build.opensuse.org/package/rawsourcefile?file=detect_sqlite3.patch&package=sqlite2&project=home%3Ajnweiger%3Abranches%3Aserver%3Adatabase&srcmd5=256b16a0f65b85e5a10fafac55283247

  jnweiger@gmail.com


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  Hello,

  Martin Godisch notes that sqlite3 CLI doesn't count multi-byte characters correctly. He uses an example with the <em>width</em> option to highlight the problem. However, I think this is the <em>column</em> mode that does not comply with UTF-8, rather than the <em>width</em> option.

  This bug has been detected by Martin Godisch in version 3.5.9 packaged for Debian (see Debian Bug report: [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=536662|#536662]) and I still found it in version 3.7.13 (packaged for Debian also). I use the sqlite3 3.7.13-1 package (see [http://patch-tracker.debian.org/package/sqlite3/3.7.13-1|debian specific patch information for this package]) for Debian testing on an x86 architecture, without modifications.

  The following script illustrates the problem:
  <verbatim>
  .header on

  create table tbl1 (col1 text primary key);

  insert into tbl1 values ('123456');
  insert into tbl1 values ('abcdef');
  insert into tbl1 values ('äöüäöü');

  select col1 from tbl1;
  select substr(col1, 1, 4) as col1 from tbl1;

  .mode column
  select col1 from tbl1;

  .width 4
  select col1 from tbl1;
  </verbatim>


  Script output:
  <verbatim>
  col1
  123456
  abcdef
  äöüäöü

  col1
  1234
  abcd
  äöüä

  col1      
  ----------
  123456    
  abcdef    
  äöüäö

  col1
  ----
  1234
  abcd
  äö
  </verbatim>

  Regards,

  Stéphane Aulery


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  I stumbled upon some problems with aliases, fully qualified identifiers, and *. Here is a non exhaustive script that illustrates the problem:

  <verbatim>
   -- Uncomment the desired parts

  .header on

  create table main.tbl1 (col1 text primary key, col2 text);
  create table main.tbl2 (col1 text primary key, col2 text);
  create table main.tbl3 (col3 text primary key, col4 text);

  insert into main.tbl1 values ('001', 'Jasmin');
  --insert into main.tbl1 values ('002', 'Crocus');
  --insert into main.tbl1 values ('003', 'Geranium');

  insert into main.tbl2 values ('001', 'Jasmin');
  --insert into main.tbl2 values ('002', 'Crocus');
  --insert into main.tbl2 values ('003', 'Geranium');

  insert into main.tbl3 values ('001', 'Jasmin');
  --insert into main.tbl3 values ('002', 'Crocus');
  --insert into main.tbl3 values ('003', 'Geranium');


  ATTACH DATABASE ':memory:' AS aux1;

  create table aux1.tbl1 (col1 text primary key, col2 text);
  create table aux1.tbl2 (col1 text primary key, col2 text);
  create table aux1.tbl3 (col3 text primary key, col4 text);

  insert into aux1.tbl1 values ('001', 'Jasmin');
  --insert into aux1.tbl1 values ('002', 'Crocus');
  --insert into aux1.tbl1 values ('003', 'Geranium');

  insert into aux1.tbl2 values ('001', 'Jasmin');
  --insert into aux1.tbl2 values ('002', 'Crocus');
  --insert into aux1.tbl2 values ('003', 'Geranium');

  insert into aux1.tbl3 values ('001', 'Jasmin');
  --insert into aux1.tbl3 values ('002', 'Crocus');
  --insert into aux1.tbl3 values ('003', 'Geranium');



  -- 1 db, 1 table (self-joined), 2 aliases (OK)
  /*
  select *                                   from main.tbl1 as t1 join main.tbl1 as t2 on t1.col1 = t2.col1;
  select t1.*, t2.*                          from main.tbl1 as t1 join main.tbl1 as t2 on t1.col1 = t2.col1;
  select t1.col1, t1.col2, t2.*              from main.tbl1 as t1 join main.tbl1 as t2 on t1.col1 = t2.col1;
  select t2.col1, t2.col2, t1.*              from main.tbl1 as t1 join main.tbl1 as t2 on t1.col1 = t2.col1;
  select t1.col1, t1.col2, t2.col1, t2.col2  from main.tbl1 as t1 join main.tbl1 as t2 on t1.col1 = t2.col1;
  */


  -- 1 db, 1 table (self-joined), 1 alias (KO)
  /*
  select *                                                 from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.*, main.tbl1.*                                 from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.col1, main.tbl1.*                              from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.col1, main.tbl1.col1                           from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.col1, t1.col2, main.tbl1.*                     from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.*, main.tbl1.col1, main.tbl1.col2              from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  select t1.col1, t1.col2, main.tbl1.col1, main.tbl1.col2  from main.tbl1 as t1 join main.tbl1 on t1.col1 = main.tbl1.col1;
  */


  -- 1 db, 2 tables (different name), 2 aliases (OK)
  /*
  select *                                   from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  select t1.*, t2.*                          from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  select t1.col1, t2.*                       from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  select t1.col1, t1.col2, t2.*              from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  select t1.*, t2.col1, t2.col2              from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  select t1.col1, t1.col2, t2.col1, t2.col2  from main.tbl1 as t1 join main.tbl2 as t2 on t1.col1 = t2.col1;
  */


  -- 1 db, 2 tables (different name), 1 alias (KO)
  /*
  select *                                                 from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, main.tbl2.*                                 from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, main.tbl2.*                              from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, main.tbl2.col1                              from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, tbl2.*                                      from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, tbl2.*                                   from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, t1.col2, main.tbl2.*                     from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, t1.col2, tbl2.*                          from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, main.tbl2.col1, main.tbl2.col2              from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, tbl2.col1, tbl2.col2                        from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, t1.col2, main.tbl2.col1, main.tbl2.col2  from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, t1.col2, tbl2.col1, tbl2.col2            from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  */


  -- 1 db, 2 tables (different name), without alias (KO)
  /*
  select *                                                               from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.*, tbl2.*                                                  from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.col1, tbl2.*                                               from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select main.tbl1.*, tbl2.*                                             from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select main.tbl1.*, main.tbl2.*                                        from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select main.tbl1.col1, main.tbl1.col2, main.tbl2.*                     from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select main.tbl1.col1, main.tbl1.col2, tbl2.*                          from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.col1, tbl1.col2, main.tbl2.*                               from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.col1, main.tbl1.col2, tbl2.*                               from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.col1, tbl1.col2, tbl2.*                                    from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select main.tbl1.col1, main.tbl1.col2, main.tbl2.col1, main.tbl2.col2  from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  select tbl1.col1, tbl1.col2, tbl2.col1, tbl2.col2                      from main.tbl1 join main.tbl2 on main.tbl1.col1 = main.tbl2.col1;
  */


  -- 1 db, 2 tables (different name and different columns), without alias (KO)
  /*
  select *                                                               from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.*, tbl3.*                                                  from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.*, tbl3.*                                             from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.*, main.tbl3.*                                        from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, main.tbl3.*                     from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, tbl3.*                          from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.col1, tbl1.col2, main.tbl3.*                               from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.col1, main.tbl1.col2, tbl3.*                               from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.col1, tbl1.col2, tbl3.*                                    from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.*, tbl3.col3, tbl3.col4                                    from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, main.tbl3.col3, main.tbl3.col4  from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.col1, tbl1.col2, tbl3.col3, main.tbl3.col4            from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select main.tbl1.col1, col2, tbl3.col3, main.tbl3.col4                 from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select tbl1.col1, tbl1.col2, tbl3.col3, tbl3.col4                      from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  select col1, col2, col3, col4                                          from main.tbl1 join main.tbl3 on main.tbl1.col1 = main.tbl3.col3;
  */


  -- 2 db, 2 tables (same name), 2 aliases (OK)
  /*
  select *                                   from main.tbl1 as t1 join aux1.tbl1 as a1 on t1.col1 = a1.col1;
  select t1.*, a1.*                          from main.tbl1 as t1 join aux1.tbl1 as a1 on t1.col1 = a1.col1;
  select t1.col1, t1.col2, a1.*              from main.tbl1 as t1 join aux1.tbl1 as a1 on t1.col1 = a1.col1;
  select t1.*, a1.col1, a1.col2              from main.tbl1 as t1 join aux1.tbl1 as a1 on t1.col1 = a1.col1;
  select t1.col1, t1.col2, a1.col1, a1.col2  from main.tbl1 as t1 join aux1.tbl1 as a1 on t1.col1 = a1.col1;
  */


  -- 2 db, 2 tables (same name), 1 alias (KO)
  /*
  select *                                                 from main.tbl1 as t1 join aux1.tbl1 on t1.col1 = aux1.tbl1.col1;
  select t1.*, aux1.tbl1.*                                 from main.tbl1 as t1 join aux1.tbl1 on t1.col1 = aux1.tbl1.col1;
  select t1.col1, t1.col2, aux1.tbl1.*                     from main.tbl1 as t1 join aux1.tbl1 on t1.col1 = aux1.tbl1.col1;
  select t1.*, aux1.tbl1.col1, aux1.tbl1.col2              from main.tbl1 as t1 join aux1.tbl1 on t1.col1 = aux1.tbl1.col1;
  select t1.col1, t1.col2, aux1.tbl1.col1, aux1.tbl1.col2  from main.tbl1 as t1 join aux1.tbl1 on t1.col1 = aux1.tbl1.col1;
  */


  -- 2 db, 2 tables (same name), without alias (KO)
  /*
  select *                                                               from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;
  select main.tbl1.*, aux1.tbl1.*                                        from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;
  select main.tbl1.*, aux1.tbl1.col1, aux1.tbl1.col2                     from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl1.*                     from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl1.col1, aux1.tbl1.col2  from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;
  */


  -- 2 db, 2 tables (different name), 2 aliases (OK)
  /*
  select *                                   from main.tbl1 as t1 join aux1.tbl2 a2 on t1.col1 = a2.col1;
  select t1.*, a2.*                          from main.tbl1 as t1 join aux1.tbl2 a2 on t1.col1 = a2.col1;
  select t1.*, a2.col1, a2.col2              from main.tbl1 as t1 join aux1.tbl2 a2 on t1.col1 = a2.col1;
  select t1.col1, t1.col2, a2.*              from main.tbl1 as t1 join aux1.tbl2 a2 on t1.col1 = a2.col1;
  select t1.col1, t1.col2, a2.col1, a2.col2  from main.tbl1 as t1 join aux1.tbl2 a2 on t1.col1 = a2.col1;
  */


  -- 2 db, 2 tables (different name), 1 alias (KO)
  /*
  select *                                                 from main.tbl1 as t1 join aux1.tbl2 on t1.col1 = aux1.tbl2.col1;
  select t1.*, aux1.tbl2.*                                 from main.tbl1 as t1 join aux1.tbl2 on t1.col1 = aux1.tbl2.col1;
  select t1.*, aux1.tbl2.col1, aux1.tbl2.col2              from main.tbl1 as t1 join aux1.tbl2 on t1.col1 = aux1.tbl2.col1;
  select t1.col1, t1.col2, aux1.tbl2.*                     from main.tbl1 as t1 join aux1.tbl2 on t1.col1 = aux1.tbl2.col1;
  select t1.col1, t1.col2, aux1.tbl2.col1, aux1.tbl2.col2  from main.tbl1 as t1 join aux1.tbl2 on t1.col1 = aux1.tbl2.col1;
  */


  -- 2 db, 2 tables (different name), without alias (KO)
  /*
  select *                                                               from main.tbl1 join aux1.tbl2 on main.tbl1.col1 = aux1.tbl2.col1;
  select main.tbl1.*, aux1.tbl2.*                                        from main.tbl1 join aux1.tbl2 on main.tbl1.col1 = aux1.tbl2.col1;
  select main.tbl1.*, aux1.tbl2.col1, aux1.tbl2.col2                     from main.tbl1 join aux1.tbl2 on main.tbl1.col1 = aux1.tbl2.col1;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl2.*                     from main.tbl1 join aux1.tbl2 on main.tbl1.col1 = aux1.tbl2.col1;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl2.col1, aux1.tbl2.col2  from main.tbl1 join aux1.tbl2 on main.tbl1.col1 = aux1.tbl2.col1;
  */


  -- 2 db, 2 tables (different name and different columns), without alias (KO)
  /*
  select *                                                               from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.*, tbl3.*                                                  from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.*, tbl3.*                                             from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.*, aux1.tbl3.*                                        from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl3.*                     from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, tbl3.*                          from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.col1, tbl1.col2, aux1.tbl3.*                               from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.col1, main.tbl1.col2, tbl3.*                               from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.col1, tbl1.col2, tbl3.*                                    from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.*, tbl3.col3, tbl3.col4                                    from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl3.col3, aux1.tbl3.col4  from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, tbl1.col2, tbl3.col3, aux1.tbl3.col4            from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, col2, tbl3.col3, aux1.tbl3.col4                 from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.col1, tbl1.col2, tbl3.col3, tbl3.col4                      from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select col1, col2, col3, col4                                          from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  */
  </verbatim>


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  A variant of the same issue:

  <verbatim>
  -- 1 db, 2 tables (different name), 1 alias (KO)

  select *                     from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, main.tbl2.*     from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.*, tbl2.*          from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, main.tbl2.*  from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  select t1.col1, tbl2.*       from main.tbl1 as t1 join main.tbl2 on t1.col1 = main.tbl2.col1;
  </verbatim>

  Regards,

  Stéphane Aulery


login: drh
mimetype: text/x-fossil-wiki
icomment:
  As best I can determine from the description above, the OP is concerned that
  wildcards of the form:

     DATABASE.TABLE.*

  are not expanded.  SQLite has never claimed to expand terms of this kind.
  So, the OP seems to be requesting a new feature.


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  Ok. This feature seemed me quite natural from the point where it is possible to attach multiple databases. In your opinion, does the following errors are also of the same lack ?

  <verbatim>
  -- 2 db, 2 tables (same name), without alias (KO)
  select *                                           from main.tbl1 join aux1.tbl1 on main.tbl1.col1 = aux1.tbl1.col1;

  -- 2 db, 2 tables (different name and different columns), without alias (KO)
  select main.tbl1.*, tbl3.*                         from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.*, aux1.tbl3.*                    from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select main.tbl1.col1, main.tbl1.col2, aux1.tbl3.* from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  select tbl1.col1, tbl1.col2, aux1.tbl3.*           from main.tbl1 join aux1.tbl3 on main.tbl1.col1 = aux1.tbl3.col3;
  </verbatim>

  Regards,

  Stéphane Aulery


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL run in a shell generates an assertion fault in the name
  resolution logic:

  <verbatim>
  ATTACH ':memory:' AS aux1;
  CREATE TABLE t1(a,b); INSERT INTO t1 VALUES(222,'x2');
  CREATE TABLE main.t2(a,b); INSERT INTO main.t2 VALUES(444,'x4');
  CREATE TABLE aux1.t2(a,b); INSERT INTO aux1.t2 VALUES(555,'x5');
  SELECT *
    FROM t1 JOIN (main.t2 JOIN aux.t2 ON aux.t2.a=main.t2.a+111)
              ON main.t2.a=t1.a+222;
  </verbatim>

  This fault has been present in the code since version 3.6.2, 2008-08-30.  The 
  fault appears to have been injected by check-in [daf730d1defa78fb8b8].  Prior
  to that check-in, the SQL above would throw an error: 
  "no such table: aux.t4".  So it appears that no prior version of SQLite would
  get the correct answer to the SQL above.


login: nobody
mimetype: text/plain
icomment:
  Having an index on 3 integer columns (column1, column2, column3), the analyse command generates the stat1 and stat3 tables. I see the following statistics for this index in the sqlite_stat1 table:

  "4600132 1289 1275 1"

  When I execute the following SQL query, this index is not used but the query optimizer prefers to use the primary index which is on the column3, which is about 60 times slower than using the index. ( 20000 ms vs 300 ms)

  "select column1, column2, column3 from table where column1 IN ( integer1, integer2, ..., integer30 ) and column2 = 1 order by column3 limit 0,200"

  When I put less integer values in the IN operator OR remove the "order by" part from the query, my index is used. 

  OR If I create an index only on the column1, then this index is used for the original query I wrote above, which is also very efficient.

  The SQLite version 3.6 does not have the same issue.

  The same problem had been discussed a while ago: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg74412.html


login: dan
mimetype: text/x-fossil-wiki
icomment:
  This sounds like a problem, but there are details missing from this bug report. Please raise this on the mailing list again.

  Running the following with 3.7.15.2:

  <verbatim>
  CREATE TABLE tbl(c1, c2, c3 INTEGER PRIMARY KEY);
  CREATE INDEX idx2 ON tbl(c1, c2, c3);

  ANALYZE sqlite_master;
  INSERT INTO sqlite_stat1 VALUES('tbl', 'idx2', '4600132 1289 1275 1');

  EXPLAIN QUERY PLAN
  SELECT c1, c2, c3 FROM tbl
  WHERE c1 IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
              ,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
  ) AND c2=1
  ORDER BY c3 LIMIT 200;
  </verbatim>

  produces:

  <verbatim>
  0|0|0|SEARCH TABLE tbl USING COVERING INDEX idx2 (c1=? AND c2=?) (~3510 rows)
  0|0|0|EXECUTE LIST SUBQUERY 1
  0|0|0|USE TEMP B-TREE FOR ORDER BY
  </verbatim>


login: anonymous
mimetype: text/x-fossil-wiki
icomment:
  The following SQL run in a shell generates a segmentation fault: 

  <pre>create view v0 as select 0 as n;
  create view v1 as select * from v0 union select * from v0;
  create view v2 as select * from v1 union select * from v1;
  create view v3 as select * from v2 union select * from v2;
  create view v4 as select * from v3 union select * from v3;
  create view v5 as select * from v4 union select * from v4;
  create view v6 as select * from v5 union select * from v5;
  create view v7 as select * from v6 union select * from v6;
  create view v8 as select * from v7 union select * from v7;
  create view v9 as select * from v8 union select * from v8;
  create view v10 as select * from v9 union select * from v9;
  create view v11 as select * from v10 union select * from v10;
  create view v12 as select * from v11 union select * from v11;
  create view v13 as select * from v12 union select * from v12;
  create view v14 as select * from v13 union select * from v13;
  create view v15 as select * from v14 union select * from v14;
  create view v16 as select * from v15 union select * from v15;
  create view v17 as select * from v16 union select * from v16;
  create view v18 as select * from v17 union select * from v17;
  create view v19 as select * from v18 union select * from v18;
  create view v20 as select * from v19 union select * from v19;</pre>

  This fault has been present in the code since version 3.7.15.2, 2013-01-09 and early.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The LIMIT on the final SELECT in the following code does not work:

  <verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(5);
  CREATE VIEW v1 AS SELECT x*2 FROM t1;
  CREATE VIEW v2 AS SELECT * FROM v1 UNION ALL SELECT * FROM v1;
  CREATE VIEW v4 AS SELECT * FROM v2 UNION ALL SELECT * FROM v2;
  SELECT * FROM v4 LIMIT 3;
  </verbatim>

  The problem seems to have been introduced by check-ins [3ef468e7046b209]
  or [a79786a961dba8] on 2008-07-01.  The problem first appeared in version
  3.6.1.  The problem was detected during internal testing and has never
  been seen in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A related problem:  The following SQL leaks memory - lookaside memory in
  particular which will result in an assert() fault when run with SQLITE_DEBUG.

  <verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(1),(2);
  SELECT z FROM (
      SELECT y AS z FROM (SELECT x AS y FROM t1)
      UNION ALL
      SELECT y+10 FROM (SELECT x AS y FROM t1)
  )
  LIMIT 3 OFFSET 2;
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script gives an error on the SELECT statement:

  <blockquote><verbatim>
  CREATE TABLE t1 (a INTEGER PRIMARY KEY);
  CREATE TABLE t2 (a INTEGER PRIMARY KEY, b INTEGER);
  CREATE TABLE t2x (b INTEGER PRIMARY KEY);
  SELECT t1.a FROM ((t1 JOIN t2 ON t1.a=t2.a) AS x JOIN t2x ON x.b=t2x.b) as y;
  </verbatim></blockquote>

  The error only occurs if the name of the third table includes the name
  of the second table as a prefix.  This problem was introduced by
  check-in [99127a669c] on 2013-01-03 and has never appeared in an official
  release.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL yields an incorrect result (zero rows) in all versions
  of SQLite between 3.6.14 and 3.7.15.2:

  <blockquote><verbatim>
  CREATE TABLE t(textid TEXT);
  INSERT INTO t VALUES('12');
  INSERT INTO t VALUES('34');
  CREATE TABLE i(intid INTEGER PRIMARY KEY);
  INSERT INTO i VALUES(12);
  INSERT INTO i VALUES(34);

  SELECT t1.textid AS a, i.intid AS b, t2.textid AS c
    FROM t t1, i, t t2
   WHERE t1.textid = i.intid
     AND t1.textid = t2.textid;
  </verbatim></blockquote>

  The correct result should be two rows, one with 12|12|12 and the other with 34|34|34.  With this bug, no rows are returned.  Bisecting shows that this bug
  was introduced with check-in [dd4d67a67454] on 2009-04-23.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The output from the following script occurs in the wrong order:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(1),(2),(3),(4),(5),(6);
  CREATE INDEX t1xd ON t1(x DESC);
  SELECT x FROM t1 WHERE x IN (2,4,6) ORDER BY x DESC;
  </verbatim></blockquote>

  The problem appears to have been introduced by check-in
  [2cef8b68f0e1216] and has never been part of an official
  release.  The problem was detected by the SQLLogicTest
  test harness during release testing. for version 3.7.16.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The second SELECT in the following code returns 34.  The
  correct answer is 3, the same as the first SELECT.

  <blockquote><verbatim>
  PRAGMA encoding=UTF16be;
  SELECT CAST('3x' AS int);
  SELECT CAST(x'003301340000' AS int);
  </verbatim></blockquote>

  A similar error occurs for casting to "real".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following code generates an assertion fault on the SELECT:

  <blockquote><verbatim>
  CREATE VIRTUAL TABLE x USING fts4(y,z);
  INSERT INTO x VALUES('aaa bbb ccc ddd', 'bbb ddd aaa ccc');
  SELECT * FROM x where y MATCH 'bbb NEAR/6 aaa';
  </verbatim></blockquote>

  This problem was introduced when the FTS4 prefix option was added with
  check-in [b1f9c1e0ac51c] on 2011-04-14 for version 3.7.7.


login: nobody
mimetype: text/x-fossil-plain
icomment:
  In the query below, there are multiple rows with the same combination of group by columns:
  select pgr.productgroupid, r.stationgroupid, sum(quantity)
  from tmp_duration d, productgrouprelation pgr, stationgrouprelation r  where d.productid = pgr.productid  and d.stationid = r.stationid group by r.stationgroupid, pgr.productgroupid

  results in
  Fun EB 13 
  Fun EB 12 
  Fun EB 1862
  GK EB 4 
  GK EB 2 
  GK EB 505 
  GK EB 5 
  GK EB 26 
  GK EB 6 
  GK EB 155 
  GK EB 4284 
  GK EB 1781 

  Reversing the group by columns to "pgr.productgroupid, r.stationgroupid" will give the correct result:
  Fun EB 1887 
  GK EB 6768 

  This effect is also in 3.7.15.2. Version 3.7.2 is ok.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The ORDER BY clause on the final SELECT of the following script does not work:

  <blockquote><verbatim>
  CREATE TABLE t1(a, b, PRIMARY KEY(a,b));
  INSERT INTO t1 VALUES(1,1),(1,2);
  CREATE TABLE t2(x, y, PRIMARY KEY(x,y));
  INSERT INTO t2 VALUES(3,3),(4,4);
  SELECT a, x FROM t1, t2 ORDER BY 1, 2;
  </verbatim></blockquote>

  This script is a reduction of the same bug that causes the problem
  in the initial bug report above.

  This problem was introduced to trunk by check-in [956e4d7f8958e706] on
  2012-10-03 and first appeared in release 3.7.15.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another test case is shown below.  This one uses single-column indices
  only.

  <blockquote><verbatim>
  CREATE TABLE t3(a);
  INSERT INTO t3 VALUES(1),(1);
  CREATE INDEX t3a ON t3(a);
  CREATE TABLE t4(x);
  INSERT INTO t4 VALUES(3),(4);
  CREATE INDEX t4x ON t4(x);
  SELECT a, x FROM t3, t4 ORDER BY 1, 2;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The final INSERT in the script below reports that the database
  is corrupt (SQLITE_CORRUPT) and aborts.  Subsequent analysis shows
  no database corruption.

  <blockquote><verbatim>
  PRAGMA page_size=512;
  CREATE TABLE t1(x INTEGER PRIMARY KEY, y);
  INSERT INTO t1 VALUES(1,randomblob(400));
  INSERT INTO t1 VALUES(2,randomblob(400));
  INSERT INTO t1 SELECT x+2, randomblob(400) FROM t1;
  INSERT INTO t1 SELECT x+4, randomblob(400) FROM t1;
  INSERT INTO t1 SELECT x+8, randomblob(400) FROM t1;
  INSERT INTO t1 SELECT x+16, randomblob(400) FROM t1;
  INSERT INTO t1 SELECT x+32, randomblob(400) FROM t1;
  INSERT INTO t1 SELECT x+64, randomblob(400) FROM t1 WHERE x<10;
  CREATE TRIGGER r1 AFTER INSERT ON t1 WHEN new.x=74 BEGIN
    DELETE FROM t1;
    INSERT INTO t1 VALUES(75, randomblob(400));
    INSERT INTO t1 VALUES(76, randomblob(400));
  END;
  INSERT INTO t1 VALUES(74, randomblob(400));
  </verbatim></blockquote>

  The problem occurs because the INSERT operation leaves its cursor open, though
  marked CURSOR_INVALID, while holding a reference to a second-level page of the
  btree.  When the trigger fires it deletes content out from under this cursor,
  which is normally OK since the cursor is invalid.  However, during the second
  INSERT inside the trigger, when a page is removed from the freelist to hold
  the new content, that page is checked to make sure it is unreferenced.  This
  check fails since the invalid INSERT cursor is still holding a reference to
  the page.  That reference makes it appear as if the page was on the freelist
  and in a btree at the same time and so an SQLITE_CORRUPT error is thrown.
  However, the page is not really on the btree since the cursor that is 
  referencing it is marked CURSOR_INVALID.

  This problem has existing in the code since late 2009.  It is very difficult
  to hit, however (which helps to explain why it has gone undetected until now).
  The problem is very sensitive to the specific constants used in the script
  shown above.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The xCheckReservedLock() method on the windows VFS can sometimes return a
  false positive, if two different processes call it on the same file at the
  same time.  This leads one of the processes (the one that got the false
  positive) to believe that the other process is taking care of recovering
  from a prior crash.  That process might then write into the database without
  first running recovery, leading to corruption.

  To see this occur, compile and run the "mptester" utility with the mptest/crash01.test script many times on a fast multi-core win8 machine using
  check-in [663f04bd48bc6f3022] and you will eventually get integrity_check
  failures.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This bug appears to have been inserted by check-in
  [61360ca6ca3448477] - the enhancements to the windows
  system interface that allowed SQLite to work with WinRT.
  The bug first appeared in release 3.7.13.

  We have not observed an occurrence of this bug in the wild.
  The problem was found during internal testing.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Further analysis shows that this race condition is not new but has
  in fact existed in the code since SQLite 3.0.0, 2004-06-18.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the first query works but the second query (the one
  with the COLLATE clause) gives a parsing error:

  <blockquote><verbatim>
  CREATE TABLE t1(x, y); INSERT INTO t1 VALUES(11,22);
  CREATE TABLE t2(y, z); INSERT INTO t2 VALUES(33,44);
  SELECT 1 AS y FROM t1, t2 ORDER BY y;
  SELECT 2 AS y FROM t1, t2 ORDER BY y COLLATE nocase;
  </verbatim></blockquote>

  This problem appears to have been introduced in version 3.7.15 by
  check-in [8542e6180d4321d45] which was a fix for another problem
  described by ticket [71e333e7d2e642af].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When a query of a virtual table uses an IN operator in the WHERE clause as
  its index and it has an ORDER BY clause, it is possible that the ORDER BY
  clause might be optimized away and the sort omitted when in fact the sort
  is required for the correct answer.

  This problem appears to have been introduced by check-in
  [3d65c70343] on [2012-12-14] for version 3.7.16.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the query optimizer omits the ORDER BY clause under
  the assumption that the output will appear in the correct order naturally,
  but this assumption is in error.  The output is not correctly ordered by
  without the sorting step and so an incorrect result is returned:

  <blockquote><verbatim>
  CREATE VIRTUAL TABLE test_fts4 USING FTS4 (word TEXT);
  INSERT INTO test_fts4 VALUES ('one two');
  INSERT INTO test_fts4 VALUES ('one three');
  INSERT INTO test_fts4 VALUES ('one two three');
  SELECT docid
    FROM test_fts4,
    (SELECT 'one two' terms UNION SELECT 'three') candidates
  WHERE
    test_fts4 MATCH candidates.terms
  ORDER BY docid;
  </verbatim></blockquote>

  This bug appears to have been introduced by the ORDER BY query optimizer
  enhancements of check-in [1e874629d7cf568] on 2012-09-28 and first appeared
  in release 3.7.15.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates either a SEGFAULT or an assertion failure
  (depending on compile-time options):

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  SELECT * FROM ((SELECT x FROM t1)) AS y;
  </verbatim></blockquote>

  This problem was traced to check-in [7fecced466d86a66] which was part of
  version 3.7.16.


login: nobody
mimetype: text/plain
icomment:
  In the command line interpreter sqlite3 for win32 I redirect output by
  .output FILENAME
  After that i.e. 
  .databases
  is redirected correctly into FILENAME while
  .tables
  is not - still written to stdout


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The UPDATE statement in the following code reads past the end of an
  array and passes an invalid pointer to the authorizer callback:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c,d,e,f,g,h);
  UPDATE t1 SET rowid=rowid+100;
  </verbatim></blockquote>

  The problem can be seen by running the script above in the sqlite3 command-line
  shell using valgrind.  This problem has been in the code since version 2.7.6
  circa 2003-01-25 and appears to have been added by check-in
  [45de93f913a18026a].  The problem has not been noticed before now because
  it is obscure and mostly harmless - unless the authorizer is used on an UPDATE 
  statement that changes the ROWID the worst
  that can happen is a single read past the end of an array.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  An additional requirement for hitting this bug is that the number of columns
  in the table being updated must be a multiple of 8.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the code shown below (which is intended to be run from the sqlite3.exe
  command-line tool) the three SELECT statements should all generate the same
  answer.  But the third one does not.  It is as if the COLLATE clause on the
  ORDER BY somehow got pulled into the EXCEPT operator.  Note that the ".print"
  commands are instructions to the sqlite3.exe shell program to output
  delimiter lines so that you can more easily tell where the output of one
  query ends and the next query begins.

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  INSERT INTO t1 VALUES('abc'),('def');
  CREATE TABLE t2(a);
  INSERT INTO t2 VALUES('DEF');

  SELECT a FROM t1 EXCEPT SELECT a FROM t2 ORDER BY a;
  .print -----
  SELECT a FROM (SELECT a FROM t1 EXCEPT SELECT a FROM t2)
   ORDER BY a COLLATE nocase;
  .print -----
  SELECT a FROM t1 EXCEPT SELECT a FROM t2 ORDER BY a COLLATE nocase;
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced in SQLite
  version 3.6.0 by check-in [8bbfa97837a74ef] on 2008-06-15.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A Skrooge developer reports that the latest SQLite gives a performance
  regression on one query.  A test script that demonstrates the change
  in the query plan between check-in [82eb7eadb8c76b3a] and
  check-in [3f87f4593b631819] is attached.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL outputs a single row when it should output zero rows:

  <blockquote><verbatim>
  CREATE TABLE t1(x); INSERT INTO t1 VALUES(1);
  CREATE TABLE t2(y INTEGER PRIMARY KEY,a,b);
  INSERT INTO t2 VALUES(1,2,3);
  SELECT * FROM t1 LEFT JOIN t2 ON a=2 OR b=3 WHERE y IS NULL;
  </verbatim></blockquote>

  The result of the above is "1,NULL,NULL,NULL".  It is behaving
  as if the "y IS NULL" term of the WHERE clause has been moved inside 
  the ON clause of the LEFT JOIN.  This effect is only seen when there
  is an OR operator inside the ON clause.

  Bisecting reveals that this problem
  was introduced by on 2011-10-07 by check-in [b23ae131874bc5c6] and
  first appeared in release 3.7.9.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If a database schema uses an application-defined collating sequence, and
  an instance of SQLite is run on that database that does not have the
  necessary collating sequence defined, then the "REINDEX" and 
  "PRAGMA integrity_check" commands run without generating an error
  and substitute the built-in BINARY
  collating sequence in place of the application-defined collating sequence.
  Running the REINDEX command this way corrupts the index (though the 
  corruption can be reversed by rerunning REINDEX in a new database connection
  where the application-defined collating sequence has been defined).
  Running PRAGMA integrity_check this way results in spurious errors.

  This problem appears to have been introduced by check-in
  [8542e6180d4321d] and first appeared in release 3.7.15.


login: drh
mimetype: text/x-fossil-plain
icomment:
  When two database connections share the same cache and one connection is in 
  the middle of a write transaction and the other connection closes (with
  sqlite3_close) or rolls back a transaction on a different attached database,
  and the first connection continues to make changes to the database file and
  then commits, the first connection can cause the database file to go corrupt.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following script, the first query (without the ORDER BY clause) gets
  the correct answer, but adding the ORDER BY clause in the second query gives
  an extra incorrect output row:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER, b INTEGER);
  INSERT INTO t1 VALUES(1,2);
  INSERT INTO t1 VALUES(3,4);
  -- CORRECT answer when ORDER BY omitted
  SELECT *
    FROM t1 AS x
         LEFT JOIN (SELECT a AS c, b AS d FROM t1) AS y ON a=c
   WHERE d=4 OR d IS NULL;
  -- INCORRECT answer when ORDER BY used
  SELECT *
    FROM t1 AS x
         LEFT JOIN (SELECT a AS c, b AS d FROM t1) AS y ON a=c
   WHERE d=4 OR d IS NULL
   ORDER BY a;
  </verbatim></blockquote>

  The correct answer is obtained in both cases prior to check-in
  [1e874629d7cf5683].  After that check-in, the second query segfaults.
  the segfault was fixed by [dd34bec389359845] (which is the fix for 
  ticket [d02e1406a58ea02]) but the wrong answer is generated.  Hence,
  version 3.7.14 was working correctly, but all subsequent versions contain
  a problem with the second query.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is a slightly different script to demonstrate the problem.  This script
  might be easier to analyze since it does not use subqueries:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  INSERT INTO t1 VALUES(1,2), (3,4);
  CREATE TABLE t2(c,d);
  INSERT INTO t2 SELECT * FROM t1;
  SELECT * FROM t1 LEFT JOIN t2 ON a=c
   WHERE d=4 OR d IS NULL
   ORDER BY a;
  </verbatim></blockquote>

  As before, the correct answer is obtained if the ORDER BY clause is omitted.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The order of the output rows in the query below does not honor the
  ORDER BY clause.  Specifically, the "y" column is out of order.

  <blockquote><verbatim>
  CREATE TABLE t1(a INT UNIQUE NOT NULL, b INT NOT NULL);
  CREATE INDEX t1ba ON t1(b,a);
  CREATE TABLE t2(x INT NOT NULL REFERENCES t1(a), y INT NOT NULL);
  CREATE UNIQUE INDEX t2xy ON t2(x,y);
  INSERT INTO t1 VALUES(1,1),(3,1);
  INSERT INTO t2 VALUES(1,13),(1,15),(3,14),(3,16);

  SELECT b, y FROM t1 CROSS JOIN t2 ON x=a ORDER BY b, y;
  </verbatim></blockquote>

  This problem was apparently introduced by check-in [abcf6a5d054559] and
  was first released in version 3.7.15.  The rewritten query planner in the
  nextgen-query-plan-exp branch gets the correct answer.  This problem was
  discovered while looking for difference plans generated by the nextgen
  query planner versus the legacy query planner.

  This problem will be fixed when the nextgen query planner is merged to
  trunk.  The purpose of this ticket is to record the bug.
  and the nextgen query planner.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The last two paragraphs of the original comment should read:

  <blockquote>
  This problem was apparently introduced by check-in [abcf6a5d054559] and was first released in version 3.7.15. The rewritten query planner in the nextgen-query-plan-exp branch gets the correct answer. This problem was discovered while looking for different plans generated by the nextgen query planner versus the legacy query planner.

  This problem will be fixed when the nextgen query planner is merged to trunk. The purpose of this ticket is to record the bug.
  </blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The sqlite3_stmt_status() interface with the 
  SQLITE_STMTSTATUS_FULLSCAN_STEP parameter can sometimes return zero when
  in fact a full scan of an index is undertaken.  The problem arises when
  a full scan of an index is used in place of a full table scan in order to
  fulfill an ORDER BY clause without sorting.  

  The problem is fixed in the NGQP.  The problem was noticed when comparing
  results between NGQP and the legacy query planner.  This ticket is merely to
  document the problem.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The sqlite3_stmt_readonly() interface returns non-zero (true) for these
  statements, even though these statements will potentially modify the database
  file:

  <blockquote><verbatim>
  PRAGMA journal_mode=WAL;
  PRAGMA wal_checkpoint;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script causes an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE p (
    id INTEGER PRIMARY KEY,
    uid VARCHAR(36),
    t INTEGER
  );

  CREATE TABLE pa (
    id INTEGER PRIMARY KEY,
    a_uid VARCHAR(36)
  );

  CREATE TABLE a (
    id INTEGER PRIMARY KEY,
    uid VARCHAR(36),
    t INTEGER
  );

  SELECT DISTINCT *
  FROM
    pa
    LEFT JOIN p ON p.uid='1234'
    LEFT JOIN a ON a.uid=pa.a_uid
  WHERE
    a.t=p.t
  ;
  </verbatim></blockquote>

  This problem was reported on the SQLite mailing list by Jens Miltner.
  Bisecting shows that the problem was introduced by check-in
  [5f4907e1c6230e3d] that added support for transitive constraints.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL should return a single row of result.  But due to the
  transitive constraint optimization, it either hits an assertion fault or
  it returns zero rows (depending on whether the SQLITE_DEBUG compile-time option
  is used.)  Note that SQLITE_ENABLE_STAT3 is required for this bug to appear.

  <blockquote><verbatim>
  CREATE TABLE t1(w INTEGER PRIMARY KEY, x);
  CREATE TABLE t2(y INTEGER, z);
  INSERT INTO t1 VALUES(1,2);
  INSERT INTO t2 VALUES(1,3);
  SELECT *
    FROM t1 CROSS JOIN t2
   WHERE w=y AND y IS NOT NULL;
  </verbatim></blockquote>

  The problem appears to have been introduced by check-in [38852f158ab].

  Low-level technical details:  The "y IS NOT NULL" constraint is translated
  into a "y>NULL" virtual constraint when SQLITE_ENABLE_STAT3 is defined.  
  The y>NULL constraint tagged to indicate that the usual not-NULL check on 
  the right-hand side should be
  skipped since this is an artificial constraint.  The normal index handler 
  knows to check for that tag and skip the not-NULL check.  But after the
  transitive constraints were added, there is now also a "w>NULL" virtual
  constraint (because w=x).  The w>NULL constraint is also tagged, but the 
  INTEGER PRIMARY KEY handlers does not know to check for that tag and to 
  skip the not-NULL check.  It sees that the right-hand side of the constraint
  is NULL and all fails the constraint, resulting in no output rows.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL script results in an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE abc(a,b,c); INSERT INTO abc VALUES(1,2,3);
  CREATE TABLE def(d,e,f); INSERT INTO def VALUES(3,4,5);
  SELECT * FROM abc WHERE 0 UNION SELECT * FROM def;
  </verbatim></blockquote>

  This bug appears to have been in the code essentially forever.  Incorrect
  VDBE code has been generated since version 3.3.7 (circa 2006) and prior to
  that the code generator segfaults.


login: nobody
mimetype: text/html
icomment:
  <p>
  In the <code>src/vtab.c</code> and the <code>sqlite3VtabSync</code> function, <code>pVtab->zErrMsg</code> is freed without then being zeroed.  This leads to a double-free error condition when <code>pVtab->zErrMsg</code> is later examined, found non-zero and freed (again).
  </p>
  <p>
  The simplest way to reproduce the problem is to create a virtual table implementation that implements <code>xSync</code>.  In <code>xSync</code>, set <code>pVtab->zErrMsg</code> to an allocated string and return an error result.  At the next callback opportunity (mine was in <code>xRollback</code>), <code>pVtab->zErrMsg</code> will still be set, when it should be zero.
  </p>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The auxiliary information associated with specific function parameters and
  registered by the sqlite3_set_auxdata() routine is not cleared when the
  function argument is a bound parameter which is modified by one of the
  sqlite3_bind_xxxx() interfaces.

  On the other hand, the auxdata is cleared immediately, even before the
  sqlite3_set_auxdata() routine returns, if a memory allocation error occurs.
  This latter behavior probably cannot be changed, but it at least needs to
  be documented.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Consider the following SQL:

  <blockquote><verbatim>
  CREATE TABLE t1(m CHAR(2));
  INSERT INTO t1 VALUES('ax');
  INSERT INTO t1 VALUES('bx');
  INSERT INTO t1 VALUES('cy');
  SELECT '1', count(*), substr(m,2,1) AS m FROM t1 GROUP BY m;
  SELECT '2', count(*), substr(m,2,1) AS mx FROM t1 GROUP BY m;
  </verbatim></blockquote>

  Both queries should give the same three-row result.  But in SQLite, the first
  query gives a two-row results where one row has a count of 2.  The expression of a GROUP BY clause should bind preferentially to the input table, not the output table.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  It has come to light that some legacy Android applications are coded
  with queries that are logically equivalent to this:

  <blockquote><verbatim>
  CREATE TABLE t1(a,x);
  CREATE TABLE t2(a,y);
  SELECT t1.a AS a, x, y FROM t1, t2 GROUP BY a;
  </verbatim></blockquote>

  The original fix for this ticket (see check-in [f2d175f975cd]) caused such
  queries to give an error:  "ambiguous column: a".  The query is indeed 
  incorrect.  Nevertheless, we need to find a way to get SQLite to run and
  return the correct result even for these incorrect, legacy queries.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Consider the following SQL:

  <blockquote><verbatim>
  CREATE TABLE t1(m CHAR(2));
  INSERT INTO t1 VALUES('az');
  INSERT INTO t1 VALUES('by');
  INSERT INTO t1 VALUES('cx');
  SELECT '1', substr(m,2) AS m FROM t1 ORDER BY m;
  SELECT '2', substr(m,2) AS m FROM t1 ORDER BY m COLLATE binary;
  </verbatim></blockquote>

  The identifier "m" used in the ORDER BY clauses might refer to either the
  input column "t1.m" or to the second column of output.  The SQL standards
  are ambiguous on which should be used.  PostgreSQL uses the output column
  for unadorned identifiers, such as found in query (1) and it uses
  the input column when m is part of an expression, such as query (2).
  Hence, PostgreSQL answers queries (1) and (2) with rows that in opposite
  orders from one another.  MS-SQL 2008 works like PosgreSQL.  Oracle appears
  to use the output column name in both cases.

  SQLite should be changed to agree with PostgreSQL.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Add to the test case above:

  <blockquote><pre>
  SELECT '3', substr(m,2) AS m FROM t1 ORDER BY lower(m);
  </pre></blockquote>

  PostgreSQL binds ORDER BY terms to output columns if the term is an ordinal
  or a simple identifier that matches the name of an output column.  If if the
  ORDER BY term is an expression, it matches against input columns instead.
  PostgreSQL considers "m COLLATE binary" to be an expression.  But for
  compatibility with SQL92 and historical compatibility with older versions
  of SQLite, we will break with PostgreSQL and bind ORDER BY terms of the
  form "m COLLATE binary" to output columns.  However, "lower(m)" is clearly an
  expression, so both PostgreSQL and SQLite will resolve the "m" variable there
  to match the input column, t1.m.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  According to documentation here: http://www.sqlite.org/datatype3.html

  "The expression "x BETWEEN y and z" is logically equivalent to two comparisons "x >= y AND x <= z" and works with respect to collating functions as if it were two separate comparisons."

  However:

  <verbatim>
  SQLite version 3.7.17 2013-05-17 12:52:58
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> CREATE TABLE t1(x);
  sqlite> INSERT INTO t1 VALUES('b');
  sqlite> INSERT INTO t1 VALUES('B');
  sqlite> SELECT * FROM t1 WHERE x COLLATE nocase BETWEEN 'a' AND 'c';
  b
  sqlite> SELECT * FROM t1 WHERE x COLLATE nocase >= 'a' AND x COLLATE nocase <= 'c';
  b
  B
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query returns zero rows in version 3.8.0.  The correct answer
  (returned by 3.7.17 and by PostgreSQL) is a single row:  1|NULL|1

  <blockquote><verbatim>
  CREATE TABLE t (id INTEGER PRIMARY KEY NOT NULL);
  INSERT INTO t VALUES (1);
  SELECT *
    FROM t AS t1
         LEFT OUTER JOIN t AS t2 ON t2.id BETWEEN 10 AND 20
         JOIN t AS t3 ON (t3.id = t1.id OR
                          t2.id IS NOT NULL AND t3.id = t2.id);
  </verbatim></blockquote>


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following query causes a buffer overflow in the NGQP module. Any query with a single table in the FROM clause that may use more than four fields of a PRIMARY KEY or UNIQUE index is similarly vulnerable.

  <verbatim>
    CREATE TABLE t1(a,b,c,d,e, PRIMARY KEY(a,b,c,d,e));
    SELECT * FROM t1 WHERE a=? AND b=? AND c=? AND d=? AND e=?;
  </verbatim>

  3.8.0 is the only version with this bug. Earlier versions are Ok.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL segfaults:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2(b,c);
  INSERT INTO t1 VALUES(1);
  SELECT DISTINCT a FROM t1 LEFT JOIN t2 ON a=b ORDER BY c IS NULL;
  </verbatim></blockquote>

  Initially reported on the sqlite-users mailing list by Alexandre Courbot.


login: dan
mimetype: text/x-fossil-plain
icomment:
  In shared-cache mode, SQLite uses read and write locks on individual tables to isolate transactions from other shared-cache users. This means that if one shared-cache user opens a transaction and modifies the contents of a table, a write-lock is taken on that table to prevent other shared-cache clients from reading the modified data before it is committed. See section 2.2 of the following for details:

     http://www.sqlite.org/sharedcache.html

  Currently, if the statement that writes to the table is a DELETE statement that uses the truncate optimization, this lock is not being obtained. Meaning a second shared-cache client is not prohibited from observing an empty table before the transaction is committed (like a read_uncommitted client would).

  To clarify, this:

    BEGIN; DELETE FROM tbl WHERE 1;

  obtains the correct shared-cache locks, but this:

    BEGIN; DELETE FROM tbl;

  does not.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The final select in the following SQL code returns 11 instead of 128 because
  it is counting the entries in the partial index rather than the entries in
  the table.  The count(*) optimization does not know to ignore partial indices.

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c,d);
  INSERT INTO t1(a) VALUES(1),(2);
  INSERT INTO t1(a) SELECT a+2 FROM t1;
  INSERT INTO t1(a) SELECT a+4 FROM t1;
  INSERT INTO t1(a) SELECT a+8 FROM t1;
  INSERT INTO t1(a) SELECT a+16 FROM t1;
  INSERT INTO t1(a) SELECT a+32 FROM t1;
  INSERT INTO t1(a) SELECT a+64 FROM t1;
  CREATE INDEX t1a ON t1(a) WHERE a BETWEEN 10 AND 20;
  SELECT count(*) FROM t1;
  </verbatim></blockquote>

  This defect was discovered by the developers during code review.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL should output a single result row, but in SQLite 3.8.1 it outputs no rows:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2(b);
  CREATE TABLE t3(c INTEGER PRIMARY KEY);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t3 VALUES(1);
  SELECT 'a row' FROM t1 LEFT JOIN t2 ON b=a JOIN t3 ON c=a;
  </verbatim></blockquote>


login: dan
mimetype: text/x-fossil-wiki
icomment:
  When processing a LEFT JOIN, the data sources to the left and right of the join may not be reordered. In other words, "t1 LEFT JOIN t2" must be implemented with "t1" as the outer loop. This is not the case with other joins, where SQLite is free to reorder the nesting of loops.

  However, it ignores this for virtual tables. For example:

  <verbatim>
  SQLite version 3.8.2 2013-11-12 01:11:56
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> CREATE TABLE t1(a);
  sqlite> CREATE VIRTUAL TABLE t2 USING rtree(b, x1,x2);
  sqlite> EXPLAIN QUERY PLAN SELECT a, b FROM t1 LEFT JOIN t2 ON (+a = +b);
  0|0|1|SCAN TABLE t2 VIRTUAL TABLE INDEX 2:
  0|1|0|SCAN TABLE t1
  sqlite> 
  </verbatim>


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Or:

  <verbatim>

  $ ./sqlite3 
  SQLite version 3.8.2 2013-11-12 01:11:56
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> CREATE TABLE t1(a);
  sqlite> CREATE VIRTUAL TABLE t2 USING rtree(b, x1,x2);
  sqlite> INSERT INTO t1 VALUES(456);
  sqlite> SELECT a, b FROM t1 LEFT JOIN t2;
  Segmentation fault

  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an array overrun in the skip-scan logic, resulting
  in memory corruption:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c,d,e,f,g,h,i);
  CREATE INDEX t1all ON t1(a,b,c,d,e,f,g,h);
  INSERT INTO t1 VALUES(1,2,3,4,5,6,7,8,9);
  ANALYZE;
  DELETE FROM sqlite_stat1;
  INSERT INTO sqlite_stat1 
    VALUES('t1','t1all','655360 163840 40960 10240 2560 640 160 40 10');
  ANALYZE sqlite_master;
  SELECT i FROM t1 WHERE h=8;
  </verbatim></blockquote>

  This problem was introduced by check-in [b0bb975c0986fe01] which was part
  of the 3.8.2 release.  The problem was reported to the developer via private
  email.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL script generates an assertion fault:

  <blockquote><pre>
  CREATE VIRTUAL TABLE t1 USING fts4(a,b);
  SELECT * FROM t1 WHERE a=b;
  </pre></blockquote>

  Bisecting shows that this problem was introduced on [2013-01-17]
  (SQLite version 3.7.16) by check-in [38852f158ab20bb4].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The logic that computes DISTINCT sometimes thinks that a zeroblob() and a
  blob of all zeros are different when they should be the same.  The following
  query illustrates the problem:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER);
  INSERT INTO t1 VALUES(3);
  INSERT INTO t1 VALUES(2);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t1 VALUES(2);
  INSERT INTO t1 VALUES(3);
  CREATE TABLE t2(x);
  INSERT INTO t2
    SELECT DISTINCT
      CASE a WHEN 1 THEN x'0000000000'
             WHEN 2 THEN zeroblob(5)
             ELSE 'xyzzy' END
      FROM t1;
  SELECT quote(x) FROM t2;
  </verbatim></blockquote>

  The final output above should only generate two rows: x'0000000000' and 'xyzzy'
  since x'0000000000' and zeroblob(5) are logically equivalent.  But it ends up
  generating three rows.  There are two x'0000000000' values in the output.

  Bisecting shows that this problem was introduced by check-in
  [45e581bff7a75db] on [2011-07-02] for SQLite version 3.7.8.

  This problem was discovered by the developers during code analysis and
  has never been reported in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  SELECT statements in a compound SELECT other than the right-most SELECT are
  unable to use common table expressions.  Example:

  <blockquote><verbatim>
  WITH RECURSIVE
    t1(x) AS (VALUES(2) UNION ALL SELECT x+2 FROM t1 WHERE x<20),
    t2(y) AS (VALUES(3) UNION ALL SELECT y+3 FROM t2 WHERE y<20)
  SELECT x FROM t1 EXCEPT SELECT y FROM t2 ORDER BY 1;
  </verbatim></blockquote>

  The query above is suppose to return all even integers less than 20 that are
  not divisible by 3.  But instead we get a message: "no such table: t1".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When SQLite is compiled using SQLITE_ENABLE_STAT3 or SQLITE_ENABLE_STAT4, it
  gets the wrong answer for the first of the two queries below:

  <blockquote><verbatim>
  CREATE TABLE nodes(
     local_relpath  TEXT PRIMARY KEY,
     moved_to  TEXT
  );
  INSERT INTO nodes VALUES('A',NULL);
  INSERT INTO nodes VALUES('A/B',NULL);
  INSERT INTO nodes VALUES('',NULL);
  INSERT INTO nodes VALUES('A/B/C-move',NULL);
  INSERT INTO nodes VALUES('A/B/C','A/B/C-move');
  INSERT INTO nodes VALUES('A/B-move',NULL);
  INSERT INTO nodes VALUES('A/B-move/C-move',NULL);
  INSERT INTO nodes VALUES('A/B-move/C','x');

  SELECT local_relpath, moved_to
   FROM nodes
  WHERE (local_relpath = 'A/B' OR ((local_relpath > 'A/B/') AND (local_relpath < 'A/B0')))
    AND moved_to IS NOT NULL;

  .print ------------------------------------------------

  SELECT local_relpath, moved_to
   FROM nodes
  WHERE (+local_relpath = 'A/B' OR ((local_relpath > 'A/B/') AND (local_relpath < 'A/B0')))
    AND moved_to IS NOT NULL;
  </verbatim></blockquote>

  The bug appears to have been inserted into trunk with the STAT4 changes
  of check-in [a32af0abe5fa6d] which occurred just after the release of
  3.8.0.  The bug first appeared in 3.8.1.

  Note that it is not necessary to run
  ANALYZE or to have sqlite_stat3 or sqlite_stat4 tables in the database.
  The only requirement is to compile with SQLITE_ENABLE_STAT3 or
  SQLITE_ENABLE_STAT4.  

  The extra unary "+" operator in the second queries disable the OR optimization
  and enables the second query to get the correct answer regardless.

  This bug was discovered in the wild by the SVN developers.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is a simpler script to illustrate the problem.  The query should return
  a single row, but it is returning no rows:

  <blockquote><verbatim>
  CREATE TABLE t1(x TEXT UNIQUE, y TEXT UNIQUE, z);
  INSERT INTO t1 VALUES('ghi','jkl','y');
  SELECT * FROM t1 WHERE (x='ghi' OR y='jkl') AND z IS NOT NULL;
  </verbatim></blockquote>

  The exact cause of the problem is still unknown.  However, it appears to have
  something to do with an adverse interaction between an IS NOT NULL constraint
  and a constraint composed of two or more OR-connected terms.  IS NOT NULL
  constraints are never considered except when SQLITE_ENABLE_STAT3 or _STAT4 are
  used, so that explains why those compile-time options are required in order to
  demonstrate the problem.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Now fixed on trunk.  Work-arounds:

    1.  Omit the SQLITE_ENABLE_STAT3 and SQLITE_ENABLE_STAT4 compile-time options.

    2.  For WHERE-clause terms of the form "column IS NOT NULL" add a unary "+"
        operator before the column name.  Like this:  "+column IS NOT NULL".

    3.  After opening each database connection (in variable db) run code like the
        following to disable the STAT3 and STAT4 logic:
  <blockquote><verbatim>
      if( strcmp(sqlite3_sourceid(),"2013-08-26 23:00")>0
       && strcmp(sqlite3_sourceid(),"2014-02-11")<0 ){
        sqlite3_test_control(SQLITE_TESTCTRL_OPTIMIZATIONS, db, 0x800);
      }
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL gives a "no such column" error while parsing the second
  query:

  <blockquote><verbatim>
  CREATE TABLE t1(a PRIMARY KEY) WITHOUT ROWID;
  INSERT INTO t1 VALUES('abc');
  CREATE TABLE t2(x);
  INSERT INTO t2 VALUES('xyz');
  SELECT t2.rowid FROM t1, t2; -- works
  SELECT t2.rowid FROM t2, t1; -- "no such column" error
  </verbatim></blockquote>

  Note that the error only arises if the second of the two tables in the
  join is a WITHOUT ROWID table.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Temp registers can sometimes be used in concurrent co-routines, resulting in a
  incorrect answer.  The attached script demonstrations one manifestation of this.
  The attachment fails with any check-in after the introduction of the NGQP.
  But the register allocation problem predates that change.  The cut-over of the
  NGQP only makes the attached test case work.  Probably other test cases can be
  made to fail for versions prior to the NGQP.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following query, the ORDER BY clause is ignored:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b DEFAULT 1);
  INSERT INTO t1(a) VALUES(1),(2),(3),(4),(5);
  CREATE TABLE t2(x INTEGER PRIMARY KEY);
  INSERT INTO t2 VALUES(1);
  SELECT * FROM t1, t2 WHERE x=1 ORDER BY random();
  </verbatim></blockquote>

  This problem appears to have been introduced on 2013-06-21 (version 3.8.0)
  with the introduction of the NGQP.


login: nobody
mimetype: text/x-fossil-wiki
icomment:
  Within a trigger program, if match for a column reference does not match any column in the outermost context and the table name is not "new" or "old", name resolution may malfunction. If assert() is enabled, an assert() may fail:

  <verbatim>
  sqlite>  CREATE TABLE x(a);
  sqlite> CREATE TRIGGER t AFTER INSERT ON x BEGIN
     ...> SELECT * FROM x WHERE abc.a = 1;
       ...>   END;
  sqlite>   INSERT INTO x VALUES('assert');
  sqlite3: /home/dan/work/sqlite/trunk/src/expr.c:2869: sqlite3ExprCodeTarget: Assertion `pExpr->iTable==0 || pExpr->iTable==1' failed.
  </verbatim>

  Or, if assert() is not enabled:

  <verbatim>
  sqlite> CREATE TABLE w(a);
  sqlite> CREATE TABLE x(a);
  sqlite> CREATE TABLE y(a);
  sqlite> CREATE TABLE z(a);
  sqlite> INSERT INTO x(a) VALUES(5);
  sqlite> INSERT INTO y(a) VALUES(10);
  sqlite> CREATE TRIGGER t AFTER INSERT ON w BEGIN
     ...> INSERT INTO z SELECT (SELECT x.a + y.a FROM y) FROM x;
     ...> END;
  sqlite> INSERT INTO w VALUES('incorrect');
  sqlite> SELECT * FROM z;

  sqlite> SELECT typeof(a) FROM z;
  null
  sqlite> -- should be an integer value 15 
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is the test script from above with the command-line shell output
  elided, for clarity of presentation and simplified copy/paste:

  <blockquote><verbatim>
  CREATE TABLE w(a);
  CREATE TABLE x(a);
  CREATE TABLE y(a);
  CREATE TABLE z(a);
  INSERT INTO x(a) VALUES(5);
  INSERT INTO y(a) VALUES(10);
  CREATE TRIGGER t AFTER INSERT ON w BEGIN
    INSERT INTO z SELECT (SELECT x.a+y.a FROM y) FROM x;
  END;
  INSERT INTO w VALUES('incorrect');
  SELECT a, typeof(a) FROM z; -- should be 15, integer
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced with the cut-over of support
  for WITHOUT ROWID tables by check-in [c80e229dd9c12] on [2013-11-07].  The
  problem appears in releases 3.8.2, 3.8.3, and 3.8.3.1.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  CREATE INDEX t1ab ON t1(b,a);
  SELECT * FROM t1 WHERE a=? AND b IN (?);
  </verbatim></blockquote>

  The problem appears to have been introduced by the NGQP cut-over on
  2013-06-26 (check-in [0fe31f60ca] for version 3.8.0).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The failing assert() statement is incorrect.  So compiling without -DSQLITE_DEBUG
  and thus omitting all assert() statements is an easy work-around.

  This problem only arises in cases where there is a multi-column index and
  the earlier column is constrained by "x IN (?1)" and the later column is
  constrained by "y=?2".  The fix on trunk is to change the parser so that
  when the right-hand side of IN is a single expression, the IN operator is
  converted into ==.  This accomplishes the same thing, and it results in 
  tighter VDBE code in the corner-case of a single value on the RHS of IN.

  Since after the change above, all IN operators have two or more elements on
  their RHS, the assert() statement is now correct and is retained.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL unexpectedly gives a single row of output:

  <blockquote><verbatim>
  SELECT 1,2,3 LIMIT 1 OFFSET 25;
  </verbatim></blockquote>

  This problem was discovered during internal testing and analysis and has
  not been observed in the wild.  The problem is believed to have been in
  the code for a very long time.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  <verbatim>
  SQLite version 3.8.5 2014-03-22 02:19:53
  Enter ".help" for usage hints.
  Connected to a transient in-memory database.
  Use ".open FILENAME" to reopen on a persistent database.
  sqlite> create table t(x);
  sqlite> insert into t values('1');
  sqlite> select typeof(x) from t;
  text
  sqlite> select typeof(x) from t where x=x+0;
  integer
  sqlite> 
  </verbatim>

  Introduced by [58704ed1f4cd78bb].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another example script:

  <blockquote><verbatim>
  CREATE TABLE t(x,y);
  INSERT INTO t VALUES('1','1');
  SELECT typeof(x), typeof(y) FROM t WHERE 1=x+0 AND y=='1';
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query answers 'wrong' even though the correct answer is 'right':

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES('right'),('wrong');
  SELECT DISTINCT x
   FROM (SELECT x FROM t1 GROUP BY x)
   WHERE x='right'
   ORDER BY x;
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced by check-in
  [1e64dd782a126f48d78c4] on 2014-02-08 and first appeared in
  SQLite version 3.8.4.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query returns two rows instead of just one:

  <blockquote><verbatim>
  create table t(c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17);
  create index tc0 on t(c0);
  create index tc1 on t(c1);
  create index tc2 on t(c2);
  create index tc3 on t(c3);
  create index tc4 on t(c4);
  create index tc5 on t(c5);
  create index tc6 on t(c6);
  create index tc7 on t(c7);
  create index tc8 on t(c8);
  create index tc9 on t(c9);
  create index tc10 on t(c10);
  create index tc11 on t(c11);
  create index tc12 on t(c12);
  create index tc13 on t(c13);
  create index tc14 on t(c14);
  create index tc15 on t(c15);
  create index tc16 on t(c16);
  create index tc17 on t(c17);

  insert into t(c0, c16) VALUES (1,1);

  select * from t where
    c0=1 or  c1=1 or  c2=1 or  c3=1 or
    c4=1 or  c5=1 or  c6=1 or  c7=1 or
    c8=1 or  c9=1 or c10=1 or c11=1 or
    c12=1 or c13=1 or c14=1 or c15=1 or
    c16=1 or c17=1;
  </verbatim></blockquote>

  This problem has been latent in the code for about 5 years, since checkin
  [b101cf70b75c9772aaf] of version 3.6.14.  The bug was found by Peter Reid
  and reported on the sqlite-dev mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script causes an assertion fault on all prior versions of
  SQLite that support partial indices:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  CREATE UNIQUE INDEX t1ab ON t1(a,b);
  CREATE INDEX t1b ON t1(b) WHERE b=1;
  INSERT INTO t1(a,b) VALUES(123,456);
  UPDATE OR REPLACE t1 SET b=789;
  </verbatim></blockquote>

  This bug was introduced with the addition of partial index support 
  by check-in [478113f18b1d2860] on 2013-08-02 for SQLite version 3.8.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The sqlite_rename_parent() SQL function is an undocumented function used as
  part of the implementation of ALTER TABLE.  This function should never be
  access by an application.  But if an application does invoke the function and
  gives it a NULL parameter in the 1st or 2nd argument, a crash might result.

  <blockquote><verbatim>
  SELECT sqlite_rename_parent(NULL,1,1);
  </verbatim></blockquote>

  The problem appears to have existed since ALTER TABLE was first enhanced to
  be able to deal with foreign key constraints, by check-in [b4a10c39e726dc190]
  on 2009-09-26 for SQLite version 3.6.19.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  For example:

  <verbatim>

  Enter ".help" for usage hints.
  Connected to a transient in-memory database.
  Use ".open FILENAME" to reopen on a persistent database.
  sqlite> CREATE TABLE t1 (x, y);
  sqlite> INSERT INTO t1 VALUES (1, 1); 
  sqlite> INSERT INTO t1 VALUES (2, 0);
  sqlite> SELECT x, y FROM t1 GROUP BY x, y ORDER BY x,y;
  1|1
  2|0
  sqlite> CREATE INDEX i1 ON t1(y, x);
  sqlite> SELECT x, y FROM t1 GROUP BY x, y ORDER BY x,y;
  2|0
  1|1

  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the query at the end of the following code, the ORDER BY clause is ignored
  and the results appear in the wrong order:

  <blockquote><verbatim>
  CREATE TABLE t1(x,y);
  INSERT INTO t1 VALUES(1,1),(2,0);
  CREATE INDEX t1yx ON t1(y,x);
  SELECT x,y FROM t1 GROUP BY x, y ORDER BY x, y;
  </verbatim></blockquote>

  This problem appears to have been introduced into trunk by the cutover to the
  next generation query planner in check-in [0fe31f60cadc5fe5a9d]
  on 2013-06-26.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL raises an assertion fault:

  <blockquote><verbatim>
  CREATE VIRTUAL TABLE fts USING fts4(content TEXT);
  CREATE TABLE t1(x TEXT PRIMARY KEY, y);
  SELECT DISTINCT fts.rowid, t1.x
    FROM fts, t1
   WHERE fts MATCH 'hello world'
     AND content=x
   ORDER BY y;
  </verbatim></blockquote>

  This problem was introduced with the Next Generation Query Planner by
  check-in [0fe31f60cadc5fe5] on 2013-06-26 for version 3.8.0.  The problem
  was first detected by RedHat
  (see [https://bugzilla.redhat.com/show_bug.cgi?id=1075889]).


login: dan
mimetype: text/x-fossil-wiki
icomment:
  If an "ALTER TABLE ... ADD COLUMN ..." statement is used to add a column to an existing table, the existing table entries are not modified. Instead, it is assumed that any fields missing from the end of such records take the default values for their respective columns. 

  The following bug is caused by copying such records to a different table with different default values:

  <verbatim>
  SQLite version 3.8.5 2014-04-25 14:42:17
  Enter ".help" for usage hints.
  Connected to a transient in-memory database.
  Use ".open FILENAME" to reopen on a persistent database.
  sqlite> CREATE TABLE t1(a);        
  sqlite> INSERT INTO t1 VALUES(1);    
  sqlite> ALTER TABLE t1 ADD COLUMN b DEFAULT 2;
  sqlite> CREATE TABLE t2(a, b);
  sqlite> INSERT INTO t2 SELECT * FROM t1;
  sqlite> SELECT * FROM t2;
  1|
  sqlite> SELECT * FROM t1;            
  1|2
  sqlite> 
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The group_concat(x,'') aggregate function returns NULL when all inputs
  are empty strings.  The correct return should be an empty string.

  <blockquote><verbatim>
  SELECT typeof(group_concat(x,''))
    FROM (SELECT '' AS x UNION ALL SELECT '');
  </verbatim></blockquote>

  This problem appears to have existed in the code since the group_concat()
  function was initially added to SQLite in version 3.5.4, 2007-12-14.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The INSERT statement at the end of the following SQL causes SQLite to enter
  an infinite loop:

  <blockquote><verbatim>
  CREATE TABLE tab1( value INTEGER);
  INSERT INTO tab1 (value, _rowid_) values( 11, 1);
  INSERT INTO tab1 (value, _rowid_) SELECT 22,2;
  </verbatim></blockquote>

  Bisecting shows that this bug was introduced by check-in [a72687699ba2af2e7]
  for version 3.8.4.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL command sequence, query 3 returns no results when it
  should return "1":

  <blockquote><verbatim>
  CREATE TABLE t1(v text);
  INSERT INTO t1 VALUES('1'),('2');
  SELECT 1, v FROM t1 WHERE v=1;
  SELECT 2, v FROM t1 WHERE v='1';
  SELECT 3, v FROM t1 WHERE v IN (1);
  SELECT 4, v FROM t1 WHERE v IN ('1');
  SELECT 5, v FROM t1 WHERE v IN (1,2);
  SELECT 6, v FROM t1 WHERE v IN ('1','2');
  </verbatim></blockquote>

  This problem was reported on the mailing list by Guillaume Fougnies.
  It appears to have been introduced by check-in [2ea4a9f75f46eaa928ba]
  on [2014-03-20] and was released in version 3.8.5.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The samples added to the sqlite_stat4 table by the ANALYZE command are
  incorrect for WITHOUT ROWID tables.  After the key columns, a BLOB that
  is the record encoding of the key is added.  This is contrary to the
  file-format documentation which says that the sample should be an exact
  copy of the index entry.  Either the file-format document needs to be
  fixed or else the samples need to be computed differently.  Or both.

  Note that the bug has been harmless up to now because the query planner
  is unable to use columns in the sample past the key and so the incorrect
  sample content was never accessed or used.  However, if the query planner
  is improved to make use of the extra sample content, older sqlite_stat4
  tables might need to be recomputed.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following script returns a single row:

  <verbatim>
    CREATE VIRTUAL TABLE t3 USING rtree(Pid, xmin, xmax);   
    INSERT INTO t3 VALUES(1,0,0);       
    INSERT INTO t3 VALUES(2,0,0);            
    SELECT * FROM t3 WHERE Pid IN (1, 2);
  </verbatim>

  But should return two. Introduced by [af2cbe64adab5f9e].

  Bug first appeared in 3.8.5.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Or the same problem using a join:

  <verbatim>
  WITH r(x) AS (
    SELECT 1 UNION ALL
    SELECT 2 UNION ALL
    SELECT 3
  )
  SELECT * FROM r CROSS JOIN t3 WHERE Pid=x;    
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The CREATE UNIQUE INDEX on the third line below does not fail even though
  the x column values are not all unique.

  <blockquote><verbatim>
  CREATE TABLE t1(x NOT NULL);
  INSERT INTO t1 VALUES(1),(2),(2),(3);
  CREATE UNIQUE INDEX t1x ON t1(x);
  </verbatim></blockquote>

  If the index is created before the INSERT, then uniqueness is enforced at
  the point of the INSERT.  Note that the NOT NULL on the indexed column
  seems to be required in order to exhibit this bug.

  "PRAGMA integrity_check" does not detect the resulting malformed database.
  That might be considered a separate issue.

  Bisecting shows that this problem was introduced by the addition of
  WITHOUT ROWID support in version 3.8.2, specifically in check-in
  [c80e229dd9c1230] on 2013-11-07.  This problem was reported on the
  mailing list by Pavel Pimenov.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL script segfaults while running the INSERT statement:

  <blockquote><verbatim>
  CREATE TABLE t1(x DEFAULT(max(1)));
  INSERT INTO t1(rowid) VALUES(1);
  </verbatim></blockquote>

  The problem appears to be the use of an aggregate function as part of
  the default value for a column.  This problem has been in the code
  since at least 2006-01-01 and probably before that.  This problem was
  detected and reported on the sqlite-users@sqlite.org mailing list by
  Zsbán Ambrus.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The callback to sqlite3_trace() is suppose to be invoked for every SQL statement
  that is run by the VDBE.  But sometimes the callback will be suppressed.  If
  the prepared statement is marked as expired (which can happen, for example,
  if the RHS of an expression like "x GLOB ?1" is rebound and X is an indexed
  text column) then the callback is not invoked.  This can be seen in the shell
  by running:

  <blockquote><verbatim>
  .trace stderr
  .tables
  </verbatim></blockquote>

  The SELECT statement that implements the ".tables" command is not traced.

  This problem appears to have existed since 2012-10-03, version 3.7.15 and
  check-in [39f763bfc04174ee0fe2cdf6a92].  The problem was discovered during
  internal testing and has not been observed in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an assertion fault while running sqlite3_prepare()
  on the DELETE statement:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  CREATE TABLE t2(y);
  CREATE VIRTUAL TABLE t3 USING rtree(a,b,c);
  CREATE TRIGGER t2del AFTER DELETE ON t2 WHEN (SELECT 1 from t1) BEGIN 
    DELETE FROM t3 WHERE a=old.y; 
  END;
  DELETE FROM t2 WHERE y=1;
  </verbatim></blockquote>

  The problem appears to have been introduced by the addition of support
  for WITHOUT ROWID tables in SQLite version 3.8.2, specifically on
  check-in [c80e229dd9c1230a] on 2013-11-07.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statement at the end of the following block of SQL outputs no rows, but it should output one row.

  <blockquote><verbatim>
  CREATE TABLE t1(a, b);
  CREATE TABLE t2(c, d);
  INSERT INTO t1 VALUES(1, 'xyz');
  INSERT INTO t2 VALUES('abc', 'not xyz');
  CREATE INDEX i2 ON t2(c) WHERE d='xyz'; 
  SELECT * FROM (SELECT * FROM t1 WHERE a=1 AND b='xyz'), t2 WHERE c='abc';\
  </verbatim></blockquote>

  A correct result is obtained if the index created on the penultimate line is convert into a complete index or if the index is omitted entirely.

  This bug appears to have been in the implementation of partial indexes from their introduction in SQLite version 3.8.0, 2013-08-26.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statement at the end of the following SQL code causes an assertion
  fault if assertions are enabled, or gives the error "1st ORDER BY term out
  of range - should be between 1 and 1" if asserts are disabled.

  <blockquote><verbatim>
  CREATE TABLE t1 (id INTEGER PRIMARY KEY, data TEXT);
  INSERT INTO t1(id,data) VALUES(9,'nine-a');
  INSERT INTO t1(id,data) VALUES(10,'ten-a');
  INSERT INTO t1(id,data) VALUES(11,'eleven-a');
  CREATE TABLE t2 (id INTEGER PRIMARY KEY, data TEXT);
  INSERT INTO t2(id,data) VALUES(9,'nine-b');
  INSERT INTO t2(id,data) VALUES(10,'ten-b');
  INSERT INTO t2(id,data) VALUES(11,'eleven-b');

  SELECT id FROM (
    SELECT id,data FROM (
       SELECT * FROM t1 UNION ALL SELECT * FROM t2
    )
    WHERE id=10 ORDER BY data
  );
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced by check-in [a49e909c8738317]
  on 2012-04-27 and first appeared in SQLite release 3.7.12.  The problem was
  originally detected by Yuanzhong Xu and reported on the sqlite-users mailing
  list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The same problem can result in queries that sort on the wrong column, thus
  giving an incorrect answer.  For example:

  <blockquote><verbatim>
  CREATE TABLE t1(id INTEGER, data TEXT);
  CREATE TABLE t2(id INTEGER, data TEXT);
  INSERT INTO t1 VALUES(4, 'a');
  INSERT INTO t2 VALUES(3, 'b');
  INSERT INTO t1 VALUES(2, 'c');
  INSERT INTO t2 VALUES(1, 'd');

  SELECT data, id FROM (
    SELECT id, data FROM (
       SELECT * FROM t1 UNION ALL SELECT * FROM t2
    ) ORDER BY data
  );
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL code causes a segfault while running the INSERT statement:

  <blockquote><verbatim>
  CREATE TABLE t1(a text, b text DEFAULT(?1));
  INSERT INTO t1(a) VALUES('hello');
  SELECT * FROM t1;
  </verbatim></blockquote>

  This problem appears to have been introduced in SQLite version 3.6.14, over
  five years ago.  The problem was reported on the sqlite-users mailing list
  by Nathaniel Trellice.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If the content of a table is deleted out from under a SELECT statement
  during a reverse order ROWID scan, an assertion fault might result.
  The following TCL script illustrates the problem:

  <blockquote><verbatim>
  sqlite3 db :memory:
  db function eval eval
  proc eval {sql} {db eval $sql; return [db changes];}
  db eval {
    CREATE TABLE t1(a INTEGER PRIMARY KEY,b);
    INSERT INTO t1(a,b) VALUES(1,'a'),(2,'b'),(3,'c');
    SELECT a, eval('DELETE FROM t1'), b FROM t1 ORDER BY a DESC;
  }
  </verbatim></blockquote>

  The problem was introduced just prior to version 3.8.0 by check-in
  [dc65ad8c4c67b21e].  This problem was discovered during internal testing
  and analysis and has not been observed in the wild.


login: dan
mimetype: text/x-fossil-plain
icomment:
  If SQLITE_DIRECT_OVERFLOW_READ is defined and a large text or blob field that is aligned with the start of an overflow page is read, SQLite may temporarily modify (and then restore) the contents of up to 4 bytes of space immediately before a buffer obtained from malloc(). Here: 

    [http://www.sqlite.org/src/artifact/ede8348a7d62?ln=4166]

  Normally this is not a problem, as the default malloc implementation used by SQLite always allocates at least 4 bytes of space for bookkeeping purposes before each buffer returned to SQLite. However, some custom malloc implemenations, or occasionally the built-in memsys3 or memsys5 implemenations, may not do this. In those cases the results are undefined.

  Problem has existed for as long as SQLITE_DIRECT_OVERFLOW_READ has. Introduced by [2ab14a8467]. First published in version 3.7.9.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL statement generates a segfault in the parser when it
  really should generate a syntax error.

  <blockquote><verbatim>
  SELECT 1 IS #1;
  </verbatim></blockquote>

  This problem has existed since the generalization of the IS and NOT IS
  operators, five years ago, in check-in [98853f6104076c50ea92175e1].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statement at the end of the following SQL generates results 
  in the wrong order.

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t1 VALUES(2);
  CREATE INDEX ix1 ON t1(x DESC);
  SELECT * FROM t1 GROUP BY x ORDER BY x;
  </verbatim></blockquote>

  This problem arises when there is a DESC index which is used to determine 
  the output order and the query has identical GROUP BY and ORDER BY clauses
  and define ASC output order.  The problem seems to have been introduced
  with the new query planner in version 3.8.0, 2013-08-26.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Beginning with version 3.8.7, the following SQL code results in an 
  assertion fault while executing the final UPDATE statement:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  INSERT INTO t1(a,b) VALUES(10,'abc');
  ALTER TABLE t1 ADD COLUMN c;
  CREATE INDEX t1c ON t1(c);
  INSERT INTO t1(a,b,c) VALUES(5,'def','ghi');

  UPDATE t1 SET c=randomblob(10) WHERE c IS NULL;
  </verbatim></blockquote>

  This problem was discovered by Mozilla while trying to incorporate SQLite
  version 3.8.7 into Firefox.  See 
  [https://bugzilla.mozilla.org/show_bug.cgi?id=1084437] for the original
  bug report.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL gives an assertion fault while
  running the SELECT in the CREATE TABLE statement:

  <blockquote><verbatim>
  CREATE TABLE t1(a TEXT);
  INSERT INTO t1(a) VALUES(1),(2);
  CREATE TABLE t2(x TEXT, y TEXT);
  INSERT INTO t2(x,y) VALUES(3,4);
  CREATE TEMP TABLE t3 AS
    SELECT a AS p, coalesce(y,a) AS q FROM t1 LEFT JOIN t2 ON a=x;
  </verbatim></blockquote>

  This problem was reported on the sqlite-users@sqlite.org mailing list by
  Nissl Reinhard.  Bisecting shows that the problem was introduced by
  check-in [36b613ccf0ddb764a].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This appears to be the same underlying problem as 
  ticket [43107840f1c02], just in a different spot in the code.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The final query in the following SQL generates an incorrect result.  It
  returns just one row when it should return two.

  <blockquote><verbatim>
  REATE TABLE t1(a);
  CREATE TABLE t2(b);
  INSERT INTO t1 VALUES(1), (1), (2);
  INSERT INTO t2 VALUES(1), (2);
  SELECT * FROM t2 WHERE b IN (SELECT DISTINCT a FROM t1 LIMIT 2);
  </verbatim></blockquote>

  This bug can be hit when an IN operator has a subquery on its right-hand
  side and the subquery contains both a DISTINCT keyword and
  a LIMIT clause.

  The cause of this problem is check-in [f4cb53651b1e352f] which is an
  optimization introduced in version 3.8.6 that suppresses the DISTINCT on
  an IN subquery.  But that optimization is not always valid, as the example
  above shows.

  This problem was discovered by Igor Stassiy and reported on the sqlite-users
  mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the final SELECT statement should return just a single
  row, but beginning in SQLite 3.8.6 it returns 3 rows:

  <blockquote><verbatim>
  CREATE TABLE x1(a);
  INSERT INTO x1 VALUES(1);
  CREATE TABLE x2(b NOT NULL);    -- Remove the NOT NULL and things work
  CREATE TABLE x3(c, d);
  INSERT INTO x3 VALUES('a', NULL);
  INSERT INTO x3 VALUES('b', NULL);
  INSERT INTO x3 VALUES('c', NULL);
  SELECT * FROM x1 LEFT JOIN x2 LEFT JOIN x3 ON x3.d = x2.b;
  </verbatim></blockquote>

  As the comment states, removing the NOT NULL constraint on the x2.b column
  causes the output to be correct.

  The problem appears to have been introduced by checkin [952868216854e83]
  on 2014-08-05.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was that the query planner was using NOT NULL constraints
  on table columns to optimize out certain operations.  However, for the
  right-hand table of a LEFT JOIN, any column can be NULL despite having
  a NOT NULL constraint.  The sqlite3ExprCanBeNull() routine in the query
  planner had to be enhanced to recognize that corner case.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The file SELECT statement in the following SQL returns results in descending
  order instead of the ascending order requested:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(3),(1),(5),(2),(6),(4),(5),(1),(3);
  CREATE INDEX t1x ON t1(x DESC);
  SELECT DISTINCT x FROM t1 ORDER BY x ASC;
  </verbatim></blockquote>

  The problem only comes up in a DISTINCT query when using a descending
  index.  The problem seems to have been introduced as part of the NGQP in
  SQLite version 3.8.0 circa 2013-06-26, checkin [0fe31f60cadc5fe5a9d].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The sqlite3KeyInfoOfIndex() routine ([/artifact/67bb05b1077?ln=4192,4199|here])
  can return a stale KeyInfo object in shared cache mode.  

  Scenario:

    *  Two open database connections (A and B) on the same cache
    *  The shared pIdx object is holding a KeyInfo object for A
    *  Connection A closes.
    *  Connection C opens, and by (bad) luck is allocated
       the same database connection pointer as A had.
    *  Connection C now tries to get a KeyInfo object for index pIdx but
       is handed the stale one from A.  The KeyInfo object itself is ref-counted
       and so it is still live.  But the KeyInfo points to CollSeq objects
       within the old A connection that are now dead and have been freed.

  This problem appears to have been introduced by check-in [c80e229dd9c1230ab]
  on 2013-11-07, which first appeared in SQLite version 3.8.2.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Michal Zalewski found several crashing bugs by running a fuzzer against
  SQLite and reported them on the mailing list, moments ago.
  Each of the following four statements causes a crash:

  <blockquote><verbatim>
  SELECT n()AND+#00;
  SELECT(SELECT strftime());
  DETACH(SELECT group_concat(q));
  CREATE TABLE t0(t); INSERT INTO t0 SELECT strftime();
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL outputs rows in the incorrect order for SQLite
  versions 3.8.7 and 3.8.8:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1(x) VALUES(1),(5),(3),(4),(2);
  SELECT
     x, 01, 02, 03, 04, 05, 06, 07, 08, 09,
    10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
    20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
    30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
    40, 41, 42, 43, 44, 45, 46, 47, 48, 49,
    50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60
  FROM t1
  ORDER BY x
  LIMIT -1;
  </verbatim></blockquote>

  In order to hit this bug, there must be an ORDER BY clause and a LIMIT
  clause and the number of columns in the query plus the number of sort
  keys needs to sum to exactly 62.  Adding or removing a single column
  from the result set yields the correct answer.

  This bug first appeared in check-in [b1c0f0bc1bd8a] which is when the
  multi-threaded sorter was introduced.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Correction: the total number of columns plus the number of sort terms needs
  to be 62 or greater, not exactly 62.  And, if some of the columns
  contain large text or blob values that use more than one byte in their type
  header, then numbers less than 62 will provoke the problem as well.

  So a better characterization of the problem:
    *  An ORDER BY clause that requires a sort (cannot use an index)
    *  A LIMIT clause
    *  There are a large number of columns in the result


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was introduced by the 
  sqlite3VdbeRecordCompare() optimizations on 2014-03-04
  checkin [3325ad5bdc2f81f63b55] (version 3.8.4).
    The KeyInfo.nXField value was
  not always accurate for emphemeral tables and so the
  sqlite3VdbeFindCompare() function might return one of the
  short-circuit comparison routines when in fact the
  generalized comparison routine was required.  The problem
  became more acute with the introduction of multi-threaded 
  sorting on on 2014-09-01 checkin [b1c0f0bc1bd8a347]
  (version 3.8.7) because
  the new sorter creates tables with lots of extra fields
  after the key.

  The problem is fixed for version 3.8.8.1 by ensuring that
  the KeyInfo.nXField is set correctly.  Two new assert()
  statements are added to verify correct KeyInfo.nXField
  values moving forward.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following (incorrect) SQL text causes SQLite to segfault:

  <blockquote><verbatim>
  SELECT c.* FROM (a,b) AS c;
  </verbatim></blockquote>

  This problem was found by Michal Zalewski, presumably using his "american fuzzy lop" fuzzer tool, and reported on the SQLite mailing list.  The bug appears to originate in check-in [99127a669c49f829] from 2013-01-03.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  <verbatim>
  CREATE TABLE t1(x);
  CREATE TABLE t2(y, z);
  SELECT ( SELECT y FROM t2 WHERE z = cnt ) FROM ( SELECT count(*) AS cnt FROM t1 );
  </verbatim>

  This is likely the same problem as reported here:

    [https://www.mail-archive.com/sqlite-users@sqlite.org/msg88569.html]


login: drh
mimetype: text/x-fossil-wiki
icomment:
  All four of the following queries should return TRUE (1) according to the
  COLLATE operator rules defined in the SQLite documentation 
  ([https://www.sqlite.org/datatype3.html#collation] section 6.1).

  <blockquote><verbatim>
  SELECT 'abc'==('ABC'||'') COLLATE nocase;
  SELECT 'abc'==('ABC'||'' COLLATE nocase);
  SELECT 'abc'==('ABC'||('' COLLATE nocase));
  SELECT 'abc'==('ABC'||upper('' COLLATE nocase));
  </verbatim></blockquote>

  However, in the 3.8.8.2 implementation (and probably all prior versions)
  the final query returns FALSE (0).  Apparently the upper() function is
  containing the COLLATE property and preventing it from propagating up to the
  comparison operator.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL has two identical joins, one with and the other without an
  ORDER BY clause.  The one without ORDER BY correctly returns two rows of 
  result.  The one with ORDER BY returns no rows.

  <blockquote><verbatim>
  CREATE TABLE A(Name text);
  CREATE TABLE Items(ItemName text , Name text);
  INSERT INTO Items VALUES('Item1','Parent');
  INSERT INTO Items VALUES('Item2','Parent');
  CREATE TABLE B(Name text);

  SELECT Items.ItemName
    FROM Items
      LEFT JOIN A ON (A.Name = Items.ItemName and Items.ItemName = 'dummy')
      LEFT JOIN B ON (B.Name = Items.ItemName)
    WHERE Items.Name = 'Parent';

  SELECT Items.ItemName
    FROM Items
      LEFT JOIN A ON (A.Name = Items.ItemName and Items.ItemName = 'dummy')
      LEFT JOIN B ON (B.Name = Items.ItemName)
    WHERE Items.Name = 'Parent'
  ORDER BY Items.ItemName;
  </verbatim></blockquote>

  This error seems to have been introduced by check-in [d95d0313c447f5]
  which was first delivered in SQLite version 3.8.8.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem goes back much further than version 3.8.8.  There is the
  potential to get an incorrect result any time the qualifying constraint
  for a partial index is inside the ON clause of a LEFT JOIN.  For example,
  the following SQL should return a single row of "1,  NULL" but instead
  returns no rows for all prior versions of SQLite that support partial
  indexes (SQLite 3.8.0 and later):

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2(b);
  INSERT INTO t1 VALUES(1);
  CREATE INDEX t1x ON t1(a) WHERE a=99;
  PRAGMA automatic_index=OFF;
  SELECT * FROM t1 LEFT JOIN t2 ON (a=99) ORDER BY a;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the first query returns no rows whereas the second
  query returns one row.  Both queries should operate as the second.

  <blockquote><verbatim>
  CREATE TABLE t1(x TEXT UNIQUE COLLATE nocase);
  INSERT INTO t1(x) VALUES(x'616263');
  SELECT 'query-1', x FROM t1 WHERE x LIKE 'a%';
  SELECT 'query-2', x FROM t1 WHERE +x LIKE 'a%';
  </verbatim></blockquote>

  This problem appears to have been introduced on 2005-08-28 (9.5 years ago)
  by check-in [ef84ff795c85e9d2].  And yet the problem has never been observed
  in the wild. The problem was discovered by the SQLite developers while
  reasoning about the validity of the LIKE optimization.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following script:

  <verbatim>
    create table t(x, y, unique(x, y));
    insert into t values(null, 1);
    insert into t values(null, 1);
    select * from t where x is null and y = 1;
  </verbatim>

  is returning one row instead of two.

  Introduced here: [90e36676476e8db0]

  First appeared in version 3.8.5

  Isolated on the mailing list here: [http://osdir.com/ml/general/2015-04/msg13265.html]


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script generates a corrupt database file:

  <blockquote><verbatim>
  CREATE TABLE t1(ii INT);
  CREATE TABLE t2(tt TEXT PRIMARY KEY, ss);
  CREATE TRIGGER t1_ai AFTER INSERT ON t1 BEGIN
    INSERT INTO t2(tt) VALUES(new.ii);
  END;
  CREATE TRIGGER t2_ai AFTER INSERT ON t2 BEGIN
    UPDATE t2 SET ss = 4;
  END;
  INSERT INTO t1(ii) VALUES('1');
  PRAGMA integrity_check;
  </verbatim></blockquote>

  This problem appears to have been introduced on 2014-08-23 (version 3.8.7)
  by check-in [25f2246be404f38b].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In order for this bug to strike, one must do an INSERT into a table that
  has an index with a TEXT key, and two nested triggers on that table must
  convert the key value from TEXT to INTEGER and back to TEXT again.

  Any index corruption that does occur because of this bug can be
  fixed by running REINDEX.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When a column has type INT then that column is allowed to store text, but
  text should only be stored if it does not look like an integer.  If the text
  looks like an integer then it should be stored as an integer.

  Sometimes a table can be created that has an INT column that contains text that
  looks like an integer.  The following code illustrates:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT);
  CREATE TABLE t2(x TEXT);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t1 VALUES('abc');
  INSERT INTO t2 VALUES(2);
  CREATE TABLE t3 AS SELECT * FROM t1 UNION SELECT * FROM t2;
  SELECT * FROM t3
   WHERE typeof(a)='text'
     AND CAST(CAST(a AS int) AS text)==a;
  </verbatim></blockquote>

  The resulting table t3 contains two rows with text.  The row that contains 'abc' is correct.  But the row containing '2' really should contain a numeric 2.

  This problem goes back to before version 3.5.1 (circa 2007).  However, the
  problem became more acute with check-in [10d851353c2abead] (2014-01-02 -
  version 3.8.3) because that check-in optimized index creation to avoid
  "unnecessary" type checking.  With version 3.8.3 and later, the following
  query gives a different answer with an index than it gives without the index:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT);
  CREATE TABLE t2(x TEXT);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t1 VALUES('abc');
  INSERT INTO t2 VALUES(2);
  CREATE TABLE t3 AS SELECT * FROM t1 UNION SELECT * FROM t2;
  SELECT 'without-index', a FROM t3 WHERE a>9;
  CREATE INDEX t3a ON t3(a);
  SELECT 'with-index', a FROM t3 WHERE a>9;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Below is a simpler test case that further illustrates the problem.  In this case,
  a table with a column of type TEXT ends up holding an INTEGER value, which
  is something that should never happen.

  <blockquote><verbatim>
  CREATE TABLE t1(a TEXT);
  CREATE TABLE t2 AS SELECT * FROM t1 UNION ALL SELECT 123;
  SELECT a, typeof(a) FROM t2;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When a CREATE TABLE AS statement is part of a larger transaction and it aborts
  due to an error condition in the SELECT on the right-hand side, this can leave
  the sqlite3_master table with a blank entry which subsequent database
  connections will interpret as a corrupt database.  The following SQL
  illustrates the problem:

  <blockquote><verbatim>
  BEGIN;
  CREATE TABLE t1 AS SELECT zeroblob(2e20);
  COMMIT;
  PRAGMA integrity_check;
  </verbatim></blockquote>

  This problem predates SQLite 3.5.1 (circa 2007).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Using any of the likely(), unlikely(), or likelihood() functions on a term in
  the ON clause of a LEFT JOIN can cause the LEFT JOIN to malfunction. Example:

  <blockquote><verbatim>
  CREATE TABLE t1(i int, x, y, z);
  INSERT INTO t1 VALUES (1,1,1,1), (2,2,2,2), (3,3,3,3), (4,4,4,4);

  CREATE TABLE t2(i int, bool char);
  INSERT INTO t2 VALUES(1,'T'), (2,'F');

  SELECT count(*) FROM t1 LEFT JOIN t2 ON t1.i=t2.i AND bool='T';
  SELECT count(*) FROM t1 LEFT JOIN t2 ON likely(t1.i=t2.i) AND bool='T';
  </verbatim></blockquote>

  The two queries in the script above should give the same answer, but due
  to this bug, they do not. The problem is that the likely() function causes
  the query planner to forget that the t1.i=t2.i term is part of the ON
  clause, making it think that the term is in the WHERE clause instead.

  This problem appears to have existed since the unlikely() function was
  first introduced in version 3.8.1.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates an assertion fault while processing the final
  SELECT statement.

  <blockquote><verbatim>
  CREATE TABLE t1(x, y, PRIMARY KEY(x,y)) WITHOUT ROWID;
  INSERT INTO t1(x,y) VALUES(1,'AB');
  INSERT INTO t1(x,y) VALUES(2,'CD');
  ANALYZE;
  DROP TABLE IF EXISTS sqlite_stat4;
  DELETE FROM sqlite_stat1;
  INSERT INTO sqlite_stat1(tbl,idx,stat) VALUES('t1','t1','1000000 100 1');
  ANALYZE sqlite_master;
  SELECT * FROM t1
   WHERE (y = 'AB' AND x <= 4)
      OR (y = 'EF' AND x = 5);
  </verbatim></blockquote>

  The problem appears to have been introduced when the OR optimization was
  added to WITHOUT ROWID tables in version 3.8.5.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Turns out the problem also involves the skip-scan optimization, which is
  why the ANALYZE was necessary to express the problem.

  So to summarize, versions of SQLite between 3.8.5 and 3.8.10.2 might get
  an incorrect result on queries against a WITHOUT ROWID table that contain
  OR terms in the WHERE clause and that employ the skip-scan optimization.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query returns a single row rather than two rows as it should:

  <blockquote><verbatim>
  SELECT *
  FROM (
      SELECT 'apple' fruit
      UNION ALL SELECT 'banana'
  ) a
  JOIN (
      SELECT 'apple' fruit
      UNION ALL SELECT 'banana'
  ) b ON a.fruit=b.fruit
  LEFT JOIN (
      SELECT 1 isyellow
  ) c ON b.fruit='banana';
  </verbatim></blockquote>

  The error seems to have been introduced by check-in [6df18e949d367629] on 2015-06-02 and first appeared in version 3.8.11.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The second statement in the following script reports an SQLITE_CORRUPT error:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2 AS SELECT name FROM sqlite_master;
  </verbatim></blockquote>

  The database is not actually corrupt.  This is a malfunction in the CREATE TABLE AS statement.

  Bisecting indicates that this problem was introduced by check-in
  [972881c6d3] on [2013-11-21], first released with version 3.8.2.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script causes an assertion fault:

  <blockquote><pre>
  PRAGMA foreign_keys=1;
  CREATE TABLE Foo (Id INTEGER PRIMARY KEY, ParentId INTEGER REFERENCES Foo(Id) ON DELETE CASCADE, C1);
  INSERT OR REPLACE INTO Foo(Id, ParentId, C1) VALUES (1, null, 'A');
  INSERT OR REPLACE INTO Foo(Id, ParentId, C1) VALUES (2, 1, 'A-2-1');
  INSERT OR REPLACE INTO Foo(Id, ParentId, C1) VALUES (3, 2, 'A-3-2');
  INSERT OR REPLACE INTO Foo(Id, ParentId, C1) VALUES (4, 3, 'A-4-3');
  INSERT OR REPLACE INTO Foo(Id, ParentId, C1) VALUES (2, 3, 'A-2-3');
  </pre></blockquote>

  This problem appears to have been introduced with version 3.8.4, specifically
  at check-in [56bc5ce8958c8e625].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A simpler test case:

  <blockquote><verbatim>
  PRAGMA foreign_keys=ON;
  CREATE TABLE t1(x INTEGER PRIMARY KEY, parent REFERENCES t1 ON DELETE CASCADE);
  INSERT INTO t1 VALUES(1, NULL),(2, 1),(3, 2);
  INSERT OR REPLACE INTO t1 VALUES(2, 3);
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In a UNION ALL query with a LIMIT and OFFSET, if the OFFSET is greater than
  or equal to the number of rows in the first SELECT, then the LIMIT is
  disabled.  For example, the following SQL outputs 5 rows instead of just 1:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES('a');
  INSERT INTO t1 VALUES('b');
  INSERT INTO t1 VALUES('c');
  INSERT INTO t1 VALUES('d');
  INSERT INTO t1 VALUES('e');
  SELECT * FROM (SELECT x, rowid FROM t1)
  UNION ALL
  SELECT * FROM (SELECT rowid, x FROM t1 ORDER BY x)
  LIMIT 1 OFFSET 5;
  </verbatim></blockquote>

  But if the OFFSET is changed to 4, then the query correctly outputs
  just a single row.

  This problem appears to have existed as far back as 3.5.1 (circa 2007).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The JSON extension is accepting and form-feed (0xc) as
  whitespace.  But RFC-7159 says that only space (0x20), tab (0x09),
  line-feed (0x0a) and carriage-return (0x0d) are whitespace for JSON.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  This:

  <verbatim>
  CREATE TABLE t1(x, z);
  INSERT INTO t1 VALUES('aaa', 4.0);
  INSERT INTO t1 VALUES('aaa', 4.0);
  CREATE VIEW vvv AS
    SELECT * FROM t1
      UNION ALL
    SELECT 0, 0 WHERE 0;

  SELECT (
    SELECT sum(z) FROM vvv WHERE x='aaa'
  ) FROM sqlite_master;
  </verbatim>

  currently returns two rows containing the single value 4.0 (instead of 8.0). The problem is that when SQLite creates an automatic index on view "vvv", it incorrectly discards duplicate rows - making the sum() expression equivalent to just 4.0 instead of (4.0 + 4.0). 

  Problem introduced here:

    http://www.sqlite.org/src/info/020b8b106fc8f840

  First appeared in release 3.8.11.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Historically, the following four table have been interpreted the same was, as
  a two column table with an INTEGER PRIMARY KEY in the first column:

  <blockquote><verbatim>
  CREATE TABLE t1(x integer NOT NULL, y test, PRIMARY KEY(x));
  CREATE TABLE t2("x" integer NOT NULL, "y" test, PRIMARY KEY("x"));
  CREATE TABLE t3([x] integer NOT NULL, [y] test, PRIMARY KEY([x]));
  CREATE TABLE t4('x' integer NOT NULL, 'y' test, PRIMARY KEY('x'));
  </verbatim></blockquote>

  However, in 3.9.0 and 3.9.1, the t4 table above is interpreted as a two column
  table with a separate PRIMARY KEY index on the first column.  Hence, if the
  database schema above is created with version 3.8.11.1 or earlier, then
  PRAGMA integrity_check will show an error with version 3.9.0 and later.

  Version 3.9.0 and later should interpret all four tables above the same way.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Imprecision in the comparison of integer and floating-point values can
  result in a corrupt index.  The following SQL demonstrates:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  CREATE INDEX t1b ON t1(b);
  INSERT INTO t1(a,b) VALUES(3, 356282677878746339);
  INSERT INTO t1(a,b) VALUES(2, 356282677878746339.0);
  INSERT INTO t1(a,b) VALUES(1, 356282677878746340);
  DELETE FROM t1 WHERE a=2;
  PRAGMA integrity_check;
  </verbatim></blockquote>

  The integrity_check reports a row missing from the index.  Examination of
  the binary for the resulting databases shows that the index has two rows
  but that the rows are out-of-order.

  The above is true when compiling with gcc on linux.  But using MSVC on Windows,
  the script works correctly.

  This bug has existed in the code since before version 3.5.1 (circa 2007).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The parser silently accepts and ignores table-valued function syntax on
  ordinary tables and views which are not table-valued functions.  This can
  lead to memory leaks.  An example:

  <blockquote><verbatim>
  CREATE TABLE t(x);
  SELECT * FROM t(0);
  WITH x AS (SELECT * FROM t) SELECT * FROM x(0);
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an ALWAYS() macro to fail:

  <blockquote><verbatim>
  SELECT (SELECT 'mmm' UNION SELECT DISTINCT max(name) ORDER BY 1) 
    FROM sqlite_master;
  </verbatim></blockquote>

  This problem appears to have been introduced by check-in [7ab0b258eabfcfb7]
  on 2015-09-05 for release 3.9.0.  The problem was discovered by 
  Kostya Serebryany using libFuzzer.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL should return a value of "1", but it
  sometimes returns "NULL" on recent versions of SQLite:

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER PRIMARY KEY);
  INSERT INTO t1 VALUES(1),(2);
  CREATE TABLE t2(y);
  INSERT INTO t2 VALUES(3),(4);
  SELECT quote((SELECT x FROM t2, t1 ORDER BY x, y));
  </verbatim></blockquote>

  This problem appears to have been introduced on [2014-08-29 14:20:46] by
  check-in [2a74129a21f], though it is unclear if that check-in actually caused
  the problem or just unmasked it.
  The problem appears to be cause by misbehavior in the
  [https://www.sqlite.org/queryplanner.html#partialsort | partial sorting by index]
  optimization.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The simple work-around to this problem is to not over-constraint ORDER BY
  clauses.  Do not write:

  <blockquote>
  SELECT x FROM ... ORDER BY x, y, z;
  </blockquote>

  The extra "y" and "z" terms on the ORDER BY are pointless as the "ORDER BY x" is sufficient to completely define the output order.  So instead write:

  <blockquote>
  SELECT x FROM ... ORDER BY x;
  </blockquote>

  SQLite ought to give the correct answer in either case.  And apparently it always does in the second case.  The bug here is that sometimes the first case gives an incorrect result.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following (equivalent) queries return true, when the correct answer is false:

  <blockquote><verbatim>
  SELECT 'き' LIKE 'ね';
  SELECT char(0x304d) LIKE char(0x306d);
  </verbatim></blockquote>

  The culprit seems to be check-in [0a99a8c4facf] on [2015-12-30 14:06:22].  Case comparison is only looking at the lower 8 bytes of unicode values.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The first column in the output of the following query contains invalid
  JSON.  The "abc" string value is not quoted.  The output is
  <nowiki><b>[1,abc]</b> instead of <b>[1,"abc"]</b></nowiki>

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(1),('abc');
  SELECT
     json_group_array(x),
     json_group_array(json_object('x',x))
    FROM t1;
  </verbatim></blockquote>

  This problem was reported by Marien Zwart on the SQLite Users
  mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The underlying problem was that the subtype information for a VDBE register
  was not being cleared when that register was overwritten and reused, which
  cause cause the subtype from a prior json_object() call to leak into the
  parameter of the json_group_array() call.  Upgraded the severity to Severe
  since this problem affects all users of sqlite3_value_subtype() and
  sqlite3_result_subtype().


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The JSON specification requires control characters in strings to be escaped,
  but the JSON1 extension does not do this.  For example:

  <blockquote><verbatim>
  SELECT json_array(char(1,8,9,10,12,13,31));
  </verbatim></blockquote>

  The output should be <nowiki><tt>["\u0001\b\t\n\f\r\u001f"]</tt></nowiki> but
  the JSON1 extension leaves the characters unescaped.

  This error arose because when I was coding up the JSON1 extension
  I missed the part in the JSON specification that says that control
  characters must be escaped.  I understood the escapes to be optional
  for all characters other than " and \.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL code generates an incorrect answer of three rows instead of two:

  <blockquote><verbatim>
  CREATE TABLE tx(id INTEGER PRIMARY KEY, a, b);
  INSERT INTO tx(a,b) VALUES(33,456);
  INSERT INTO tx(a,b) VALUES(33,789);

  SELECT DISTINCT t0.id, t0.a, t0.b
    FROM tx AS t0, tx AS t1
   WHERE t0.a=t1.a AND t1.a=33 AND t0.b=456
  UNION
  SELECT DISTINCT t0.id, t0.a, t0.b
    FROM tx AS t0, tx AS t1
   WHERE t0.a=t1.a AND t1.a=33 AND t0.b=789
   ORDER BY 1;
  </verbatim></blockquote>

  Prior to and including check-in [93121d3097a43997] (version 3.8.3, 2013-12-17) 
  the correct answer was obtained.  After that, the test script above generates
  an assertion fault.  At check-in [c0fa0c0e2de50d7e] (version 3.8.4, 2014-03-03)
  the assertion faults go away, but the incorrect answer is returned.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was two co-routines using the same temporary registers and
  expecting those temporary registers to be preserved across a Yield.  The
  solution is to ensure that every co-routine has its own set of temporary
  registers that are not shared with the main thread or other co-routines.

  In the demonstration query shown in the problem description, each SELECT
  statement is evaluated by a separate co-routine that includes an ORDER BY.
  The main thread handles the UNION by running each co-routine long enough
  to generate a single row, then choosing the smaller of the two
  rows to output and running the corresponding co-routine to generate the
  next row for that SELECT.  Each co-routine evaluates its query as a nested
  loop with t0 being the outer loop and t1 being the inner loop.  The t0.a
  and t0.b values are stored in temporary registers in the outer loop, and
  those value are expected to remain intact even though a Yield occurs after
  each iteration of the inner loop.  Prior to the fix of
  check-in [ca72be8618e5d466] both co-routines were using the same temporary
  registers for their t0.a and t0.b values, which means that running one
  co-routine would disrupt the other.  After check-in [ca72be8618e5d466] the
  two co-routines use different temporary registers for holding t0.a and
  t0.b so they no longer interfere with one another.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Assuming the REGEXP operator implementation uses sqlite3_set_auxdata() to store a compiled version of its regular expression (as the bundled extension does), test 1.1 of the following tcl test script fails:

  <verbatim>
  do_execsql_test 1.0 {
    CREATE TABLE t1(a, b, c);
    CREATE TABLE x1(x, y, z);
    CREATE TABLE x2(x, y, z);

    CREATE TRIGGER tr1 AFTER INSERT ON t1 BEGIN
      INSERT INTO x1 VALUES(
          new.a REGEXP 'abc',
          new.b REGEXP 'abc',
          new.c REGEXP 'abc'
      );
    END;

    CREATE TRIGGER tr2 AFTER INSERT ON t1 BEGIN
      INSERT INTO x2 VALUES(
          new.a REGEXP 'def',
          new.b REGEXP 'def',
          new.c REGEXP 'def'
      );
    END;

    INSERT INTO t1 VALUES('abc', 'def', 'abc');
    SELECT * FROM t1;
  } {abc def abc}

  do_execsql_test 1.1 { SELECT * FROM x1 } {1 0 1}
  do_execsql_test 1.2 { SELECT * FROM x2 } {0 1 0}
  </verbatim>

  The trouble is that the REGEXP operators in trigger "tr1" are incorrectly using the compiled regular expressions for trigger "tr2" - causing the operators in "tr1" to populate table x1 with the same values as table x2.

  If the two triggers were using different operators or user-functions that both used sqlite3_set_auxdata() a crash or other error might have occurred instead.

  Reported on mailing list here: http://osdir.com/ml/general/2016-02/msg26665.html


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates a single row of output where it should generate
  an empty set:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY);
  INSERT INTO t1(a) VALUES(1),(2);
  CREATE TABLE t2(b INTEGER PRIMARY KEY);
  INSERT INTO t2(b) VALUES(1);
  SELECT a FROM t1 WHERE a NOT IN (SELECT a FROM t2);
  </verbatim></blockquote>

  Apparently, the subquery "(SELECT a FROM t2)" is being compiled as if it
  where "(SELECT b FROM t2)".  This problem appears to go back to before
  version 3.6.1, circa 2008.

  Problem reported at 
  [http://stackoverflow.com/questions/35891605/why-do-i-receive-an-unexpected-result-set-in-sqlite-when-excluding-rows-via-sub|stackoverflow]
  and relayed to the sqlite-users mailing list by Clemens Ladisch.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A defect in the in-memory journal logic can leave the read cursor for the
  in-memory journal in an inconsistent state, which can result in a segfault.
  No simple test case is (currently) available for this problem.  One must
  make heavy use of the SAVEPOINT operation to make it happen.

  This problem was discovered by Django testing.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem only comes up when the size of a savepoint journal (which is a
  multiple of 1028 bytes) is an exact multiple of the in-memory journal buffer
  size, which defaults to 1016 bytes on 64-bit machines and 1020 bytes on
  32-bit machines.  And even then, you have to do just the right sequence of
  SAVEPOINT operations to get the cursor to be reused.

  The following scripts will segfault 64-bit builds of SQLite 3.12.0:

  <blockquote><verbatim>
  PRAGMA page_size=1024;
  BEGIN;
  CREATE TABLE t1(x INTEGER PRIMARY KEY, y TEXT);
  WITH RECURSIVE c(x) AS (VALUES(1) UNION SELECT x+1 FROM c WHERE x<250)
  INSERT INTO t1(x,y) SELECT x*10, printf('%04d%.800c',x,'*') FROM c;
  SAVEPOINT one;
    SELECT count(*) FROM t1;
    WITH RECURSIVE c(x) AS (VALUES(1) UNION SELECT x+1 FROM c WHERE x<250)
    INSERT INTO t1(x,y) SELECT x*10+1, printf('%04d%.800c',x,'*') FROM c;
  ROLLBACK TO one;
    SELECT count(*) FROM t1;
    SAVEPOINT twoB;
      WITH RECURSIVE c(x) AS (VALUES(1) UNION SELECT x+1 FROM c WHERE x<10)
      INSERT INTO t1(x,y) SELECT x*10+2, printf('%04d%.800c',x,'*') FROM c;
    ROLLBACK TO twoB;
  RELEASE one;
  COMMIT;
  </verbatim></blockquote>

  For 32-bit builds, change the two "250" constants to "251".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  For some builds (depending on compile-time settings) it might be necessary
  to add

  <blockquote><verbatim>
  PRAGMA temp_store=MEMORY;
  </verbatim></blockquote>

  to the top of the test script in the previous comment.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The DELETE operation on the penultimate line of the script below fails to
  delete every row for which b==2, as shown by the SELECT on the last line:

  <blockquote><verbatim>
  PRAGMA page_size=1024;
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
  CREATE INDEX x1 ON t1(b, c);
  INSERT INTO t1(a,b,c) VALUES(1, 1, zeroblob(80));
  INSERT INTO t1(a,b,c) SELECT a+1, 1, c FROM t1;
  INSERT INTO t1(a,b,c) SELECT a+2, 1, c FROM t1;
  INSERT INTO t1(a,b,c) SELECT a+10, 2, c FROM t1 WHERE b=1;
  INSERT INTO t1(a,b,c) SELECT a+20, 3, c FROM t1 WHERE b=1;
  PRAGMA reverse_unordered_selects = ON;
  DELETE FROM t1 WHERE b=2;
  SELECT a FROM t1 WHERE b=2;
  </verbatim></blockquote>

  The underlying cause of this malfunction appears to be a defect in the
  OP_Prev opcode of the VDBE.  However, we have so far been unable to cause
  incorrect behavior except by using reverse_unordered_selects and the
  multi-row on-pass delete optimization that was introduced in version 3.9.0
  (check-in [8b93cc5937]).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  SQLite as always allowed the datatype of a column to be quoted.  For
  example:

  <blockquote><verbatim>
  CREATE TABLE t1(
    a "INTEGER" PRIMARY KEY,
    b "TEXT"
  );
  </verbatim></blockquote>

  But beginning with version 3.12.0, quoting the "INTEGER" datatype causes
  SQLite to fail to recognize that the column is the INTEGER PRIMARY KEY that
  has special properties.  This means that legacy database files that were
  created using INTEGER PRIMARY KEYs where the "INTEGER" keywords was quoted
  become unreadable.

  The fact that legacy versions of SQLite allowed datatype names to be quoted
  is undocumented and untested, and was a fact unknown to the developers,
  which is how this problem managed to slip through testing.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query works and gives the correct answer for SQLite version 3.8.10
  but fails to compile with a "misuse of aggregate" error for 3.8.11.

  <blockquote><verbatim>
  CREATE TABLE t1(a int, b int);
  INSERT INTO t1 VALUES(1,2),(1,18),(2,19);
  SELECT x, y FROM (
    SELECT a AS x, sum(b) AS y FROM t1 GROUP BY a
    UNION
    SELECT 98, 99
  ) AS w WHERE y>=20;
  </verbatim></blockquote>

  The problem seems to have been introduced by check-in [6df18e949d36] on [2015-06-02].
  The problem was reported on the sqlite-users mailing list by
  Jean-Luc Hainaut.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The DELETE statement on the third line of the example below only deletes a
  single row, rather than two rows as expected.  The correct behavior, confirmed by running on PostgreSQL, is to delete two rows, such that the final answer is a single row with value "1".

  <blockquote><verbatim>
  CREATE TABLE t1(x INT);
  INSERT INTO t1 VALUES(1),(2),(3);
  DELETE FROM t1 WHERE EXISTS(SELECT 1 FROM t1 AS v WHERE v.x=t1.x-1);
  SELECT * FROM t1;
  </verbatim></blockquote>

  Bisecting shows that this problem results from the one-pass optimization
  at check-in [8b93cc59370].  The problem was reported on the sqlite-users
  mailing list by Rob Golsteijn.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following causes an assertion fault, or segfaults if asserts are disabled.
  The problem occurs on the DELETE statement.

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER PRIMARY KEY, y TEXT);
  INSERT INTO t1(x,y) VALUES(1,'zebra');
  CREATE INDEX t1x ON t1(x);
  DELETE FROM t1 WHERE x IS NULL OR x<2;
  </verbatim></blockquote>

  This problem was introduced in version 3.12.0 by check-in [96ea990942].
  A simple work-around is to drop the redundant "t1x" index.  As far as
  I can tell, this problem only comes up when there is an index on the
  INTEGER PRIMARY KEY, as illustrated by t1x above.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another problem is that the SELECT statement below does a full table scan:

  <blockquote><verbatim>
  CREATE TABLE t2(x INTEGER PRIMARY KEY, y TEXT);
  SELECT * FROM t2 WHERE x IS NULL;
  </verbatim></blockquote>

  If this optimization is fixed, the situation that causes the bug in this
  ticket would never come up, as far as I can tell.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another crashing example that does not use the degenerate index:

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER PRIMARY KEY, y TEXT);
  INSERT INTO t1(x,y) VALUES(1,'zebra');
  CREATE INDEX t1x ON t1(abs(x));
  DELETE FROM t1 WHERE abs(x)=99 OR abs(x)<2;
  </verbatim></blockquote>

  These seem to be the requirements to cause the crash:
    1.  A rowid table with an integer primary key
    2.  No indexes reference any columns of the table other than the primary key
    3.  DELETE with an OR in the WHERE clause

  The OR clause uses the OP_Seek opcode to do a deferred seek of the main table.
  That deferred seek is resolved at the first OP_Column.  But if none of the
  indexes ever reference a column (because they all only reference the integer
  primary key) then the seek does not occur prior to the OP_Delete.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statements in the following SQL provoke a NULL pointer deference:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY);
  CREATE TABLE t2(b INTEGER PRIMARY KEY, x1 INT COLLATE NOCASE);
  CREATE TABLE t3(x2 INT);
  SELECT * FROM t3, t2, t1 WHERE x2=b AND x1=a AND a=1;
  SELECT * FROM t3, t2, t1 WHERE b=x2 AND a=x1 AND 1=a;
  </verbatim></blockquote>

  This problem was originally reported against System.Data.SQLite at
  [https://system.data.sqlite.org/index.html/info/dd30ecb89d423c4c].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The change to ALTER TABLE in [e1d8ec85546c] can cause indexes to go corrupt.
  Sample code:

  <blockquote><verbatim>
  PRAGMA legacy_file_format=ON;
  CREATE TABLE t1(a,b,c);
  CREATE INDEX t1a ON t1(a DESC);
  INSERT INTO t1 VALUES(1,2,3);
  INSERT INTO t1 VALUES(2,3,4);
  ALTER TABLE t1 ADD COLUMN d;
  PRAGMA integrity_check;
  </verbatim></blockquote>

  If the schema version is initially less than 4, then the DESC keyword on 
  indexes is ignored.  If ALTER TABLE then changes the schema version to 4 from
  a smaller value, then preexisting DESC indexes will be in the wrong order.

  Corruption caused by this bug can be fixed by running the REINDEX command.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the same query is run twice, once before and once after
  a CREATE INDEX.  This is clearly a bug since a CREATE INDEX should never 
  change the answer, only make the answer appear more quickly.  The fact that
  the index and the query are both a bit goofy should not matter.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b INTEGER);
  INSERT INTO t1(a,b) VALUES(500,123);
  SELECT min(a) FROM t1 WHERE b='123' AND a>'456';
  CREATE INDEX t1ba ON t1(b, a);
  SELECT min(a) FROM t1 WHERE b='123' AND a>'456';
  </verbatim></blockquote>

  A quick initial assessment of this problem suggests that it is a bug in the
  min/max optimization.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The index need not by goofy.  The following is another example of the
  failures that omits the PRIMARY KEY constraint:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER, b INTEGER);
  INSERT INTO t1(a,b) VALUES(500,123),(600,123);
  SELECT min(a) FROM t1 WHERE b='123' AND a>'456';
  CREATE INDEX t1ba ON t1(b, a);
  SELECT min(a) FROM t1 WHERE b='123' AND a>'456';
  </verbatim></blockquote>

  The use of the min() operator is required to generate the error however.
  The problem is in the logic that skips initial NULL entries on the column
  that is the argument to min().


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the SQL below, the DELETE operation is a no-op.

  <blockquote><verbatim>
  CREATE TABLE t1(a INT UNIQUE, b INT);
  INSERT INTO t1(a,b) VALUES('1','2');
  SELECT * FROM t1 WHERE a='1' AND b='2';
  DELETE FROM t1 WHERE a='1' AND b='2';
  SELECT * FROM t1;
  </verbatim></blockquote>

  Even the SELECT on the third line outputs the row, and the DELETE has the
  exact same WHERE clause, the DELETE does not delete the row.

  Bisecting suggests that this problem was introduced by 
  check-in [85c467041c9378ca] on 2016-01-30 for version 3.11.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the SELECT statement occurs twice, once before and once 
  after the index is created.  The two SELECTs give different answers.
  This should never be.
  An index will hopefully make the answer appear more quickly,
  but it should never change the answer.

  <blockquote><verbatim>
  CREATE TABLE b1(a TEXT); INSERT INTO b1 VALUES(1);
  CREATE TABLE b2(x BLOB); INSERT INTO b2 VALUES(1);
  SELECT count(*) FROM b1 WHERE a IN (SELECT x FROM b2);
  CREATE UNIQUE INDEX b1a ON b1(a);
  SELECT count(*) FROM b1 WHERE a IN (SELECT x FROM b2);
  </verbatim></blockquote>

  This problem has been in the code since before version 3.3.7 (2006-08-12), and
  and hence gone unnoticed for over ten years.  For that reason, it is given a
  low severity.

  This problem was discovered during internal code reviews of
  the row-value feature
  planned for the 3.15 release.  There have been no reports of this problem in
  the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This is related to [93fb9f89d6a69dc3d8], though that prior ticket dealt
  with joins instead of IN operators.  Consider:

  <blockquote><verbatim>
  CREATE TABLE b1(a TEXT); INSERT INTO b1 VALUES(1);
  CREATE TABLE b2(x BLOB); INSERT INTO b2 VALUES(1);
  SELECT count(*) FROM b1 WHERE a IN (SELECT x FROM b2);
  SELECT count(*) FROM b1, b2 WHERE a=x;
  CREATE UNIQUE INDEX b1a ON b1(a);
  SELECT count(*) FROM b1 WHERE a IN (SELECT x FROM b2);
  SELECT count(*) FROM b1, b2 WHERE a=x;
  </verbatim></blockquote>

  The code above used to return "0 0 1 1" prior to check-in [e72186f2d6]
  which was the fix for ticket [93fb9f89d6a69dc3d8].  After that check-in
  (circa 2009-08-13) the answer returned was "0 0 1 0".  The correct answer
  should be "0 0 0 0".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the first query gives the correct 3 lines of result
  in the correct order.  But the second query returns the second line of result,
  not the first line as it should:

  <blockquote><pre>
  CREATE TABLE t1(a,b,c);
  CREATE INDEX t1x ON t1(a,b,c);
  INSERT INTO t1 VALUES(0,1,99),(0,1,0),(0,0,0);
  SELECT * FROM t1 WHERE a=0 AND (c=0 OR c=99) ORDER BY c DESC;
  SELECT * FROM t1 WHERE a=0 AND (c=0 OR c=99) ORDER BY c DESC LIMIT 1;
  </pre></blockquote>

  This problem seems to have been introduced by check-in [559733b09e96]
  on 2016-05-20 and was released with version 3.14.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL exhibits O(N*N) behavior:

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER UNIQUE PRIMARY KEY NOT NULL, a,b,c,d);
  CREATE TABLE t2(y INT);
  explain query plan SELECT count(*) FROM t1 WHERE x IN t2;
  </verbatim></blockquote>

  The EXPLAIN QUERY PLAN shows that the subquery against t2 is run twice.  One
  of those two is inside the t1 loop, causing the O(N*N) behavior.

  Bisecting shows this problem was introduced by check-in
  [96ea990942] (2016-03-16, version 3.12.0) and was fixed by
  check-in [061b800603] (2016-07-26).  That fix was later merged to trunk at
  [ddb5f0558c] but has not yet been released.

  In other words, this problem has already been fixed on trunk.
  This ticket is created to an historical record of the problem,
  and to allow for study of the root cause, and to make improvements
  to the test suite to avoid a regression.

  This problem was first reported on the sqlite-users mailing list
  by Takasumi Iwamoto.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query causes an assertion fault on trunk:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER UNIQUE PRIMARY KEY, b VARCHAR(200));
  SELECT count(*) FROM t1 WHERE a IN (1, 2, 3);
  </verbatim></blockquote>

  The root cause of this failure and the original O(N*N) performance is 
  the redundant UNIQUE constraint on the INTEGER PRIMARY KEY.  That extra
  unique index is confusing the query planner, apparently.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If integer arithmetic overflows in SQLite, it is suppose to convert the
  answer to floating point.  But sometimes it returns a large negative number:

  <blockquote><verbatim>
  SELECT 3037000500*3037000500;
  </verbatim></blockquote>

  Experiments on SqlFiddle.com show that 
  PostgreSQL, MySQL, and SQL Server all throw integer overflow exceptions for
  the above.  Oracle overflows to floating point.

  This problem has been in SQLite since before version 3.5.1 (2007-10-04).

  Problem reported on the mailing list by Scott Hess from Google.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The sqlite_sequence entry for table t2 is not initialized by the final
  INSERT in the following SQL:

  <blockquote><pre>
  CREATE TABLE t1(a INTEGER PRIMARY KEY AUTOINCREMENT, b UNIQUE);
  INSERT INTO t1 VALUES(888,9999);
  CREATE TABLE t2(x INTEGER PRIMARY KEY AUTOINCREMENT, y UNIQUE);
  INSERT INTO t2 SELECT * FROM t1;
  SELECT * FROM sqlite_sequence;
  </pre></blockquote>

  Bisecting shows that the problem was introduced on on 2009-06-23 by
  check-in [1330993de8] and first appeared in version 3.6.16.

  This problem was reported on the mailing list by Clemens Ladisch.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The fourth SELECT statement in the example below outputs one row where t1.a is
  NULL.  The first three SELECT statements demonstrate that the fourth
  SELECT gives an is an incorrect result.

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  INSERT INTO t1 VALUES(NULL),(1);
  SELECT '1:', typeof(a), a FROM t1 WHERE a<10;
  SELECT '2:', typeof(a), a FROM t1 WHERE a+0<10;
  CREATE INDEX t1x1 ON t1(a);
  CREATE INDEX t1x2 ON t1(a+0);
  SELECT '3:', typeof(a), a FROM t1 WHERE a<10;
  SELECT '4:', typeof(a), a FROM t1 WHERE a+0<10;
  </verbatim></blockquote>


login: dan
mimetype: text/x-fossil-wiki
icomment:
  When there is an index on an expression that matches the first term of an ORDER BY expression-list, SQLite is ignoring all subsequent expressions. For example:

  <verbatim>
    sqlite> CREATE TABLE t1(x, y);
    sqlite> INSERT INTO t1 VALUES(1, 1);
    sqlite> INSERT INTO t1 VALUES(1, 2);
    sqlite> INSERT INTO t1 VALUES(2, 2);
    sqlite> INSERT INTO t1 VALUES(2, 1);
    sqlite> SELECT * FROM t1 ORDER BY x+0,y;
    1|1
    1|2
    2|1
    2|2
    sqlite> CREATE INDEX i1 ON t1(x+0);
    sqlite> SELECT * FROM t1 ORDER BY x+0, y;
    1|1
    1|2
    2|2
    2|1
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The second query in the following SQL returns an incorrect result:

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER PRIMARY KEY, y INT);
  INSERT INTO t1(x,y) VALUES(1,1),(2,1),(3,1),(4,1),(5,5),(6,6),(7,4);
  CREATE TABLE t2(z INT);
  INSERT INTO t2(z) VALUES(1),(2),(3),(4),(5),(6),(7);
  SELECT 1,x,y FROM t1 WHERE x IN (SELECT z FROM t2) ORDER BY y DESC;
  SELECT 2,x,y FROM t1 WHERE x IN (SELECT z FROM t2) ORDER BY y DESC LIMIT 3;
  </verbatim></blockquote>

  This problem first appeared with the ORDER BY LIMIT optimization of
  check-in [559733b09e9630fa] on 2016-05-20 and first appeared in the 3.14.0
  release.  The problem was reported on the mailing list by Keith Medcalf.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  With the current trunk:

  <verbatim>
  SQLite version 3.16.0 2016-10-26 13:58:47
  Enter ".help" for usage hints.
  Connected to a transient in-memory database.
  Use ".open FILENAME" to reopen on a persistent database.
  sqlite> CREATE TABLE y1(a, b);
  sqlite> CREATE TABLE y2(x, y);
  sqlite> INSERT INTO y1 VALUES(1, 1);
  sqlite> INSERT INTO y2 VALUES(3, 3);
  sqlite> SELECT * FROM y1 LEFT JOIN y2 ON ((x=1 AND y=b) OR (x=2 AND y=b));
  1|1||
  sqlite> CREATE INDEX y2xy ON y2(x, y);
  sqlite> SELECT * FROM y1 LEFT JOIN y2 ON ((x=1 AND y=b) OR (x=2 AND y=b));
  1|1|3|3
  </verbatim>

  The first result (without the index) is correct.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem was introduced by the Next Generation Query Planner cutover
  at check-in [0fe31f60cadc5fe5] on 2013-06-26 and first appeared in 
  version 3.8.0 - over three years ago.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The second SELECT statement below gives an incorrect answer, even though
  it is logically identical to the first SELECT statement:

  <blockquote><verbatim>
  CREATE TABLE t1(a, b);
  INSERT INTO t1 VALUES(1, 2);
  CREATE TABLE t2(a, b);
  INSERT INTO t2 VALUES(3, 4);
  SELECT * FROM t1 LEFT JOIN t2 ON t2.a = t1.a AND t2.b = t1.b;
  SELECT * FROM t1 LEFT JOIN t2 ON (t2.a, t2.b) = (t1.a, t1.b);
  </verbatim></blockquote>

  Since row values are a new feature beginning with version 3.15.0, this bug
  could not have existed in any prior versions.

  Reported by Ján Hric on the sqlite-users mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following (syntactically incorrect) statement causes a NULL pointer
  deference in SQLite 3.15.1:

  <blockquote><verbatim>
  DETACH x IS #1;
  </verbatim></blockquote>

  The problem was introduced by parser changes associated with the introduction
  of the row-value logic in check-in [ddb5f0558c445699] on 2016-09-07.  The
  problem first appeared in SQLite 3.15.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL results in a segfault due to an attempt to access a
  non-existent cursor:

  <blockquote><verbatim>
  REATE TABLE T1(a TEXT);
  INSERT INTO T1(a) VALUES ('aaa');
  CREATE TABLE T2(a TEXT PRIMARY KEY,n INT UNIQUE);
  INSERT INTO T2(a, n) VALUES('aaa',0);
  SELECT * FROM T2
  WHERE (a,n) IN (SELECT T1.a, V.n FROM T1 JOIN (SELECT * FROM (SELECT 0 n)) V);
  </verbatim></blockquote>

  The problem might be caused by query flattening on the V table of
  the join on the RHS of the vector IN operator, but this is not
  a definitive diagnosis.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Each of the following two queries generates aa assertion fault in
  SQLite 3.15.1:

  <blockquote><verbatim>
  SELECT 1 WHERE (2,2) BETWEEN (1,1) AND (3,3);
  SELECT CASE (2,2) WHEN (1,1) THEN 1 ELSE 2 END;
  </verbatim></blockquote>

  This problem was originally reported by Mark Brand on the SQLite mailing list.  The problem is specific to 3.15.0 and 3.15.1, since it requires the use of row value expressions which were introduced in version 3.15.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The result of the first SELECT in the SQL below is correct.  But in the
  second version, where the left-hand side of the LEFT JOIN is coming from
  an equivalent subquery, no rows are returned.

  <blockquote><verbatim>
  CREATE TABLE x1(x,y,z);
  INSERT INTO x1 VALUES(0,0,1);
  CREATE TABLE x2(a);
  SELECT avg(z) FROM x1 LEFT JOIN x2 ON x GROUP BY y;
  SELECT avg(z) FROM (SELECT * FROM x1) LEFT JOIN x2 ON x GROUP BY y;
  </verbatim></blockquote>

  This problem has existed in the SQLite query planner for over 10 years,
  since before version 3.3.0 (2006-01-11).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another test case that does not involve an aggregate query:

  <blockquote><verbatim>
  CREATE TABLE x1(x,y,z);
  INSERT INTO x1 VALUES(0,0,1);
  CREATE TABLE x2(a);
  SELECT z FROM (SELECT * FROM x1) LEFT JOIN x2 ON x;
  </verbatim></blockquote>

  One row of result should be returned above, but all versions of SQLite from
  3.15.2 back to 3.3.0 (and likely before that too) return no rows.  Except,
  when the query flattener optimization is turned off using the
  sqlite3_test_control() interface, the correct answer is obtained in all cases.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Row-value UPDATE statements inside of TRIGGERs cause segfaults, memory leaks,
  and/or assertion faults.  Example:

  <blockquote><pre>
  CREATE TABLE t1(a,b,c);
  INSERT INTO t1 VALUES(1,2,3);
  CREATE TABLE t2(x);
  CREATE TRIGGER r2 AFTER INSERT ON t2 BEGIN
    UPDATE t1 SET (a,b,c)=(SELECT 4,5,6);
  END;
  INSERT INTO t2 VALUES(1);
  SELECT * FROM t1;
  SELECT * FROM t2;
  </pre></blockquote>

  This problem has existed ever since row value were added in version 3.15.0,
  about two months ago.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  If "PRAGMA recursive_triggers=ON" is configured, then DELETE triggers should fire for any rows deleted by REPLACE conflict handling. This is broken for WITHOUT ROWID tables in 3.16:

  <verbatim>
   PRAGMA recursive_triggers = 1;
   CREATE TABLE t1(a PRIMARY KEY, b) WITHOUT ROWID;
   CREATE TABLE del(a, b);
   CREATE TRIGGER t1d AFTER DELETE ON t1 BEGIN
     INSERT INTO del VALUES(old.a, old.b);
   END;
   INSERT INTO t1 VALUES(1, 'one');
   REPLACE INTO t1 VALUES(1, 'two');  -- causes DELETE, should fire trigger but does not
   SELECT * FROM del;

  </verbatim>

  With version 3.15, the final SELECT correctly returns a single row - {1, 'one'}. 3.16 returns zero rows.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem was introduced by check-in [925840cf] and first appeared in
  the 3.16.0 release.  The problem was first detected by the Firefox regression
  tests (see [https://bugzilla.mozilla.org/show_bug.cgi?id=1328161]) shortly
  after the 3.16.0 release.

  As far as we know at this time, 
  the minimum conditions necessary to invoke this problem are:

    1.  Recursive triggers enabled by "PRAGMA recursive_triggers=ON".
    2.  A WITHOUT ROWID table.
    3.  A DELETE trigger on the WITHOUT ROWID table.
    4.  Run an INSERT OR REPLACE operation on the WITHOUT ROWID table that
        overwrites a preexisting row.

  The DELETE trigger is suppose to fire in that case, but it does not in
  version 3.16.0.

  Check-in [925840cf] is a performance optimization that helps REPLACE operations
  run faster by directly overwriting the existing row rather than doing separate
  delete and insert operations.  But fewer b-tree manipulations are required
  if the row is simply overwritten.  Unfortunately, optimizing out the delete 
  operation also seems to have disabled the DELETE trigger.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The same optimization from check-in [925840cf] also causes problems with
  foreign key constraint enforcements on REPLACE statements for WITHOUT ROWID
  tables that lack secondary indexes.

  The optimization that causes all the trouble is seen here:
  [https://www.sqlite.org/src/artifact/0db6364a6a64e521?ln=1561-1564].
  That block of code causes the separate OP_Delete operation on REPLACE to 
  be omitted, causing the record to be directly overwritten, for WITHOUT ROWID
  tables that lack secondary indexes.

  The correct optimization is here:
  [https://www.sqlite.org/src/artifact/7af46a3be2656f5e?ln=1552-1569].
  The problem is that the original change in 3.16.0 and 3.16.1 omitted
  conditions 4 and 5, and thus omitted the separate OP_Delete operation in
  cases where it was actually needed.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2(b);
  CREATE TABLE t3(c);
  CREATE VIEW vvv AS SELECT b FROM t2 ORDER BY 1;
  SELECT 123 FROM t1, (SELECT b FROM vvv UNION ALL SELECT c from t3);
  </verbatim></blockquote>

  Bisecting isolates the problem to check-in [b9727e6bbfc3c] (2015-05-22).
  The problem first appeared in version 3.8.11 on 2015-07-27.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the SELECT on the last line of the SQL below, the ON clause of the LEFT JOIN
  refers to a column in the bbb table which is further to the right of the ON
  clause.  This causes the LEFT JOIN to behave as an INNER JOIN in the current
  implementation.  PostgreSQL throws an error.

  <blockquote><verbatim>
  CREATE TABLE aaa (a INTEGER);
  INSERT INTO aaa VALUES (1);
  INSERT INTO aaa VALUES (2);

  CREATE TABLE bbb (b INTEGER);
  INSERT INTO bbb VALUES (1);
  INSERT INTO bbb VALUES (2);

  CREATE TABLE ccc (c INTEGER);
  INSERT INTO ccc VALUES (2);

  SELECT * FROM aaa LEFT JOIN ccc ON c=bbb.b CROSS JOIN bbb ON a=b;
  </verbatim></blockquote>

  This behavior has been a part of SQLite since before version 3.0.0 (2004-06-18).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  PostgreSQL throws an error.  Another option is to treat the query as if
  it were the following:

  <blockquote><verbatim>
  SELECT * FROM aaa LEFT JOIN (ccc INNER JOIN bbb) ON a=b AND c=b;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the script below, the first query which uses automatic indexes gets
  a different and incorrect answer than the other three:

  <blockquote><verbatim>
  CREATE TABLE map_integer (id INT, name);
  INSERT INTO map_integer VALUES(1,'a');
  CREATE TABLE map_text (id TEXT, name);
  INSERT INTO map_text VALUES('4','e');
  CREATE TABLE data (id TEXT, name);
  INSERT INTO data VALUES(1,'abc');
  INSERT INTO data VALUES('4','xyz');
  CREATE VIEW idmap as
      SELECT * FROM map_integer
      UNION SELECT * FROM map_text;
  CREATE TABLE mzed AS SELECT * FROM idmap;

  .print ------ idmap:
  SELECT * FROM data JOIN idmap USING(id);
  .print ------ mzed:
  SELECT * FROM data JOIN mzed USING(id);
  .print ------ id_map1 with auto-index off:
  PRAGMA automatic_index=off;
  SELECT * FROM data JOIN idmap USING(id);
  .print ------ mzed with auto-index off:
  SELECT * FROM data JOIN mzed USING(id);
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced by check-in
  [0b3174e0b1] on 2011-06-23 and first appeared in SQLite version 3.7.7.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If ALTER TABLE ADD COLUMN is run to add a column to a table, and then later
  the sqlite3_blob_open() interface is run successfully on that new column, 
  and then sqlite3_blob_reopen() is run to move the sqlite3_blob object to a
  different row where the added column has never been initialized, then the
  sqlite3_blob_reopen() routine does not detect that the column it is
  attempting to open contains a NULL.  This can lead to further problems
  downstream.  A C program to demonstrate the issue is as follows:

  <blockquote><verbatim>
  #include <stdio.h>
  #include "sqlite3.h"

  int main(int argc, char **argv){
    sqlite3 *db;
    sqlite3_blob *pBlob = 0;
    int rc;

    sqlite3_open(":memory:", &db);
    sqlite3_exec(db,
      "CREATE TABLE t1(a INTEGER PRIMARY KEY);"
      "INSERT INTO t1(a) VALUES(1),(2),(5),(7);"
      "ALTER TABLE t1 ADD COLUMN b;"
      "ALTER TABLE t1 ADD COLUMN c;"
      "ALTER TABLE t1 ADD COLUMN d;"
      "UPDATE t1 SET d=zeroblob(15) WHERE a=1;",
    0, 0, 0);
    rc = sqlite3_blob_open(db, 0, "t1", "d", 1, 1, &pBlob);
    printf("sqlite3_blob_open() returns %d (%s)\n", rc, sqlite3_errmsg(db));
    rc = sqlite3_blob_reopen(pBlob, 2);
    printf("sqlite3_blob_reopen() returns %d (%s)\n", rc, sqlite3_errmsg(db));
    sqlite3_blob_close(pBlob);
    sqlite3_close(db);
    return 0;
  }
  </verbatim></blockquote>

  In the program above, the sqlite3_blob_reopen() should fail since it is
  attempting to open a NULL value.  But it returns SQLITE_OK.

  This problem appears to have always existed in the sqlite3_blob_reopen()
  interface since it was first added in SQLite 3.7.4 (2010-12-07).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the second query - the one after the CREATE INDEX -
  gets an incorrect answer:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  INSERT INTO t1 VALUES(1,'coffee'),(2,'COFFEE'),(3,'stress'),(4,'STRESS');
  SELECT '1:', a FROM t1 WHERE substr(b,4)='ess' COLLATE nocase;
  CREATE INDEX t1b ON t1(substr(b,4));
  SELECT '2:', a FROM t1 WHERE substr(b,4)='ess' COLLATE nocase;
  </verbatim></blockquote>

  The problem appears to be that the index is being used even though it uses
  a different collation.

  This problem was discovered during internal analysis and has never been
  observed in the wild - probably because nobody has ever though to create
  an index on an expression with non-BINARY collation.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Beginning with check-in [768c9859b21a3c8] the SQL

  <blockquote><verbatim>
  SELECT datetime(2457754, 'start of day');
  </verbatim></blockquote>

  is return NULL instead of '2016-12-31 00:00:00'.  Issue reported on the
  SQLite mailing list by Mark Brand.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  For example:

  <verbatim>
    PRAGMA recursive_triggers = 1;

    CREATE TABLE t1(a);
    CREATE INDEX i1 ON t1(a);
    CREATE TABLE t3(c);

    CREATE TRIGGER tr AFTER INSERT ON t3 BEGIN
      INSERT INTO t3 SELECT new.c+1 WHERE new.c<5;
      SELECT * FROM t1 WHERE a IN (1, 2, 3, 4);
    END;

    INSERT INTO t3 VALUES(2);
  </verbatim>

  The trigger above uses an ephemeral table for the IN(...) results. Running the sample above causes a segfault (or assert() failure if SQLITE_DEBUG is defined).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was introduced by check-in [6bf5ba10d28f1b0a32] on 2016-09-18
  and first appeared in version 3.15.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The [http://www.rfc-editor.org/rfc/rfc7159.txt|JSON spec] says that numeric
  values may not have leading zeros.  The value can consist of a single zero
  or a zero before the decimal point, but otherwise the first digit must be
  between 1 and 9.  The SQLite JSON extension does not enforce this and allows
  numeric values with an unlimited number of leading zeros.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The JSON spec requires that control characters are always escaped.  However,
  the SQLite JSON1 extension allows unescaped control characters in input strings
  to the "json_valid()" function.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query causes a stack overflow in the JSON parser:

  <blockquote><verbatim>
  WITH RECURSIVE c(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM c WHERE x<400000)
  SELECT json_valid(group_concat('[', '')) FROM c;
  </verbatim></blockquote>

  This problem was reported on the SQLite mailing list by Ralf Junker and is
  based on the n_structure_100000_opening_arrays.json test case from the
  [https://github.com/nst/JSONTestSuite|JSONTestSuite].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL results in three rows of output instead of just two:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT);
  INSERT INTO t1(b) VALUES ('one'), ('two');
  SELECT a FROM t1 WHERE a IN (1,2,1,2);
  </verbatim></blockquote>

  This bug was originally reported on the System.Data.SQLite mailing list.
  Bisect traces the problem back to check-in [925840cfdb] so that the problem
  was introduced in release 3.16.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL might cause an assertion fault.  If it does not, it results
  in two rows of the t table having the same rowid, which is database corruption.

  <blockquote><pre>
  CREATE TABLE t(i INTEGER PRIMARY KEY, a TEXT);
  INSERT INTO t VALUES (NULL, 'generates row 1');
  REPLACE INTO t VALUES (NULL, 'generates row 2'), (1, 'replaces row 1');
  SELECT * FROM t;
  </pre></blockquote>

  The expected output is:

  <blockquote><pre>
  1|replaces row 1
  2|generates row 2
  </pre></blockquote>

  The output generated (if there is no assertion fault) is:

  <blockquote><pre>
  1|generates row 1
  1|replaces row 1
  </pre></blockquote>

  This problem has existed in the code since version 3.8.4 (2014-03-10).  The
  problem was reported on the mailing list by E. Pasma.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes SQLite 3.19.0 to segfault:

  <blockquote><pre>
  SELECT *
  FROM (SELECT 1 a) s
  LEFT JOIN (
      SELECT 1 b, * FROM (
          SELECT * FROM (
              SELECT 1 c
          ) x
      ) x
  ) x;
  </pre></blockquote>

  The problem is the new LEFT JOIN flattening optimization in the just released
  SQLite 3.19.0.  No prior versions of SQLite are affected.
  The problem was reported by Mark Brand.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another test case:

  <blockquote><verbatim>
  SELECT *
  FROM (SELECT 1 a) s
  LEFT JOIN (
      SELECT c+1 b, * FROM (
          SELECT * FROM (
              SELECT 1 c
          ) x
      ) x
  ) x;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Further test cases for faults in the LEFT JOIN query flattener:

  <blockquote><verbatim>
  CREATE TABLE t1(c PRIMARY KEY,a TEXT(10000),b TEXT (10000));
  SELECT * FROM (SELECT 111) LEFT JOIN (SELECT c+222 FROM t1) GROUP BY 1;
  </verbatim></blockquote>

  And

  <blockquote><verbatim>
  CREATE TABLE t1(c PRIMARY KEY) WITHOUT ROWID;
  SELECT * FROM (SELECT 111) LEFT JOIN (SELECT c+222 FROM t1) GROUP BY 1;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL causes an assertion fault in the
  OP_IfNullRow opcode, which is part of the new LEFT JOIN flattening optimization:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  INSERT INTO t1 VALUES(1),(2),(3);
  CREATE VIEW v2 AS SELECT a, 1 AS b FROM t1;
  CREATE TABLE t3(x);
  INSERT INTO t3 VALUES(2),(4);

  SELECT * FROM t3 LEFT JOIN v2 ON a=x WHERE b=1;
  </verbatim></blockquote>

  This bug was reported via private email from an SQLite user.  The problem
  appears to be that the TK_IF_NULL_ROW nodes of the expression tree that are
  inserted as part of the LEFT JOIN query flattening transformation are being
  identified as constants (assuming that have all constant substructure) and
  are hence being factored out of the query evaluation loop to a point where
  the cursor for the LEFT JOIN table is not valid.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates a corrupt database file:

  <blockquote><verbatim>
  PRAGMA page_size = 1024;
  PRAGMA auto_vacuum = 1;
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  INSERT INTO t1 VALUES(25, randomblob(104));
  REPLACE INTO t1 VALUES(25, randomblob(1117));
  PRAGMA integrity_check;
  </verbatim></blockquote>

  The corruption only happens on an auto-vacuumed database.  No data is lost
  and the corruption can be repaired by running VACUUM.

  The problem was introduced by check-in [0b86fbca6615ccf1] on 2016-12-09 
  as part of an optimization to the b-tree row insertion logic. The optimization
  was to overwrite an existing row rather than doing a separate delete and
  insert.  The problem arises because the pointer map pages were not correctly
  updated if the overwriting row had the same amount of content on the btree
  leaf page but a different number of overflow pages.

  The problem first appeared in the 3.16.0 release.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When SQLite is compiled with SQLITE_ENABLE_STAT4, the following SQL causes
  an assertion fault or segfault:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  CREATE INDEX t1x ON t1(x);
  INSERT INTO t1 VALUES(1),(2),(3);
  ANALYZE;
  SELECT * FROM t1 WHERE x = -(-1);
  </verbatim></blockquote>

  This bug appears to have been in the STAT4 code since its STAT4 was
  first introduced in SQLite version 3.8.1 (2013-10-17).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL returns a different result in SQLite version 3.19.3 compared 
  to SQLite version 3.18.0 and PostgreSQL.  The result is also different if
  automatic indexes are disabled using "PRAGMA automatic_index=OFF".

  <blockquote><verbatim>
  CREATE TABLE t1(id INTEGER PRIMARY KEY);
  CREATE TABLE t2(id INTEGER PRIMARY KEY, c2 INTEGER);
  CREATE TABLE t3(id INTEGER PRIMARY KEY, c3 INTEGER);
  INSERT INTO t1(id) VALUES(456);
  INSERT INTO t3(id) VALUES(1),(2);

  SELECT t1.id, x2.id, x3.id
  FROM t1
  LEFT JOIN (SELECT * FROM t2) AS x2 ON t1.id=x2.c2
  LEFT JOIN t3 AS x3 ON x2.id=x3.c3;
  </verbatim></blockquote>

  This problem was introduced by the LEFT JOIN subquery flattener optimization
  of check-in [41c27bc0ff] (which is also the cause of other bugs such as
  [cad1ab4cb7b0] and [7fde638e94287d]).  The problem first appeared in the
  3.19.0 release (2017-05-22).

  This problem was reported on the SQLite users mailing list by Eric Sink.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The Query Planner Stability Guarantee (QPSG) states that SQLite will
  always generate the same query plan given the same source SQL text, as long
  as certain conditions are met, such as that there are no changes in the
  database schema.  The QPSG is important for many embedded systems.

  Unfortunately the QPSG is violated in SQLite 3.19.3 by the query at the
  end of the following SQL:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  CREATE INDEX t1b ON t1(b);
  SELECT a FROM t1 WHERE b GLOB ?1;
  </verbatim></blockquote>

  If the ?1 parameter is bound with "xyz*" then the query uses the t1b index.
  But if the ?1 parameter is bound to "0xyz*" then a full table scan is run.
  This is a violation of the QPST.

  The problem appears to have been introduced by the LIKE optimization
  enhancement of check-in [158290c0ab] which landed in SQLite 3.18.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Further testing shows that this problem is much older than originally thought.
  If the column passed to the GLOB operator has type TEXT, like this:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT);
  CREATE INDEX t1b ON t1(b);
  SELECT a FROM t1 WHERE b GLOB ?1;
  </verbatim></blockquote>

  Then the trouble dates back to check-in [50136840d5] circa 2009-10-16
  which first landed on version 3.6.20.

  This is the test script used for bisecting:

  <blockquote><verbatim>
  sqlite3 db :memory:
  db eval {
    CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT);
    CREATE INDEX t1b ON t1(b);
  }
  set x "abc*"
  set orig [db eval {explain SELECT a FROM t1 WHERE b GLOB $x}]
  set x "*abc"
  set new [db eval {explain SELECT a FROM t1 WHERE b GLOB $x}]
  if {$new!=$orig} {
    puts "different plan on reprepare"
  } else {
    puts "same plan!"
  }
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The PRAGMA integrity_check at the end of the following sequence of code
  reports a "wrong # of entries" error, even though the database is intact
  and correct.

  <blockquote><verbatim>
  CREATE TABLE Table1 (col TEXT NOT NULL PRIMARY KEY);
  ATTACH ':memory:' AS db2;
  CREATE TABLE db2.Table2(col1 INTEGER, col2 INTEGER, col3 INTEGER, col4 INTEGER);
  CREATE UNIQUE INDEX db2.idx_col1_unique ON Table2 (col1);
  CREATE UNIQUE INDEX db2.idx_col23_unique ON Table2 (col2, col3);
  CREATE INDEX db2.idx_col2 ON Table2 (col2);
  INSERT INTO Table2 VALUES(1,2,3,4);
  PRAGMA integrity_check;
  </verbatim></blockquote>

  Bisecting shows that the problem first appeared with check-in [88439a866]
  on 2016-0319 and was first
  released with SQLite 3.12.0 on 2016-03-29

  This problem was reported on the sqlite-users mailing list by
  Brendan E. Coughlan.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When generating byte-code for an integrity_check on a connection with
  ATTACHed databases, the temporary register cache was not being reset
  after each database had been checked.  If the schema for "main" was much
  simpler than the schema for the first attached database, this could
  cause cached temporary registers that overlapped with index row counters
  to be used, thus messing up the index row counts and leading to the 
  "wrong # of rows" report.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The same LEFT JOIN query gives different results depending on whether or
  not automatic indexes are enabled.  The code below illustrates:

  <blockquote><verbatim>
  CREATE TABLE t1(w int);
  CREATE TABLE t2(y int);
  INSERT INTO t1 VALUES(NULL);
  INSERT INTO t2 VALUES(1);
  CREATE INDEX t2y ON t2(y);
  SELECT * FROM t1 LEFT JOIN t2 WHERE t2.y IS t1.w;
  PRAGMA automatic_index=off;
  SELECT * FROM t1 LEFT JOIN t2 WHERE t2.y IS t1.w;
  </verbatim></blockquote>

  Both queries should return no rows.  But when automatic indexes are turned on
  (the default) a single row of two NULLs is returned.

  Bisecting shows this problem was introduced by check-in [3428043cd0029445]

  This problem was discovered during internal testing and analysis and has
  never been reported in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The name of the output column on the SELECT statement below as reported by the
  sqlite3_column_name() interface changed in between versions 3.18.0 and 3.19.0.

  <blockquote><verbatim>
  CREATE TABLE x(a,b);
  INSERT INTO x VALUES(1,2);
  SELECT x.a FROM x LEFT JOIN (SELECT * FROM x) AS y;
  </verbatim></blockquote>

  SQLite version 3.18.0 and earlier reports
  the column name as "a".
  But in SQLite 3.19.0 and 3.20.0, the
  column name is reported as "x.a".

  The SQLite documentation clearly states that, if there is no AS clause,
  the names of output columns are indeterminate, arbitrary, and subject 
  to change.  The column names are intended for display purposes only.
  And yet, many applications depend on particular column names.
  This name shift resulted at least one application error.

  The problem was introduced by check-in [41c27bc0ff1d3135], which added a new
  optimization for doing query flattening on the right operand of a LEFT JOIN.

  <b>Analysis:</b>

  Without an AS clause, if the output column is a direct reference to one
  of the input table columns, then the output column name will be the same
  as the input column name.  Except, if query flattening occurs, the output
  column name becomes an exact copy of the input SQL text.

  The exception explains the name shift.  Prior to version 3.19.0, the query
  flattener would not operate on a LEFT JOIN.  Hence the output column name
  was derived from the input column name and was "a".  But once the LEFT JOIN
  query flattener optimization was added by check-in [41c27bc0ff1d3135], the
  column naming exception mentioned in the previous paragraph kicked in and
  caused the column to be named as it was typed in the input SQL:  "x.a".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  <b>Column Naming Rules</b>

  The names of the columns in the result set of a query can take three forms:

    1.  If there is an AS clause, the label to the right of the AS keyword
        becomes the column name.  This has always worked and has never been an
        issue.  When it doubt, the programmer can always put an AS clause onto
        the result expression to define the name of a column.

    2.  If the output column is a direct reference to a column of an input table,
        the the output column name is the name of the column in the input table,
        as it appears in the CREATE TABLE (or CREATE VIEW) statement for the
        input table.  (The previous sentence assumes that PRAGMA short_column_names
        if ON and PRAGMA full_column_names is OFF, which are the default settings.
        As both of those pragmas are deprecated, we will not consider the other
        cases.)  For example, if the table is "CREATE TABLE t1(x)", and if the
        SELECT statement refers to the column of t1 as "x" or "X" or "(X)" or
        "t1.x" or "main.t1.x" or "[main].[t1].[x]" or anything else, the output
        column name will always be just "x", as it appears in the CREATE TABLE
        statement.

    3.  If neither of the above apply, then the column name is a copy of the part
        of the original SQL statement that specified the column.  Examples:
        In the query "SELECT 5+11;" the name of the column is "5+11".  In the
        query "SELECT +x FROM t1" the "+x" is not a direct reference to the t1.x
        column because of the "+" operator, so the name of the output column is "+x".

  Columns are named using form (1) if it applies, otherwise
  use form (2) if it applies, and finally fall back to form (3) if none of the
  other forms apply.

  Except, since about 2009, if the query flattener
  ([https://sqlite.org/optoverview.html#flattening]) is involved, then columns of
  the query that would otherwise use form (2) shift to using form (3).

  The exception is a hack that was inserted in order to give sensible column
  names when working with views.  The problem that the hack aimed to address
  is demonstrated by this SQL:

  <blockquote><verbatim>
  CREATE TABLE x(a,b);
  INSERT INTO x VALUES(1,2);
  CREATE VIEW y(c,d) AS SELECT a,b FROM x;
  SELECT c, d FROM y;
  </verbatim></blockquote>

  On expects the column names from the final query to be "c" and "d".  But the
  query flattener runs in this case and transforms the input query into 
  "SELECT a,b FROM x".  With no adjustments, the names of the output columns
  for the revised query would be "a" and "b".  The hack to fix this was to
  change the column naming rule from (2) to (3) so that the columns would continue
  to be called "c" and "d" as they were shown in the input text.

  <b>The Problem</b>

  That exception to the column naming rules that kicks in when the query flattener
  runs is the cause of the bug reported in this ticket.  The original query
  used the form "t1.x" to specify the column in the result set.  In SQLite 3.18.0
  and earlier, rule (2) above applied and the output column name was just "x".
  Then in SQLite 3.19.0, the query flattener was enhanced so that it could be
  applied to the query in question, making the query run faster.  But a side-effect
  was that the column naming rule shifted from (2) to (3) and the output column
  name became "t1.x".

  <b>Proposed Solution</b>

  The proposed solution (implemented by check-in [09834279aeca3bda]) is to eliminate
  the exception to the column naming rules associated with the query flattener.
  That means that the query that provoked this ticket would continue to use
  column naming rule (2) and would continue to see the column name as "x" as
  the application expects.  This change is accomplished by moving the column
  naming step ahead of the query flattener step, so that column naming happens
  before query flattening, not afterwards.

  <b>Possible Problems</b>

  The proposed solution makes column names more consistent and easier to understand.
  It also makes the implementation smaller and simpler and helps it to run a little
  faster.  However, there may be legacy programs that depended on the old behavior
  (shifting the column naming rule from (2) to (3) when using the query flattener)
  and those legacy programs could potentially break due to this change.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL leads to an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(x INTEGER PRIMARY KEY UNIQUE, b) WITHOUT ROWID;
  CREATE INDEX t1x ON t1(x);
  </verbatim></blockquote>

  This problem appears in all versions of SQLite that support WITHOUT ROWID
  tables, which is to say all versions since 3.8.2 (2013-12-06).

  This problem was discovered by [https://github.com/google/oss-fuzz|OSSFuzz].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When assert()s are enabled (which is to say when SQLite is compiled using
  -DSQLITE_DEBUG) the following SQL causes an assertion fault in the 
  byte-code engine:

  <blockquote><verbatim>
  PRAGMA reverse_unordered_selects=on;
  CREATE TABLE t1(a,b);
  INSERT INTO t1 VALUES(1,2);
  CREATE INDEX t1b ON t1(b);
  SELECT a FROM t1 WHERE b=-99 OR b>1;
  </verbatim></blockquote>

  Bisecting shows that this bug was introduced with check-in [663473850c] 
  on 2016-11-15 and first appeared in release 3.16.0.  As far as we can
  determine at this time, the bug is harmless in the sense that SQLite
  functions normally and computes the correct answer when assert()s are
  disabled.  However, the assertion fault does indicate that the logic
  is confused and needs to be refactored for long-term maintainability.

  This problem was discovered by [https://github.com/google/oss-fuzz|OSSFuzz].


login: anonymous
mimetype: text/plain
icomment:
  In the lsm1 extension, lsm_insert(): insert a new key/value pair into the database, overwriting any existing entry with the same key (https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki#writing_to_a_database), When I tried to test this case (insert existing entry with the same key) the database size increasing without overwriting any existing entry with the same key. The below test c code illustrate the case:

  void test(void)
  {
  	lsm_db *db;
  	int rc = lsm_new(0, &db);
  	if( rc != LSM_OK )
  		return;

  	rc = lsm_open(db, "test.lsmdb");
  	if( rc != LSM_OK )
  		return;

  	DWORD Key = 0x12345678;
  	char Val_1[] = "12345678";
  	char Val_2[] = "ABCDEFGH";

  	lsm_begin(db, 1);

  	lsm_insert(db, (void *)&Key, sizeof(Key), Val_1, strlen(Val_1)+1);

  	lsm_insert(db, (void *)&Key, sizeof(Key), Val_2, strlen(Val_2)+1);

  	lsm_commit(db, 0);

  	lsm_cursor *csr;
  	if( lsm_csr_open(db, &csr) != LSM_OK )
  		return;

  	int seekrc = lsm_csr_seek(csr, &Key, sizeof(Key), LSM_SEEK_EQ);
  	int validrc = lsm_csr_valid(csr);
  	if( seekrc == LSM_OK && validrc )
  	{
  		char *pKeyPtr, *pValPtr;
  		int nKey, nVal;
  		lsm_csr_key(csr, (const void **)&pKeyPtr, &nKey);
  		lsm_csr_value(csr, (const void **)&pValPtr, &nVal);

  		printf("key = %08X, val = %s\n", pKeyPtr, pValPtr);
  	}
  	lsm_csr_close(csr);

  	lsm_close(db);
  }


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQLite causes an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(x TEXT);
  CREATE INDEX t1x1 ON t1(1);
  SELECT 1 IN (SELECT 2) FROM t1;
  </verbatim></blockquote>

  Notice the goofy index - an index on a constant expression.  An index on
  a constant expression appears to be required to expression this problem.

  Bisecting shows that this problem was introduced by check-in [a47efb7c8520a011]
  (2017-04-11) and first appeared in the 3.19.0 release (2017-05-22).

  This problem was discovered by OSSFuzz.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Another reproduction:

  <verbatim>
    CREATE TABLE t1 ( x );
    CREATE INDEX i1 ON t1 ( upper(x) );
    SELECT 1 FROM t1 dfs WHERE upper(x)=1 AND upper(x) IN ('a', 'b', 'c');
  </verbatim>


login: anonymous
mimetype: text/plain
icomment:
  In the lsm1 extension, lsm_insert(): insert a new key/value pair into the database, overwriting any existing entry with the same key (https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki#writing_to_a_database), When I tried to test this case (insert existing entry with the same key) the database size increasing without overwriting any existing entry with the same key. The below test c code illustrate the case:

  void test(void)
  {
  	lsm_db *db;
  	int rc = lsm_new(0, &db);
  	if( rc != LSM_OK )
  		return;

  	rc = lsm_open(db, "test.lsmdb");
  	if( rc != LSM_OK )
  		return;

  	DWORD Key = 0x12345678;
  	char Val_1[] = "12345678";
  	char Val_2[] = "ABCDEFGH";

  	lsm_begin(db, 1);

  	lsm_insert(db, (void *)&Key, sizeof(Key), Val_1, strlen(Val_1)+1);

  	// below insert should update the same key with a new value
  	lsm_insert(db, (void *)&Key, sizeof(Key), Val_2, strlen(Val_2)+1);

  	lsm_commit(db, 0);

  	lsm_cursor *csr;
  	if( lsm_csr_open(db, &csr) != LSM_OK )
  		return;

  	int seekrc = lsm_csr_seek(csr, &Key, sizeof(Key), LSM_SEEK_EQ);
  	int validrc = lsm_csr_valid(csr);
  	if( seekrc == LSM_OK && validrc )
  	{
  		char *pKeyPtr, *pValPtr;
  		int nKey, nVal;
  		lsm_csr_key(csr, (const void **)&pKeyPtr, &nKey);
  		lsm_csr_value(csr, (const void **)&pValPtr, &nVal);

  		// the below printf will display the old value rather than new updated value
  		printf("key = %08X, val = %s\n", pKeyPtr, pValPtr);
  	}
  	lsm_csr_close(csr);

  	lsm_close(db);
  }


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If a virtual table invokes sqlite3_step() as part of its xConnect method
  (as RTREE does) and if that virtual table is used inside of a trigger and
  if an external schema change occurs on the first use of that trigger, then
  the xConnect method will provoke a schema reset that will delete a pointer
  to the trigger out from under the query planner, resulting in a use-after-free
  and a probable segfault.  The following C code demonstrates the problems:

  <blockquote><verbatim>
  #include <stdio.h>
  #include <unistd.h>
  #include "sqlite3.h"

  int main(int argc, char *argv){
    sqlite3 *db1, *db2;
    unlink("test.db");
    printf("VERSION: %s\n", sqlite3_libversion());
    sqlite3_open("test.db", &db1);
    sqlite3_exec(db1, 
     "CREATE VIRTUAL TABLE r1 USING rtree(id, x1, x2, y1, y2);\n"
     "CREATE TABLE t1(id, x1, x2, y1, y2);\n"
     "CREATE TABLE log(l);\n"
     "CREATE TRIGGER tr1 AFTER INSERT ON t1 BEGIN\n"
     "  INSERT INTO r1 VALUES(new.id, new.x1, new.x2, new.y1, new.y2);\n"
     "  INSERT INTO log VALUES('r1: ' || new.id);\n"
     "END;", 0, 0, 0);
    sqlite3_close(db1);
    sqlite3_open("test.db", &db1);
    sqlite3_open("test.db", &db2);
    sqlite3_exec(db1, "INSERT INTO log VALUES('startup');", 0, 0, 0);
    sqlite3_exec(db2, "CREATE TABLE newtab(a,b);", 0, 0, 0);
    sqlite3_exec(db1, "INSERT INTO t1 VALUES(1,2,3,4,5);", 0, 0, 0);
    sqlite3_close(db1);
    sqlite3_close(db2);
    return 0;
  }
  </verbatim></blockquote>

  This problem appears to have existed in the virtual table implementation
  forever.  The test program above first began to fail with
  check-in [ebc9433f] on 2010-02-16 (SQLite version 3.6.23) because that
  check-in was the first to cause RTREE to invoke sqlite3_step() from within
  its xConnect method.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  With 3.20.0, this script:

  <verbatim>
    CREATE TABLE t2(x);
    INSERT INTO t2 VALUES('.ABC');
    INSERT INTO t2 VALUES('.abcd');
    INSERT INTO t2 VALUES('.DEF');

    SELECT substr(x, 2) FROM t2 ORDER BY substr(x, 2) COLLATE nocase;
    .print ------
    CREATE INDEX i2 ON t2( substr(x, 2) );
    SELECT substr(x, 2) FROM t2 ORDER BY substr(x, 2) COLLATE nocase;
  </verbatim>

  returns:

  <verbatim>
    ABC
    abcd
    DEF
    ------
    ABC
    DEF
    abcd
  </verbatim>

  The problem is that in the second query SQLite uses the index to optimize the ORDER BY clause, even though the collation sequence of the index does not match that of the ORDER BY expression.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  This problem has been present since 3.9.0 - when the indexes-on-expressions feature was added to SQLite.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Use of the sqlite3_result_pointer() interface can, on some obscure queries,
  result in an assertion fault on this line: [https://sqlite.org/src/artifact/2f9d672af?ln=38].

  This problem was discovered by the Thunderbird project when they upgraded
  to SQLite 3.20.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script causes the command-line shell to crash:

  <blockquote><verbatim>
  PRAGMA empty_result_callbacks=1;
  .dump
  </verbatim></blockquote>

  This is a problem in the command-line shell program, not the the core SQLite
  library.  The problem seems to have been introduced by check-in
  [a0028d4808275c] on 2009-05-21 and first appeared in SQLite 3.6.15.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL results in a segfault:

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  SELECT * FROM t1 WHERE 1 IN (SELECT value FROM json_each(x));
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced by check-in [712267c9c0] on 2017-06-23 and was first released with SQLite 3.20.0.  Preliminary analysis
  suggests that the problem has nothing to do with the JSON1 extension, but is
  instead a code-generator fault of some kinds that causes an OP_Column opcode
  to be run on a cursor for the "t1" table before that cursor has been opened.


  This problem was reported on the public mailing list by Martin Thierer.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The DROP TABLE statement does not work on a WITHOUT ROWID table when
  SQLite is compiled with -DSQLITE_OMIT_AUTOVACUUM.  C code below
  demonstrates the malfunction.

  This problem has existed ever since WITHOUT ROWID tables were added
  in release 3.8.2, 2013-12-06.  As SQLITE_OMIT_AUTOVACUUM is not a
  supported or tested configuration, this ticket is given a low priority.

  This problem was reported on the sqlite-users mailing list by Bernard Schurdevin.

  <blockquote><verbatim>
  #include <stdio.h>
  #include <unistd.h>
  #include "sqlite3.h"

  static int showResults(void *notUsed, int argc, char **argv, char **colv){
    int i;
    for(i=0; i<argc; i++){
      printf("%s = %s\n", colv[i], argv[i] ? argv[i] : "NULL");
    }
    return 0;
  }

  int main(int argc, char **argv){
    sqlite3 *db;
    int rc;
    char *zErrmsg = 0;
    unlink("x1.db");
    rc = sqlite3_open("x1.db", &db);
    if( rc ){
      fprintf(stderr, "cannot open 'x1.db'\n");
      return 1;
    }
    sqlite3_exec(db, 
      "CREATE TABLE foo (num INTEGER PRIMARY KEY,comment TEXT COLLATE NOCASE)"
      " WITHOUT ROWID;\n"
      "PRAGMA integrity_check;\n"
      "DROP TABLE foo;", showResults, 0, &zErrmsg);
    if( zErrmsg ){
      fprintf(stderr, "ERROR: %s\n", zErrmsg);
      sqlite3_free(zErrmsg);
      zErrmsg = 0;
    }
    sqlite3_exec(db,"PRAGMA integrity_check;", showResults, 0, &zErrmsg);
    if( zErrmsg ){
      fprintf(stderr, "ERROR: %s\n", zErrmsg);
      sqlite3_free(zErrmsg);
      zErrmsg = 0;
    }
    sqlite3_close(db);
    return 0;    
  }
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A DELETE statement that includes a LIMIT clause (which requires that
  SQLite be built with the -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT compile-time option)
  does not work correctly for a WITHOUT ROWID table.  No rows are deleted.  The
  DELETE statement might report SQLITE_CORRUPT, even though the database is
  not actually corrupt.
  Example:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c,d TEXT, PRIMARY KEY(a,b)) WITHOUT ROWID;
  INSERT INTO t1(a,b,c,d) VALUES(1,2,3,4),(5,6,7,8),(9,10,11,12);
  DELETE FROM t1 WHERE a=5 AND b=6 LIMIT 2;
  </verbatim></blockquote>

  The final DELETE statement above causes an SQLITE_CORRUPT error.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query in the SQL shown below gives an incorrect answer after
  ANALYZE has been run:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b INTEGER);
  CREATE INDEX t1b ON t1(b);
  CREATE TABLE t2(x INTEGER PRIMARY KEY, y INTEGER);
  CREATE INDEX t2y ON t2(y);
  WITH RECURSIVE c(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM c WHERE x<49)
    INSERT INTO t1(b) SELECT x/10 - 1 FROM c;
  WITH RECURSIVE c(x) AS (VALUES(-1) UNION ALL SELECT x+1 FROM c WHERE x<19)
    INSERT INTO t2(x,y) SELECT x, 1 FROM c;
  SELECT DISTINCT y FROM t1, t2 WHERE b=x AND b<>-1;
  ANALYZE;
  SELECT DISTINCT y FROM t1, t2 WHERE b=x AND b<>-1;
  </verbatim></blockquote>

  This problem seems to have been introduced by the skip-ahead-distinct
  optimization of check-in [f489b5bb6b35665], first included in SQLite
  version 3.19.0.  The problem was reported on the public mailing list
  by David Raymond.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Two more test cases:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b INTEGER);
  CREATE INDEX t1b ON t1(b);
  CREATE TABLE t2(x INTEGER PRIMARY KEY, y INTEGER);
  CREATE INDEX t2y ON t2(y);
  WITH RECURSIVE c(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM c WHERE x<49)
    INSERT INTO t1(b) SELECT -(x/10 - 1) FROM c;
  WITH RECURSIVE c(x) AS (VALUES(-1) UNION ALL SELECT x+1 FROM c WHERE x<19)
    INSERT INTO t2(x,y) SELECT -x, 1 FROM c;
  SELECT DISTINCT y FROM t1, t2 WHERE b=x AND b<>1 ORDER BY y DESC;
  ANALYZE;
  SELECT DISTINCT y FROM t1, t2 WHERE b=x AND b<>1 ORDER BY y DESC;
  </verbatim></blockquote>

  <blockquote><verbatim>
  CREATE TABLE t1(a, b);
  CREATE INDEX t1a ON t1(a, b);
  -- Lots of rows of (1, 'no'), followed by a single (1, 'yes').
  WITH RECURSIVE c(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM c WHERE x<100)
    INSERT INTO t1(a, b) SELECT 1, 'no' FROM c;
  INSERT INTO t1(a, b) VALUES(1, 'yes');
  CREATE TABLE t2(x PRIMARY KEY);
  INSERT INTO t2 VALUES('yes');
  SELECT DISTINCT a FROM t1, t2 WHERE x=b;
  ANALYZE;
  SELECT DISTINCT a FROM t1, t2 WHERE x=b;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query in the SQL below gets a different answer after the index is
  created.  The answer without the index is correct.

  <blockquote><verbatim>
  CREATE TABLE t1(a, b);  INSERT INTO t1 VALUES(1,2);
  CREATE TABLE t2(x, y);  INSERT INTO t2 VALUES(1,3);
  SELECT y FROM t1, t2 WHERE a=x AND b<=y ORDER BY b DESC;
  CREATE INDEX t1ab ON t1(a,b);
  SELECT y FROM t1, t2 WHERE a=x AND b<=y ORDER BY b DESC;
  </verbatim></blockquote>

  This problem appears to have been introduced by check-in [559733b09e] on
  2016-05-20, 572 days ago.  The problem first appeared in version 3.14.0.
  The problem was discovered in the wild and reported to the SQLite developers
  via private email.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the SQL below, the result-set column name for the first SELECT is "a" but
  the column name in the table created by the CREATE TABLE AS on an identical
  SELECT is "x".

  <blockquote><verbatim>
  CREATE TABLE t1(x INT);
  INSERT INTO t1(x) VALUES(1);
  .headers on
  SELECT a FROM (SELECT x AS a FROM t1);
  CREATE TABLE t2 AS
  SELECT a FROM (SELECT x AS a FROM t1);
  PRAGMA table_info(t2);
  </verbatim></blockquote>

  This problem first appeared with check-in [ade7ddf1998190b2] which is a
  fix for a prior ticket [de3403bf5ae5f72ed].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Another example:

  <blockquote><verbatim>
  .headers on
  CREATE TABLE t1(aaa INT);
  INSERT INTO t1(aaa) VALUES(1);
  SELECT BBb FROM (SELECT aaa AS Bbb FROM t1);
  CREATE TABLE t2 AS
  SELECT BBb FROM (SELECT aaa AS Bbb FROM t1);
  PRAGMA table_info(t2);
  </verbatim></blockquote>

  For the straight SELECT statement, recent code gives a column name of Bbb and a column name of aaa for the t2 table.  Older code gives the name BBb for both.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an assertion fault on the REPLACE statement:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT PRIMARY KEY, b INT UNIQUE);
  CREATE INDEX t1ab ON t1(a*b);
  INSERT INTO t1(a,b) VALUES(1,2);
  REPLACE INTO t1(a,b) VALUES(1,3);
  </verbatim></blockquote>

  This problem appears to go back to the introduction of indexes on expressions
  with SQLite release 3.9.0 on 2015-10-14.  The problem was discovered by OSSFuzz.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The UPDATE at the end of the following sequence of SQL statements enters
  an infinite loop.

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  CREATE INDEX t1b ON t1(a);
  CREATE INDEX t1c ON t1(b);
  INSERT INTO t1 VALUES(1,2);
  UPDATE t1 SET a = a+2 WHERE a>0 OR b>0;
  </verbatim></blockquote>

  Bisecting shows that this problem was introduced by the single-pass update
  optimization of check-in [7ae6104a3e0d1d2] on 2017-01-11 and first released
  with version 3.17.0.  The problem was discovered by Harald Klimach and 
  reported on the SQLite mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statement below if valid syntax and ought to work, but SQLite rejects
  it and refuses to evaluate it:

  <blockquote><verbatim>
  CREATE TABLE x(a, b, PRIMARY KEY (a,b));
  CREATE TABLE y(a);
  CREATE TABLE z(a, b);
  SELECT * FROM x JOIN y ON y.a = x.a  WHERE (x.a, x.b) IN (SELECT a, b FROM z);
  </verbatim></blockquote>

  This problem has been in the code since support for row-values was added
  with version 3.15.0 (2016-10-14).  SQLite does not generate an incorrect
  answer because of this bug.  Rather, SQLite merely refuses to process what
  ought to be valid SQL syntax.

  Note that this problem was fixed prior to the ticket being created.  The
  ticket was added after-the-fact for documentation purposes.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The SELECT statement at the end of the SQL shown below should return a two-row result, but instead returns
  an empty set.

  <blockquote><verbatim>
  CREATE TABLE t1(id, json);
  INSERT INTO t1(id,json) VALUES(1,'{"items":[3,5]}');
  CREATE TABLE t2(id, json);
  INSERT INTO t2(id,json) VALUES(2,'{"value":2}');
  INSERT INTO t2(id,json) VALUES(3,'{"value":3}');
  INSERT INTO t2(id,json) VALUES(4,'{"value":4}');
  INSERT INTO t2(id,json) VALUES(5,'{"value":5}');
  INSERT INTO t2(id,json) VALUES(6,'{"value":6}');
  SELECT * FROM t2 CROSS JOIN t1
   WHERE EXISTS(SELECT 1 FROM json_each(t1.json,'$.items') AS Z
                 WHERE Z.value==t2.id);
  </verbatim></blockquote>

  The problem is that the query planner fails to recognize that the EXISTS
  expression in the WHERE clause has a dependency on table t1, and so it
  generates code for the EXISTS expression after the outer loop over t2
  and before the inner loop over t1 has been run.  This means that the
  t1.json value for the EXISTS expression is incorrect, and an incorrect
  answer results.

  This problem appears to have been in the code ever since table-valued
  functions were introduced with version 3.9.0 (2015-10-14).

  This problem was reported on the sqlite-users mailing list by
  Jens Alfke.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script shows a query that gives a different answer with and
  without the t1b index.  The difference only appears when SQLite is compiled
  using the SQLITE_ENABLE_STAT4 option.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  INSERT INTO t1(a,b) VALUES(1,1),(2,NULL),(3,NULL);
  CREATE VIEW err AS
    SELECT a
      FROM t1 X
    WHERE 2 > (
      SELECT COUNT(*) FROM t1 Y
       WHERE ( X.b IS NOT NULL AND Y.b IS NULL )
          OR ( Y.b < X.b )
          OR ( X.b IS Y.B AND Y.a > X.a )
      );
  SELECT * FROM err;
  .print -----------------
  CREATE INDEX t1b ON t1(b);
  SELECT * FROM err;
  </verbatim></blockquote>

  Bisecting shows this problem first appeared with check-in
  [712267c9c08fdcef] and subsequently in release 3.20.0
  on 2017-08-01. The problem was reported on the sqlite-users
  mailing list by Kenichi Ishigaki.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is another demonstration of the same problem that uses two separate
  tables and omits the WHERE clause.  In this example, both queries return
  a single row, but the answer is different.  The correct answer is 2, but
  when the index is created the answer becomes 0.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  INSERT INTO t1(a,b) VALUES(1,1);
  CREATE TABLE t2(aa INTEGER PRIMARY KEY, bb);
  INSERT INTO t2(aa,bb) VALUES(1,1),(2,NULL),(3,NULL);
  CREATE VIEW err AS
    SELECT (
      SELECT COUNT(*) FROM t2
       WHERE ( t1.b IS NOT NULL AND t2.bb IS NULL )
          OR ( t2.bb < t1.b )
          OR ( t1.b IS t2.bb AND t2.aa > t1.a )
      )
      FROM t1;
  SELECT * FROM err;
  .print -----------------
  CREATE INDEX t2b ON t2(bb);
  SELECT * FROM err;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The integrity_check at the end of the following block of SQL reports
  index corruption.

  <blockquote><verbatim>
  CREATE TABLE t1 (a INTEGER);
  CREATE INDEX idx1 ON t1 (lower(a));
  INSERT INTO t1 VALUES('0001234');
  PRAGMA integrity_check;
  </verbatim></blockquote>

  Running REINDEX clears the problem.

  This problem appears to have been in the code since indexes on expressions
  were first introduced with version 3.9.0 (2015-10-14).

  This problem was discovered in the wild and reported to the SQLite developers
  through private channels.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is deeper discussion of the problem:

  Internally, SQLite tracks values as Mem objects.  (The sqlite3_value object
  in the public interface is just a renamed Mem object.)  Each Mem object is
  able to hold both a numeric value and its equivalent string representation
  at the same time.  Hence sqlite3_value_int() and 
  sqlite3_value_text() can both be answered for the same object without having
  to do any conversions.

  The mapping between string representations and numeric representations is
  many to one.  There are many different strings that map to the same numeric
  value.  For example, '123', '0123', and '+123' all map into numeric 123.
  Every numeric value has a canonical string representation, which is what is
  generated when SQLite converts numeric to string.  So a numeric-to-string
  conversion is one-to-one.  But a string-to-numeric conversion is many-to-one.

  If you have situation where a table is storing a numeric value, and an
  index is based on the string representation of that numeric value, it is
  critical that the string representation used by the index be the canonical
  representation of the numeric value, and not one of the other strings that
  happen to map into that numeric value.  Otherwise, the index will be
  inconsistent with the table.

  This bug arose when a non-canonical string representation was used to build
  the index.

  The fix was to add logic to SQLite that ensures that when the same Mem object
  holds both a string representation and a numeric representation, that the
  string rep is the canonical string rep.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL generates an assertion fault in FTS4:

  <blockquote><verbatim>
  CREATE VIRTUAL TABLE ft USING fts4(a, b, c);
  INSERT INTO ft(rowid, a, b, c) VALUES(1, 'unus duo', NULL, NULL);
  INSERT INTO ft(rowid, a, b, c) VALUES(2, NULL, NULL, NULL);
  BEGIN;
    UPDATE ft SET b='septum' WHERE rowid = 1;
    UPDATE ft SET c='octo' WHERE rowid = 1;
  COMMIT;
  </verbatim></blockquote>

  Bisecting shows that this problem first appeared with
  check-in [a1d08fd3d0419da8] on 2015-09-29 and was released
  with version 3.9.0.

  This is a simplification of a problem discovered in the wild and reported
  to SQLite developers through private channels.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL returns no rows.  It ought to return four rows:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY,b);
  INSERT INTO t1(a,b) VALUES(1,11),(2,22),(3,33),(4,44);
  SELECT * FROM t1 WHERE (a,b)>(0,0);
  </verbatim></blockquote>

  Changing the comparison operator from &gt; to &gt;= gives the correct result.
  The correct result also results from &lt; and &lt;= operators assuming the values
  on the RHS are adjusted accordingly.

  The problem appears to have been in the code ever since row-values were
  introduced in version 3.15.0 on 2016-10-14.  The problem was reported
  on the sqlite-users mailing list by an anonymous users.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Root cause was an incorrect table lookup when translating the TK_GT expression
  node operator from the parse tree into the OP_SeekGE opcode for the virtual
  machine.  This was probably caused by insufficient caffeine.  The problem
  should have been detected by subsequent assert() statement, except those
  assert()s were omitted.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  SQLite does not have a separate Boolean type.  Instead, it interprets
  integer, floating-point, text, and blob values as true or false.  But
  the rules for determining truth are not consistent.

  The ambiguity arises with floating-point values.  Some operations consider
  floating point values to be true if they are non-zero.  Others consider
  floating point values to be true if their integer part os non-zero.
  Example:

  <blockquote><verbatim>
  CREATE TABLE t1(x ANY);
  INSERT INTO t1 VALUES(0.5);
  SELECT x OR (8==9), NOT NOT x, CASE WHEN x THEN 1 ELSE 0 END FROM t1 WHERE x;
  </verbatim></blockquote>

  The WHERE clause and the CASE statement both consider the value 0.5 to 
  be true, but the OR and NOT operators consider 0.5 to be false.

  This confusion dates back to before version 3.6.1 (2008-08-06).  The
  problem was seen during internal source code analysis and has never
  caused problems in the wild, as far as we are aware.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The fix is to interpret floating-point values as true if they are non-zero,
  even if their integer part is zero.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  SQLite version 3.23.0 added the new LEFT JOIN strength reduction optimization
  that converts a LEFT JOIN into an ordinary JOIN if it can be proven that some
  column of the right-hand table of the LEFT JOIN must be non-NULL on every output
  row.  But the prover is overzealous and believes that equality constraints
  a virtual table column require elements of the equality to be non-NULL.  That
  may or may not be the case, depending on the virtual table.  Here is an
  example using the generate_series() table-valued function where it is not:

  <blockquote><verbatim>
  CREATE TABLE t1(x); INSERT INTO t1 VALUES(1),(2);
  CREATE TABLE t2(y,z); INSERT INTO t2 VALUES(2,1);
  SELECT * FROM t1 LEFT JOIN t2 ON x=y JOIN generate_series
   WHERE start=z AND stop=2;
  </verbatim></blockquote>

  Because of the way the generate_series virtual table works, 
  the WHERE clause term start=z
  can be true even if z is NULL.  But the prover does not realize that and assumes
  that z must never be NULL, and hence the LEFT JOIN is allowed to change
  into a JOIN.  An incorrect answer results.

  One could argue that this is a bug in the generate_series virtual table
  implementation, not in the LEFT JOIN strength reduction prover.  Nevertheless,
  we think this kind of problem is common with virtual tables, so it seems
  good to enhance the prover to assume that constraints against a virtual
  table might be true even if elements of the expression are NULL.

  This problem was discovered by E.Pasma and reported on the SQLite users
  mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Further investigation confirms that the problem is really in the
  generate_series virtual table.  If the series.c file is compiled with
  the -DSQLITE_SERIES_CONSTRAINT_VERIFY=1 flag, then the generate_series
  virtual table does the right thing, and the correct answer is obtained.

  Meanwhile, Peter Nichvolodov observes that the test case can be
  simplified using CTEs:

  <blockquote><verbatim>
  WITH
    t1(x) AS (VALUES(1),(2)),
    t2(y,z) AS (VALUES(2,1))
  SELECT x, z, value, 'x' FROM t1 LEFT JOIN t2 ON x=y JOIN generate_series
   WHERE start=z AND stop=2;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  I am increasingly viewing this problem as a long-standing bug in the
  generate_series implementation that was merely exposed by the LEFT JOIN
  strength reduction optimization (LJSRO), and not a bug in the LJSRO.
  For that reason, the severity of this ticket is downgraded to "Important".


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the virtual machine references an uninitialized
  register.  This is harmless, but it still needs to be fixed.

  <blockquote><verbatim>
  CREATE TABLE t1(b, c, PRIMARY KEY(b,c)) WITHOUT ROWID;
  CREATE UNIQUE INDEX t1b ON t1(b);
  UPDATE t1 SET b=1 WHERE b='';
  </verbatim></blockquote>

  This problem has existed in the code since support for WITHOUT ROWID
  tables was added with SQLite 3.8.2 (2013-12-06).  The problem was
  discovered by OSSFuzz.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The xBestIndex method of the FTS5 virtual table misbehaves in the following
  input:

  <blockquote><verbatim>
  CREATE VIRTUAL TABLE ft USING fts5(x);
  SELECT * FROM ft WHERE ft.rowid IS NULL;
  </verbatim></blockquote>

  The above results in an error:  "ft.xBestIndex malfunction".  In other
  queries, this malfunction might result in memory leaks or even
  NULL pointer dereferences.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The new LEFT JOIN strength reduction optimization is sometimes applied in
  cases where it should not be, resulting in an error.  The following
  query demonstrates:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  CREATE TABLE t2(x INTEGER PRIMARY KEY, y);
  INSERT INTO t1(a,b) VALUES(1,null);

  SELECT a FROM t1 LEFT JOIN t2 ON (b=x) 
   WHERE NOT ( b IS NOT NULL AND y='xyz' );
  </verbatim></blockquote>

  The problem was caused by the NOT operator, which the prover did not
  recognize as inverting the logic of its operand, effectively converting
  the AND into an OR.

  The problem was discovered by Raphael Michel and reported on the sqlite-users
  mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query below gets a different result with and without the index.  The
  result without the index is correct.

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(1);
  CREATE TABLE t2(y, z);
  SELECT ifnull(z, '!!!') FROM t1 LEFT JOIN t2 ON (x=y);
  CREATE INDEX t2i ON t2(y, ifnull(z, '!!!'));
  SELECT ifnull(z, '!!!') FROM t1 LEFT JOIN t2 ON (x=y);
  </verbatim></blockquote>

  This problem appears to have been introduced by check-in
  [a47efb7c8520a01110ce3] on 2017-04-11, and first released with
  version 3.19.0.  The problem was discovered during internal testing
  and has not been observed in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL demonstrates how an index on an expression that can be
  non-NULL even if all its arguments are NULL can lead to an incorrect answer
  in a LEFT JOIN.

  <blockquote><verbatim>
  CREATE TABLE t1(x);
  CREATE TABLE t2(y, z);
  INSERT INTO t1 VALUES('key');
  INSERT INTO t2 VALUES('key', -1);
  SELECT count(*) FROM t1 LEFT JOIN t2 ON (x=y) WHERE ifnull(z, 0) >=0;
  CREATE INDEX t2i ON t2(y, ifnull(z, 0));
  SELECT count(*) FROM t1 LEFT JOIN t2 ON (x=y) WHERE ifnull(z, 0) >=0;
  </verbatim></blockquote>

  This problem has been in the code ever since support for indexes on
  expression was added by check-in [2131a5ca53f0e9b0b98a9dd9a20e495]
  on 2015-09-04.  The problem first appeared in the 3.9.0 release.

  This problem was discovered during internal testing and analysis
  and has not been observed in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL code causes a VDBE fault on the final UPDATE statement,
  caused by the use of a register that was previously initialized by SCopy from
  another register that subsequently changed.  This is completely harmless in
  the example shown, but could cause problems if the value passed was a string or
  blob rather than an integer.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b int, c DEFAULT 0)WITHOUT ROWID;
  CREATE TABLE record(x TEXT, y TEXT);
  CREATE TRIGGER r3 BEFORE UPDATE ON t1 BEGIN
    INSERT INTO record(x,y)
        VALUES('before-update',printf('%d,%d,%d/%d,%d,%d',
                      old.a,old.b,old.c,new.a,new.b,new.c));
  END;
  INSERT INTO t1(a,b) VALUES(1,2);
  UPDATE t1 SET c=a WHERE a=1;
  </verbatim></blockquote>

  The problem manifests as an assertion fault here:
  [/artifact/066a4e1de2ed83e2?ln=2813].

  Bisecting shows this problem began with the optimization at check-in
  [871e091df651b2275a672c] on 2015-10-15 which was first released in
  version 3.10.0 on 2016-01-06.  The problem was detected during internal
  testing (specifically during AFL testing of the new UPSERT capability
  to be added for 3.24.0) and has never been seen in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Simplified test case:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT PRIMARY KEY, b, c)WITHOUT ROWID;
  INSERT INTO t1(a,b,c) VALUES(1,2,3);
  CREATE TRIGGER r3 BEFORE UPDATE ON t1 BEGIN SELECT new.b; END;
  UPDATE t1 SET c=b WHERE a=1;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The ".fullschema" command in the CLI incorrectly formats the STAT1 and STAT4
  data.  Bisecting shows that this problem originated in check-in
  [72e8f529ac8a268611f16456]


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When the entry in the sqlite_master table that describes the
  sqlite_sequence is corrupted in various ways, subsequent writes to an
  autoincrement table result in an assertion fault or crash.  Where are
  two ways to reproduce the problem from the CLI:

  <blockquote><verbatim>
  .open -new test.db
  CREATE TABLE fake_sequence(name TEXT PRIMARY KEY,seq) WITHOUT ROWID;
  PRAGMA writable_schema=on;
  UPDATE sqlite_master SET
    sql=replace(sql,'fake_','sqlite_'),
    name='sqlite_sequence',
    tbl_name='sqlite_sequence'
   WHERE name='fake_sequence';
  .open test.db
  CREATE TABLE t1(a INTEGER PRIMARY KEY AUTOINCREMENT, b TEXT);
  INSERT INTO t1(b) VALUES('one');
  </verbatim></blockquote>

  <blockquote><verbatim>
  .open -new test.db
  CREATE TABLE t1(a INTEGER PRIMARY KEY AUTOINCREMENT, b TEXT);
  INSERT INTO t1(b) VALUES('one');
  PRAGMA writable_schema=on;
  UPDATE sqlite_master SET
    sql=replace(sql,'sqlite_','x_'),
    name='x_sequence',
    tbl_name='x_sequence'
   WHERE name='sqlite_sequence';
  .open test.db
  INSERT INTO t1(b) VALUES('two');
  </verbatim></blockquote>

  There are probably many other reproduction scripts.

  The problem has existed ever since autoincrement tables were first
  added to SQLite version 3 with version 3.1.0 on 2005-01-21.  This problem
  was reported on the SQLite users mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script causes a segfault in the CLI:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  INSERT INTO t1 VALUES(1,2),(99,100),(4,3);
  .eqp full
  .stats on
  SELECT * FROM t1;
  </verbatim></blockquote>

  This problem appears to be a case of using a prepared statement
  after it has been finalized.  The problem was introduced by
  check-in [ae34edb7f4fa7b37] on 2018-03-20 and first appeared in
  the version 3.23.0 release.

  To be clear:  This is a problem with the command-line shell auxiliary
  program that uses SQLite.  It is not a defect in SQLite itself.

  This problem was discovered during internal testing and has not been
  reported in the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes an assertion fault in SQLite 3.19.0 and later:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY,b UNIQUE);
  CREATE INDEX x1 ON t1(-b=b);
  INSERT INTO t1 VALUES(1,5),(6,2);
  REPLACE INTO t1 SELECT b,0 FROM t1;
  </verbatim></blockquote>

  The assertion faults begin with check-in [e6e36b288fdf2].  However, further
  analysis suggests that that check-in merely exposed the problem and is not
  the actual cause of the problem.  The underlying problem appears to be that
  the column-cache is attempting to reuse the value of a register that was
  never initialized to that column in the first place.

  This fault was found by OSSFuzz.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  An UPSERT discovered by OSSFuzz results in a corrupt database file:

  <blockquote><verbatim>
  CREATE TABLE t1(
    a INT,
    b INT,
    c INT UNIQUE,
    PRIMARY KEY(a,b)
  ) WITHOUT ROWID;
  INSERT INTO t1(a,b) VALUES(json(3),4),(1,2) ON CONFLICT(a,b) DO NOTHING;
  PRAGMA integrity_check;
  </verbatim></blockquote>

  The corruption is a missing entry from the unique index and is easily
  fixed using REINDEX.  Nevertheless, this should not be happening.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is a simplified script to demonstrate the problem:

  <blockquote><verbatim>
  CREATE TABLE t1(b UNIQUE, a INT PRIMARY KEY) WITHOUT ROWID;
  INSERT INTO t1(a) VALUES('1'),(1) ON CONFLICT(a) DO NOTHING;
  PRAGMA integrity_check;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was that the affinity change from '1' to 1 on the PRIMARY KEY
  column was happening on the first constraint check in bytecode order, which
  is the UNIQUE constraint.  But the PRIMARY KEY constraint was checked first,
  and hence was checked before the affinity change.  The fix was to move the
  affinity change to before the branch that reorders the constraint checks
  for the UPSERT.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If a query that uses a correlated subquery is running, and midway through that
  query a CREATE INDEX statement is run, then the next time the correlated
  subquery needs to run within the main query, 
  the main query aborts with an "abort due to ROLLBACK"
  error.  There are two problems with this.  (1) It does not need to abort,
  since the query could continue running.  Certainly it ought to reprepare
  before it runs again, to possibly take advantage of the new index, but for
  the current cycle it should just keep on going.  (2) The error message
  (and the SQLITE_ABORT_ROLLBACK error code) do not accurately describe what
  is going wrong.

  The following TCL script demonstrates the problem:

  <blockquote><verbatim>
  sqlite3 db :memory:
  db eval {
    CREATE TABLE t1(a INTEGER PRIMARY KEY);
    CREATE TABLE t2(d INTEGER PRIMARY KEY,e,f);
    INSERT INTO t1(a) VALUES(1),(2),(3),(4);
  }
  db eval {SELECT a FROM t1 WHERE NOT EXISTS(SELECT 1 FROM t2 WHERE d=a)} {
    if {$a==3} {
      puts "CREATE INDEX"
      db eval {CREATE INDEX t2e ON t2(e);}
    }
    puts a=$a
  }
  </verbatim></blockquote>

  In the script above the output shows:

  <blockquote><verbatim>
  a=1
  a=2
  CREATE INDEX
  a=3
  </verbatim></blockquote>

  Then the "abort due to ROLLBACK" error occurs.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The upsert in the following SQL raises an error in SQLite, as it is designed
  to do and as it is documented.  However, in PostgreSQL, the upsert works, and
  leaves behind a single row with values 1,2,33,4.  The behavior of SQLite should
  be modified so that it gets the same answer as PostgreSQL.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b INT, c INT, d INT);
  CREATE UNIQUE INDEX t1b ON t1(b);
  INSERT INTO t1(a,b,c,d) VALUES(1,2,3,4);
  INSERT INTO t1(a,b,c,d) VALUES(1,2,33,44)
      ON CONFLICT(b) DO UPDATE SET c=excluded.c;
  SELECT * FROM t1;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following also works on PostgreSQL but raises a constraint error on
  SQLite:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT, b INT, c INT, d INT);
  CREATE UNIQUE INDEX t1a ON t1(a);
  CREATE UNIQUE INDEX t1b ON t1(b);
  INSERT INTO t1(a,b,c,d) VALUES(1,2,3,4);
  INSERT INTO t1(a,b,c,d) VALUES(1,2,33,44)
    ON CONFLICT(b) DO UPDATE SET c=excluded.c;
  SELECT * FROM t1;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A confluence of multiple defects in the code generator associated with the
  ORDER BY LIMIT optimization causes the prepared statement to enter an infinite
  loop while running the final query in the following SQL:

  <blockquote><verbatim>
  CREATE TABLE t1(aa VARCHAR PRIMARY KEY NOT NULL,bb,cc,x VARCHAR(400));
  INSERT INTO t1(aa,bb,cc) VALUES('maroon','meal','lecture');
  INSERT INTO t1(aa,bb,cc) VALUES('reality','meal','catsear');
  CREATE TABLE t2(aa VARCHAR PRIMARY KEY, dd INT DEFAULT 1, ee, x VARCHAR(100));
  INSERT INTO t2(aa,dd,ee) VALUES('maroon',0,'travel'),('reality',0,'hour');
  CREATE INDEX t2x1 ON t2(dd,ee);
  ANALYZE;
  DROP TABLE IF EXISTS sqlite_stat4;
  DELETE FROM sqlite_stat1;
  INSERT INTO sqlite_stat1 VALUES
    ('t2','t2x1','3 3 3'),
    ('t2','sqlite_autoindex_t2_1','3 1'),
    ('t1','sqlite_autoindex_t1_1','2 1');
  ANALYZE sqlite_master;
  PRAGMA vdbe_trace=on;
  SELECT *
    FROM t1 LEFT JOIN t2 ON t1.aa=t2.aa
   WHERE t1.bb='meal'
   ORDER BY t2.dd DESC
   LIMIT 1;
  </verbatim></blockquote>

  The infinite loop is only entered if the command-line shell is compiled
  without -DSQLITE_DEBUG or without -DSQLITE_ENABLE_MODULE_COMMENTS.  If both
  of those compile-time options are enabled, then another bug in the code
  generator causes the ORDER BY LIMIT optimization to be disabled, and so the
  infinite loop does not occur.

  This problem was originally reported on the SQLite users mailing list.
  It took considerable effort to reproduce the problem and then boil it down
  to the repro script shown above.

  The script above first began failing with check-in [02ba8a7ba7ba71cd7abd5d]
  which added the omit-left-join optimization.  That optimization does not
  seem to be the root cause, but merely a changed that allowed
  it to be expressed with the current script.  The ORDER BY LIMIT optimization
  that probably is the root cause was introduced in version 3.14 (2016-08-08)
  by check-in [559733b09e9630fac9d].


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Different manifestation of one of the same problems causes SQLite to return incorrect data:

  <verbatim>
  CREATE TABLE t1(aa, bb);
  INSERT INTO t1 VALUES('maroon','meal');

  CREATE TABLE t2(cc, dd, ee, x VARCHAR(100));
  INSERT INTO t2(cc,dd,ee) VALUES('maroon',1,'one');
  INSERT INTO t2(cc,dd,ee) VALUES('maroon',2,'two');
  INSERT INTO t2(cc,dd,ee) VALUES('maroon',0,'zero');

  CREATE INDEX t2ddee ON t2(dd,ee);
  CREATE INDEX t2cc ON t2(cc);

  ANALYZE;

  ---------- returns "maroon|0|zero" (correct)
  SELECT t2.cc, t2.dd, t2.ee FROM t1 CROSS JOIN t2 ON t1.aa=t2.cc
  ORDER BY t2.dd LIMIT 1;

  ---------- returns "maroon|1|one" (incorrect)
  SELECT t2.cc, t2.dd, t2.ee FROM t1 CROSS JOIN t2 ON t1.aa=t2.cc
  WHERE t1.bb='meal'
  ORDER BY t2.dd LIMIT 1;
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query returns no rows where it should return one row:

  <blockquote><verbatim>
  CREATE TABLE t1(x INT COLLATE NOCASE UNIQUE);
  INSERT INTO t1 VALUES('/abc');
  SELECT x FROM t1 WHERE x LIKE '/%';
  </verbatim></blockquote>

  The problem is that a numeric column is used with a LIKE operator and the LIKE
  pattern begins with the character '/'.  The '/' character is one less than '0'
  (in ASCII) and so the LIKE optimization converts the expression into:

      ... x>='/' AND x<'0'

  But because column x is numeric, the x<'0' is convert into x<0 and there are
  not values that are both greater than '/' and less then zero.

  Of course, a reasonable schema should not do a LIKE string comparison on a
  numeric column.  This problem was reported (via private) email against a
  schema that uses the type "STRING" for the column x, which is actually
  a numeric type in SQLite.  (The fact that STRING is a numeric type is a
  separate issue that cannot be fixed without breaking backwards compatibility.)


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A similar problem can arise if the first character of the pattern is
  the ESCAPE character.  The following SQL returns an empty set but it should
  return one row.

  <blockquote><verbatim>
  CREATE TABLE t1(x INT COLLATE NOCASE UNIQUE);
  INSERT INTO t1 VALUES('123');
  SELECT x FROM t1 WHERE x LIKE '^1%' ESCAPE '^';
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The newly enhanced ALTER TABLE in SQLite 3.25.0 hits an assertion fault when
  there are operations on an rtree virtual table in the body of the trigger.

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
  CREATE VIRTUAL TABLE t2 USING rtree(id,x0,x1);
  CREATE TRIGGER r1 AFTER UPDATE ON t1 BEGIN
     DELETE FROM t2 WHERE id = OLD.a;
  END;
  ALTER TABLE t1 RENAME TO t3;
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Further information:

  The assert() seems harmless.  But after removing it the ALTER TABLE statement
  returns an error.

  In version 3.25.0, the ALTER TABLE statement has been enhanced to run a
  test-parse of the whole schema after it alters the schema, to make sure that
  the edits it performed on the schema didn't break anything.  There is a bug
  in this test-parse logic that gives a false positive when a virtual table is
  used inside the body of a trigger.

  So, the ALTER TABLE statement seemed to work correctly, but the extra sanity
  checking logic that runs at the end found a false-positive which causes the
  ALTER TABLE to rollback.  This makes ALTER TABLE unusable for schemas that
  have triggers in which virtual tables are used in the body of the trigger.
  Because the failed ALTER TABLE is run inside a nested transaction which
  rollbacks after the failure, the failure does not cause corruption or otherwise
  harm the database file in any way.  The ALTER TABLE simply fails to work.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following query hits an assert() in the byte-code engine, or if assert()s
  are disabled, goes into an endless loop in the byte code.

  <blockquote><verbatim>
  CREATE TABLE t1(id, b, c);
  INSERT INTO t1 VALUES(1, 'A', 'one');
  INSERT INTO t1 VALUES(2, 'B', 'two');
  INSERT INTO t1 VALUES(3, 'C', 'three');
  INSERT INTO t1 VALUES(4, 'D', 'one');
  INSERT INTO t1 VALUES(5, 'E', 'two');
  SELECT id, b, lead(c,1) OVER(ORDER BY c) AS x FROM t1 WHERE id>1 ORDER BY b LIMIT 1;
  </verbatim></blockquote>

  This problem appears to have been introduced by check-in
  [206720129ed2fa8875a2] which was a fix for ticket
  [9936b2fa443fec03ff25] which was the previous problem of an
  endless loop due to the ORDER BY LIMIT optimization.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Calling sqlite3_declare_vtab() with schema that includes both
  INTEGER PRIMARY KEY and WITHOUT ROWID causes an assertion fault.

  The problem does not arise if the virtual table is INT PRIMARY KEY
  or if the WITHOUT ROWID is omitted.  Because of these simple work-arounds
  and because the problem normally requires C-level API calls, it is not
  considered serious.

  This problem was introduced by check-in [c601d128ff1f1dd6d221ee9f55595a95b]
  on 2017-08-10, which is the check-in that added the ability for WITHOUT
  ROWID virtual tables to be writable.  The first release containing this
  problem was version 3.21.0 on 2017-10-24.

  This problem was originally reported by an identified post on the
  sqlite-users mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL results in an assertion fault:

  <blockquote><verbatim>
  WITH RECURSIVE
    t(id, parent) AS (VALUES(1,2)),
    q(id, parent, rn) AS (
      VALUES(1,2,3)
      UNION ALL
      SELECT t.*, ROW_NUMBER() OVER (ORDER BY t.id) AS rn
        FROM q JOIN t ON t.parent = q.id
    )
  SELECT * FROM q;
  </verbatim></blockquote>

  As the SQL makes use of window functions, this defect only affects SQLite
  version 3.25 and its patches.  The problem was reported on the sqlite-users
  mailing list by Yaroslav Schekin.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When the typeof() or length() functions are used on a column of a
  virtual table that makes use of the sqlite3_vtab_nochange() optimization,
  those functions always return NULL rather than the appropriate value.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The ALTER TABLE statement on the last line of the following script
  fails:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c);
  CREATE VIEW v1 AS SELECT c,b,a FROM t1;
  CREATE TRIGGER v1r1 INSTEAD OF INSERT ON v1 WHEN new.a!=0 BEGIN
    INSERT INTO t1(a,b,c) VALUES(new.a,new.b,new.c);
  END;
  ALTER TABLE t1 RENAME a TO aa;
  </verbatim></blockquote>

  The ALTER TABLE returns SQLITE_ERROR and leave a message:
  "error in trigger v1r1 after rename: no such column: new.a".
  The database is unchanged, so no lasting harm is done, but
  there is no way to rename the "t1.a" column.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The column names on the view are not defined by the schema.  So this schema
  is incorrect.  SQLite is free to choose any names it wants for the columns of
  the view.

  To make this schema work, rewrite the view with explicit column names
  (shown in bold):

  <blockquote><pre>
  CREATE VIEW v1<b>(c,b,a)</b> AS SELECT c,b,a FROM t1;
  </pre></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following script, the second query is 8 times faster than the
  first query:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
  WITH RECURSIVE c(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM c WHERE x<5000)
    INSERT INTO t1(a,b,c) SELECT x, random()%5000, random()%5000 FROM c;
  CREATE TABLE t2(d,e,f);
  WITH RECURSIVE c(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM c WHERE x<50000)
    INSERT INTO t2(d,e,f) SELECT
       NULLIF(0, random()%2), random()%5000, random()%5000
       FROM c;
  ANALYZE;
  SELECT * FROM sqlite_stat1;

  .timer on
  .output /dev/null
  SELECT a FROM t1 JOIN t2
   WHERE a IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
     AND a=CASE WHEN d IS NOT NULL THEN e ELSE f END
   ORDER BY a;

  SELECT a FROM t1 JOIN t2
   WHERE a IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
     AND a=CASE WHEN d IS NOT NULL THEN e ELSE f END
   ORDER BY +a;
  </verbatim></blockquote>

  The only difference in the last two queries is that the second query forces
  a separate sorting pass on the output by putting a "+" in front of the
  sort column.  For the first query, the query planner arranges the loops
  so that the output naturally emerges in sorted order:

  <blockquote><verbatim>
  QUERY PLAN
  |--SEARCH TABLE t1 USING INTEGER PRIMARY KEY (rowid=?)
  `--SCAN TABLE t2
  </verbatim></blockquote>

  For the second query, since a separate sort is required, the full scan of table
  t2 goes first:

  <blockquote><verbatim>
  QUERY PLAN
  |--SCAN TABLE t2
  |--SEARCH TABLE t1 USING INTEGER PRIMARY KEY (rowid=?)
  `--USE TEMP B-TREE FOR ORDER BY
  </verbatim></blockquote>

  Thus, for the first query, the t2 table must be scanned multiple times,
  once for each entry in the IN clause, whereas the t2 table is scanned only once.
  Since t2 is the larger of the two tables, you would think that scanning t2
  once would be the preferred algorithm.  But somehow, the query planner
  feels that sorting is more expensive than scanning the t2 table multiple
  times.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The ALTER TABLE statement in the following script returns an error:

  <blockquote><verbatim>
  CREATE TABLE t1(a);
  CREATE TABLE t2(b);
  CREATE TRIGGER r1 AFTER INSERT ON t2 BEGIN
    INSERT INTO t1(a) VALUES(99);
  END;
  DROP TABLE t1;
  ALTER TABLE t2 RENAME TO t3;
  </verbatim></blockquote>

  The DROP TABLE removes the table t1, which leaves a dangling reference to t1
  in the trigger.  Then during the ALTER TABLE, the trigger is reparsed, but the
  reparse fails due to the dangling reference.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the following SQL, the query with the duplicate "user.active=1" constraint
  returns an incorrect result.  The one with a single "user.active=1" constraint
  works correctly.

  <blockquote><verbatim>CREATE TABLE user ( id INTEGER NOT NULL, active INTEGER NOT NULL, account INTEGER, PRIMARY KEY(id) );
  CREATE TABLE account ( id INTEGER NOT NULL, name TEXT, PRIMARY KEY(id) );
  INSERT INTO user VALUES (1, 0, 1);
  INSERT INTO account VALUES (1, 'test');
  SELECT user.id FROM user INNER JOIN account ON user.account = account.id WHERE user.active = 1 AND user.active = 1;
  SELECT user.id FROM user INNER JOIN account ON user.account = account.id WHERE user.active = 1;
  </verbatim></blockquote>

  Bisecting shows that the problem was introduced by the
  WHERE-clause constraint propagation optimization added by
  check-in [f4229707ac08d66c5b0f5348] on 2018-07-27.

  This problem was reported on the mailing list by Sebastian Zwack.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL results in an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,PRIMARY KEY(b,b));
  SELECT * FROM t1 WHERE (a,b) IN (VALUES(1,2));
  </verbatim></blockquote>

  The problem seems to be associated with the duplicate columns
  in the PRIMARY KEY.  Detailed analysis of the problem is pending.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When the SQLITE_ENABLE_STAT4 compile-time option is used, the SELECT statement
  at the end of the following SQL returns an incorrect answer.  The correct
  answer is 5, but the SELECT statement returns 1:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT);
  CREATE INDEX t1a ON t1(a);
  INSERT INTO t1(a) VALUES(NULL),(NULL),(42),(NULL),(NULL);
  CREATE TABLE t2(dummy INT);
  SELECT count(*) FROM t1 LEFT JOIN t2 ON a IS NOT NULL;
  </verbatim></blockquote>

  This problem was originally reported against System.Data.SQLite (SDS).
  [https://system.data.sqlite.org/index.html/info/03cb660e3b18105a].
  The test case above is a simplification of the test case provided on that
  earlier ticket.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem first appeared in SQLite version 3.7.6 (2011-04-12)
  and is associated with
  check-in [499edcbc8ab70fcf] that added the STAT2 changes to the query planner.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL gives an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE artists (
    id integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    name varchar(255)
  );
  CREATE TABLE albums (
    id integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    name varchar(255),
    artist_id integer REFERENCES artists
  );
  INSERT INTO artists (name) VALUES ('Ar');
  INSERT INTO albums (name, artist_id) VALUES ('Al', 1);
  SELECT artists.*
  FROM artists
  INNER JOIN artists AS 'b' ON (b.id = artists.id)
  WHERE (artists.id IN (
    SELECT albums.artist_id
    FROM albums
    WHERE ((name = 'Al')
      AND (albums.artist_id IS NOT NULL)
      AND (albums.id IN (
        SELECT id
        FROM (
          SELECT albums.id,
                 row_number() OVER (
                   PARTITION BY albums.artist_id
                   ORDER BY name
                 ) AS 'x'
          FROM albums
          WHERE (name = 'Al')
        ) AS 't1'
        WHERE (x = 1)
      ))
      AND (albums.id IN (1, 2)))
  ));
  </verbatim></blockquote>

  This problem was reported on the mailing list by Jeremy Evans.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This appears to have nothing to do with window functions.  The use of window
  functions in the example above merely disables the query flattener.  The
  problem can also be seen in the following query:

  <blockquote><verbatim>
  CREATE TABLE t1 (a);
  CREATE TABLE t2 (b);
  CREATE TABLE t3 (c);
  CREATE TABLE t4 (d);
  INSERT INTO t1 (a) VALUES (104);
  INSERT INTO t2 (b) VALUES (104);
  INSERT INTO t3 (c) VALUES (104);
  INSERT INTO t4 (d) VALUES (104);

  -- Disable the query-flattener
  .testctrl optimizations 0x01

  SELECT *
  FROM t1 CROSS JOIN t2 ON (t1.a = t2.b) WHERE t2.b IN (
    SELECT t3.c
    FROM t3
    WHERE t3.c IN (
      SELECT d FROM (SELECT d FROM t4) AS innermost WHERE innermost.d=104
    )
  );
  </verbatim></blockquote>

  The query above fails in all versions of SQLite from 3.8.11 and later.  The
  check-in that causes the failure is [020b8b106fc8f840f6b506e1c9c3b] on
  2015-05-29


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is another test case that does not require manually disabling the
  query flattener:

  <blockquote><verbatim>
  CREATE TABLE t5(a, b, c, d);
  CREATE INDEX t5a ON t5(a);
  CREATE INDEX t5b ON t5(b);
  CREATE TABLE t6(e);
  INSERT INTO t6 VALUES(1);
  INSERT INTO t5 VALUES(1,1,1,1), (2,2,2,2);
  SELECT * FROM t5 WHERE (a=1 OR b=2) AND c IN (
    SELECT e FROM (SELECT DISTINCT e FROM t6) WHERE e=1
  );
  </verbatim></blockquote>

  The failure in this test case also goes back to check-in [020b8b106fc8f840f6b506e1c9c3bc75676daec3] on 2015-05-29.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is a variant of the second test case that does not require the manual
  disabling of the query flattener:

  <blockquote><verbatim>
  CREATE TABLE t1 (a); INSERT INTO t1 (a) VALUES (104);
  CREATE TABLE t2 (b); INSERT INTO t2 (b) VALUES (104);
  CREATE TABLE t3 (c); INSERT INTO t3 (c) VALUES (104);
  CREATE TABLE t4 (d); INSERT INTO t4 (d) VALUES (104);
  SELECT *
  FROM t1 CROSS JOIN t2 ON (t1.a = t2.b) WHERE t2.b IN (
    SELECT t3.c
    FROM t3
    WHERE t3.c IN (
      SELECT d FROM (SELECT DISTINCT d FROM t4) AS x WHERE x.d=104
    )
  );
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Here is another test case that does not make use of the IN operator.
  The common factor is now the use of co-routines to implement a subquery and
  then evaluating that subquery more than once.

  <blockquote><verbatim>
  CREATE TABLE t1(a1, a2, a3);
  CREATE INDEX t1a2 ON t1(a2, a1);
  CREATE INDEX t1a3 ON t1(a3, a1);
  CREATE TABLE t2(d);
  INSERT INTO t1 VALUES(1, 1, 1), (2, 2, 2);
  INSERT INTO t2 VALUES(22);
  SELECT * FROM t1 WHERE (a2=1 OR a3=2) AND a1 = (
    SELECT d FROM (SELECT DISTINCT d FROM t2) WHERE d=22
  );
  </verbatim></blockquote>

  This new test case demonstrates that checkin [8d663bfaaa4656c4f4ff1] is
  not a solution to the problem.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the latest version of SQLite (3.28.0), I could use the following statement sequence to cause a segfault:

  CREATE TABLE t0 (c0, c1, PRIMARY KEY (c0, c1));
  CREATE TABLE t1 (c0);
  INSERT INTO t1 VALUES (2);
  SELECT * FROM t0, t1 WHERE (t0.c1 >= 1 OR t0.c1 < 1) AND t0.c0 IN (1, t1.c0) ORDER BY 1;


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Note that I sent the test case that triggers the segfault via a personal email to the SQLite developers. The bug was fixed by check-in [778b1224a318d013] shortly after and before I attached the bug report as part of this ticket.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL leads to an assertion fault.

  <blockquote><verbatim>
  SELECT * FROM(
      SELECT * FROM (SELECT 1 AS c) WHERE c IN (
          SELECT (row_number() OVER()) FROM (VALUES (0))
      )
  );
  </verbatim></blockquote>

  This problem has existed since window functions were added in 3.25.0.
  Incorrect bytecode is being generated for the row_number() function.

  This problem was originally reported on the mailing list Gene Sacky.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following script returns no rows before the partial index is created,
  and one row afterwards.  The correct answer is no rows:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  INSERT INTO t1 VALUES(1,1);
  INSERT INTO t1 VALUES(2,2);
  CREATE TABLE t2(x);
  INSERT INTO t2 VALUES(1);
  INSERT INTO t2 VALUES(2);
  SELECT 'one', * FROM t2 WHERE x NOT IN (SELECT a FROM t1);
  CREATE INDEX t1a ON t1(a) WHERE b=1;
  SELECT 'two', * FROM t2 WHERE x NOT IN (SELECT a FROM t1);
  </verbatim></blockquote>

  This problem appears to have been in the code since partial indexes were first
  introduced with version 3.8.0 (2013-08-26).  The problem was first reported on 
  the user mailing list by Deon Brewis.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  WHERE terms of the form:

  <verbatim>
    ... WHERE func1(cols) = ?1 OR func2(cols) = ?2
  </verbatim>

  are being incorrectly transformed to:

  <verbatim>
    ... WHERE func2(cols) IN (?1, ?2)
  </verbatim>

  For example:

  <verbatim>
    CREATE TABLE t1(a, b);
    INSERT INTO t1 VALUES('abc', 'ABC');
    INSERT INTO t1 VALUES('def', 'DEF');

    .print "---- without indexes:"
    SELECT * FROM t1 WHERE upper(a) = 'ABC' OR lower(b) = 'def';

    .print "---- with indexes:"
    CREATE INDEX i1 ON t1( upper(a) );
    CREATE INDEX i2 ON t1( lower(b) );
    SELECT * FROM t1 WHERE upper(a) = 'ABC' OR lower(b) = 'def';
  </verbatim>

  The first query above returns both table rows, but the second only returns the ('def', 'DEF') tuple.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem has existed in all versions of SQLite since support for
  indexes on expressions was first added, with version 3.9.0 on 2015-10-14.
  This problem is a previously undetected
  error in the implementation of indexes on expressions.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  In 3.26.0 and earlier, a foreign key may malfunction if the parent and child are the same WITHOUT ROWID table, and the parent key is not the same as the primary key.

  The problem can occur either when deleting (no constraint failure when there should be one) or inserting (deferred constraint failure when there should not be one) a row.

  In 3.26.0 and earlier, the last SQL statement in the following fails to report the constraint error.

  <verbatim>
    PRAGMA foreign_keys=ON;
    CREATE TABLE t2(
      a PRIMARY KEY, b, c, d, e,
      FOREIGN KEY(b, c) REFERENCES t2(d, e)
    ) WITHOUT ROWID;
    CREATE UNIQUE INDEX idx ON t2(d, e);

    INSERT INTO t2 VALUES(1, 'one', 'one', 'one', 'one'); -- row is parent of self
    INSERT INTO t2 VALUES(2, 'one', 'one', 'one', NULL);  -- parent is row 1

    DELETE FROM t2 WHERE a=1;
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The REPLACE statement is able to put a NULL value into a NOT NULL column.
  If the DEFAULT value for a NOT NULL column is NULL, and if you try to insert a NULL,
  then the inserted NULL is detected and the "REPLACE" algorithm kicks in and
  substitutes the default value.  But if the default is also NULL, the second
  NULL is never detected.  For example:

  <blockquote><verbatim>
  CREATE TABLE t1(x NOT NULL DEFAULT NULL);
  REPLACE INTO t1 DEFAULT VALUES;
  SELECT quote(x) FROM t1;
  </verbatim></blockquote>

  This bug has existed in the code since before SQLite version 3.1.0 (2005-01-21).
  The problem was detected by the Chromium developers using a fuzzer.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Using a window function in a correlated subquery with a "ROWS" window that is not "BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW" (the default) causes an assert to fail. In non-debug builds valgrind also shows problems. Example:

  <verbatim>
      CREATE TABLE t1(x);
      INSERT INTO t1 VALUES('a'), ('b'), ('c');

      CREATE TABLE t2(a, b);
      INSERT INTO t2 VALUES('X', 1), ('X', 2), ('Y', 2), ('Y', 3);

      SELECT x, (
        SELECT sum(b)
          OVER (PARTITION BY a ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
        FROM t2 WHERE b<x
      ) FROM t1;
  </verbatim>

  This problem was discovered and reported to the mailing list by Nicolas Roy-Renaud here:

    http://sqlite.1065341.n5.nabble.com/Bug-in-3-25-2-RECURSIVE-CTE-window-function-tp103958p105246.html


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The ephemeral table used for the partition table was being reopened with each
  iteration of the correlated subquery.  However, there were OP_OpenDup cursors
  on that table that were not being reopened.  The solution was to enhance the
  OP_OpenEphemeral opcode so that on second and subsequent invocations it merely
  deletes any content from the table and reuses it rather than opening a new table.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  OSSFuzz found the following counter-example to the fix described above:

  <blockquote><verbatim>
  SELECT(WITH c AS(VALUES(1))SELECT '' FROM c,c) x WHERE x+x;
  </verbatim></blockquote>

  The query above fails with check-in [4678cb1044f0b4dc8] but works before then.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If a CREATE TABLE statement has redundant UNIQUE constraints, then an
  ALTER TABLE RENAME COLUMN fails.

  <blockquote><verbatim>
  CREATE TABLE t1(aaa, UNIQUE(aaA), UNIQUE(aAa), UNIQUE(aAA), CHECK(Aaa>0));
  ALTER TABLE t1 RENAME aaa TO bbb; --- fails
  </verbatim><blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was detected by a fuzzer run by the Chrome developers.  There have
  been no reports of this problem causing problems in the wild, perhaps because
  few applications have redundant UNIQUE constraints.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When a table has an index on an expression, then inserting a zeroblob into
  an indexed column of that table might cause the zeroblob to be truncated.

  <blockquote><verbatim>
  CREATE TABLE t1(a,b,c);
  CREATE INDEX t1bbc ON t1(b, b+c);
  INSERT INTO t1(a,b,c) VALUES(1,zeroblob(8),3);
  SELECT a, b, length(b), c FROM t1;
  </verbatim></blockquote>

  The correct answer is

  <blockquote><verbatim>
  1,X'0000000000000000',8,3
  </verbatim></blockquote>

  But the answer returned is:

  <blockquote><verbatim>
  1,X'',0,3
  </verbatim></blockquote>

  This issue was discovered during internal testing.  There have been no
  reports of this from the wild.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL returns no rows, but it should return
  one row.  The correct answer (one row) is confirmed by running the same
  SQL against PostgreSQL.

  <blockquote><verbatim>
  CREATE TABLE t1(a INT);
  INSERT INTO t1(a) VALUES(1);
  CREATE TABLE t2(b INT);
  SELECT a, b
    FROM t1 LEFT JOIN t2 ON true
   WHERE (b IS NOT NULL) IS false;
  </verbatim></blockquote>

  Another manifestation of the problem which runs on historical versions
  of SQLite but which does not work on PostgreSQL is this:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT);
  INSERT INTO t1(a) VALUES(1);
  CREATE TABLE t2(b INT);
  SELECT a, b
    FROM t1 LEFT JOIN t2 ON 0
   WHERE (b IS NOT NULL)=0;
  </verbatim></blockquote>

  This problem first appeared with check-in [dd568c27b1d76563] - the
  LEFT JOIN strength reduction optimization - and first appeared in
  release 3.23.0 (2018-04-02).  The problem was reported
  on the SQLite users mailing list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query at the end of the following SQL crashes due to a NULL pointer
  dereference.

  <blockquote><verbatim>
  CREATE TABLE t1(aa, bb);
  CREATE INDEX t1x1 on t1(abs(aa), abs(bb));
  INSERT INTO t1 VALUES(-2,-3),(+2,-3),(-2,+3),(+2,+3);
  SELECT * FROM (t1) 
   WHERE ((abs(aa)=1 AND 1=2) OR abs(aa)=2)
     AND abs(bb)=3;
  </verbatim></blockquote>

  This ticket derives from a separate ticket that was reported against
  System.Data.SQLite:
  [https://system.data.sqlite.org/index.html/info/b0778fcc041fdca4]

  Bisecting shows that the problem was introduced for SQLite version
  3.20.0 (2017-08-01) by check-in [712267c9c08fdcef] (2017-06-23).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The real problem here is in the optimization of check-in [a47efb7c8520a0111]
  that tries to make use of expression values pulled from columns of an
  expression index, rather than recomputing the value of the expression.
  When processing a multi-index OR clause, the transformations on the expressions
  to convert them from their original expressions into a reference to the index
  column become invalid when processing moves on to the next index of a multi-index
  OR.  The solution is to simply disable that optimization for a multi-index OR.

  The optimization at [712267c9c08fdcef] that was found by bisect is blameless
  here, I think.  That optimization merely enabled this particular test case
  to work, and was not the underlying cause of the problem.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The query in the following SQL hits an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE IF NOT EXISTS t1(id INTEGER PRIMARY KEY);
  INSERT INTO t1 VALUES(1);
  SELECT a.id FROM t1 AS a JOIN t1 AS b ON a.id=b.id WHERE a.id IN (1,2,3);
  </verbatim></blockquote>

  The problem was reported on the SQLite users mailing list by
  Ignacio Losiggio.  Ignacio ran a bisect and found that the problem
  was introduced by check-in [e130319317e76119], which means that the
  problem is new for SQLite version 3.27.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The problem was that the same IN operator was being used twice,
  once as IN_INDEX_LOOP to drive a loop and later as IN_INDEX_EPH to
  do a membership test.  The first case uses a table-btree and the second
  case uses an index-btree.  The optimization to reuse the RHS of IN
  operators generated a table-btree for the first case, then tried to reuse
  that table-btree for the second case, which will not work, and hence the
  error.

  The first fix (now on a closed branch) was to disable the IN-operator RHS
  reuse optimization for IN_INDEX_LOOP.  That worked, but it also reduced
  the number of cases where the IN-operator RHS reuse optimization would apply,
  resulting in some slower queries.  The second fix (now on trunk) marks the
  IN operator as TERM_CODED after it is used as an IN_INDEX_LOOP, then preventing
  it from being reused later for a membership test.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Reopened because the previous fix caused a new problem:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY);
  INSERT INTO t1(a) VALUES(1),(2),(3);
  CREATE TABLE t2(x INTEGER PRIMARY KEY, y INT);
  INSERT INTO t2(y) VALUES(2),(3);
  SELECT * FROM t1, t2 WHERE a=y AND y=3;
  </verbatim></blockquote>

  The previous fix caused the final SELECT statement in the code above to
  output a row where y<>3, in clear violation of the WHERE clause.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL statement causes a crash due to attempt to free memory
  that was not obtained from malloc():

  <blockquote><verbatim>
  SELECT +sum(0)OVER() ORDER BY +sum(0)OVER();
  </verbatim></blockquote>

  This problem has existed in all prior versions of SQLite that support
  window functions, beginning with version 3.25.0 (2018-09-15).


login: dan
mimetype: text/x-fossil-wiki
icomment:
  <verbatim>
    CREATE TABLE t1(a, b);
    CREATE TABLE t3(e, f);
    CREATE TRIGGER tr1 AFTER INSERT ON t1 BEGIN
      INSERT INTO t2 VALUES(new.a, new.b);
    END;
    BEGIN;
      ALTER TABLE t3 RENAME TO t4;
    COMMIT;
    SELECT * FROM sqlite_master WHERE type='table' AND name!='t1';
  </verbatim>

  The output of the above script is:

  <verbatim>
    table|t3|t3|3|CREATE TABLE "t4"(e, f) 
  </verbatim>

  The schema is now corrupt - as the table name in the SQL does not match the table names in the "name" and "tbl_name" columns.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following sequence of SQL statements results in an sqlite_master table that
  has one name for the table in the "name" column but a different name in the
  "sql" column:

  <blockquote><verbatim>
  CREATE TABLE "sqlite1234" ("id" integer);
  ALTER TABLE "sqlite1234" RENAME TO "User";
  SELECT name, sql FROM sqlite_master WHERE sql IS NOT NULL;
  </verbatim></blockquote>

  This issue was reported on the mailing list by Semih Hazar.  The problem
  appears to have originated in SQLite version 3.25.0.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following sequence of SQL statements results in a corrupt index:

  <blockquote><verbatim>
  CREATE TABLE test (c1 TEXT PRIMARY KEY) WITHOUT ROWID;
  CREATE INDEX index_0 ON test(c1 COLLATE NOCASE);
  INSERT INTO test(c1) VALUES ('A');
  INSERT INTO test(c1) VALUES ('a');
  PRAGMA integrity_check;
  </verbatim></blockquote>

  This problem goes all the way back to the introduction of WITHOUT ROWID
  tables in version 3.8.2 (2013-12-06).  The problem was reported on the
  mailing list by Manuel Rigger.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The "PRAGMA case_sensitive_like" command changes the behavior of the
  LIKE operator.  But that means that if the LIKE operator is used in the
  schema, for example in a CHECK constraint or in a partial index or an
  index on an expression, then a change in behavior of LIKE can cause
  the database content to disagree with the schema.

  This issue was brought to our attention by a mailing list post
  from Manuel Rigger.

  This problem is unusual in that it is a defect in the design of SQLite,
  not a defect in the implementation.

  It is unclear what can or should be done to resolve this issue.  Possible
  resolutions include, but are not limited to, the following:

    1.  Mark the LIKE operator as non-deterministic so that it cannot
        be used in the schema.  This solution runs into problems of
        what to do with historical schemas that already use the LIKE
        operator.  Note that (due to a separate coding error) the LIKE operator
        already is marked as non-deterministic *after* the case_sensitive_like
        pragma has been run, just not before.

    2.  Disallow "PRAGMA case_sensitive_like=on" if the schema
        contains a LIKE operator.

    3.  Cause LIKE operators in the schema to ignore case (the default
        behavior) regardless of the case_sensitive_like setting.  In
        other words, change case_sensitive_like to only affect LIKE
        operators in transient queries.

    4.  Simply document the fact that "PRAGMA case_sensitive_like" can
        cause a corrupt database.  Perhaps also disable
        case_sensitive_like when SQLITE_DBCONFIG_DEFENSIVE is enabled.

    5.  Automatically run REINDEX on indexes that contain a LIKE
        operator when the case_sensitive_like setting changes.  This
        would only fix problems in indexes, though, not in CHECK
        constraints.

    6.  Deprecate or disable the case_sensitive_like pragma.  Replace it
        with a C-language API of some kind.  This prevents the database
        from being corrupted by an SQL script.  The database schema can
        still get out-of-sync with the database content by C code, but
        that is also true if the C code does something like redefine
        another built-in function using the sqlite3_create_function()
        interface.

    7.  If the schema contains a LIKE operator and the case_sensitive_like
        pragma is turned on, disable all indexes that use LIKE and mark
        the whole database as read-only.  This prevents index corruption,
        but does nothing to fix CHECK constraints that are rendered
        invalid.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  For now, this issue is merely documented.  No change are made to the
  code, except for a new compile-time option to complete disable the
  PRAGMA case_sensitive_like command.  We might revisit this choice later.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The REINDEX statement in the following SQL gives an error indicating that
  the index is not unique:

  <blockquote><verbatim>
  CREATE TABLE t1(a TEXT, b REAL);
  CREATE UNIQUE INDEX t1x1 ON t1(a GLOB b);
  INSERT INTO t1(a,b) VALUES('0','1'),('1','1');
  REINDEX;
  </verbatim></blockquote>

  This problem was reported on the SQLite mailing list by Manuel Rigger.
  This ticket is given a low priority since we not (yet) been able to devise
  a scenario where this quirk could cause problems in a practical,
  real-world database.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the SQL below, the first INSERT statement fails due to a uniqueness
  constraint failure.  But the next two inserts actually work, even though
  they collectively insert the same content.

  <blockquote><verbatim>
  CREATE TABLE test (c0, c1 TEXT);
  CREATE UNIQUE INDEX IF NOT EXISTS index_0 ON test(c1 == FALSE);
  CREATE INDEX IF NOT EXISTS index_1 ON test(c0 || FALSE) WHERE c1;
  INSERT INTO test(c0, c1) VALUES('a',TRUE),('a',FALSE); -- fails
  INSERT INTO test(c0, c1) VALUES('a',TRUE);
  INSERT INTO test(c0, c1) VALUES('a',FALSE); -- works
  </verbatim></blockquote>

  This issue arose from a mailing list report by Manual Rigger.  As the
  anomalous behavior depends on having a boolean expression as a term
  in a UNIQUE constraint, and since that is a unlikely construct to occur
  in practice, this ticket is given a low priority.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The final SELECT statement in the following SQL returns no rows, but it should
  return a single row with a NULL value:

  <blockquote><verbatim>
  CREATE TABLE t0(c0);
  CREATE INDEX index_0 ON t0(c0) WHERE c0 NOT NULL;
  INSERT INTO t0(c0) VALUES (NULL);
  SELECT * FROM t0 WHERE c0 OR 1;
  </verbatim></blockquote>

  Bisecting this error lands on check-in [fe1874321ba31cec].  But I suspect that
  is just where the symptoms first appeared and that the problem is a deeper and
  older error in the theorem prover that (incorrectly) establishes that c0 is
  not null when "c0 OR 1" is true.

  This issue was discovered by Manuel Rigger and reported on the SQLite mailing
  list.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Confirmed: the error was in the theorem prover.  The error goes all the
  way back to when partial index support was first added to SQLite in
  version 3.8.0 (2013-08-26).  The problem is fixed by check-in
  [c2e439bccc40825e211bf] (2019-05-04).


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following script ends up with the table b-tree for table "parent" containing the integer value 123, and the index b-tree for the PRIMARY KEY of "parent" containing text value "123". A corrupt database.

  <verbatim>
  PRAGMA foreign_keys = true;
  CREATE TABLE parent(
    p TEXT PRIMARY KEY
  );
  CREATE TABLE child(
    c INTEGER UNIQUE,
    FOREIGN KEY(c) REFERENCES parent(p) DEFERRABLE INITIALLY DEFERRED
  );

  BEGIN;
    INSERT INTO child VALUES(123);
    INSERT INTO parent VALUES('123');
  COMMIT;
  PRAGMA integrity_check;
  </verbatim>

  The problem is that right before inserting the record into the table b-tree for "parent", the equivalent of "SELECT count(*) FROM child WHERE c='123'" is run to check for foreign key constraint violation resolutions. When integer affinity is applied to the '123', it is also applied to the value that will be inserted into table "parent".


login: dan
mimetype: text/x-fossil-wiki
icomment:
  In the following, the second SELECT statement returns 0 rows:

  <verbatim>
    CREATE TABLE t14(x INTEGER PRIMARY KEY);
    INSERT INTO t14(x) VALUES (100);
    SELECT * FROM t14 WHERE x < 'a' ORDER BY rowid ASC;
    SELECT * FROM t14 WHERE x < 'a' ORDER BY rowid DESC;
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem was originally reported on the SQLite mailing list
  by Manuel Rigger.

  The original problem report used PRAGMA
  reverse_unordered_selects.  However, as the script above shows,
  the problem has nothing to do with reverse_unordered_selects.
  The problem was a defect in the implementation of some of the
  (rarely used) comparison opcodes in the bytecode engine.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following sequence of SQL statements results in a database where the
  primary key index has fewer rows than the table:

  <blockquote><verbatim>
  CREATE TABLE t1 (c0, c1 REAL PRIMARY KEY);
  INSERT INTO t1(c0, c1) VALUES (0, 9223372036854775807), (0, 0);
  UPDATE t1 SET c0 = NULL;
  PRAGMA integrity_check;
  </verbatim></blockquote>

  This problem was discovered by Manuel Rigger and reported on the SQLite
  mailing list. Bisecting shows the problem being introduced by check-in
  [849e686da2d20fd7] which was a fix for ticket [38a97a87a6e4e83].  This
  was on 2015-11-06 and first appeared in the 3.10.0 release of SQLite.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following should return 1 row. But instead returns 0:

  <verbatim>
    CREATE TABLE IF NOT EXISTS t0 (c0);
    CREATE INDEX IF NOT EXISTS i0 ON t0(1) WHERE c0 NOT NULL;
    INSERT INTO t0(c0) VALUES(NULL);
    SELECT * FROM t0 WHERE t0.c0 IS NOT 1; -- returns no row
  </verbatim>

  This issue was discovered by Manuel Rigger and reported on the SQLite mailing list.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The error goes all the way back to when partial index support was first added to SQLite in version 3.8.0 (2013-08-26). Use the following index to demonstrate the problem with versions that do not support indexes on expressions:

  <verbatim>
    CREATE INDEX IF NOT EXISTS i0 ON t0(c0, c1) WHERE c0 NOT NULL;
  </verbatim>


login: dan
mimetype: text/x-fossil-wiki
icomment:
  For example:

  <verbatim>
    CREATE TABLE t0 (c0 INTEGER PRIMARY KEY DESC, c1 UNIQUE DEFAULT NULL) WITHOUT ROWID;
    INSERT INTO t0(c0) VALUES (1), (2), (3), (4), (5);
    REINDEX;
    PRAGMA integrity_check;
  </verbatim>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem arises from an oversight in the implementation of WITHOUT ROWID.
  The original error dates from 2013, and cannot be fixed without breaking
  backwards compatibility.

  A UNIQUE constraint is just a UNIQUE index on the columns named.

  Every index is a btree that is keyed by the columns being indexed followed
  by the primary key for the main table.  (Special rules apply if the primary
  key of the table is one of the columns being indexed.)  Normally, the sort
  order of the primary key column(s) in the index match the sort order of the
  primary key column(s) in the main table.  However, if

    1.  The table is a WITHOUT ROWID table, and
    2.  The PRIMARY KEY of the table uses DESC sort order, and
    3.  The index results from a UNIQUE constraint in the original
        CREATE TABLE statement, not from a separate CREATE INDEX 
        or CREATE UNIQUE INDEX statement,

  Then the primary key columns in the index will use ASC sort order and
  will thus have a different sort order from the main table.  Call
  this situation the "WITHOUT ROWID DESC PRIMARY KEY UNIQUE constraint
  index anomaly".

  I suspect that this anomaly does not occur in the wild, since I do not
  recall ever seeing a real-world schema that included a PRIMARY KEY
  with a DESC sort order.

  Normally, the anomaly is without consequence.  All queries still work
  correctly.

  However, the REINDEX command was trying to take advantage of the fact
  that the keys in the main table are in the same order as the keys in the
  index.  Normally this is true and the REINDEX runs much faster because it
  could avoid some unnecessary seek operations.  But this condition is not true
  for the obscure indexes described above, and will cause the REINDEX to
  compute an incorrect index.

  The fix here is to omit the REINDEX optimization in the corner case where
  the WITHOUT ROWID DESC PRIMARY KEY UNIQUE constraint index anomaly
  occurs.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  It seems that depending on the FROM clause, a different floating-point rounding is performed as demonstrated in the example below. Although programmers should not compare floating-point numbers using equality operators, the result should not depend on the FROM clause either. It is not certain that this can be reproduced on all machines.

  CREATE TABLE t0 (c0); <br />
  CREATE TABLE t1 (c1 REAL); <br />

  INSERT INTO t1(c1) VALUES (8366271098608253588); <br />
  INSERT INTO t0(c0) VALUES ('a'); <br />

  SELECT * FROM t1 WHERE (t1.c1 = CAST(8366271098608253588 AS REAL)); -- fetches row <br />
  SELECT * FROM t0, t1 WHERE (t1.c1 = CAST(8366271098608253588 AS REAL)); -- fetches no row <br />
  SELECT * FROM t0, t1 WHERE (t1.c1 >= CAST(8366271098608253588 AS REAL) AND t1.c1 <= CAST(8366271098608253588 AS REAL)); -- fetches row


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Same problem:

  <verbatim>
    CREATE TABLE t1(a REAL);
    INSERT INTO t1 VALUES( 836627109860825358 );
    SELECT * FROM t1 WHERE a = CAST(836627109860825358 AS REAL); -- returns 1 row

    CREATE INDEX i1 ON t1(a);
    SELECT * FROM t1 WHERE a = CAST(836627109860825358 AS REAL); -- same query now returns 0 rows
  </verbatim>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  PRAGMA journal_mode=OFF;
  PRAGMA main.cache_size=0;
  CREATE TABLE IF NOT EXISTS t0 (c0);
  CREATE INDEX i0 ON t0(1);
  DROP INDEX "i0";
  INSERT OR IGNORE INTO t0(c0) VALUES (1), (2);
  CREATE UNIQUE INDEX i0 ON t0(1); -- UNIQUE constraint failed: index 'i0'
  CREATE UNIQUE INDEX i0 ON t0(1); -- database disk image is malformed
  </pre>

  Is this expected? The documentation mentions that the <i>ROLLBACK</i> statement cannot be used with <i>journal_mode=OFF</i>, but it does not mention failed index creation.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This is the expected behavior when there is no journal available to
  rollback a partially completed operation that fails due to a constraint.
  The documentation has been strengthened to try to make this clear.
  And PRAGMA journal_mode=OFF is now disallowed if
  SQLITE_DBCONFIG_DEFENSIVE is turned on.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the example below:

  <pre>
  CREATE TABLE t0(c1, c2);
  INSERT INTO t0(c1, c2) VALUES  ('a', 1);
  CREATE INDEX i0 ON t0("C3");
  ALTER TABLE t0 RENAME COLUMN c1 TO c3;
  SELECT DISTINCT * FROM t0; -- fetches C3|1 rather than a|1
  </pre>

  It is possible to create an index on a column that does not exist yet. When the column is then created, the index results in the name of the column being returned, instead of a value stored in it.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the example below:

  <pre>
  CREATE TABLE t0(c0);
  INSERT INTO t0(c0) VALUES ('val');
  SELECT * FROM t0 WHERE (
   (
   		(0 IS NOT FALSE) -- 0
  	OR -- 0 
  		NOT (0 IS FALSE OR (t0.c0 IN (-1))) -- 0
  		
   ) -- should be 0 (but is 1)
   IS 0
  ); -- fetches no row
  </pre>

  I expect the query to fetch 'val', but it is not fetched. This is due to an incorrect result being computed by the expression. This can be verified by replacing subexpressions by their results. For example, consider the left side of the OR expression:

  <pre>
  SELECT (0 IS NOT FALSE); -- 0
  </pre>

  When replacing this subexpression above by 0, which should be equivalent, the query fetches the row.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Same problem demonstrated here:

  <verbatim>
  CREATE TABLE t1(x, y);
  INSERT INTO t1 VALUES(1, 1);
  INSERT INTO t1 VALUES(0, 0);
  .print -- before index
  SELECT * FROM t1 WHERE y IS TRUE ORDER BY x;

  .print -- after index
  CREATE INDEX i1 ON t1(x) WHERE y IS NOT TRUE;
  SELECT * FROM t1 WHERE y IS TRUE ORDER BY x;
  </verbatim>

  First query above works, second query returns the wrong row.

  The problem is that the compiler sometimes thinks the expressions "x IS TRUE/FALSE" is the same as "x IS NOT TRUE/FALSE".


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Thanks for the explanation and the fix! I've started assigning a severity level of "cosmetic" to new bug reports, so feel free to adjust it.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem goes back to the introduction of the IS TRUE and IS FALSE
  operators in SQLite version 3.23.0 (2018-04-02).


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  It seems that an integer overflow that occurs when creating an index does not prevent the index from being created, when the <i>CREATE</i> statement is executed as part of an explicit transaction:

  <pre>
  CREATE TABLE IF NOT EXISTS t0(c0);
  INSERT INTO t0(c0) VALUES (-9223372036854775808);
  BEGIN TRANSACTION;
  CREATE INDEX i0 ON t0(ABS(c0)); -- integer overflow (expected)
  COMMIT; -- unexpected: the index is still created
  CREATE INDEX i0 ON t0(1); -- malformed database schema (i0) - index i0 already exists
  </pre>

  That the schema is malformed can also be verified by <i>pragma integrity_check</i>. Without the <i>BEGIN TRANSACTION</i> and <i>COMMIT</i>, the index is not created as expected.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  This bug seems serious, since the index can also not be dropped:

  <pre>
  CREATE TABLE IF NOT EXISTS t0(c0);
  INSERT INTO t0(c0) VALUES (-9223372036854775808);
  BEGIN TRANSACTION;
  CREATE INDEX i0 ON t0(ABS(c0)); -- integer overflow (expected)
  COMMIT; -- unexpected: the index is still created
  DROP INDEX i0; -- no such index: i0
  CREATE INDEX i0 ON t0(1); -- malformed database schema (i0) - index i0 already exists
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  If two views both use a Common Table Expression with the same name,
  and both views are used in a join, and the query flattening optimization
  runs, then the query planner can confuse the names of the two CTEs resulting
  in an incorrect results.  The following SQL demontrates:

  <blockquote><verbatim>
  CREATE TABLE t1(id INTEGER NULL PRIMARY KEY, name Text);
  INSERT INTO t1 VALUES (1, 'john');
  INSERT INTO t1 VALUES (2, 'james');
  INSERT INTO t1 VALUES (3, 'jingle');
  INSERT INTO t1 VALUES (4, 'himer');
  INSERT INTO t1 VALUES (5, 'smith');
  CREATE VIEW v2 AS
    WITH t4(Name) AS (VALUES ('A'), ('B'))
    SELECT Name Name FROM t4;
  CREATE VIEW v3 AS
    WITH t4(Att, Val, Act) AS (VALUES
      ('C', 'D', 'E'),
      ('F', 'G', 'H')
    )
    SELECT D.Id Id, P.Name Protocol, T.Att Att, T.Val Val, T.Act Act
    FROM t1 D
    CROSS JOIN v2 P
    CROSS JOIN t4 T;
  SELECT * FROM v3;
  </verbatim></blockquote>

  The problem bisects to check-in [9e35c89dbe744312] (2017-05-02) which was
  first released with version 3.19.0 (2017-05-22).  This problem was reported
  on the sqlite-users mailing list by Stephen Hunt.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  SELECT CAST('-' AS NUMERIC); -- unexpected: computes 0.0
  </pre>

  I would expect that 0 is computed by the expression, as, for example, in the expressions below:

  <pre>
  SELECT CAST('-0' AS NUMERIC); -- 0
  SELECT CAST('+' AS NUMERIC); -- 0
  SELECT CAST('/' AS NUMERIC); -- 0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the example below:

  <pre>
  SELECT '' - 2851427734582196970; -- actual: -28514277345821967<b>36</b>, expected: -28514277345821969<b>70</b>
  </pre>

  I expect the subtraction to only affect the number's sign, since I expect the empty string to be converted to zero. However, the subtraction also changes the result's last two least significant digits, even though the result fits into a 64 bit integer. This is not the case for smaller numbers, or when subtracting the value from 0:

  <pre>
  SELECT 0 - 2851427734582196970; -- expected: -2851427734582196970
  SELECT '' - 1; -- expected: -1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  In the SQL below, the "LIMIT 1" clause ends up being applied to the
  entire query, rather than to just the one subquery to which it is attached.

  <blockquote><verbatim>
  CREATE TABLE t1 AS VALUES(1),(2);
  CREATE TABLE t2 AS VALUES(3);
  SELECT * FROM (
    SELECT * FROM (SELECT * FROM t1 LIMIT 1)
    UNION ALL
    SELECT * from t2);
  </verbatim></blockquote>


  Bisect shows the error was introduced by check-in [67bfd59d9087a987f] (about 5.3 years
  ago) and first appeared in SQLite version 3.8.4 (2014-03-10).

  This problem was reported on the mailing list by Marco Foit.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  CREATE TABLE t0 (c0 TEXT);
  INSERT INTO t0(c0) VALUES ('1.0');
  SELECT CAST(c0 AS NUMERIC) FROM t0; -- expected: 1, actual: 1.0
  </pre>

  From the following part of the documentation, I would expect that the the value '1.0' is converted to 1:

  <pre>
  Casting a TEXT or BLOB value into NUMERIC first does a forced conversion into REAL but then further converts the result into INTEGER if and only if the conversion from REAL to INTEGER is lossless and reversible.
  </pre>

  I remember that this previously worked as I would expect. Could this be a regression caused by [67a68af5578f08d2]?


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following case:

  <pre>
  CREATE TABLE t0(c0);
  CREATE INDEX i0 ON t0('1' IN ());
  ALTER TABLE t0 RENAME TO t1; -- error in index i0: no such column: 1
  </pre>

  It seems that the empty list on the right side of the IN operator causes '1' to be interpreted as a column name during the ALTER TABLE statement, but not when the index is created, thus resulting in an error when renaming the table.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  CREATE TABLE t0(c0 INT PRIMARY KEY COLLATE NOCASE);
  INSERT INTO t0 VALUES (' 1-');
  SELECT * FROM t0 WHERE t0.c0 LIKE ' 1-'; -- expected: ' 1-', actual: no row is fetched
  </pre>

  Unexpectedly, no row is fetched, even though the WHERE condition should evaluate to 1:

  <pre>
  SELECT t0.c0 LIKE (' 1-') FROM t0; -- 1
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t1 (c0);
  CREATE INDEX i0 ON t1((LIKELIHOOD(c0, 100) IN ())); -- unexpected: no error
  ALTER TABLE t1 RENAME COLUMN c0 TO c1; -- error occurs only here: second argument to likelihood() must be a constant between 0.0 and 1.0
  </pre>

  I would expect that the index cannot be created, since an illegal argument is passed to LIKELIHOOD. However, this is not the case and only when an ALTER TABLE is executed does the illegal parameter result in an error.

  Note that this problem also applies to SELECT queries:

  <pre>
  SELECT * FROM t1 WHERE LIKELIHOOD(c0, 100) IN (); -- unexpected: also no error
  SELECT * FROM t1 WHERE LIKELIHOOD(c0, 100) IS 1 -- expected: second argument to likelihood() must be a constant between 0.0 and 1.0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  I found another, similar bug that was not addressed by the fix:

  <pre>
  CREATE TABLE t0(c0);
  CREATE INDEX i0 ON t0(((LIKELIHOOD(1, 2)) AND ((1 IN ())))); -- unexpected: no error
  ALTER TABLE t0 RENAME TO t1; -- -- error occurs only here: second argument to likelihood() must be a constant between 0.0 and 1.0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following statement:

  <pre>
  SELECT -'.'; -- expected: 0, actual: 0.0
  </pre>

  It seems that the '.' is confused with a floating-point number when used with the unary minus. This is also the case for other arithmetic operators (e.g., see below).

  <pre>
  SELECT '.' + 0; -- expected: 0, actual: 0.0
  </pre>

  This is unexpected, since according to [https://www.sqlite.org/syntax/numeric-literal.html], the '.' should be followed by at least one digit to be a valid number.

  Previously, this worked as expected and still works as expected when casting '.' to NUMERIC:

  <pre>
  SELECT -CAST('.' AS NUMERIC); -- 0
  </pre>

  See [4c2d7639f0], where CAST('-' AS NUMERIC) previously computed 0.0.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following statement:

  <pre>
  SELECT ((CAST(1 as INT)) COLLATE BINARY) == '1'; -- expected: 0, actual: 1
  </pre>

  The expression <i>CAST(1 as INT)</i> has INT affinity. Since <i>COLLATE BINARY</i> is a postfix operator attached to the CAST, I expect that the left side of the comparison has no affinity. However, the comparison yields 1, which suggests that the left side was assigned an INT affinity and the right side converted to INT.

  Note that having a column with the respective affinity has the same effect. The documentation explicitly states that operators on columns result in the expression having no affinity:

  <pre>
  Any operators applied to column names, including the no-op unary "+" operator, convert the column name into an expression which always has no affinity.
  </pre>

  This works as expected for other postfix operators (i.e., they resulting expression has no affinity):

  <pre>
  SELECT ((CAST(1 as INT)) NOTNULL) == '1'; -- 0
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This is a documentation error, fixed by check-in
  [https://www.sqlite.org/docsrc/info/722f0828b3074e89]


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the example below, I expect the conversion of '1.0' to an INT to yield 1, however, it yields 1.0:

  <pre>
  SELECT -'1.0'; -- expected: -1, actual: -1.0
  </pre>

  This is also the case for other floating-point strings whose conversion to INT would be lossless. I expect the conversion to behave as a CAST to NUMERIC, as stated in the documentation:

  <pre>
  All mathematical operators (+, -, *, /, %, <<, >>, &, and |) cast both operands to the NUMERIC storage class prior to being carried out. 
  </pre>


  The conversion works as expected for CASTs:

  <pre>
  SELECT -CAST('1.0' AS NUMERIC); -- -1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Documentation updated to align with the implementation.
  See [https://www.sqlite.org/docsrc/info/e0f700bb6c8142c5].
  Test cases added to TH3.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following statement:

  <pre>
  SELECT (1 IN (CAST('1' as TEXT) COLLATE NOCASE)); -- expected: 0, actual: 1
  </pre>

  I expect the right side to not have an affinity, as stated in the documentation:

  <pre>
  The right-hand operand of an IN or NOT IN operator has no affinity if the operand is a list [...]
  </pre>

  However, it seems that using a COLLATE operator results in an affinity conversion, since the expression above yields 1 rather than 0. Without the COLLATE, no affinity conversion is performed:

  <pre>
  SELECT (1 IN (CAST('1' as TEXT))); -- 0
  </pre>

  If the list has more than one element, also no affinity conversion is performed, as expected:

  <pre>
  SELECT (1 IN (CAST('1' as TEXT) COLLATE NOCASE, 2)); -- 0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following statement:

  <pre>
  SELECT CAST('8.2250617031974513E18' AS NUMERIC); -- expected: 8225061703197451300, unexpected: 8.22506170319745e+18
  </pre>

  I would expect that the floating-point number is parsed/converted to an INTEGER value, since the corresponding value would fit into a 64 bit integer. Instead, the value is stored as an imprecise floating-point number that, when cast back, does not yield the same value:
  <pre>
  SELECT CAST(CAST('8.2250617031974513E18' AS NUMERIC) AS INT); -- 8225061703197451264
  </pre>

  When not using an E notation, the value is converted to an INTEGER as expected:

  <pre>
  SELECT typeof(CAST('8225061703197451300' AS NUMERIC)); -- integer
  </pre>

  The conversion also works as expected for small numbers:

  <pre>
  SELECT CAST('8.225061703197E12' AS NUMERIC); -- 8225061703197
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The CAST to NUMERIC documentation has been clarified at
  [https://www.sqlite.org/docsrc/info/7a51b32537ac7e95] in an attempt to show
  why this is a non-issue.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the statements below:

  <pre>
  SELECT LIKELY(CAST(1 AS INT)) = '1'; -- expected: 0, actual: 1
  SELECT UNLIKELY(CAST(1 AS INT)) = '1'; -- expected: 0, actual: 1
  SELECT LIKELIHOOD(CAST(1 AS INT), 0.5) = '1'; -- expected: 0, actual: 1
  </pre>

  All statements return 1, which is somehow unexpected, since functions are expected to have no affinity; however, it seems that for these functions, the affinity of the function argument is assumed (for LIKELIHOOD, the affinity of the first argument is taken).

  This bug report is similar to [d60b3cd7cb], where it was reported that the COLLATE operator takes the affinity of the expression applied to, and resulted in a documentation update.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  It seems that using IS TRUE together with a COLLATE operator that is part of the right side of the expression results in an incorrect result:

  <pre>
  SELECT 0.5 IS TRUE COLLATE NOCASE; -- expected: 1, actual: 0
  SELECT 0.5 IS TRUE COLLATE RTRIM; -- expected: 1, actual: 0
  SELECT 0.5 IS TRUE COLLATE BINARY; -- expected: 1, actual: 0
  </pre>

  Without the COLLATE, or if the COLLATE is on the left side, the result is as expected:

  <pre>
  SELECT 0.5 IS TRUE; -- 1
  SELECT 0.5 COLLATE NOCASE IS TRUE; -- 1
  </pre>

  The IS FALSE operator does not seem to be affected:
  <pre>
  SELECT 0.5 IS FALSE COLLATE NOCASE; -- 0
  SELECT 0.5 IS FALSE COLLATE RTRIM; -- 0
  SELECT 0.5 IS FALSE COLLATE BINARY; -- 0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the example below, the value is not converted to an INTEGER, although the conversion would be lossless:

  <pre>
  SELECT CAST('-0.0' AS NUMERIC); -- expected: 0, unexpected: 0.0
  </pre>

  In similar situations, the value is converted to an INTEGER as expected:

  <pre>
  SELECT CAST('0.0' AS NUMERIC); -- 0
  SELECT CAST('+0.0' AS NUMERIC); -- 0
  SELECT CAST('-1.0' AS NUMERIC); -- -1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The IEEE754 floating point format distinguishes between +0.0 and -0.0 - those
  are different numbers.  So the statement in the initial description of this
  ticket that "-0.0 can be losslessly converted into 0" is wrong.  If -0.0 is
  converted into 0, then you lose the fact that you have a negative zero.

  Open questions:

    1.  Should the CAST operator convert -0.0 into 0 as a special case, in spite of
        the fact that the conversion losses information?

    2.  Should SQLite be enhanced to display a -0.0 as "-0.0" instead of (as it
        currently does) conflating the -0.0 and +0.0 values into a single display
        of just "0.0"?


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  I think that what "lossless" mean in this context is up to interpretation. In some sense, all conversions are lossful. For example, both '1.0' and '1' are converted to 1 and it is not possible to determine whether the original string was '1.0' or '1'.

  <pre>
  SELECT CAST('1.0' AS NUMERIC); -- 1
  SELECT CAST('1' AS NUMERIC); -- 1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The CAST of '-0.0' into numeric now yields 0 as a special case, which resolves
  this ticket.

  The issue of whether or not to display the minus sign when rendering a -0.0
  value into text is still open.  I observe that other database engines,
  and in particular PostgreSQL, do *not* display the minus sign.  So, for now,
  SQLite will follow PostgreSQL's lead and behave the same.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  CREATE TABLE t0(c0 COLLATE NOCASE);
  INSERT INTO t0(c0) VALUES ('a');
  SELECT * FROM t0 WHERE CAST(t0.c0 AS TEXT) = 'A'; -- expected: no row is fetched, actual: a
  </pre>

  I would expect the comparison to yield 0, since the left operand is 'a', and the right one 'A'. Neither operand has an explicit collating function assigned and only t0.c0 has an implicit one. It seems that the comparison takes the implicit collating function of c0, although the column is surrounded by the cast. The documentation on implicit collating functions, however, only mentions these cases:

  <pre>
  If either operand is a column, then the collating function of that column is used with precedence to the left operand. For the purposes of the previous sentence, a column name preceded by one or more unary "+" operators is still considered a column name.
  </pre>

  This works as expected for other functions and operators, for example:

  <pre>
  CREATE TABLE t0(c0 COLLATE NOCASE);
  INSERT INTO t0(c0) VALUES ('a');
  SELECT * FROM t0 WHERE LOWER(t0.c0) = 'A'; -- no row is fetched
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider this example:

  <pre>
  CREATE TABLE t0(c0 INT UNIQUE COLLATE NOCASE);
  INSERT INTO t0(c0) VALUES ('.1%');
  SELECT * FROM t0 WHERE t0.c0 LIKE '.1%'; -- expected: '.1%', actual: no row is fetched
  </pre>

  Although the LIKE expression should yield 1 (see below), no row is fetched:

  <pre>
  SELECT t0.c0 LIKE '.1%' FROM t0; -- 1
  </pre>

  It seems that this problem only applies to COLLATE NOCASE as well as to numeric columns (INT, REAL, and NUMERIC).


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following statements:

  <pre>
  CREATE TABLE t0(c0 COLLATE RTRIM, c1 BLOB UNIQUE, PRIMARY KEY (c0, c1)) WITHOUT ROWID;
  INSERT INTO t0 VALUES (123, 3), (' ', 1), ('	', 2), ('', 4);
  SELECT * FROM t0 WHERE c1 = 1; -- expected: ' ', 1, actual: no row is fetched
  </pre>

  I would expect a row to be fetched, which is not the case. Note that the whitespace seems to be significant: the first one is a space, the second one a tab, and the third one an empty string.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Alternative test case:

  <blockquote><verbatim>
  SELECT ' '<char(31) COLLATE rtrim;
  </verbatim></blockquote>

  The above should give and answer of true since a zero-length string is less than
  a string of length 1.  But the answer returned is false.

  This problem has existing in the RTRIM collating sequence since it was first
  introduced by check-in [0bf4e7fefdbbf7be] on 2008-01-20 for SQLite version 3.5.5.
  Fixed 11 years later by check-in [86fa0087cd1f5c79] 11 years later.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  CREATE TABLE t0 (c3 TEXT);
  INSERT INTO t0(c3) VALUES ('0');
  SELECT * FROM t0 WHERE (t0.c3 COLLATE NOCASE) BETWEEN 1 AND '5'; -- expected: no row is fetched, actual: row is fetched
  </pre>

  I expect that the BETWEEN operation yields 0, since the first comparison should yield 0 (the affinity conversion should convert 1 to TEXT and '0' >= '1' is 0). However, the row is fetched, which indicates that the expression yields 1. This also applies to the other COLLATE operators.

  The documentation states the following: The expression "x BETWEEN y and z" is logically equivalent to two comparisons "x >= y AND x <= z" and <b>works with respect to collating functions as if it were two separate comparisons</b>.

  Accordingly, the following query should be equivalent to the one above:

  <pre>
  SELECT * FROM t0 WHERE (t0.c3 COLLATE NOCASE >= 1) AND (t0.c3 COLLATE NOCASE <= '5'); -- no row is fetched
  </pre>

  As expected for the first query, this one fetches no rows.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  <verbatim>
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES('ALFKI');
  INSERT INTO t1 VALUES('ANATR');

  CREATE TABLE t2(y, z);
  CREATE INDEX t2y ON t2 (y);
  INSERT INTO t2 VALUES('ANATR', '1997-08-08 00:00:00');
  INSERT INTO t2 VALUES('ALFKI', '1997-08-25 00:00:00');

  SELECT (
    SELECT y FROM t2 WHERE x = y ORDER BY y, z
  )
  FROM t1;
  </verbatim>

  The script above should return two rows - one with 'ANATR' and a second with 'ALFKI'. Instead it returns 'ANATR' and then NULL.

  Problem introduced by commit [4678cb1044f0b].


login: dan
mimetype: text/x-fossil-wiki
icomment:
  For example, this:

  <verbatim>
    CREATE TABLE t1(a BLOB, b INTEGER, c COLLATE nocase);
    INSERT INTO t1 VALUES(1, 2, 'abc');
    INSERT INTO t1 VALUES(3, 4, 'ABC');

    SELECT c=='Abc', 0 FROM t1;
  </verbatim>

  returns two tuples: (1, 0) and (1, 0).

  But this:

  <verbatim>
    SELECT c=='Abc', rank() OVER (ORDER BY b) FROM t1
  </verbatim>

  returns (0, 1) and (0, 2). The default collation sequence of column "c" was lost when the window function was added to the query.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Discovered and reported to the mailing list by Yuriy Garin:

  [https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg115803.html]


login: drh
mimetype: text/x-fossil-wiki
icomment:
  A valgrind warning is generated by the following SQL:

  <blockquote><verbatim>
  CREATE TABLE t1(a INT, PRIMARY KEY(a,a COLLATE NOCASE)) WITHOUT ROWID;
  EXPLAIN INSERT INTO t1(a) VALUES(1);
  </verbatim></blockquote>

  This problem seems to have been introduced by check-in
  [1b1dd4d48cd79a58] which was a fix for ticket [3182d3879020ef3b].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  When a CHECK constraint contains a CASE or BETWEEN operator with a left-hand
  side that has a COLLATE operator, incorrect byte code is generated resulting
  in an assertion fault.  Examples:

  <blockquote><verbatim>
  CREATE TABLE t1(a CHECK(1 COLLATE BINARY BETWEEN 1 AND 1));
  INSERT INTO t1 VALUES('abc');
  </verbatim></blockquote>

  <blockquote><verbatim>
  CREATE TABLE t1(a CHECK(CASE 'abc' COLLATE nocase WHEN a THEN 1 END));
  INSERT INTO t1 VALUES('abc');
  </verbatim></blockquote>

  This problem appears to have arisen from check-in [54110870487f7801]
  (2019-06-16) which was a fix for ticket [e1e07ef2028c3c22].


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The BETWEEN example above requires two inserts before it fails:

  <blockquote><verbatim>
  CREATE TABLE t1(a CHECK(1 COLLATE BINARY BETWEEN 1 AND 1));
  INSERT INTO t1 VALUES('abc');
  INSERT INTO t1 VALUES('abc');
  </verbatim></blockquote>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The dbsqlfuzz fuzzer (as augmented recently with enhancements to the structural
  mutator component) found the following case that causes an assertion fault in
  the query planner:

  <blockquote><verbatim>
  CREATE TABLE t1(c);
  CREATE TABLE t2(x PRIMARY KEY, y);
  SELECT x FROM t2, t1 WHERE x BETWEEN c AND null OR x AND
  x IN ((SELECT x FROM (SELECT x FROM t2, t1 
  WHERE x BETWEEN (SELECT x FROM (SELECT x COLLATE rtrim 
  FROM t2, t1 WHERE x BETWEEN c AND null
  OR x AND x IN (c)), t1 WHERE x BETWEEN c AND null
  OR x AND x IN (c)) AND null
  OR NOT EXISTS(SELECT -4.81 FROM t1, t2 WHERE x BETWEEN c AND null
  OR x AND x IN ((SELECT x FROM (SELECT x FROM t2, t1
  WHERE x BETWEEN (SELECT x FROM (SELECT x BETWEEN c AND null
  OR x AND x IN (c)), t1 WHERE x BETWEEN c AND null
  OR x AND x IN (c)) AND null
  OR x AND x IN (c)), t1 WHERE x BETWEEN c AND null
  OR x AND x IN (c)))) AND x IN (c)
  ), t1 WHERE x BETWEEN c AND null
  OR x AND x IN (c))));
  </verbatim></blockquote>

  The query above has been manually simplified to the following:

  <blockquote><verbatim>
  CREATE TABLE t1(c);
  CREATE TABLE t2(x PRIMARY KEY);
  SELECT 1 FROM t1 WHERE (
    SELECT 2 FROM t2 WHERE (
      SELECT 3 FROM (
        SELECT x FROM t2 WHERE x=c OR x=(SELECT x FROM (VALUES(0)))
      ) WHERE x>c OR x=c
    )
  );
  </verbatim></blockquote>

  Bisecting shows that the problem was introduced by check-in
  [e130319317e76119].  Hence, the problem exists in version 3.27, 3.28, and 3.29.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Oh boo.  All that work to try to simplify the test case and to isolate the
  point of introduction, and I never took the time to actually look at the assert()
  that is failing:

    *  [/artifact/9817d812b13c956a?ln=5868-5873]

  The assert() is benign.  Completely harmless.

  I've changed the severity on this ticket to "minor".
  We'll try to deal with it as we have opportunity.  The solution might be to
  simply delete the assert().


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the test case below:

  <pre>
  CREATE TABLE t0(c0 COLLATE RTRIM);
  INSERT INTO t0(c0) VALUES (' ');
  SELECT DISTINCT(t0.c0) != '' FROM t0; -- expected: 1, actual: 0
  </pre>

  I would expect the SELECT to fetch a row with a value 1, since the DISTINCT(t0.c0) should be equal to ' ', and ' ' != '' should be true. However, the result is 0. I suspect that DISTINCT(t0.c0) assumes the COLLATE RTRIM. This is unexpected, since the documentation mentions that only the plus operator and CAST in combination with a column still assume the column's COLLATE, rather than having no implicit COLLATE:

  If either operand is a column, then the collating function of that column is used with precedence to the left operand. For the purposes of the previous sentence, a column name preceded by one or more unary "+" operators and/or CAST operators is still considered a column name.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The DISTINCT keyword is a modifier on SELECT.  It is not a function that
  is applied to parts of the result set.  The query actually parses out like
  this:

    SELECT DISTINCT ((t0.c0) != '') FROM t0;

  Hence, the RTRIM collating sequence does apply to the LHS of the comparison,
  and the result is 0.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the test case below:

  <pre>
  CREATE TABLE t0 (c0 REAL, c1);
  CREATE UNIQUE INDEX i0 ON t0(c1, 0 | c0);
  INSERT INTO t0(c0) VALUES (4750228396194493326), (0);
  UPDATE OR REPLACE t0 SET c0 = 'a', c1 = '';
  SELECT * FROM t0 ORDER BY t0.c1; -- unexpected: database disk image is malformed
  </pre>

  Unexpectedly, the query results in an error "database disk image is malformed".


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the test case below, only one row is fetched, rather than two:

  <pre>
  CREATE TABLE t0 (c0, c1, c2, PRIMARY KEY (c0, c1));
  CREATE TABLE t1 (c2);
  INSERT INTO t0(c2) VALUES (0), (1), (3), (4), (5), (6), (7), (8), (9), (10), (11);
  INSERT INTO t0(c1) VALUES ('a');
  INSERT INTO t1(c2) VALUES (0);
  ANALYZE;
  SELECT DISTINCT t0.c0, t1._rowid_, t0.c1 FROM t1 CROSS JOIN t0 ON TRUE ORDER BY t0.c0; -- expected: |1|, |1|a, actual: |1|
  </pre>

  When removing the ANALYZE, the query works as expected. It also works when using JOIN or FROM t1, t0 rather than a CROSS JOIN.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the test case below:

  <pre>
  CREATE TABLE t1 (c1 , c2, c3, c4 , PRIMARY KEY (c4, c3));
  INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0), (0), (0), (0), (0), (NULL), (1), (0);
  UPDATE t1 SET c2 = 0;
  INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);
  ANALYZE t1;
  UPDATE t1 SET c3 = 1;
  SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1; -- expected: |0|1|, 0||1|, ||1|, actual: |0|1|
  </pre>

  I would expect the query to fetch the three distinct rows, however, only one is fetched. If the ANALYZE is omitted, then all three distinct rows are fetched.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0 UNIQUE, c1);
  INSERT INTO t0(c0, c1) VALUES (NULL, 1);
  SELECT MIN(t0.c0), t0.c1 FROM t0; -- expected: NULL | 1, actual: NULL | NULL
  </pre>

  Unexpectedly, a fabricated NULL value for the bare column c1 is returned. Removing the UNIQUE constraint on c0 does not trigger this bug. This also works as expected for other aggregate functions, such as MAX().


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the test case below, the result seems to be incorrect:

  <pre>
  CREATE TABLE t0 (c0, c1);
  CREATE INDEX i0 ON t0(c1, c1 + 1 DESC);
  INSERT INTO t0(c0) VALUES (1);
  SELECT MIN(t0.c1), t0.c0 FROM t0 WHERE t0.c1 ISNULL; -- expected: NULL | 1, actual: NULL | NULL
  </pre>

  When omitting the index or the WHERE clause, the query works as expected.

  This bug report is similar to [41866dc373], which has already been addressed by a fix.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  A bit simpler:

  <verbatim>
    CREATE TABLE t1 (a, b);
    INSERT INTO t1 VALUES(123, NULL);
    CREATE INDEX i1 ON t1(a, b DESC);
    SELECT MIN(a) FROM t1 WHERE a=123;
  </verbatim>

  Should return integer value 123, but currently returns NULL.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem is apparently over 11 years old.

  Bisecting shows that this problem was introduced on 2008-07-08 by
  check-in [fa07c360b708324c] and first appeared in release 3.6.0.
  The second script above (Dan's reduction) causes an assertion fault
  starting with the 2008-07-08 check-in if SQLite is compiled with -DSQLITE_DEBUG,
  but it give the incorrect NULL result if asserts are disabled.  The
  assertion fault went away with check-in [778e91ddb834f608] on
  2008-12-21 but the incorrect NULL result has persisted ever since then.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0, c1 TEXT);
  CREATE VIEW v0(c0) AS SELECT SUM(t0.c1) FROM t0;
  INSERT INTO t0(c0, c1) VALUES ('a', 1);
  SELECT * FROM v0, t0 WHERE t0.c1 <= v0.c0; -- unexpected: row is not fetched
  </pre>

  Unexpectedly, the WHERE clause yields false. t0.c1 has TEXT affinity. v0.c0 refers to SUM(t0.c1) and should have no affinity. This can also be verified using the following PRAGMA:

  <pre>
  PRAGMA table_info(v0); -- 0|c0||0||0
  </pre>

  Thus, I would expect that v0.c0 is converted to TEXT. However, in this case, the row should have been fetched:

  <pre>
  SELECT * FROM v0, t0 WHERE t0.c1 <= CAST(v0.c0 AS TEXT); -- 1|a|1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Documentation changed [https://sqlite.org/docsrc/info/07b7749da88d54e5|&#91;1&#93;] to
  reflect the actual behavior of the code for the past 10 years.  Current behavior
  was established in 2009 by check-in
  [33cf83591e6e1387] but the
  documentation was 8 years later in 2017 at 
  [https://sqlite.org/docsrc/timeline?c=524ba450fb959a75|&#91;524ba450fb959a75&#93;].

  Changing the code to match the documentation risk breaking programs that
  are already in the field.  Hence, it is better to change the documentation
  to match the code.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the test case below:

  <pre>
  CREATE TABLE t0(c0 TEXT, c1);
  INSERT INTO t0(c0, c1) VALUES (-1, 0);
  CREATE VIEW v0(c0, c1) AS SELECT t0.c0, AVG(t0.c1) FROM t0;
  SELECT * FROM v0 WHERE v0.c1 < v0.c0; -- unexpected: row is not fetched
  </pre>

  Unexpectedly, the row is not fetched. Interestingly, but as expected, the row is also not fetched when negating the WHERE condition:

  <pre>
  SELECT * FROM v0 WHERE NOT(v0.c1 < v0.c0); -- expected: -1|0.0, actual: row is not fetched
  </pre>

  Evaluating the condition yields TRUE:

  <pre>
  SELECT v0.c1 < v0.c0 FROM v0; -- 1
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the test case below, the EXPLAIN PRAGMA seems to execute the PRAGMA, which is unexpected:

  <pre>
  CREATE TABLE t0 (c0 CHECK(FALSE));
  EXPLAIN PRAGMA ignore_check_constraints = true; 
  INSERT INTO t0(c0) VALUES (1); -- expected: CHECK constraint failed: t0, actual: no error
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  Improved documentation to explain how EXPLAIN affects run-time but that some
  PRAGMA statements operate at prepare-time, and hence some PRAGMA statements
  are unaffected by EXPLAIN.

  [https://www.sqlite.org/docsrc/info/e167521765c0e7c7]


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0 TEXT);
  CREATE VIEW v0(c0) AS SELECT t0.c0 FROM t0;
  INSERT INTO t0(c0) VALUES ('0');
  SELECT 0 IN (c0) FROM v0; -- expected: 0, actual: 1
  </pre>

  The expression 0 in (c0) unexpectedly yields TRUE, although c0 is a string and 0 an integer; according to the documentation, values in the right list should have no affinity, so I would not expect an affinity conversion to be performed. When fetching from the table, rather than the view, the result is as expected:

  <pre>
  SELECT 0 IN (c0) FROM t0; -- 0
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0 UNIQUE, c1);
  INSERT INTO t0(c1) VALUES (0);
  INSERT INTO t0(c0) VALUES (0);
  CREATE VIEW v0(c0, c1) AS SELECT t0.c1, t0.c0 FROM t0 WHERE CAST(t0.rowid AS INT) = 1;
  SELECT v0.c0, MIN(v0.c1) FROM v0; -- expected: 0|, actual: |
  </pre>

  The view holds a row with the values 0|:

  <pre>
  SELECT c0, c1 FROM v0; -- 0|
  </pre>

  Thus, it is unexpected that fetching from it returns |.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  This problem was introduced on 2012-12-13 by check-in [52e755943f87354f]
  and first appeared in release 3.7.16 on 2013-03-18


login: dan
mimetype: text/x-fossil-wiki
icomment:
  Another example:

  <verbatim>
  CREATE TABLE t1(a, b, c);
  INSERT INTO t1 VALUES(NULL, 1, 'x');
  CREATE INDEX i1 ON t1(a);

  SELECT min(a), b, c FROM t1 WHERE c='x';  -- returns NULL|1|x

  INSERT INTO t1 VALUES(1,    2, 'y');

  SELECT min(a), b, c FROM t1 WHERE c='x';  -- returns NULL|NULL|NULL
  </verbatim>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the test case below:

  <pre>
  CREATE TABLE t0(c0);
  INSERT INTO t0(c0) VALUES (0);
  CREATE INDEX i0 ON t0(NULL > c0) WHERE (NULL NOT NULL);
  SELECT * FROM t0 WHERE ((NULL IS FALSE) IS FALSE); -- expected: row is fetched: actual: row is not fetched
  </pre>

  I would expect the row in the table being fetched. However, it is only fetched if the index is not created.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL reports database corruption on the REPLACE:

  <blockquote><verbatim>
  CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT);
  CREATE INDEX i1 ON t1(b);
  CREATE TABLE t2(a, b);
  INSERT INTO t2 VALUES(4, randomblob(31000));
  INSERT INTO t2 VALUES(4, randomblob(32000));
  INSERT INTO t2 VALUES(4, randomblob(33000));
  REPLACE INTO t1 SELECT a, b FROM t2;
  </verbatim></blockquote>

  The problem seems to be caused by a failure to clear the overflow page
  cache.  The problem seems to have been introduced by check-in
  [2312eb6a9eb31051] which first appeared in release 3.8.5 on 2014-06-04.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following example:

  <pre>
  CREATE TABLE t0(c0);
  CREATE TRIGGER tr0 AFTER DELETE ON t0 WHEN c0 BEGIN DELETE FROM t0; END;
  DELETE FROM t0; -- unexpected: no such column: c0
  </pre>

  The error that no column c0 exists is surprising, since it has not been deleted.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  This seems to be working as designed. The "no such column" error refers to the "c0" in the trigger. In this context "old.c0" would work, but there is no column that can be referred to as just "c0".

  The error occurs when the DELETE is prepared, not when the trigger is created, as SQLite does not resolve column or table references when triggers and views are created, only when they are used. This can be surprising.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following script:

  <verbatim>
    CREATE TABLE t0(c0);
    INSERT INTO t0(c0) VALUES (0);
    SELECT * FROM t0 WHERE EXISTS (SELECT MIN(c0)  OVER (), CUME_DIST()
    OVER () FROM t0) BETWEEN 1 AND 1;
  </verbatim>

  causes a null pointer dereference.

  This bug was discovered and reported by Manuel Rigger (user mrigger).


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The following SQL causes a division by zero in the query planner:

  <blockquote><verbatim>
  ANALYZE;
  CREATE TABLE t1(a PRIMARY KEY);
  INSERT INTO sqlite_stat1 VALUES('t1',null,'sz=0');
  ANALYZE sqlite_master;
  SELECT 0 FROM t1 WHERE a IN(1,2,3);
  </verbatim></blockquote>

  The problem is the sz=0 parameter on the sqlite_stat1 entry for the t1 table.
  The value of sz needs to be 2 or greater, and this is not enforced.  The
  problem was introduced by check-in [90e36676476e8db0] on 2014-04-30 and first
  appeared in version 3.8.5.

  This problem was reported on the SQLite mailing list by Xingwei Lin.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0);
  CREATE VIEW v0(c0) AS SELECT TYPEOF(1) FROM t0;
  INSERT INTO t0(c0) VALUES (0), (1);
  SELECT * FROM t0 LEFT JOIN v0 ON t0.c0 WHERE NOT(v0.c0 = 'a'); -- unexpected: fetches no row
  </pre>

  I would expect the last query to be equivalent to the following query:

  <pre>
  SELECT * FROM t0 LEFT JOIN v0 ON t0.c0 WHERE v0.c0 != 'a'; -- fetches two rows
  </pre>

  While the first query does not fetch any rows, the second one fetches two rows.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The expression "typeof(1)" appears to be constant, always returning the
  string 'integer'.  However, because it originates from the right-hand side
  of a LEFT JOIN, it might also sometimes be NULL, and so it is not really
  constant.  The logic that factors out constant expressions (to avoid
  evaluating them multiple times) failed to detect this and factored out
  the "typeof(1)" expression.  The fix is to disable the constant-expression
  factoring optimization when evaluating expressions that originate on the
  right-hand side of a LEFT JOIN.

  Check-in [500c9152daaf11cf]


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0);
  CREATE VIEW v0(c0) AS SELECT LOWER(CAST('1e500' AS TEXT)) FROM t0;
  INSERT INTO t0(c0) VALUES (NULL);
  SELECT v0.c0 FROM v0, t0 WHERE t0.rowid NOT IN (0, 0, v0.c0); -- expected: '1e500', actual: Inf
  </pre>

  I think that the conversion to a floating-point number should not be performed, as with the same query without the WHERE clause:

  <pre>
  SELECT v0.c0 FROM v0, t0; -- '1e500'
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  <pre>
  CREATE TABLE t0 (c0, c1);
  CREATE INDEX i0 ON t0(CAST(c0 AS NUMERIC));
  INSERT INTO t0(c0, c1) VALUES ('a', -1);
  SELECT * FROM t0 INDEXED BY i0 WHERE CAST(t0.c0 AS NUMERIC) > LOWER(t0.c1) GROUP BY t0.rowid; -- expected: row is fetched, actual: no row is fetched
  </pre>

  In the test case above, the INDEXED by causes the row not to be fetched. If it is omitted, the row is fetched, as expected:

  <pre>
  SELECT * FROM t0 WHERE CAST(t0.c0 AS NUMERIC) > LOWER(t0.c1) GROUP BY t0.rowid;
  </pre>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  In the following test case, a row is unexpectedly not fetched.
  <pre>
  CREATE TABLE t0 (c0, c1 NOT NULL DEFAULT 1, c2, PRIMARY KEY (c0, c1));
  INSERT INTO t0(c2) VALUES (NULL), (NULL), (NULL), (NULL), (NULL), (NULL), (NULL), (NULL), (NULL), (NULL), (NULL);
  INSERT INTO t0(c2) VALUES ('a');
  ANALYZE t0;
  SELECT DISTINCT * FROM t0 WHERE NULL IS t0.c0; -- unexpected: |1|a is not part of the result set
  </pre>

  Note that changing NULL IS t0.c0 to t0.c0 IS NULL results in the query working correctly.


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0 INTEGER PRIMARY KEY, c1 TEXT);
  INSERT INTO t0(c0, c1) VALUES (1, 'a');
  SELECT * FROM t0 WHERE '-1' BETWEEN 0 AND t0.c0; -- expected: 1|a, actual: no row is fetched
  </pre>

  I would expect the query to be equivalent to the following:

  <pre>
  SELECT * FROM t0 WHERE '-1' >= 0 AND '-1' <= t0.c0;
  </pre>

  However, for the first query no row is fetched, while one is fetched for the second one.


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The malfunction only occurs if the t0.c0 column in an INTEGER PRIMARY KEY
  or rowid in the table.  This same anomalous behavior is displayed by every
  version of SQLite going back to 3.1.0 (2005-01-21).


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0(c0 TEXT UNIQUE, c1);
  INSERT INTO t0(c0) VALUES (-1);
  SELECT * FROM t0 WHERE - x'ce' >= t0.c0; -- unexpected: row is not fetched
  </pre>

  I would expect the row to be fetched, since the following returns TRUE:

  <pre>
  SELECT - x'ce' >= t0.c0 FROM t0;
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  False positive in the sqlite3ExprNeedsNoAffinityChange() optimization.

  The error first appeared in version 3.8.1 (2013-10-17).


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Consider the following test case:

  <pre>
  CREATE TABLE t0 (c0);
  INSERT INTO t0(c0) VALUES ('a');
  SELECT * FROM t0 WHERE LIKELY(t0.rowid) <= '0'; -- unexpected: row is not fetched
  </pre>

  Unexpectedly, no row is fetched. The expression LIKELY(t0.rowid) <= '0' should evaluate to TRUE:

  <pre>
  SELECT LIKELY(t0.rowid) <= '0' FROM t0; -- 1
  </pre>


login: drh
mimetype: text/x-fossil-wiki
icomment:
  The UPDATE statement in the following code fragment causes an assertion fault:

  <blockquote><verbatim>
  CREATE TABLE t1(a,b);
  INSERT INTO t1 VALUES(1,2);
  UPDATE t1 SET (a,a,a,b)=(SELECT 99,100);
  </verbatim></blockquote>

  This issue was discovered during internal testing.  There have been no reports
  of problems in the wild.  Bisecting shows that the error was introduced by
  check-in [36944be6be5c4209] on 2017-01-03 and first appeared in release
  3.17.0.


login: dan
mimetype: text/x-fossil-wiki
icomment:
  The following causes a segfault. Or an assert() failure if assert is enabled.

  <verbatim>
    ATTACH '' AS aux;
    CREATE TABLE aux.t1(a, b, c);
    CREATE TEMP TRIGGER tmp AFTER INSERT ON t1 BEGIN
      SELECT 'abc';
    END;

    DETACH aux;
    DROP TRIGGER tmp;     
  </verbatim>


login: mrigger
mimetype: text/x-fossil-wiki
icomment:
  Unexpectedly, in the test case below, a row is fetched that should not be fetched:

  <pre>
  CREATE TABLE t0(c0 INT UNIQUE);
  INSERT INTO t0(c0) VALUES (1);
  SELECT * FROM t0 WHERE '1' IN (t0.c0); -- unexpected: fetches row
  </pre>

  The WHERE condition should yield FALSE, as demonstrated by the following statement:

  <pre>
  SELECT '1' IN (t0.c0) FROM t0; -- expected: 0
  </pre>

  When removing the UNIQUE constraint on the column, the query works as expected.


