Use sqlite-vec extension instead of chromadb for embeddings (#14163)

* swap sqlite_vec for chroma in requirements

* load sqlite_vec in embeddings manager

* remove chroma and revamp Embeddings class for sqlite_vec

* manual minilm onnx inference

* remove chroma in clip model

* migrate api from chroma to sqlite_vec

* migrate event cleanup from chroma to sqlite_vec

* migrate embedding maintainer from chroma to sqlite_vec

* genai description for sqlite_vec

* load sqlite_vec in main thread db

* extend the SqliteQueueDatabase class and use peewee db.execute_sql

* search with Event type for similarity

* fix similarity search

* install and add comment about transformers

* fix normalization

* add id filter

* clean up

* clean up

* fully remove chroma and add transformers env var

* readd uvicorn for fastapi

* readd tokenizer parallelism env var

* remove chroma from docs

* remove chroma from UI

* try removing custom pysqlite3 build

* hard code limit

* optimize queries

* revert explore query

* fix query

* keep building pysqlite3

* single pass fetch and process

* remove unnecessary re-embed

* update deps

* move SqliteVecQueueDatabase to db directory

* make search thumbnail take up full size of results box

* improve typing

* improve model downloading and add status screen

* daemon downloading thread

* catch case when semantic search is disabled

* fix typing

* build sqlite_vec from source

* resolve conflict

* file permissions

* try build deps

* remove sources

* sources

* fix thread start

* include git in build

* reorder embeddings after detectors are started

* build with sqlite amalgamation

* non-platform specific

* use wget instead of curl

* remove unzip -d

* remove sqlite_vec from requirements and load the compiled version

* fix build

* avoid race in db connection

* add scale_factor and bias to description zscore normalization

docker/main/build_sqlite_vec.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+SQLITE_VEC_VERSION="0.1.3"
+
+cp /etc/apt/sources.list /etc/apt/sources.list.d/sources-src.list
+sed -i 's|deb http|deb-src http|g' /etc/apt/sources.list.d/sources-src.list
+apt-get update
+apt-get -yqq build-dep sqlite3 gettext git
+
+mkdir /tmp/sqlite_vec
+# Grab the sqlite_vec source code.
+wget -nv https://github.com/asg017/sqlite-vec/archive/refs/tags/v${SQLITE_VEC_VERSION}.tar.gz
+tar -zxf v${SQLITE_VEC_VERSION}.tar.gz -C /tmp/sqlite_vec
+
+cd /tmp/sqlite_vec/sqlite-vec-${SQLITE_VEC_VERSION}
+
+mkdir -p vendor
+wget -O sqlite-amalgamation.zip https://www.sqlite.org/2024/sqlite-amalgamation-3450300.zip
+unzip sqlite-amalgamation.zip
+mv sqlite-amalgamation-3450300/* vendor/
+rmdir sqlite-amalgamation-3450300
+rm sqlite-amalgamation.zip
+
+# build loadable module
+make loadable
+
+# install it
+cp dist/vec0.* /usr/local/lib
+