chris/frigate
1
0
Fork 0
forked from Github/frigate
Code Pull Requests Packages Activity

Use sqlite-vec extension instead of chromadb for embeddings (#14163)

Browse Source 
* swap sqlite_vec for chroma in requirements

* load sqlite_vec in embeddings manager

* remove chroma and revamp Embeddings class for sqlite_vec

* manual minilm onnx inference

* remove chroma in clip model

* migrate api from chroma to sqlite_vec

* migrate event cleanup from chroma to sqlite_vec

* migrate embedding maintainer from chroma to sqlite_vec

* genai description for sqlite_vec

* load sqlite_vec in main thread db

* extend the SqliteQueueDatabase class and use peewee db.execute_sql

* search with Event type for similarity

* fix similarity search

* install and add comment about transformers

* fix normalization

* add id filter

* clean up

* clean up

* fully remove chroma and add transformers env var

* readd uvicorn for fastapi

* readd tokenizer parallelism env var

* remove chroma from docs

* remove chroma from UI

* try removing custom pysqlite3 build

* hard code limit

* optimize queries

* revert explore query

* fix query

* keep building pysqlite3

* single pass fetch and process

* remove unnecessary re-embed

* update deps

* move SqliteVecQueueDatabase to db directory

* make search thumbnail take up full size of results box

* improve typing

* improve model downloading and add status screen

* daemon downloading thread

* catch case when semantic search is disabled

* fix typing

* build sqlite_vec from source

* resolve conflict

* file permissions

* try build deps

* remove sources

* sources

* fix thread start

* include git in build

* reorder embeddings after detectors are started

* build with sqlite amalgamation

* non-platform specific

* use wget instead of curl

* remove unzip -d

* remove sqlite_vec from requirements and load the compiled version

* fix build

* avoid race in db connection

* add scale_factor and bias to description zscore normalization
This commit is contained in:
Josh Hawkins
2024-10-07 15:30:45 -05:00
committed by GitHub
parent 757150dec1
commit 24ac9f3e5a
42 changed files with 951 additions and 533 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
web/src/utils/logUtil.ts
View File

@@ -128,46 +128,6 @@ export function parseLogLines(logService: LogType, logs: string[]) {
};
})
.filter((value) => value != null) as LogLine[];
} else if (logService == "chroma") {
return logs
.map((line) => {
const match = frigateDateStamp.exec(line);
if (!match) {
const infoIndex = line.indexOf("[INFO]");
if (infoIndex != -1) {
return {
dateStamp: line.substring(0, 19),
severity: "info",
section: "startup",
content: line.substring(infoIndex + 6).trim(),
};
}
return null;
}
const startup =
line.indexOf("Starting component") !== -1 ||
line.indexOf("startup") !== -1 ||
line.indexOf("Started") !== -1 ||
line.indexOf("Uvicorn") !== -1;
const api = !!httpMethods.exec(line);
const tag = startup ? "startup" : api ? "API" : "server";
return {
dateStamp: match.toString().slice(1, -1),
severity: pythonSeverity
.exec(line)
?.at(0)
?.toString()
?.toLowerCase() as LogSeverity,
section: tag,
content: line.substring(match.index + match[0].length).trim(),
};
})
.filter((value) => value != null) as LogLine[];
}
return [];