add links in docs to other sites

Based on issue #1976 - specify explicitly that these fields can include environment variables to avoid interpretation that environment variables could be used anywhere.
I am participating in #hacktoberfest, so I would appreciate if you could add the 'hacktoberfest-accepted' label (or add #hacktoberfest topic to your repo). Thanks!

.dockerignore

@@ -7,4 +7,5 @@ config/
 .git
 core
 *.mp4
-*.db
+*.db
+*.ts

.gitignore

@@ -6,6 +6,7 @@ debug
 config/config.yml
 models
 *.mp4
+*.ts
 *.db
 frigate/version.py
 web/build

Makefile

@@ -3,7 +3,7 @@ default_target: amd64_frigate
 COMMIT_HASH := $(shell git log -1 --pretty=format:"%h"|tail -1)
 
 version:
-	echo "VERSION='0.9.2-$(COMMIT_HASH)'" > frigate/version.py
+	echo "VERSION='0.9.3-$(COMMIT_HASH)'" > frigate/version.py
 
 web:
 	docker build --tag frigate-web --file docker/Dockerfile.web web/

docs/docs/configuration/camera_specific.md

@@ -15,8 +15,8 @@ Note that mjpeg cameras require encoding the video into h264 for recording, and
 
 ```yaml
 output_args:
-  record: -f segment -segment_time 10 -segment_format ts -reset_timestamps 1 -strftime 1 -c:v libx264
-  rtmp: -c:v libx264 -f flv
+  record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
+  rtmp: -c:v libx264 -an -f flv
 ```
 
 ### RTMP Cameras

docs/docs/configuration/index.md

@@ -48,8 +48,8 @@ mqtt:
   # Optional: user
   user: mqtt_user
   # Optional: password
-  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
-  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
+  # NOTE: MQTT password can be specified with an environment variables that must begin with 'FRIGATE_'.
+  #       e.g. password: '{FRIGATE_MQTT_PASSWORD}'
   password: password
   # Optional: tls_ca_certs for enabling TLS using self-signed certs (default: None)
   tls_ca_certs: /path/to/ca.crt
@@ -140,7 +140,7 @@ ffmpeg:
     # Optional: output args for detect streams (default: shown below)
     detect: -f rawvideo -pix_fmt yuv420p
     # Optional: output args for record streams (default: shown below)
-    record: -f segment -segment_time 10 -segment_format ts -reset_timestamps 1 -strftime 1 -c copy
+    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
     # Optional: output args for rtmp streams (default: shown below)
     rtmp: -c copy -f flv
 
@@ -319,7 +319,7 @@ cameras:
       # Required: A list of input streams for the camera. See documentation for more information.
       inputs:
         # Required: the path to the stream
-        # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}
+        # NOTE: path may include environment variables, which must begin with 'FRIGATE_' and be referenced in {}
         - path: rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@10.0.10.10:554/cam/realmonitor?channel=1&subtype=2
           # Required: list of roles for this stream. valid values are: detect,record,rtmp
           # NOTICE: In addition to assigning the record, and rtmp roles,

docs/docs/faqs.md

@@ -13,7 +13,7 @@ A solid green image means that frigate has not received any frames from ffmpeg.
 
 ### How can I get sound or audio in my recordings?
 
-The recommended audio codec is `aac`. Not all audio codecs are supported by RTMP, so you may need to re-encode your audio with `-c:a aac`. The default ffmpeg args are shown [here](configuration/index#full-configuration-reference).
+By default, Frigate removes audio from recordings to reduce the likelihood of failing for invalid data. If you would like to include audio, you need to override the output args to remove `-an` for where you want to include audio. The recommended audio codec is `aac`. Not all audio codecs are supported by RTMP, so you may need to re-encode your audio with `-c:a aac`. The default ffmpeg args are shown [here](configuration/index#full-configuration-reference).
 
 ### My mjpeg stream or snapshots look green and crazy
 

docs/docs/installation.md

@@ -17,6 +17,61 @@ Frigate runs best with docker installed on bare metal debian-based distributions
 
 Windows is not officially supported, but some users have had success getting it to run under WSL or Virtualbox. Getting the GPU and/or Coral devices properly passed to Frigate may be difficult or impossible. Search previous discussions or issues for help.
 
+### Storage
+
+Frigate uses the following locations for read/write operations in the container. Docker volume mappings can be used to map these to any location on your host machine.
+
+- `/media/frigate/clips`: Used for snapshot storage. In the future, it will likely be renamed from `clips` to `snapshots`. The file structure here cannot be modified and isn't intended to be browsed or managed manually.
+- `/media/frigate/recordings`: Internal system storage for recording segments. The file structure here cannot be modified and isn't intended to be browsed or managed manually.
+- `/media/frigate/frigate.db`: Default location for the sqlite database. You will also see several files alongside this file while frigate is running. If moving the database location (often needed when using a network drive at `/media/frigate`), it is recommended to mount a volume with docker at `/db` and change the storage location of the database to `/db/frigate.db` in the config file.
+- `/tmp/cache`: Cache location for recording segments. Initial recordings are written here before being checked and converted to mp4 and moved to the recordings folder.
+- `/dev/shm`: It is not recommended to modify this directory or map it with docker. This is the location for raw decoded frames in shared memory and it's size is impacted by the `shm-size` calculations below.
+- `/config/config.yml`: Default location of the config file.
+
+#### Common docker compose storage configurations
+
+Writing to a local disk or external USB drive:
+
+```yaml
+version: "3.9"
+services:
+  frigate:
+    ...
+    volumes:
+      - /path/to/your/config.yml:/config/config.yml:ro
+      - /path/to/your/storage:/media/frigate
+      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
+        target: /tmp/cache
+        tmpfs:
+          size: 1000000000
+    ...
+```
+
+Writing to a network drive with database on a local drive:
+
+```yaml
+version: "3.9"
+services:
+  frigate:
+    ...
+    volumes:
+      - /path/to/your/config.yml:/config/config.yml:ro
+      - /path/to/network/storage:/media/frigate
+      - /path/to/local/disk:/db
+      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
+        target: /tmp/cache
+        tmpfs:
+          size: 1000000000
+    ...
+```
+
+frigate.yml
+
+```yaml
+database:
+  path: /db/frigate.db
+```
+
 ### Calculating required shm-size
 
 Frigate utilizes shared memory to store frames during processing. The default `shm-size` provided by Docker is 64m.

docs/docusaurus.config.js

@@ -29,6 +29,16 @@ module.exports = {
           label: 'Docs',
           position: 'left',
         },
+        {
+          href: 'https://frigate.video',
+          label: 'Website',
+          position: 'right',
+        },
+        {
+          href: 'https://demo.frigate.video',
+          label: 'Demo',
+          position: 'right',
+        },
         {
           href: 'https://github.com/blakeblackshear/frigate',
           label: 'GitHub',

frigate/config.py

@@ -298,13 +298,14 @@ RECORD_FFMPEG_OUTPUT_ARGS_DEFAULT = [
     "-segment_time",
     "10",
     "-segment_format",
-    "ts",
+    "mp4",
     "-reset_timestamps",
     "1",
     "-strftime",
     "1",
     "-c",
     "copy",
+    "-an",
 ]
 
 
@@ -564,16 +565,9 @@ class CameraConfig(FrigateBaseModel):
                 else self.ffmpeg.output_args.record.split(" ")
             )
 
-            # backwards compatibility check for segment_format change from mp4 to ts
-            record_args = (
-                " ".join(record_args)
-                .replace("-segment_format mp4", "-segment_format ts")
-                .split(" ")
-            )
-
             ffmpeg_output_args = (
                 record_args
-                + [f"{os.path.join(CACHE_DIR, self.name)}-%Y%m%d%H%M%S.ts"]
+                + [f"{os.path.join(CACHE_DIR, self.name)}-%Y%m%d%H%M%S.mp4"]
                 + ffmpeg_output_args
             )
 
@@ -800,6 +794,21 @@ class FrigateConfig(FrigateBaseModel):
 
             config.cameras[name] = camera_config
 
+            # check runtime config
+            for name, camera in config.cameras.items():
+                assigned_roles = list(
+                    set([r for i in camera.ffmpeg.inputs for r in i.roles])
+                )
+                if camera.record.enabled and not "record" in assigned_roles:
+                    raise ValueError(
+                        f"Camera {name} has record enabled, but record is not assigned to an input."
+                    )
+
+                if camera.rtmp.enabled and not "rtmp" in assigned_roles:
+                    raise ValueError(
+                        f"Camera {name} has rtmp enabled, but rtmp is not assigned to an input."
+                    )
+
         return config
 
     @validator("cameras")
@@ -810,23 +819,6 @@ class FrigateConfig(FrigateBaseModel):
                 raise ValueError("Zones cannot share names with cameras")
         return v
 
-    @validator("cameras")
-    def ensure_cameras_are_not_missing_roles(cls, v: Dict[str, CameraConfig]):
-        for name, camera in v.items():
-            assigned_roles = list(
-                set([r for i in camera.ffmpeg.inputs for r in i.roles])
-            )
-            if camera.record.enabled and not "record" in assigned_roles:
-                raise ValueError(
-                    f"Camera {name} has record enabled, but record is not assigned to an input."
-                )
-
-            if camera.rtmp.enabled and not "rtmp" in assigned_roles:
-                raise ValueError(
-                    f"Camera {name} has rtmp enabled, but rtmp is not assigned to an input."
-                )
-        return v
-
     @classmethod
     def parse_file(cls, config_file):
         with open(config_file) as f:

frigate/record.py

@@ -48,7 +48,9 @@ class RecordingMaintainer(threading.Thread):
         recordings = [
             d
             for d in os.listdir(CACHE_DIR)
-            if os.path.isfile(os.path.join(CACHE_DIR, d)) and d.endswith(".ts")
+            if os.path.isfile(os.path.join(CACHE_DIR, d))
+            and d.endswith(".mp4")
+            and not d.startswith("clip_")
         ]
 
         files_in_use = []
@@ -111,30 +113,9 @@ class RecordingMaintainer(threading.Thread):
             file_name = f"{start_time.strftime('%M.%S.mp4')}"
             file_path = os.path.join(directory, file_name)
 
-            ffmpeg_cmd = [
-                "ffmpeg",
-                "-y",
-                "-i",
-                cache_path,
-                "-c",
-                "copy",
-                "-movflags",
-                "+faststart",
-                file_path,
-            ]
-
-            p = sp.run(
-                ffmpeg_cmd,
-                encoding="ascii",
-                capture_output=True,
-            )
-
-            Path(cache_path).unlink(missing_ok=True)
-
-            if p.returncode != 0:
-                logger.error(f"Unable to convert {cache_path} to {file_path}")
-                logger.error(p.stderr)
-                continue
+            # copy then delete is required when recordings are stored on some network drives
+            shutil.copyfile(cache_path, file_path)
+            os.remove(cache_path)
 
             rand_id = "".join(
                 random.choices(string.ascii_lowercase + string.digits, k=6)
@@ -150,8 +131,11 @@ class RecordingMaintainer(threading.Thread):
 
     def run(self):
         # Check for new files every 5 seconds
-        while not self.stop_event.wait(5):
+        wait_time = 5
+        while not self.stop_event.wait(wait_time):
+            run_start = datetime.datetime.now().timestamp()
             self.move_files()
+            wait_time = max(0, 5 - (datetime.datetime.now().timestamp() - run_start))
 
         logger.info(f"Exiting recording maintenance...")
 
@@ -290,9 +274,7 @@ class RecordingCleanup(threading.Thread):
 
         # find all the recordings older than the oldest recording in the db
         try:
-            oldest_recording = (
-                Recordings.select().order_by(Recordings.start_time.desc()).get()
-            )
+            oldest_recording = Recordings.select().order_by(Recordings.start_time).get()
 
             p = Path(oldest_recording.path)
             oldest_timestamp = p.stat().st_mtime - 1
@@ -301,7 +283,7 @@ class RecordingCleanup(threading.Thread):
 
         logger.debug(f"Oldest recording in the db: {oldest_timestamp}")
         process = sp.run(
-            ["find", RECORD_DIR, "-type", "f", "-newermt", f"@{oldest_timestamp}"],
+            ["find", RECORD_DIR, "-type", "f", "!", "-newermt", f"@{oldest_timestamp}"],
             capture_output=True,
             text=True,
         )

frigate/test/test_config.py

@@ -702,7 +702,11 @@ class TestConfig(unittest.TestCase):
                         "inputs": [
                             {
                                 "path": "rtsp://10.0.0.1:554/video",
-                                "roles": ["detect", "clips"],
+                                "roles": ["detect"],
+                            },
+                            {
+                                "path": "rtsp://10.0.0.1:554/video2",
+                                "roles": ["clips"],
                             },
                         ]
                     },
@@ -717,6 +721,37 @@ class TestConfig(unittest.TestCase):
 
         self.assertRaises(ValidationError, lambda: FrigateConfig(**config))
 
+    def test_fails_on_missing_role(self):
+
+        config = {
+            "mqtt": {"host": "mqtt"},
+            "cameras": {
+                "back": {
+                    "ffmpeg": {
+                        "inputs": [
+                            {
+                                "path": "rtsp://10.0.0.1:554/video",
+                                "roles": ["detect"],
+                            },
+                            {
+                                "path": "rtsp://10.0.0.1:554/video2",
+                                "roles": ["record"],
+                            },
+                        ]
+                    },
+                    "detect": {
+                        "height": 1080,
+                        "width": 1920,
+                        "fps": 5,
+                    },
+                    "rtmp": {"enabled": True},
+                }
+            },
+        }
+
+        frigate_config = FrigateConfig(**config)
+        self.assertRaises(ValueError, lambda: frigate_config.runtime_config)
+
     def test_global_detect(self):
 
         config = {
@@ -958,6 +993,34 @@ class TestConfig(unittest.TestCase):
         runtime_config = frigate_config.runtime_config
         assert runtime_config.cameras["back"].rtmp.enabled
 
+    def test_global_rtmp_default(self):
+
+        config = {
+            "mqtt": {"host": "mqtt"},
+            "rtmp": {"enabled": False},
+            "cameras": {
+                "back": {
+                    "ffmpeg": {
+                        "inputs": [
+                            {
+                                "path": "rtsp://10.0.0.1:554/video",
+                                "roles": ["detect"],
+                            },
+                            {
+                                "path": "rtsp://10.0.0.1:554/video2",
+                                "roles": ["record"],
+                            },
+                        ]
+                    },
+                }
+            },
+        }
+        frigate_config = FrigateConfig(**config)
+        assert config == frigate_config.dict(exclude_unset=True)
+
+        runtime_config = frigate_config.runtime_config
+        assert not runtime_config.cameras["back"].rtmp.enabled
+
     def test_global_live(self):
 
         config = {