increment motionless_count

docker/Dockerfile.dev

@@ -22,3 +22,5 @@ RUN pip3 install pylint black
 # Install Node 14
 RUN curl -sL https://deb.nodesource.com/setup_14.x | bash - \
     && apt-get install -y nodejs
+
+RUN npm install -g npm@latest

docs/docs/configuration/camera_specific.md

@@ -61,8 +61,8 @@ cameras:
           roles:
             - detect
     detect:
-      width: 640
-      height: 480
+      width: 896
+      height: 672
       fps: 7
 ```
 

docs/docs/configuration/index.md

@@ -381,7 +381,7 @@ cameras:
       #       camera.
       front_steps:
         # Required: List of x,y coordinates to define the polygon of the zone.
-        # NOTE: Coordinates can be generated at https://www.image-map.net/
+        # NOTE: Presence in a zone is evaluated only based on the bottom center of the objects bounding box.
         coordinates: 545,1077,747,939,788,805
         # Optional: List of objects that can trigger this zone (default: all tracked objects)
         objects:

docs/docs/configuration/nvdec.md

@@ -97,15 +97,3 @@ processes:
 |    0   N/A  N/A     12827      C   ffmpeg                            417MiB |
 +-----------------------------------------------------------------------------+
 ```
-
-To further improve performance, you can set ffmpeg to skip frames in the output,
-using the fps filter:
-
-```yaml
-output_args:
-  - -filter:v
-  - fps=fps=5
-```
-
-This setting, for example, allows Frigate to consume my 10-15fps camera streams on
-my relatively low powered Haswell machine with relatively low cpu usage.

docs/docs/configuration/zones.md

@@ -3,7 +3,9 @@ id: zones
 title: Zones
 ---
 
-Zones allow you to define a specific area of the frame and apply additional filters for object types so you can determine whether or not an object is within a particular area. Zones cannot have the same name as a camera. If desired, a single zone can include multiple cameras if you have multiple cameras covering the same area by configuring zones with the same name for each camera.
+Zones allow you to define a specific area of the frame and apply additional filters for object types so you can determine whether or not an object is within a particular area. Presence in a zone is evaluated based on the bottom center of the bounding box for the object. It does not matter how much of the bounding box overlaps with the zone.
+
+Zones cannot have the same name as a camera. If desired, a single zone can include multiple cameras if you have multiple cameras covering the same area by configuring zones with the same name for each camera.
 
 During testing, enable the Zones option for the debug feed so you can adjust as needed. The zone line will increase in thickness when any object enters the zone.
 

docs/docs/guides/getting_started.md

@@ -62,6 +62,8 @@ cameras:
           roles:
             - detect
             - rtmp
+    rtmp:
+      enabled: False # <-- RTMP should be disabled if your stream is not H264
     detect:
       width: 1280 # <---- update for your camera's resolution
       height: 720 # <---- update for your camera's resolution
@@ -71,7 +73,9 @@ cameras:
 
 At this point you should be able to start Frigate and see the the video feed in the UI.
 
-If you get a green image from the camera, this means ffmpeg was not able to get the video feed from your camera. Check the logs for error messages from ffmpeg. The default ffmpeg arguments are designed to work with RTSP cameras that support TCP connections. FFmpeg arguments for other types of cameras can be found [here](/configuration/camera_specific).
+If you get a green image from the camera, this means ffmpeg was not able to get the video feed from your camera. Check the logs for error messages from ffmpeg. The default ffmpeg arguments are designed to work with H264 RTSP cameras that support TCP connections. If you do not have H264 cameras, make sure you have disabled RTMP. It is possible to enable it, but you must tell ffmpeg to re-encode the video with customized output args.
+
+FFmpeg arguments for other types of cameras can be found [here](/configuration/camera_specific).
 
 ### Step 5: Configure hardware acceleration (optional)
 

docs/docs/guides/ha_notifications.md

@@ -25,6 +25,30 @@ automation:
             when: '{{trigger.payload_json["after"]["start_time"]|int}}'
 ```
 
+Note that iOS devices support live previews of cameras by adding a camera entity id to the message data.
+
+```yaml
+automation:
+  - alias: Security_Frigate_Notifications
+    description: ""
+    trigger:
+      - platform: mqtt
+        topic: frigate/events
+        payload: new
+        value_template: "{{ value_json.type }}"
+    action:
+      - service: notify.mobile_app_iphone
+        data:
+          message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
+          data:
+            image: >-
+              https://your.public.hass.address.com/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg
+            tag: '{{trigger.payload_json["after"]["id"]}}'
+            when: '{{trigger.payload_json["after"]["start_time"]|int}}'
+            entity_id: camera.{{trigger.payload_json["after"]["camera"]}}
+    mode: single
+```
+
 ## Conditions
 
 Conditions with the `before` and `after` values allow a high degree of customization for automations.

docs/docs/installation.md

@@ -177,6 +177,15 @@ HassOS users can install via the addon repository.
 6. Start the addon container
 7. (not for proxy addon) If you are using hardware acceleration for ffmpeg, you may need to disable "Protection mode"
 
+There are several versions of the addon available:
+
+| Addon Version                  | Description                                                |
+| ------------------------------ | ---------------------------------------------------------- |
+| Frigate NVR                    | Current release with protection mode on                    |
+| Frigate NVR (Full Access)      | Current release with the option to disable protection mode |
+| Frigate NVR Beta               | Beta release with protection mode on                       |
+| Frigate NVR Beta (Full Access) | Beta release with the option to disable protection mode    |
+
 ## Home Assistant Supervised
 
 :::tip

docs/docs/integrations/home-assistant.md

@@ -45,11 +45,14 @@ that card.
 
 ## Configuration
 
-When configuring the integration, you will be asked for the following parameters:
+When configuring the integration, you will be asked for the `URL` of your frigate instance which is the URL you use to access Frigate in the browser. This may look like `http://<host>:5000/`. If you are using HassOS with the addon, the URL should be one of the following depending on which addon version you are using. Note that if you are using the Proxy Addon, you do NOT point the integration at the proxy URL. Just enter the URL used to access frigate directly from your network.
 
-| Variable | Description                                                                                                                                                                                                                                                                                                                                                                      |
-| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| URL      | The `URL` of your frigate instance, the URL you use to access Frigate in the browser. This may look like `http://<host>:5000/`. If you are using HassOS with the addon, the URL should be `http://ccab4aaf-frigate:5000` (or `http://ccab4aaf-frigate-beta:5000` if your are using the beta version of the addon). Live streams required port 1935, see [RTMP streams](#streams) |
+| Addon Version                  | URL                                    |
+| ------------------------------ | -------------------------------------- |
+| Frigate NVR                    | `http://ccab4aaf-frigate:5000`         |
+| Frigate NVR (Full Access)      | `http://ccab4aaf-frigate-fa:5000`      |
+| Frigate NVR Beta               | `http://ccab4aaf-frigate-beta:5000`    |
+| Frigate NVR Beta (Full Access) | `http://ccab4aaf-frigate-fa-beta:5000` |
 
 <a name="options"></a>
 

docs/docs/integrations/mqtt.md

@@ -55,7 +55,9 @@ Message published for each changed event. The first message is published when th
     "entered_zones": ["yard", "driveway"],
     "thumbnail": null,
     "has_snapshot": false,
-    "has_clip": false
+    "has_clip": false,
+    "motionless_count": 0, // number of frames the object has been motionless
+    "position_changes": 2 // number of times the object has changed position
   },
   "after": {
     "id": "1607123955.475377-mxklsc",
@@ -75,7 +77,9 @@ Message published for each changed event. The first message is published when th
     "entered_zones": ["yard", "driveway"],
     "thumbnail": null,
     "has_snapshot": false,
-    "has_clip": false
+    "has_clip": false,
+    "motionless_count": 0, // number of frames the object has been motionless
+    "position_changes": 2 // number of times the object has changed position
   }
 }
 ```

docs/package.json

@@ -12,13 +12,13 @@
     "clear": "docusaurus clear"
   },
   "dependencies": {
-    "@docusaurus/core": "^2.0.0-beta.6",
-    "@docusaurus/preset-classic": "^2.0.0-beta.6",
-    "@mdx-js/react": "^1.6.21",
+    "@docusaurus/core": "^2.0.0-beta.15",
+    "@docusaurus/preset-classic": "^2.0.0-beta.15",
+    "@mdx-js/react": "^1.6.22",
     "clsx": "^1.1.1",
     "raw-loader": "^4.0.2",
-    "react": "^16.8.4",
-    "react-dom": "^16.8.4"
+    "react": "^16.14.0",
+    "react-dom": "^16.14.0"
   },
   "browserslist": {
     "production": [
@@ -31,5 +31,8 @@
       "last 1 firefox version",
       "last 1 safari version"
     ]
+  },
+  "devDependencies": {
+    "@types/react": "^16.14.0"
   }
 }

frigate/app.py

@@ -8,6 +8,7 @@ import threading
 from logging.handlers import QueueHandler
 from typing import Dict, List
 
+import traceback
 import yaml
 from peewee_migrate import Router
 from playhouse.sqlite_ext import SqliteExtDatabase
@@ -320,6 +321,7 @@ class FrigateApp:
                 print("***    Config Validation Errors                           ***")
                 print("*************************************************************")
                 print(e)
+                print(traceback.format_exc())
                 print("*************************************************************")
                 print("***    End Config Validation Errors                       ***")
                 print("*************************************************************")

frigate/config.py

@@ -476,7 +476,7 @@ class CameraLiveConfig(FrigateBaseModel):
 
 
 class CameraConfig(FrigateBaseModel):
-    name: Optional[str] = Field(title="Camera name.", regex="^[a-zA-Z0-9_]+$")
+    name: Optional[str] = Field(title="Camera name.", regex="^[a-zA-Z0-9_-]+$")
     ffmpeg: CameraFfmpegConfig = Field(title="FFmpeg configuration for the camera.")
     best_image_timeout: int = Field(
         default=60,
@@ -836,14 +836,18 @@ class FrigateConfig(FrigateBaseModel):
                     camera_config.record.retain.days = camera_config.record.retain_days
 
             # warning if the higher level record mode is potentially more restrictive than the events
+            rank_map = {
+                RetainModeEnum.all: 0,
+                RetainModeEnum.motion: 1,
+                RetainModeEnum.active_objects: 2,
+            }
             if (
                 camera_config.record.retain.days != 0
-                and camera_config.record.retain.mode != RetainModeEnum.all
-                and camera_config.record.events.retain.mode
-                != camera_config.record.retain.mode
+                and rank_map[camera_config.record.retain.mode]
+                > rank_map[camera_config.record.events.retain.mode]
             ):
                 logger.warning(
-                    f"Recording retention is configured for {camera_config.record.retain.mode} and event retention is configured for {camera_config.record.events.retain.mode}. The more restrictive retention policy will be applied."
+                    f"{name}: Recording retention is configured for {camera_config.record.retain.mode} and event retention is configured for {camera_config.record.events.retain.mode}. The more restrictive retention policy will be applied."
                 )
             # generage the ffmpeg commands
             camera_config.create_ffmpeg_cmds()

frigate/http.py

@@ -364,7 +364,13 @@ def best(camera_name, label):
             box_size = 300
             box = best_object.get("box", (0, 0, box_size, box_size))
             region = calculate_region(
-                best_frame.shape, box[0], box[1], box[2], box[3], box_size, multiplier=1.1
+                best_frame.shape,
+                box[0],
+                box[1],
+                box[2],
+                box[3],
+                box_size,
+                multiplier=1.1,
             )
             best_frame = best_frame[region[1] : region[3], region[0] : region[2]]
 
@@ -711,7 +717,15 @@ def vod_event(id):
         end_ts = (
             datetime.now().timestamp() if event.end_time is None else event.end_time
         )
-        return vod_ts(event.camera, event.start_time, end_ts)
+        vod_response = vod_ts(event.camera, event.start_time, end_ts)
+        # If the recordings are not found, set has_clip to false
+        if (
+            type(vod_response) == tuple
+            and len(vod_response) == 2
+            and vod_response[1] == 404
+        ):
+            Event.update(has_clip=False).where(Event.id == id).execute()
+        return vod_response
 
     duration = int((event.end_time - event.start_time) * 1000)
     return jsonify(

frigate/objects.py

@@ -106,6 +106,7 @@ class ObjectTracker:
     def update_frame_times(self, frame_time):
         for id in self.tracked_objects.keys():
             self.tracked_objects[id]["frame_time"] = frame_time
+            self.tracked_objects[id]["motionless_count"] += 1
 
     def match_and_update(self, frame_time, new_objects):
         # group by name

frigate/record.py

@@ -230,7 +230,7 @@ class RecordingMaintainer(threading.Thread):
                 [
                     o
                     for o in frame[1]
-                    if not o["false_positive"] and o["motionless_count"] > 0
+                    if not o["false_positive"] and o["motionless_count"] == 0
                 ]
             )
 
@@ -285,6 +285,7 @@ class RecordingMaintainer(threading.Thread):
                 end_time=end_time.timestamp(),
                 duration=duration,
                 motion=motion_count,
+                # TODO: update this to store list of active objects at some point
                 objects=active_count,
             )
         except Exception as e:

frigate/util.py

@@ -567,6 +567,9 @@ class EventsPerSecond:
         # compute the (approximate) events in the last n seconds
         now = datetime.datetime.now().timestamp()
         seconds = min(now - self._start, last_n_seconds)
+        # avoid divide by zero
+        if seconds == 0:
+            seconds = 1
         return (
             len([t for t in self._timestamps if t > (now - last_n_seconds)]) / seconds
         )
@@ -601,6 +604,7 @@ def add_mask(mask, mask_img):
     )
     cv2.fillPoly(mask_img, pts=[contour], color=(0))
 
+
 def load_labels(path, encoding="utf-8"):
     """Loads labels from file (with or without index numbers).
     Args:
@@ -620,6 +624,7 @@ def load_labels(path, encoding="utf-8"):
         else:
             return {index: line.strip() for index, line in enumerate(lines)}
 
+
 class FrameManager(ABC):
     @abstractmethod
     def create(self, name, size) -> AnyStr:

frigate/video.py

@@ -153,10 +153,10 @@ def capture_frames(
         try:
             frame_buffer[:] = ffmpeg_process.stdout.read(frame_size)
         except Exception as e:
-            logger.info(f"{camera_name}: ffmpeg sent a broken frame. {e}")
+            logger.error(f"{camera_name}: Unable to read frames from ffmpeg process.")
 
             if ffmpeg_process.poll() != None:
-                logger.info(
+                logger.error(
                     f"{camera_name}: ffmpeg process is not running. exiting capture thread..."
                 )
                 frame_manager.delete(frame_name)
@@ -221,12 +221,11 @@ class CameraWatchdog(threading.Thread):
 
             if not self.capture_thread.is_alive():
                 self.logger.error(
-                    f"FFMPEG process crashed unexpectedly for {self.camera_name}."
+                    f"Ffmpeg process crashed unexpectedly for {self.camera_name}."
                 )
                 self.logger.error(
                     "The following ffmpeg logs include the last 100 lines prior to exit."
                 )
-                self.logger.error("You may have invalid args defined for this camera.")
                 self.logpipe.dump()
                 self.start_ffmpeg_detect()
             elif now - self.capture_thread.current_frame.value > 20:
@@ -492,212 +491,219 @@ def process_frames(
             logger.info(f"{camera_name}: frame {frame_time} is not in memory store.")
             continue
 
-        if not detection_enabled.value:
-            fps.value = fps_tracker.eps()
-            object_tracker.match_and_update(frame_time, [])
-            detected_objects_queue.put(
-                (camera_name, frame_time, object_tracker.tracked_objects, [], [])
-            )
-            detection_fps.value = object_detector.fps.eps()
-            frame_manager.close(f"{camera_name}{frame_time}")
-            continue
-
         # look for motion
         motion_boxes = motion_detector.detect(frame)
 
-        # get stationary object ids
-        # check every Nth frame for stationary objects
-        # disappeared objects are not stationary
-        # also check for overlapping motion boxes
-        stationary_object_ids = [
-            obj["id"]
-            for obj in object_tracker.tracked_objects.values()
-            # if there hasn't been motion for 10 frames
-            if obj["motionless_count"] >= 10
-            # and it isn't due for a periodic check
-            and (
-                detect_config.stationary_interval == 0
-                or obj["motionless_count"] % detect_config.stationary_interval != 0
-            )
-            # and it hasn't disappeared
-            and object_tracker.disappeared[obj["id"]] == 0
-            # and it doesn't overlap with any current motion boxes
-            and not intersects_any(obj["box"], motion_boxes)
-        ]
+        regions = []
 
-        # get tracked object boxes that aren't stationary
-        tracked_object_boxes = [
-            obj["box"]
-            for obj in object_tracker.tracked_objects.values()
-            if not obj["id"] in stationary_object_ids
-        ]
-
-        # combine motion boxes with known locations of existing objects
-        combined_boxes = reduce_boxes(motion_boxes + tracked_object_boxes)
-
-        region_min_size = max(model_shape[0], model_shape[1])
-        # compute regions
-        regions = [
-            calculate_region(
-                frame_shape,
-                a[0],
-                a[1],
-                a[2],
-                a[3],
-                region_min_size,
-                multiplier=random.uniform(1.2, 1.5),
-            )
-            for a in combined_boxes
-        ]
-
-        # consolidate regions with heavy overlap
-        regions = [
-            calculate_region(
-                frame_shape, a[0], a[1], a[2], a[3], region_min_size, multiplier=1.0
-            )
-            for a in reduce_boxes(regions, 0.4)
-        ]
-
-        # if starting up, get the next startup scan region
-        if startup_scan_counter < 9:
-            ymin = int(frame_shape[0] / 3 * startup_scan_counter / 3)
-            ymax = int(frame_shape[0] / 3 + ymin)
-            xmin = int(frame_shape[1] / 3 * startup_scan_counter / 3)
-            xmax = int(frame_shape[1] / 3 + xmin)
-            regions.append(
-                calculate_region(
-                    frame_shape, xmin, ymin, xmax, ymax, region_min_size, multiplier=1.2
-                )
-            )
-            startup_scan_counter += 1
-
-        # resize regions and detect
-        # seed with stationary objects
-        detections = [
-            (
-                obj["label"],
-                obj["score"],
-                obj["box"],
-                obj["area"],
-                obj["region"],
-            )
-            for obj in object_tracker.tracked_objects.values()
-            if obj["id"] in stationary_object_ids
-        ]
-
-        for region in regions:
-            detections.extend(
-                detect(
-                    object_detector,
-                    frame,
-                    model_shape,
-                    region,
-                    objects_to_track,
-                    object_filters,
-                )
-            )
-
-        #########
-        # merge objects, check for clipped objects and look again up to 4 times
-        #########
-        refining = len(regions) > 0
-        refine_count = 0
-        while refining and refine_count < 4:
-            refining = False
-
-            # group by name
-            detected_object_groups = defaultdict(lambda: [])
-            for detection in detections:
-                detected_object_groups[detection[0]].append(detection)
-
-            selected_objects = []
-            for group in detected_object_groups.values():
-
-                # apply non-maxima suppression to suppress weak, overlapping bounding boxes
-                boxes = [
-                    (o[2][0], o[2][1], o[2][2] - o[2][0], o[2][3] - o[2][1])
-                    for o in group
-                ]
-                confidences = [o[1] for o in group]
-                idxs = cv2.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4)
-
-                for index in idxs:
-                    obj = group[index[0]]
-                    if clipped(obj, frame_shape):
-                        box = obj[2]
-                        # calculate a new region that will hopefully get the entire object
-                        region = calculate_region(
-                            frame_shape, box[0], box[1], box[2], box[3], region_min_size
-                        )
-
-                        regions.append(region)
-
-                        selected_objects.extend(
-                            detect(
-                                object_detector,
-                                frame,
-                                model_shape,
-                                region,
-                                objects_to_track,
-                                object_filters,
-                            )
-                        )
-
-                        refining = True
-                    else:
-                        selected_objects.append(obj)
-            # set the detections list to only include top, complete objects
-            # and new detections
-            detections = selected_objects
-
-            if refining:
-                refine_count += 1
-
-        ## drop detections that overlap too much
-        consolidated_detections = []
-
-        # if detection was run on this frame, consolidate
-        if len(regions) > 0:
-            # group by name
-            detected_object_groups = defaultdict(lambda: [])
-            for detection in detections:
-                detected_object_groups[detection[0]].append(detection)
-
-            # loop over detections grouped by label
-            for group in detected_object_groups.values():
-                # if the group only has 1 item, skip
-                if len(group) == 1:
-                    consolidated_detections.append(group[0])
-                    continue
-
-                # sort smallest to largest by area
-                sorted_by_area = sorted(group, key=lambda g: g[3])
-
-                for current_detection_idx in range(0, len(sorted_by_area)):
-                    current_detection = sorted_by_area[current_detection_idx][2]
-                    overlap = 0
-                    for to_check_idx in range(
-                        min(current_detection_idx + 1, len(sorted_by_area)),
-                        len(sorted_by_area),
-                    ):
-                        to_check = sorted_by_area[to_check_idx][2]
-                        # if 90% of smaller detection is inside of another detection, consolidate
-                        if (
-                            area(intersection(current_detection, to_check))
-                            / area(current_detection)
-                            > 0.9
-                        ):
-                            overlap = 1
-                            break
-                    if overlap == 0:
-                        consolidated_detections.append(
-                            sorted_by_area[current_detection_idx]
-                        )
-            # now that we have refined our detections, we need to track objects
-            object_tracker.match_and_update(frame_time, consolidated_detections)
-        # else, just update the frame times for the stationary objects
+        # if detection is disabled
+        if not detection_enabled.value:
+            object_tracker.match_and_update(frame_time, [])
         else:
-            object_tracker.update_frame_times(frame_time)
+            # get stationary object ids
+            # check every Nth frame for stationary objects
+            # disappeared objects are not stationary
+            # also check for overlapping motion boxes
+            stationary_object_ids = [
+                obj["id"]
+                for obj in object_tracker.tracked_objects.values()
+                # if there hasn't been motion for 10 frames
+                if obj["motionless_count"] >= 10
+                # and it isn't due for a periodic check
+                and (
+                    detect_config.stationary_interval == 0
+                    or obj["motionless_count"] % detect_config.stationary_interval != 0
+                )
+                # and it hasn't disappeared
+                and object_tracker.disappeared[obj["id"]] == 0
+                # and it doesn't overlap with any current motion boxes
+                and not intersects_any(obj["box"], motion_boxes)
+            ]
+
+            # get tracked object boxes that aren't stationary
+            tracked_object_boxes = [
+                obj["box"]
+                for obj in object_tracker.tracked_objects.values()
+                if not obj["id"] in stationary_object_ids
+            ]
+
+            # combine motion boxes with known locations of existing objects
+            combined_boxes = reduce_boxes(motion_boxes + tracked_object_boxes)
+
+            region_min_size = max(model_shape[0], model_shape[1])
+            # compute regions
+            regions = [
+                calculate_region(
+                    frame_shape,
+                    a[0],
+                    a[1],
+                    a[2],
+                    a[3],
+                    region_min_size,
+                    multiplier=random.uniform(1.2, 1.5),
+                )
+                for a in combined_boxes
+            ]
+
+            # consolidate regions with heavy overlap
+            regions = [
+                calculate_region(
+                    frame_shape, a[0], a[1], a[2], a[3], region_min_size, multiplier=1.0
+                )
+                for a in reduce_boxes(regions, 0.4)
+            ]
+
+            # if starting up, get the next startup scan region
+            if startup_scan_counter < 9:
+                ymin = int(frame_shape[0] / 3 * startup_scan_counter / 3)
+                ymax = int(frame_shape[0] / 3 + ymin)
+                xmin = int(frame_shape[1] / 3 * startup_scan_counter / 3)
+                xmax = int(frame_shape[1] / 3 + xmin)
+                regions.append(
+                    calculate_region(
+                        frame_shape,
+                        xmin,
+                        ymin,
+                        xmax,
+                        ymax,
+                        region_min_size,
+                        multiplier=1.2,
+                    )
+                )
+                startup_scan_counter += 1
+
+            # resize regions and detect
+            # seed with stationary objects
+            detections = [
+                (
+                    obj["label"],
+                    obj["score"],
+                    obj["box"],
+                    obj["area"],
+                    obj["region"],
+                )
+                for obj in object_tracker.tracked_objects.values()
+                if obj["id"] in stationary_object_ids
+            ]
+
+            for region in regions:
+                detections.extend(
+                    detect(
+                        object_detector,
+                        frame,
+                        model_shape,
+                        region,
+                        objects_to_track,
+                        object_filters,
+                    )
+                )
+
+            #########
+            # merge objects, check for clipped objects and look again up to 4 times
+            #########
+            refining = len(regions) > 0
+            refine_count = 0
+            while refining and refine_count < 4:
+                refining = False
+
+                # group by name
+                detected_object_groups = defaultdict(lambda: [])
+                for detection in detections:
+                    detected_object_groups[detection[0]].append(detection)
+
+                selected_objects = []
+                for group in detected_object_groups.values():
+
+                    # apply non-maxima suppression to suppress weak, overlapping bounding boxes
+                    boxes = [
+                        (o[2][0], o[2][1], o[2][2] - o[2][0], o[2][3] - o[2][1])
+                        for o in group
+                    ]
+                    confidences = [o[1] for o in group]
+                    idxs = cv2.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4)
+
+                    for index in idxs:
+                        obj = group[index[0]]
+                        if clipped(obj, frame_shape):
+                            box = obj[2]
+                            # calculate a new region that will hopefully get the entire object
+                            region = calculate_region(
+                                frame_shape,
+                                box[0],
+                                box[1],
+                                box[2],
+                                box[3],
+                                region_min_size,
+                            )
+
+                            regions.append(region)
+
+                            selected_objects.extend(
+                                detect(
+                                    object_detector,
+                                    frame,
+                                    model_shape,
+                                    region,
+                                    objects_to_track,
+                                    object_filters,
+                                )
+                            )
+
+                            refining = True
+                        else:
+                            selected_objects.append(obj)
+                # set the detections list to only include top, complete objects
+                # and new detections
+                detections = selected_objects
+
+                if refining:
+                    refine_count += 1
+
+            ## drop detections that overlap too much
+            consolidated_detections = []
+
+            # if detection was run on this frame, consolidate
+            if len(regions) > 0:
+                # group by name
+                detected_object_groups = defaultdict(lambda: [])
+                for detection in detections:
+                    detected_object_groups[detection[0]].append(detection)
+
+                # loop over detections grouped by label
+                for group in detected_object_groups.values():
+                    # if the group only has 1 item, skip
+                    if len(group) == 1:
+                        consolidated_detections.append(group[0])
+                        continue
+
+                    # sort smallest to largest by area
+                    sorted_by_area = sorted(group, key=lambda g: g[3])
+
+                    for current_detection_idx in range(0, len(sorted_by_area)):
+                        current_detection = sorted_by_area[current_detection_idx][2]
+                        overlap = 0
+                        for to_check_idx in range(
+                            min(current_detection_idx + 1, len(sorted_by_area)),
+                            len(sorted_by_area),
+                        ):
+                            to_check = sorted_by_area[to_check_idx][2]
+                            # if 90% of smaller detection is inside of another detection, consolidate
+                            if (
+                                area(intersection(current_detection, to_check))
+                                / area(current_detection)
+                                > 0.9
+                            ):
+                                overlap = 1
+                                break
+                        if overlap == 0:
+                            consolidated_detections.append(
+                                sorted_by_area[current_detection_idx]
+                            )
+                # now that we have refined our detections, we need to track objects
+                object_tracker.match_and_update(frame_time, consolidated_detections)
+            # else, just update the frame times for the stationary objects
+            else:
+                object_tracker.update_frame_times(frame_time)
 
         # add to the queue if not full
         if detected_objects_queue.full():