Multi-rate capture

This is a beta feature. Please provide feedback on your experiences and suggestions after trying out the new Multi-rate capture.

Background

Multi-rate capture allows a video to be captured with three different frame rates, as shown below. Multi-rate capture is intended to provide additional information on what happened before and after the event of interest. For example, imagine a video of a baseball pitch. A pitch can be broken down into a (1) windup, (2) pitch, and (3) follow through. The camera is triggered right as the pitch starts.

Multi-rate capture solves several issues:

• The need to reduce save time so repetitive events, such as baseball pitches, can all be captured and saved without losing any events of interest.
• The need to have context around the even of interest, where the context doesn't need to have such fine detail.

Multi-rate capture example

The image below shows a timeline with video frames being captured at three different frame rates; pretrigger frame rate, post trigger 1 frame rate, and post trigger 2 frame rate. Each vertical bar represents the start-of-frame capture time. This is when the image sensor starts collecting photons based on the configured exposure time. The blue vertical line represents when the trigger event occurs. The trigger debounce delay in detecting the trigger event is not shown in the diagram.

The example times in the diagram are based on a multi-rate capture configuration as follows:

• pre-trigger frame rate: 1000 fps
• post trigger 1 frame rate: 500 fps
• post trigger 2 frame rate: 200 fps

The numbers in the middle of the diagram (-4, -3, -2, -1, 0, 1, 2, 3) are the frame numbers, where 0 is trigger frame. You can configure overlay to show the frame number on each frame in the video.

Each saved video also has an accompanying saved metadata file. In the metadata file you can find the value for the trigger time and the trigger delay. The trigger delay is the time from when the trigger event is detected until the start-of-frame for frame 0, the trigger frame. Notice the trigger event always occurs at some point during frame -1 interval. Keep in mind that the default value for trigger debounce is enabled, which adds a 5 ms delay from when a trigger event occurs until the camera detects the trigger event.

If you have set your camera's clock accurately (by using NTP for example), then it is possible to calculate the start-of-frame time for any frame in a captured video. For this example, we assume:

• The frame of interest is the 4th post trigger 2 frame.
• Trigger debounce is enabled
• There are 1000 post trigger 1 frames, i.e. power trigger one duration is set to 2 seconds
• From the metadata file, we have
  • Trigger time: 1681065361.550394 (sec since 1970)
  • Trigger delay: 0.000387642 sec (time from trigger input to start of triggered frame)
• Recall:
  • Post trigger 1 frame rate is 500 fps, or 2 ms per frame
  • Post trigger 2 frame rate is 200 fps, or 5 ms per frame

To calculate the time when the start-of-frame for the 4th frame in post trigger 2 video capture occurs, you can use the following formula.

$t_{frame\ 4\ post\ trigger\ 2}=t_{trigger}+t_{debounce}+t_{trigger\ delay}+t_{post\ 1\ frame\ duration}*N_{post\ 1\ frame\ count}+t_{post\ 2\ frame\ duration}*(N_{post\ 2\ desired\ frame}-1)t_{frame\ 4\ post\ trigger\ 2}=1681065361.550394+0.005+0.000387642+0.002*1000+0.005*(4-1)t_{frame\ 4\ post\ trigger\ 2}=1681065363.57t_{frame\ 4\ post\ trigger\ 2}=Sunday,April9,20236:36:03.57PMGMT$

The value has been rounded based on the assumption that the camera's clock was set accurately +/- 10ms and can maintain it time to that level of accuracy.

Frame rate, duration, and frame count

There are several ways you can think about multi-rate capture. Normally we think in time. As an example, a pitch consists of 2.5 second of wind up, 1.2 second to throw the ball, and then 3 seconds of follow through.

You can also think about multi-rate capture in terms of frames. The video will have 1280 x 720 size color frames. The camera tells me the 16GB memory will hold 13,806 frames. If a queue of 10 captures is needed (e.g. shot count equals 10), then each capture will have at most 1380 frames. With three capture rates available, if the wind up and follow through are 60 fps for 2.5 and 3 seconds respectively, 60 * (2.5 + 3) equals 330 frames, leaving 1380 - 330 frames or 1,050 frames for capturing the pitch. With a

Configuring multi-rate capture

Capturing multi-rate videos

Once you have configured multi-rate capture, actually capturing the video is easy; simple trigger the camera. Based on your save mode and multi-shot settings, you can either review the captured video or continue to capture more videos.

Implementation details

Waterfall configuration model

The requested values the user enters into the webUI settings modal follow a waterfall configuration style where a requested value specified higher in the list can change the allowed value of a configuration setting lower in the list.

The camera processes the parameters that affect how a video is captured in the following order:

• Sensitivity
• Shutter (exposure)
• Pre-trigger frame rate
• Post1-trigger frame rate
• Post2-trigger frame rate
• Horizontal resolution
• Vertical resolution
• Pre-trigger duration
• Post1-trigger duration
• Post2-trigger duration
• Shot count
• Overclocking
• Sub-Sampling
• Genlock

Handling unspecified configuration values

If you have a setting lower in the waterfall that is more important to you that a value higher in the waterfall, you leave the requested values higher in the waterfall blank/empty/unspecified and then the camera will provide allowed values, doing its best to meet the values you did specify, limited by the camera's hardware capabilities.

For multi-rate capture, the following table describes how unspecified requested values affect allowed values.

Pre-trigger frame rate	Post1-trigger frame rate	Post2-trigger frame rate	Horizontal resolution	Vertical resolution	Pre-trigger duration	Post1-trigger duration	Post2-trigger duration	Shot count	Description
blank	any	any	any	any	non-zero	any	any	any	Fastest possible frame rate is used for allowed pre-trigger
any	blank	any	any	any	any	non-zero	any	any	Allowed pre-trigger frame rate is used for post1-trigger
any	any	blank	any	any	any	any	non-zero	any	Allowed post1-trigger frame rate is used for post2-trigger

Backward compatibility

The common camera configuration supports specifying a single frame rate, duration, and pre-trigger percentage. When in advanced camera configuration those three common capture configuration parameters are not used. Instead advanced camera configuration displays multi-rate capture settings, which consists of a three sets of (frame rate, duration) values, one set for each of of the pre-trigger, post1-trigger and post2-trigger multi-rate capture regions.

The camera determines if it is in backward compatibility mode by checking if any of the configure_camera() or run() CAMAPI dictionary parameters keys frame_rate, duration, or pretrigger are defined. If any of those three keys are defined, then the camera will be calculate the allowed values based on those keys. When in backward compatibility mode, the six keys related to multi-rate capture are ignored.