Extending edgertronic capabilities - Customizing video processing

From edgertronic high speed video camera
Jump to: navigation, search


Home

Mp4 video file format support




If you are developing software to control the camera, the following information describes how to supply your own video preview and video encode pipelines. Future versions of the edgertronic camera software may not be compatible with the pipeline files syntax defined below.

You can also change overlay font size, positioning, and color in a user supplied pipeline file.

Pipeline file

A pipeline file is a text file containing an overlay section and one or more video pipeline sections each containing five strings. Those strings control how the camera captures, streams, and saves video frames. You can have as many video pipeline sections as you like, with each section having five name:value entries. You can also have more than one pipeline file as long as each file has a unique name. The Web UI custom encoding quality setting will select the file named pipelines_custom.txt and will select the [DEFAULT] named video pipeline section.

The format of the text file complies with the python configuration file format. In summary, a python configuration file consists of sections, led by a [section] header and followed by name=value entries.

The camera uses the Open Source GStreamer audio video streaming middleware. The contents of the pipeline file video pipeline section are details used by GStreamer to construct the video streams. There is also a human readable description that is put in the metadata file so you know which pipeline was used for each captured video.

Each video pipeline section need the following named entries:

Name Description
description Text description made available via CAMAPI get_camstatus() and also stored in the video metadata file.
source_format Text description of the GStreamer capabilities for the video frame format.
movie_extension Text string that is appended to the end of the movie filename. Typical value is mov
preview_during_save Set to either true or false. Default is true - preview during save is enabled. The idea was to reduce the time to save a video, but disabling JPEG preview during save only reduces save time by a couple of percent.
source_bin GStreamer pipeline segment for acquiring frames and doing initial processing
stream_bin GStreamer pipeline segment used for live preview
save_bin GStreamer pipeline segment for encoding video frames, putting them in a multiplexer and saving to storage.

GStreamer uses pipelines to stream data from a source element to a sink element. The edgertronic camera bundles several elements together into a GStreamer bin. These bins are connected in a pipeline as follows:

  • During live preview and video capture:
 source_bin ! stream_bin
  • During video save:
 source_bin ! tee name=srctee ! stream_bin .srctee ! save_bin

which is GStreamer syntax for saying for live preview the source bin is connected to the stream bin and during save the source bin is connected to both the stream_bin and the save_bin (using a tee element).

Other logic in the camera is controlling if the source element (v4l2src in this case) is getting frames from the high speed video sensor for live preview or from video frames previously saved in the DDR3 captured video memory when saving a captured video.

Example pipeline file

The following shows the pipelines_custom.txt file that ships with the camera. It contains both the [OVERLAY] section and [DEFAULT] section.

# Sanstreak Corporation provides this source code solely on the terms and
# conditions set forth in the edgertronic by Sanstreak Usage Agreement and
# on the condition that you accept and comply with them.  You may view the
# most recent version of the Usage Agreement at:
# http://wiki.edgertronic.com/index.php/Usage_Agreement.
[OVERLAY]

overlay_notes = text-offseth=10 text-offsetv=&y text-font-height=&F text-color=0xFFFFFF text-border=0x000000

overlay_settings = text2-offseth=10 text2-offsetv=&y text2-font-height=&F text2-color=0xFFFFFF text2-border=0x000000 text2=ISO:\ &i\ EXP:\ &s\ FPS:\ &f

overlay_frame_number = frame-offseth=10 frame-offsetv=&y frame-font-height=&F frame-color=0xFFFFFF frame-border=0x000000 frame-number=-100

overlay_logo = logo-offseth=-10 logo-offsetv=-10 logo-transparency=false logo-scale=&I logo-autofit=true logo=/usr/share/edgertronic/edgertronic-logo.png

[DEFAULT]
description = Customer supplied pipeline

source_format = video/x-raw-yuv,format=(fourcc)NV12

source_bin = v4l2src always-copy=false name=camera queue-size=8 ! capsfilter name=filter ! dmaiaccel ! emboverlay name=overlay ! tee name=srctee

stream_bin = queue ! dmaienc_jpeg qValue=75 ! multifilesink next-file=0 async=false sync=false location=/tmp/images/image.jpg

save_bin = queue name=video-queue ! dmaienc_h264 encodingpreset=2 ratecontrol=4 intraframeinterval=15 idrinterval=15 entropy=1 qpintra=23 qpinter=23 outputBufferSize=8294400 fixTimestamp=true single-nalu=true ! qtmux name=mux0 ! filesink name=fsink location=/video.mov async=false

Required element naming

During runtime, the camera changes various GStreamer element properties. This is done using an element naming convention:

Element
Name
Element
Property
Setting
overlay [1] Many settings as described in the emboverlay section.
srctee Used to connect video frames to the stream and save bins.
filter caps Change frame resolution.
camera Used to flush the pipeline.
mux0 Lock the multiplexer while a file is open.
fsink location Set filename

When creating or modifying a pipeline file, you must us the element naming convention listed above.

Notes

  • Normally GStreamer runs synchronously. Because of the capture then save video acquisition model supported by the camera, GStreamer buffer timing is meaningless. the video file is encoded with a 30 fps frame rate.

CAMAPI pipeline file support

You can select a pipeline from the set of factory provided pipelines or you provide your own. The most common change is simply modifying the H.264 encoder setting properties.

Start by looking at both the factory provided pipelines http://10.11.12.13/static/etc/pipelines_factory.txt and a sample user provided custom pipeline http://10.11.12.13/static/sdk/pipelines/pipelines_custom.txt

Once you have your custom pipeline file, you can save it on a storage device (such as USB hard drive or SD card) and make that storage device available to the camera.

The camera supports installing a variety of Configuration Files including the pipelines file. The pipeline file will be copied into the camera's non-volatile storage area when you power on the camera. You can see all the installed pipeline files by

browsing to http://10.11.12.13/static/etc/pipelines.

replacing 10.11.12.13 with your camera's IP address as necessary.

Pipeline file names should follow the convention pipelines_*.txt where the asterisk can be any descriptive word you choose.

Overlay support

There is an OVERLAY section in the pipelines file containing four keys:

Key Camera Setting Control Description
overlay_notes requested_overlay_notes Mapping to emboverlay element text overlay string.
overlay_settings requested_overlay_settings Mapping to emboverlay element text2 overlay string.
overlay_frame_number requested_overlay_frame Mapping to emboverlay element frame overlay string.
overlay_logo requested_overlay_logo Mapping to emboverlay element logo overlay graphic.

Repeating the able [OVERLAY] section from pipelines_custom.txt with the text containing substitution parameters is shown below:

overlay_notes = text-offseth=10 text-offsetv=&y text-font-height=&F text-color=0xFFFFFF text-border=0x000000

overlay_settings = text2-offseth=10 text2-offsetv=&y text2-font-height=&F text2-color=0xFFFFFF text2-border=0x000000 text2=ISO:\ &i\ EXP:\ &s\ FPS:\ &f

overlay_frame_number = frame-offseth=10 frame-offsetv=&y frame-font-height=&F frame-color=0xFFFFFF frame-border=0x000000 frame-number=-100

overlay_logo = logo-offseth=-10 logo-offsetv=-10 logo-transparency=false logo-scale=&I logo-autofit=true logo=/usr/share/edgertronic/edgertronic-logo.png

The full list of substitution parameters, including the special &F and &y', which only make sense to use with text font height and vertical offset is show below.

Variable Camera Setting
&d Trigger date (DD-MMM-YY)
&t Trigger time (HH-MM-SS)
&T Trigger time (seconds since Jan 1, 1970)
&i Sensitivity (ISO)
&s Shutter
&f Frame rate
&S Sub-sampling
&o Overclock
&e Extended dynamic range
&g Genlock
&b Multishot buffer
&n Serial number
&N Name of file
&F Scaled font size
&y Camera calculated y (vertical) position

Camera operation

You activate a section in a pipeline file using the CAMAPI run() method. Example:

 requested["requested_pipeline"] = "H264HIGH:pipelines_factory.txt"
 allowed = cam.configure_camera(requested)
 ret = cam.run(allowed)

 # ...

 requested["requested_pipeline"] = "DEFAULT:pipelines_custom.txt"
 allowed = cam.configure_camera(requested)
 ret = cam.run(allowed)

Where H264HIGH is the section name and pipelines_factory.txt is the name of the file. If no path is specified, then the camera will look for the pipeline file in the /etc/pipelines directory.

If the camera is unable to use the requested pipeline, say due to a typo in the pipelines file, then the camera will use the H264HIGH:pipelines_factory.txt pipeline.

Metadata pipeline information

The description text for the pipeline from the section in the pipeline file that was activated is saved in the metadata file. The line it the metadata file is labelled Active pipeline.

emboverlay element

The following is the output of gst-inspect emboverlay listing the supported element parameters.

  font                : Font typeface such as Sans
                        flags: readable, writable
                        String. Default: null Current: "Sans"
  logo                : Overlayed image path, must be a png file
                        flags: readable, writable
                        String. Default: null Current: null
  logo-offseth        : Overlayed image horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  logo-offsetv        : Overlayed image vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  logo-transparency   : Enable or disable logo transparency.
                        flags: readable, writable
                        Boolean. Default: false Current: false
  logo-scale          : Scale factor for overlay image.  Scale less than 1.0 shrinks image, range 0.1 to 10.0
                        flags: readable, writable
                        Float. Range:             0.1 -              10 Default:               1 Current:               1
  logo-autofit        : Enable or disable auto logo placement so it is fully visible
                        flags: readable, writable
                        Boolean. Default: true Current: false
  logo-enable         : Enable or disable logo rendering
                        flags: readable, writable
                        Boolean. Default: false Current: false
  text                : Overlayed text to display
                        flags: readable, writable
                        String. Default: null Current: null
  text-offseth        : Overlayed text horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-offsetv        : Overlayed text vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-font-height    : Overlayed text font height
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-color          : Overlayed text color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-border         : Overlayed text border color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-enable         : Enable or disable text rendering
                        flags: readable, writable
                        Boolean. Default: false Current: false
  text2               : Overlayed text2 to display
                        flags: readable, writable
                        String. Default: null Current: null
  text2-offseth       : Overlayed text2 horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text2-offsetv       : Overlayed text2 vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text2-font-height   : Overlayed text2 font height
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text2-color         : Overlayed text2 color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text2-border        : Overlayed text2 border color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text2-enable        : Enable or disable text2 rendering
                        flags: readable, writable
                        Boolean. Default: false Current: false
  time-pattern        : Overlayed time pattern %b %d %Y; %H:%M:%S will render: month day year; hour:minutes:seconds
                        flags: readable, writable
                        String. Default: null Current: null
  time-offseth        : Overlayed time horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-offsetv        : Overlayed time vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-font-height    : Overlayed time font size
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-color          : Overlayed time font color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-border         : Overlayed time border color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-enable         : Enable or disable time rendering
                        flags: readable, writable
                        Boolean. Default: false Current: false
  frame-number        : Starting frame number to overlay
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 1
  frame-offseth       : Overlayed frame horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  frame-offsetv       : Overlayed frame vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  frame-font-height   : Overlayed frame font size
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  frame-color         : Overlayed frame font color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  frame-border        : Overlayed frame border color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  frame-enable        : Enable or disable frame rendering
                        flags: readable, writable
                        Boolean. Default: false Current: false

Debugging GStreamer pipelines

Telnet into the camera and enable GStreamer debug output using


GST_DEBUG=GST_REFCOUNTING:5 app

To see what GST_DEBUG settings are possible, run

gst-launch --gst-debug-help

The GStreamer debugging levels include:

0 none No debug information is output.
1 ERROR Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error.
2 WARNING Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen.
3 INFO Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level.
4 DEBUG Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, change of parameters.
5 LOG Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions.
9 buffer
dump
Hex dump of buffer contents.

Monitoring pipeline performance with dmaiperf

Change the save bin by adding dmaiperf similar to qtmux name=mux0 ! dmaiperf ! filesink. You can do this via

rw # make root file system read/write
vi /etc/pipelines_factory.txt

Then run the camera application in the foreground

GST_DEBUG=dmaiperf:5 app

Trigger the camera and when the save starts, you should see console debug output similar to:

DEBUG:sms.py:313: bus message: <gst.Message GstMessageInfo, gerror=(GError)NULL, debug=(string)"Timestamp:\ 0:51:43.534209280\;\ bps:\ 279824\;\ fps:\ 54.72\;\ "; from dmaiperf0 at 0x44c8d8>.





Home

Mp4 video file format support