Extending edgertronic capabilities - Customizing video processing
| ← |
⌂ Home |
→ Mp4 video file format support |
If you are developing software to control the camera, the following information describes how to supply your own video preview and video encode pipelines. Future versions of the edgertronic camera software may not be compatible with the pipeline files syntax defined below.
You can also change overlay font size, positioning, and color in a user supplied pipeline file.
Pipeline file
A pipeline file is a text file containing an overlay section and one or more video pipeline sections each containing five strings. Those strings control how the camera captures, streams, and saves video frames. You can have as many video pipeline sections as you like, with each section having five name:value entries. You can also have more than one pipeline file as long as each file has a unique name. The Web UI custom encoding quality setting will select the file named pipelines_custom.txt and will select the [DEFAULT] named video pipeline section.
The format of the text file complies with the python configuration file format. In summary, a python configuration file consists of sections, led by a [section] header and followed by name=value entries.
The camera uses the Open Source GStreamer audio video streaming middleware. The contents of the pipeline file video pipeline section are details used by GStreamer to construct the video streams. There is also a human readable description that is put in the metadata file so you know which pipeline was used for each captured video.
Each video pipeline section need the following named entries:
| Name | Description |
|---|---|
| description | Text description made available via CAMAPI get_camstatus() and also stored in the video metadata file. |
| source_format | Text description of the GStreamer capabilities for the video frame format. |
| movie_extension | Text string that is appended to the end of the movie filename. Typical value is mov |
| preview_during_save | Set to either true or false. Default is true - preview during save is enabled. The idea was to reduce the time to save a video, but disabling JPEG preview during save only reduces save time by a couple of percent. |
| source_bin | GStreamer pipeline segment for acquiring frames and doing initial processing |
| stream_bin | GStreamer pipeline segment used for live preview |
| save_bin | GStreamer pipeline segment for encoding video frames, putting them in a multiplexer and saving to storage. |
GStreamer uses pipelines to stream data from a source element to a sink element. The edgertronic camera bundles several elements together into a GStreamer bin. These bins are connected in a pipeline as follows:
- During live preview and video capture:
source_bin ! stream_bin
- During video save:
source_bin ! tee name=srctee ! stream_bin .srctee ! save_bin
which is GStreamer syntax for saying for live preview the source bin is connected to the stream bin and during save the source bin is connected to both the stream_bin and the save_bin (using a tee element).
Other logic in the camera is controlling if the source element (v4l2src in this case) is getting frames from the high speed video sensor for live preview or from video frames previously saved in the DDR3 captured video memory when saving a captured video.
Example pipeline file
The following shows the pipelines_custom.txt file that ships with the camera. It contains both the [OVERLAY] section and [DEFAULT] section.
# Sanstreak Corporation provides this source code solely on the terms and # conditions set forth in the edgertronic by Sanstreak Usage Agreement and # on the condition that you accept and comply with them. You may view the # most recent version of the Usage Agreement at: # http://wiki.edgertronic.com/index.php/Usage_Agreement. [OVERLAY] overlay_notes = text-offseth=10 text-offsetv=&y text-font-height=&F text-color=0xFFFFFF text-border=0x000000 overlay_settings = text2-offseth=10 text2-offsetv=&y text2-font-height=&F text2-color=0xFFFFFF text2-border=0x000000 text2=ISO:\ &i\ EXP:\ &s\ FPS:\ &f overlay_frame_number = frame-offseth=10 frame-offsetv=&y frame-font-height=&F frame-color=0xFFFFFF frame-border=0x000000 frame-number=-100 overlay_logo = logo-offseth=-10 logo-offsetv=-10 logo-transparency=false logo-scale=&I logo-autofit=true logo=/usr/share/edgertronic/edgertronic-logo.png [DEFAULT] description = Customer supplied pipeline source_format = video/x-raw-yuv,format=(fourcc)NV12 source_bin = v4l2src always-copy=false name=camera queue-size=8 ! capsfilter name=filter ! dmaiaccel ! emboverlay name=overlay ! tee name=srctee stream_bin = queue ! dmaienc_jpeg qValue=75 ! multifilesink next-file=0 async=false sync=false location=/tmp/images/image.jpg save_bin = queue name=video-queue ! dmaienc_h264 encodingpreset=2 ratecontrol=4 intraframeinterval=15 idrinterval=15 entropy=1 qpintra=23 qpinter=23 outputBufferSize=8294400 fixTimestamp=true single-nalu=true ! qtmux name=mux0 ! filesink name=fsink location=/video.mov async=false
Required element naming
During runtime, the camera changes various GStreamer element properties. This is done using an element naming convention:
| Element Name |
Element Property |
Setting |
|---|---|---|
| overlay | [1] | Many settings as described in the emboverlay section. |
| srctee | Used to connect video frames to the stream and save bins. | |
| filter | caps | Change frame resolution. |
| camera | Used to flush the pipeline. | |
| mux0 | Lock the multiplexer while a file is open. | |
| fsink | location | Set filename |
When creating or modifying a pipeline file, you must us the element naming convention listed above.
Notes
- Normally GStreamer runs synchronously. Because of the capture then save video acquisition model supported by the camera, GStreamer buffer timing is meaningless. the video file is encoded with a 30 fps frame rate.
CAMAPI pipeline file support
You can select a pipeline from the set of factory provided pipelines or you provide your own. The most common change is simply modifying the H.264 encoder setting properties.
Start by looking at both the factory provided pipelines http://10.11.12.13/static/etc/pipelines_factory.txt and a sample user provided custom pipeline http://10.11.12.13/static/sdk/pipelines/pipelines_custom.txt
Once you have your custom pipeline file, you can save it on a storage device (such as USB hard drive or SD card) and make that storage device available to the camera.
The camera supports installing a variety of Configuration Files including the pipelines file. The pipeline file will be copied into the camera's non-volatile storage area when you power on the camera. You can see all the installed pipeline files by
browsing to http://10.11.12.13/static/etc/pipelines.
replacing 10.11.12.13 with your camera's IP address as necessary.
Pipeline file names should follow the convention pipelines_*.txt where the asterisk can be any descriptive word you choose.
Overlay support
There is an OVERLAY section in the pipelines file containing four keys:
| Key | Camera Setting Control | Description |
|---|---|---|
| overlay_notes | requested_overlay_notes | Mapping to emboverlay element text overlay string. |
| overlay_settings | requested_overlay_settings | Mapping to emboverlay element text2 overlay string. |
| overlay_frame_number | requested_overlay_frame | Mapping to emboverlay element frame overlay string. |
| overlay_logo | requested_overlay_logo | Mapping to emboverlay element logo overlay graphic. |
Repeating the able [OVERLAY] section from pipelines_custom.txt with the text containing substitution parameters is shown below:
overlay_notes = text-offseth=10 text-offsetv=&y text-font-height=&F text-color=0xFFFFFF text-border=0x000000 overlay_settings = text2-offseth=10 text2-offsetv=&y text2-font-height=&F text2-color=0xFFFFFF text2-border=0x000000 text2=ISO:\ &i\ EXP:\ &s\ FPS:\ &f overlay_frame_number = frame-offseth=10 frame-offsetv=&y frame-font-height=&F frame-color=0xFFFFFF frame-border=0x000000 frame-number=-100 overlay_logo = logo-offseth=-10 logo-offsetv=-10 logo-transparency=false logo-scale=&I logo-autofit=true logo=/usr/share/edgertronic/edgertronic-logo.png
The full list of substitution parameters, including the special &F and &y', which only make sense to use with text font height and vertical offset is show below.
| Variable | Camera Setting |
|---|---|
| &d | Trigger date (DD-MMM-YY) |
| &t | Trigger time (HH-MM-SS) |
| &T | Trigger time (seconds since Jan 1, 1970) |
| &i | Sensitivity (ISO) |
| &s | Shutter |
| &f | Frame rate |
| &S | Sub-sampling |
| &o | Overclock |
| &e | Extended dynamic range |
| &g | Genlock |
| &b | Multishot buffer |
| &n | Serial number |
| &N | Name of file |
| &F | Scaled font size |
| &y | Camera calculated y (vertical) position |
Camera operation
You activate a section in a pipeline file using the CAMAPI run() method. Example:
requested["requested_pipeline"] = "H264HIGH:pipelines_factory.txt" allowed = cam.configure_camera(requested) ret = cam.run(allowed) # ... requested["requested_pipeline"] = "DEFAULT:pipelines_custom.txt" allowed = cam.configure_camera(requested) ret = cam.run(allowed)
Where H264HIGH is the section name and pipelines_factory.txt is the name of the file. If no path is specified, then the camera will look for the pipeline file in the /etc/pipelines directory.
If the camera is unable to use the requested pipeline, say due to a typo in the pipelines file, then the camera will use the H264HIGH:pipelines_factory.txt pipeline.
Metadata pipeline information
The description text for the pipeline from the section in the pipeline file that was activated is saved in the metadata file. The line it the metadata file is labelled Active pipeline.
emboverlay element
The following is the output of gst-inspect emboverlay listing the supported element parameters.
font : Font typeface such as Sans
flags: readable, writable
String. Default: null Current: "Sans"
logo : Overlayed image path, must be a png file
flags: readable, writable
String. Default: null Current: null
logo-offseth : Overlayed image horizontal offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
logo-offsetv : Overlayed image vertical offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
logo-transparency : Enable or disable logo transparency.
flags: readable, writable
Boolean. Default: false Current: false
logo-scale : Scale factor for overlay image. Scale less than 1.0 shrinks image, range 0.1 to 10.0
flags: readable, writable
Float. Range: 0.1 - 10 Default: 1 Current: 1
logo-autofit : Enable or disable auto logo placement so it is fully visible
flags: readable, writable
Boolean. Default: true Current: false
logo-enable : Enable or disable logo rendering
flags: readable, writable
Boolean. Default: false Current: false
text : Overlayed text to display
flags: readable, writable
String. Default: null Current: null
text-offseth : Overlayed text horizontal offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text-offsetv : Overlayed text vertical offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text-font-height : Overlayed text font height
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text-color : Overlayed text color. RGB format 0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text-border : Overlayed text border color. RGB format 0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text-enable : Enable or disable text rendering
flags: readable, writable
Boolean. Default: false Current: false
text2 : Overlayed text2 to display
flags: readable, writable
String. Default: null Current: null
text2-offseth : Overlayed text2 horizontal offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text2-offsetv : Overlayed text2 vertical offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text2-font-height : Overlayed text2 font height
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text2-color : Overlayed text2 color. RGB format 0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text2-border : Overlayed text2 border color. RGB format 0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
text2-enable : Enable or disable text2 rendering
flags: readable, writable
Boolean. Default: false Current: false
time-pattern : Overlayed time pattern %b %d %Y; %H:%M:%S will render: month day year; hour:minutes:seconds
flags: readable, writable
String. Default: null Current: null
time-offseth : Overlayed time horizontal offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
time-offsetv : Overlayed time vertical offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
time-font-height : Overlayed time font size
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
time-color : Overlayed time font color. RGB format color=0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
time-border : Overlayed time border color. RGB format color=0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
time-enable : Enable or disable time rendering
flags: readable, writable
Boolean. Default: false Current: false
frame-number : Starting frame number to overlay
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 1
frame-offseth : Overlayed frame horizontal offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
frame-offsetv : Overlayed frame vertical offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
frame-font-height : Overlayed frame font size
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
frame-color : Overlayed frame font color. RGB format color=0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
frame-border : Overlayed frame border color. RGB format color=0xRRGGBB
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
frame-enable : Enable or disable frame rendering
flags: readable, writable
Boolean. Default: false Current: false
Debugging GStreamer pipelines
Telnet into the camera and enable GStreamer debug output using
GST_DEBUG=GST_REFCOUNTING:5 app
To see what GST_DEBUG settings are possible, run
gst-launch --gst-debug-help
The GStreamer debugging levels include:
| 0 | none | No debug information is output. |
| 1 | ERROR | Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. |
| 2 | WARNING | Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. |
| 3 | INFO | Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. |
| 4 | DEBUG | Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, change of parameters. |
| 5 | LOG | Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions. |
| 9 | buffer dump |
Hex dump of buffer contents. |
Monitoring pipeline performance with dmaiperf
Change the save bin by adding dmaiperf similar to qtmux name=mux0 ! dmaiperf ! filesink. You can do this via
rw # make root file system read/write vi /etc/pipelines_factory.txt
Then run the camera application in the foreground
GST_DEBUG=dmaiperf:5 app
Trigger the camera and when the save starts, you should see console debug output similar to:
DEBUG:sms.py:313: bus message: <gst.Message GstMessageInfo, gerror=(GError)NULL, debug=(string)"Timestamp:\ 0:51:43.534209280\;\ bps:\ 279824\;\ fps:\ 54.72\;\ "; from dmaiperf0 at 0x44c8d8>.
| ← |
⌂ Home |
→ Mp4 video file format support |