342 lines
12 KiB
ReStructuredText
342 lines
12 KiB
ReStructuredText
.. SPDX-License-Identifier: GPL-2.0
|
||
|
||
Media Controller devices
|
||
------------------------
|
||
|
||
Media Controller
|
||
~~~~~~~~~~~~~~~~
|
||
|
||
The media controller userspace API is documented in
|
||
:ref:`the Media Controller uAPI book <media_controller>`. This document focus
|
||
on the kernel-side implementation of the media framework.
|
||
|
||
Abstract media device model
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
Discovering a device internal topology, and configuring it at runtime, is one
|
||
of the goals of the media framework. To achieve this, hardware devices are
|
||
modelled as an oriented graph of building blocks called entities connected
|
||
through pads.
|
||
|
||
An entity is a basic media hardware building block. It can correspond to
|
||
a large variety of logical blocks such as physical hardware devices
|
||
(CMOS sensor for instance), logical hardware devices (a building block
|
||
in a System-on-Chip image processing pipeline), DMA channels or physical
|
||
connectors.
|
||
|
||
A pad is a connection endpoint through which an entity can interact with
|
||
other entities. Data (not restricted to video) produced by an entity
|
||
flows from the entity's output to one or more entity inputs. Pads should
|
||
not be confused with physical pins at chip boundaries.
|
||
|
||
A link is a point-to-point oriented connection between two pads, either
|
||
on the same entity or on different entities. Data flows from a source
|
||
pad to a sink pad.
|
||
|
||
Media device
|
||
^^^^^^^^^^^^
|
||
|
||
A media device is represented by a struct media_device
|
||
instance, defined in ``include/media/media-device.h``.
|
||
Allocation of the structure is handled by the media device driver, usually by
|
||
embedding the :c:type:`media_device` instance in a larger driver-specific
|
||
structure.
|
||
|
||
Drivers initialise media device instances by calling
|
||
:c:func:`media_device_init()`. After initialising a media device instance, it is
|
||
registered by calling :c:func:`__media_device_register()` via the macro
|
||
``media_device_register()`` and unregistered by calling
|
||
:c:func:`media_device_unregister()`. An initialised media device must be
|
||
eventually cleaned up by calling :c:func:`media_device_cleanup()`.
|
||
|
||
Note that it is not allowed to unregister a media device instance that was not
|
||
previously registered, or clean up a media device instance that was not
|
||
previously initialised.
|
||
|
||
Entities
|
||
^^^^^^^^
|
||
|
||
Entities are represented by a struct media_entity
|
||
instance, defined in ``include/media/media-entity.h``. The structure is usually
|
||
embedded into a higher-level structure, such as
|
||
:c:type:`v4l2_subdev` or :c:type:`video_device`
|
||
instances, although drivers can allocate entities directly.
|
||
|
||
Drivers initialize entity pads by calling
|
||
:c:func:`media_entity_pads_init()`.
|
||
|
||
Drivers register entities with a media device by calling
|
||
:c:func:`media_device_register_entity()`
|
||
and unregistered by calling
|
||
:c:func:`media_device_unregister_entity()`.
|
||
|
||
Interfaces
|
||
^^^^^^^^^^
|
||
|
||
Interfaces are represented by a
|
||
struct media_interface instance, defined in
|
||
``include/media/media-entity.h``. Currently, only one type of interface is
|
||
defined: a device node. Such interfaces are represented by a
|
||
struct media_intf_devnode.
|
||
|
||
Drivers initialize and create device node interfaces by calling
|
||
:c:func:`media_devnode_create()`
|
||
and remove them by calling:
|
||
:c:func:`media_devnode_remove()`.
|
||
|
||
Pads
|
||
^^^^
|
||
Pads are represented by a struct media_pad instance,
|
||
defined in ``include/media/media-entity.h``. Each entity stores its pads in
|
||
a pads array managed by the entity driver. Drivers usually embed the array in
|
||
a driver-specific structure.
|
||
|
||
Pads are identified by their entity and their 0-based index in the pads
|
||
array.
|
||
|
||
Both information are stored in the struct media_pad,
|
||
making the struct media_pad pointer the canonical way
|
||
to store and pass link references.
|
||
|
||
Pads have flags that describe the pad capabilities and state.
|
||
|
||
``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.
|
||
``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.
|
||
|
||
.. note::
|
||
|
||
One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must
|
||
be set for each pad.
|
||
|
||
Links
|
||
^^^^^
|
||
|
||
Links are represented by a struct media_link instance,
|
||
defined in ``include/media/media-entity.h``. There are two types of links:
|
||
|
||
**1. pad to pad links**:
|
||
|
||
Associate two entities via their PADs. Each entity has a list that points
|
||
to all links originating at or targeting any of its pads.
|
||
A given link is thus stored twice, once in the source entity and once in
|
||
the target entity.
|
||
|
||
Drivers create pad to pad links by calling:
|
||
:c:func:`media_create_pad_link()` and remove with
|
||
:c:func:`media_entity_remove_links()`.
|
||
|
||
**2. interface to entity links**:
|
||
|
||
Associate one interface to a Link.
|
||
|
||
Drivers create interface to entity links by calling:
|
||
:c:func:`media_create_intf_link()` and remove with
|
||
:c:func:`media_remove_intf_links()`.
|
||
|
||
.. note::
|
||
|
||
Links can only be created after having both ends already created.
|
||
|
||
Links have flags that describe the link capabilities and state. The
|
||
valid values are described at :c:func:`media_create_pad_link()` and
|
||
:c:func:`media_create_intf_link()`.
|
||
|
||
Graph traversal
|
||
^^^^^^^^^^^^^^^
|
||
|
||
The media framework provides APIs to traverse media graphs, locating connected
|
||
entities and links.
|
||
|
||
To iterate over all entities belonging to a media device, drivers can use
|
||
the media_device_for_each_entity macro, defined in
|
||
``include/media/media-device.h``.
|
||
|
||
.. code-block:: c
|
||
|
||
struct media_entity *entity;
|
||
|
||
media_device_for_each_entity(entity, mdev) {
|
||
// entity will point to each entity in turn
|
||
...
|
||
}
|
||
|
||
Helper functions can be used to find a link between two given pads, or a pad
|
||
connected to another pad through an enabled link
|
||
(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`,
|
||
:c:func:`media_entity_remote_source_pad_unique()` and
|
||
:c:func:`media_pad_remote_pad_unique()`).
|
||
|
||
Use count and power handling
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
Due to the wide differences between drivers regarding power management
|
||
needs, the media controller does not implement power management. However,
|
||
the struct media_entity includes a ``use_count``
|
||
field that media drivers
|
||
can use to track the number of users of every entity for power management
|
||
needs.
|
||
|
||
The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by
|
||
media drivers and must not be
|
||
touched by entity drivers. Access to the field must be protected by the
|
||
:c:type:`media_device`.\ ``graph_mutex`` lock.
|
||
|
||
Links setup
|
||
^^^^^^^^^^^
|
||
|
||
Link properties can be modified at runtime by calling
|
||
:c:func:`media_entity_setup_link()`.
|
||
|
||
Pipelines and media streams
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
A media stream is a stream of pixels or metadata originating from one or more
|
||
source devices (such as a sensors) and flowing through media entity pads
|
||
towards the final sinks. The stream can be modified on the route by the
|
||
devices (e.g. scaling or pixel format conversions), or it can be split into
|
||
multiple branches, or multiple branches can be merged.
|
||
|
||
A media pipeline is a set of media streams which are interdependent. This
|
||
interdependency can be caused by the hardware (e.g. configuration of a second
|
||
stream cannot be changed if the first stream has been enabled) or by the driver
|
||
due to the software design. Most commonly a media pipeline consists of a single
|
||
stream which does not branch.
|
||
|
||
When starting streaming, drivers must notify all entities in the pipeline to
|
||
prevent link states from being modified during streaming by calling
|
||
:c:func:`media_pipeline_start()`.
|
||
|
||
The function will mark all the pads which are part of the pipeline as streaming.
|
||
|
||
The struct media_pipeline instance pointed to by the pipe argument will be
|
||
stored in every pad in the pipeline. Drivers should embed the struct
|
||
media_pipeline in higher-level pipeline structures and can then access the
|
||
pipeline through the struct media_pad pipe field.
|
||
|
||
Calls to :c:func:`media_pipeline_start()` can be nested.
|
||
The pipeline pointer must be identical for all nested calls to the function.
|
||
|
||
:c:func:`media_pipeline_start()` may return an error. In that case,
|
||
it will clean up any of the changes it did by itself.
|
||
|
||
When stopping the stream, drivers must notify the entities with
|
||
:c:func:`media_pipeline_stop()`.
|
||
|
||
If multiple calls to :c:func:`media_pipeline_start()` have been
|
||
made the same number of :c:func:`media_pipeline_stop()` calls
|
||
are required to stop streaming.
|
||
The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last
|
||
nested stop call.
|
||
|
||
Link configuration will fail with ``-EBUSY`` by default if either end of the
|
||
link is a streaming entity. Links that can be modified while streaming must
|
||
be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.
|
||
|
||
If other operations need to be disallowed on streaming entities (such as
|
||
changing entities configuration parameters) drivers can explicitly check the
|
||
media_entity stream_count field to find out if an entity is streaming. This
|
||
operation must be done with the media_device graph_mutex held.
|
||
|
||
Link validation
|
||
^^^^^^^^^^^^^^^
|
||
|
||
Link validation is performed by :c:func:`media_pipeline_start()`
|
||
for any entity which has sink pads in the pipeline. The
|
||
:c:type:`media_entity`.\ ``link_validate()`` callback is used for that
|
||
purpose. In ``link_validate()`` callback, entity driver should check
|
||
that the properties of the source pad of the connected entity and its own
|
||
sink pad match. It is up to the type of the entity (and in the end, the
|
||
properties of the hardware) what matching actually means.
|
||
|
||
Subsystems should facilitate link validation by providing subsystem specific
|
||
helper functions to provide easy access for commonly needed information, and
|
||
in the end provide a way to use driver-specific callbacks.
|
||
|
||
Pipeline traversal
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
Once a pipeline has been constructed with :c:func:`media_pipeline_start()`,
|
||
drivers can iterate over entities or pads in the pipeline with the
|
||
:c:macro:´media_pipeline_for_each_entity` and
|
||
:c:macro:´media_pipeline_for_each_pad` macros. Iterating over pads is
|
||
straightforward:
|
||
|
||
.. code-block:: c
|
||
|
||
media_pipeline_pad_iter iter;
|
||
struct media_pad *pad;
|
||
|
||
media_pipeline_for_each_pad(pipe, &iter, pad) {
|
||
/* 'pad' will point to each pad in turn */
|
||
...
|
||
}
|
||
|
||
To iterate over entities, the iterator needs to be initialized and cleaned up
|
||
as an additional steps:
|
||
|
||
.. code-block:: c
|
||
|
||
media_pipeline_entity_iter iter;
|
||
struct media_entity *entity;
|
||
int ret;
|
||
|
||
ret = media_pipeline_entity_iter_init(pipe, &iter);
|
||
if (ret)
|
||
...;
|
||
|
||
media_pipeline_for_each_entity(pipe, &iter, entity) {
|
||
/* 'entity' will point to each entity in turn */
|
||
...
|
||
}
|
||
|
||
media_pipeline_entity_iter_cleanup(&iter);
|
||
|
||
Media Controller Device Allocator API
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
When the media device belongs to more than one driver, the shared media
|
||
device is allocated with the shared struct device as the key for look ups.
|
||
|
||
The shared media device should stay in registered state until the last
|
||
driver unregisters it. In addition, the media device should be released when
|
||
all the references are released. Each driver gets a reference to the media
|
||
device during probe, when it allocates the media device. If media device is
|
||
already allocated, the allocate API bumps up the refcount and returns the
|
||
existing media device. The driver puts the reference back in its disconnect
|
||
routine when it calls :c:func:`media_device_delete()`.
|
||
|
||
The media device is unregistered and cleaned up from the kref put handler to
|
||
ensure that the media device stays in registered state until the last driver
|
||
unregisters the media device.
|
||
|
||
**Driver Usage**
|
||
|
||
Drivers should use the appropriate media-core routines to manage the shared
|
||
media device life-time handling the two states:
|
||
1. allocate -> register -> delete
|
||
2. get reference to already registered device -> delete
|
||
|
||
call :c:func:`media_device_delete()` routine to make sure the shared media
|
||
device delete is handled correctly.
|
||
|
||
**driver probe:**
|
||
Call :c:func:`media_device_usb_allocate()` to allocate or get a reference
|
||
Call :c:func:`media_device_register()`, if media devnode isn't registered
|
||
|
||
**driver disconnect:**
|
||
Call :c:func:`media_device_delete()` to free the media_device. Freeing is
|
||
handled by the kref put handler.
|
||
|
||
API Definitions
|
||
^^^^^^^^^^^^^^^
|
||
|
||
.. kernel-doc:: include/media/media-device.h
|
||
|
||
.. kernel-doc:: include/media/media-devnode.h
|
||
|
||
.. kernel-doc:: include/media/media-entity.h
|
||
|
||
.. kernel-doc:: include/media/media-request.h
|
||
|
||
.. kernel-doc:: include/media/media-dev-allocator.h
|