Linux kernel

On the 22 of August 2009 an unofficial meeting took place with the purpose to discuss a new RFC for a new V4L API. Participants were (alphabetically sorted): Hans de Goede, Laurent Pinchart, Hans Verkuil and myself. The main discussion of this API will take place during the Linux Plumbers Conference, but since not everyone, interested in this development, including the author of these pages, will be able to attend the conference, it has been decided to hold a small unofficial preliminary meeting on this topic. The following implications on the soc-camera migration to v4l2-subdev have been decided upon:

1. We will have to remove bus-parameter auto-negotiation and force every board to specify a complete list of bus parameters, like signal polarities and sensing edge.
   
2. Will consider using the .notify() member of struct v4l2_device in sensor drivers, that change their output pixel clock frequency dynamically, e.g., depending on the frame format, instead of using the current sense object.
   
3. Maybe use a list of video subdevices as a parameter to soc-camera host platform device to avoid allocating otherwise "useless" per-source platform devices.
   
4. Use a new enumeration for various pixel data representations on the video source - video host data bus, similar to the current fourcc codes for the in-memory presentation of pixel formats.

We will have to give a bit more thinking to items 2 and 3 though.

All soc-camera drivers have been updated to comply with the V4L2 API scaling and cropping definitions. The main differences to the original soc-camera implementation are:

1. VIDIOC_S_CROP ioctl now accepts rectangle parameters in sensor input window pixel units
   
2. VIDIOC_S_CROP ioctl now tries to preserve scales, and not the output window
   
3. VIDIOC_S_FMT ioctl now preserves the sensor input window
   

The patch stack is uploaded here and is based on the 2.6.31-rc4 kernel.

A short announcement: my talk about the soc-camera framework has been accepted for the ELC-E 2009 conference in October 2009 in Grenoble, France. So, maybe, see you there.

While adding more hardware support and functionality to soc-camera, it became apparent, that its cropping / scaling concept was wrong. Trying to fix the situation by reading the API specs and trying to understand it revealed, that the documentation is indeed quite cumbersome and that even experienced V4L kernel developers do not understand completely how VIDIOC_S_CROP and VIDIOC_S_FMT ioctls shall configure cropping and scaling in camera drivers. As the situation slowly clarifies all soc-camera drivers and the core have to be modified to comply with an interpretation of the V4L2 API, which will hopefully result from the currently running on the mailing list discussion.

Porting soc-camera with all its drivers and platforms to the v4l2-subdev framework confirmed to be a complicated task. Having prepared the patches in a way to make changes as isolated as possible, they now have to be applied to the mainline in a certain order avoiding any intermittent breakage. After the 2.6.30 kernel has been released earlier this week, two first conversion steps shall be pushed upstream during the 2.6.31 merge window: a preparatory forward-compatibility patch to the soc-camera core, and multiple patches to convert all platforms currently in the mainline to the future platform-binding scheme. The actual incomplete conversion patches will then have to wait until 2.6.32. Then we will be completing the conversion some time in the 2.6.33 timeframe. For details see my post at the linux-media mailing list.

Following the "01/01: soc-camera: prepare for the platform driver conversion" patch from 07.05 6 further patches have been submitted to ARM and SuperH trees to convert all 5 currently in the mainline soc-camera platforms to the new scheme.

After that the platform-driver and v4l2-subdev conversion patches have been updated, posted to the linux-media mailing list on 15.05 and uploaded to http://download.open-technology.de/soc-camera/20090515/. See this thread for a detailed description.

Following patches have been pushed to v4l-dvb hg:

01/02: soc-camera: Fix section mismatch warning
02/02: soc_camera: depends on I2C

Following patch has been pushed to v4l-dvb hg:

01/01: soc-camera: prepare for the platform driver conversion

Following patches have been pushed to v4l-dvb hg:

01/08: ARM: update pcm990-baseboard.c to match mainline
02/08: soc-camera: add a free_bus method to struct soc_camera_link
03/08: soc-camera: host-driver cleanup
04/08: soc-camera: remove an extra device generation from struct soc_camera_host
05/08: soc-camera: simplify register access routines in multiple sensor drivers
06/08: soc-camera: link host drivers after clients
07/08: mx3_camera: Fix compilation with CONFIG_PM
08/08: pxa_camera: Documentation of the FSM

Completed step 1 of soc-camera conversion to the v4l2 framework. This means, all drivers and platforms have been converted to a scheme, where camera instances are registered by platforms as platform devices, and the soc-camera registers I2C devices later (see also my earlier posts). I also removed struct device from struct soc_camera_host, making camera instances direct children of host platform devices. In the meantime soc-camera framework covers four architectures: PXA270, SH7722, i.MX31, and i.MX1, is used by 6 in-tree platforms: pcm037, pcm990, em-x270, mioa701, ap325rxa, migor, and supports 6 client devices: mt9m001, mt9m111, mt9t031, mt9v022, ov772x, tw9910.

Following patches have been pushed:

01/09: pxa_camera: Enforce YUV422P frame sizes to be 16 multiples
02/09: pxa_camera: Remove YUV planar formats hole
03/09: pxa_camera: Redesign DMA handling
04/09: pxa_camera: Fix overrun condition on last buffer
05/09: pxa-camera: simplify the .buf_queue path by merging two loops
06/09: ov772x: wrong pointer for soc_camera_link is modified
07/09: soc-camera: fix breakage caused by 1fa5ae857bb14f6046205171d98506d8112dd74e
08/09: mt9m001: fix advertised pixel clock polarity
09/09: ov772x: add edge contrl support

The first step of the soc-camera framework conversion to the v4l2-device API is the transformation of the soc-camera core driver to a platform driver. This way the bond between soc-camera and single (sensor, decoder,...) I2C device drivers is becoming weaker and the double probing of those I2C drivers is eliminated.

The reason for the double probing previously has been, that many camera sensors have their I2C interfaces disabled in the absence of the master clock. Therefore the actual I2C probing routine could not access the chip and only registered the device with the soc-camera framework and reported success unconditionally. Then, when the soc-camera framework found a match between a camera device and a host, it registered respective devices and the second stage probing has been initiated, which switched the master clock of the camera interface on and made the actual I2C probing possible.

With the new platform-device approach first soc-camera core probe method is called, which switches the master clock on, and when the camera device driver becomes available, its I2C probe method is called and it can immediately access device I2C registers and perform the actual device identification and initialisation.

The first part of this conversion has been to convert a specific configuration to this scheme. The chosen configuration is the i.MX31 host and a MT9T031 camera, which have been successfully converted and an RFC patch has been posted to the V4L mailing list.

Following patches have been pushed:

01/12: soc-camera: separate S_FMT and S_CROP operations
02/12: soc-camera: configure drivers with a default format on open
03/12: sh-mobile-ceu-camera: set field to the value, configured at open()
04/12: soc-camera: configure drivers with a default format at probe time
05/12: ov772x: use soft sleep mode in stop_capture
06/12: video: use videobuf_waiton() in sh_mobile_ceu free_buffer()
07/12: soc-camera: add board hook to specify the buswidth for camera sensors
08/12: pcm990 baseboard: add camera bus width switch setting
09/12: mt9m001: allow setting of bus width from board code
10/12: mt9v022: allow setting of bus width from board code
11/12: soc-camera: remove now unused gpio member of struct soc_camera_link
12/12: mt9t031 bugfix

I originally developed the soc-camera V4L2 API in 2007-2008 as a part of a customer project, in 2008 I became the maintainer of the API in the Linux kernel. This API supports video interfaces on various Systems on Chip (SoC) and multiple video sources like CMOS sensors, TV-decoders. This API is in the mainline Linux kernel since version 2.6.25 (released in July 2008), it currently supports PXA270, SH7722, and i.MX31 SoCs, various video source devices from Aptina (formerly Micron), OmniVision, Techwell, out-of-tree patches for i.MX27 and PXA310 SoCs exist. The typical use of this API is on embedded video-enabled devices running Linux.

After the completion of the original customer project, development and maintenance of the API continued on an unpaid voluntary basis. As the API evolves and support for more hardware platforms and software standards is added work on supporting the API grows and it is becoming increasingly difficult to continue the work in my free time.

One of the bigger pending tasks in this area is integrating the soc-camera API with the new v4l2-device framework. This integration would allow seamless reuse of drivers for video data source components between embedded systems and drivers for USB and PCI video cards. For example, a USB camera using an MT9M111 sensor from Aptina would be able to use the same driver as an embedded PXA270 system with this sensor directly connected to the SoC.

Other planned tasks include support for the AVR32 (AP700x) and AT9 SoCs from Atmel, other PXA SoCs from Marvell, support for more features in existing sensor drivers. Maintenance of the API, which includes review and integration of patches submitted by single authors, keeping in sync with constantly developing kernel and video APIs is also taking a considerable amount of time. For example, soc-camera contribution to the 2.6.28 Linux kernel consisted of about 40 patches.

This increasing amount of work motivated me to look for sponsors. Sponsorship can be performed on a task or time basis. A sponsor can volunteer to finance the whole soc-camera work based on my reports and bills, describing work performed and hours spent. Sponsorship offers for specific parts of the work or with limited budgets would also be gratefully considered. Specific Linux kernel and user-space system development orders in video or other areas are also possible. Offers can be directed to web@liakhovetski.de.