Image Processing and Generation from a Patent Perspective

by Michael M. Fischer (DE),

German and European Patent Attorney and Computer Scientist

With the advent of smart phones at the latest, the field of image processing and generation has become ubiquitous in our daily lives. Our smartphones recognize faces and improve colors if we take pictures on a dull day. While the field is interesting from a technical perspective last but not least by its different fields of industrial applications (e.g. finding the teats of a milk animal as in T 754/03) and its close relationship to the field of artificial intelligence and mathematics, especially the latter makes the field challenging for the patent practitioner. Inventions in the field of image processing and generation belong to the computer-implemented inventions which face many obstacles on both sides of the patent world: patentability and enforceability. The author of this article has dealt with hundreds of inventions in this field and wants to share his knowledge with the interested reader. The first part of the article defines the technical field, the second part deals with patentability before the EPO and the third part deals with the enforceability of patents in this technical field. The article is rounded off by a conclusion as a fourth part.

A. Nomenclature

(Digital) image processing means the use of computer algorithms to perform image processing on (digital) images. It typically deals with classification of images, feature extraction and pattern recognition. It typically uses digital image transformations (applying kernels, masks to images, etc.) as a first step to extract features from an image and then applies approaches from artificial intelligence (neural networks, etc.) for subsequent classification. In this article, the term "image processing" also encompasses the field of image compression which uses computer algorithms to reduce the cost for storage and transmission of images. The term "image generation" is sometimes synonymously used with the term computer graphics and denotes the creation/rendering (texturing, colouring, etc.) of images (or films) using a computer. It should be noted that the International Patent Classification makes a distinction between image processing (without compression) and generation under class G06T ("image data processing or generation, in general") on one hand and image compression, which is primarily dealt with under class H03M (coding, decoding or code conversion in general) and partly under class H04N (pictorial communication, e.g. television), on the other hand.[1]

B. Patentability before the EPO

Without any doubt, the milestone decision in the field of image processing is the VICOM decision (T 208/84) from the year 1986. The method relates to "a method of digitally processing images in the form of a two-dimensional data array having elements arranged in rows and columns in which an operator matrix of a size substantially smaller than the size of the data array is convolved with the data array...". In other words, the method relates to a typical image processing method (as described above) which applies a mathematical operator (a kernel) to an image which is defined in the form of a two-dimensional data array. The headnotes of the decision read:

"I. Even if the idea underlying an invention may be considered to reside in a mathematical method a claim directed to a technical process in which the method is used does not seek protection for the mathematical method as such.

II. A computer of known type set up to operate according to a new program cannot be considered as forming part of the state of the art as defined by Article 54(2) EPC.

III. A claim directed to a technical process which process is carried out under the control of a program (whether by means of hardware or software), cannot be regarded as relating to a computer program as such.

IV. A claim which can be considered as being directed to a computer set up to operate in accordance with a specified program (whether by means of hardware or software) for controlling or carrying out a technical process cannot be regarded as relating to a computer program as such."

Mathematical Methods & Programs for Computers

Headnote I , III and IV deal with two exclusions from patentability. Art. 52 (2) (a) EPC excludes mathematical methods and Art. 52 (2) (c) EPC excludes programs for computers from patentability. Art. 52(3) EPC clarifies that the exclusions only apply "to such subject-matter or activities as such". By these headnotes, decision T 208/84 overruled the negative decision of the Examining Division which refused the patent application and opened up a liberal patent practice in the field of image processing in particular and in the field of computer implemented inventions in general.

The Board held that a "method for digitally filtering data" remains an abstract notion not distinguished from a mathematical method so long as it is not specified what physical entity is represented by the data and forms the subject of a technical process". According to the Board, the image data in the form of a two-dimensional data array do not seem to be an abstract notion. When examining whether the invention related to a mathematical method or a computer program as such (which would be non-patentable), the Board saw the basic difference between a non-patentable mathematical or algorithm and a patentable technical process in the fact that a mathematical method or mathematical algorithm is merely carried out on numbers and provides a result in numerical form whereas, if a mathematical method is used within a technical process, that process is carried on a physical entity, in the specific case a stored image, by some technical means implementing the method and provides as its result a certain change in this physical entity. Although the author of this article agrees with the Board's point of view, an image could equally be regarded as an abstract entity since it is nothing else than stored pixel values (numbers).

Similarly, a method of encoding audio information in a communication system may aim to reduce distortion induced by channel noise. Although the idea underlying such a method may be considered to reside in a mathematical method, the encoding method as a whole is not a mathematical method as such, and hence is not excluded from patentability by Art. 52 (2) (a) and (3) EPC. A method of encrypting/decrypting or signing electronic communications may be regarded as a technical method, even if it is essentially based on a mathematical method (see T 1326/06).

In this context T 1227/05 should be mentioned which says that a procedural step (e.g. a mathematical algorithm) may contribute to the technical character of a claimed method only if it serves an adequately defined technical purpose of the method. In particular, specific technical applications of computer-implemented simulation methods, even if involving mathematical formulae, are to be regarded as modern technical methods which form an essential part of the fabrication process. Such simulation methods cannot be denied a technical effect merely on the ground that they do not yet incorporate the physical end product. However, the meta-specification of an undefined technical purpose (for example, the simulation of a "technical system"), could not be considered adequate.

It should be mentioned that the decision T 208/84 comes to the conclusion that the method defined in claim 1 is not barred from patent protection under Art. 52 (2) (a), (c) and 52 (3) EPC but does not deal with novelty and inventive step. This is due to the fact that the decision was rendered long before the COMVIK approach (T 641/00) which says that if a claim contains at least one technical feature, it is not excluded from patentability under Art. 52 EPC any more. One can, however, assume that the Board considered enhancing or restoring an image, without adding to its information content to be a technical effect.

Industrial Application

Although this was not disputed by the Examining Division or the Applicant, the Board held that claim 1 also complied with the requirement of industrial application (Art. 57 EPC) by saying:

"Clearly a method for obtaining and/or reproducing an image of a physical object or even an image of a simulated object (as in computer-aided design/computer-aided manufacturing (CAD/CAM) systems) may be used e.g. in investigating properties of the object or designing an industrial article and is therefore susceptible of industrial application. Similarly, a method for enhancing or restoring such an image, without adding to its informational content, has to be considered as susceptible of industrial application within the meaning of Article 57 EPC."

Moreover, the Board held that it does not make any difference whether an invention is embodied in hardware or in software:

"In arriving at this conclusion the Board has additionally considered that making a distinction between embodiments of the same invention carried out in hardware or in software is inappropriate as it can fairly be said that the choice between these two possibilities is not of an essential nature but is based on technical and economical considerations which bear no relationship to the inventive concept as such. Generally speaking, an invention which would be patentable in accordance with conventional patentability criteria should not be excluded from protection by the mere fact that for its implementation modern technical means in the form of a computer program are used. Decisive is what technical contribution the invention as defined in the claim when considered as a whole makes to the known art. Finally, it would seem illogical to grant protection for a technical process controlled by a suitably programmed computer but not for the computer itself when set up to execute the control."

Presentation of Information

Unfortunately, T 208/84 and other decisions in the field of image processing and generation do not make any comments on "presentation of information" as an exclusion from patentability under Art. 52 (2) (d) EPC. The reason is probably that there is no doubt that an image itself is defined as nothing more than a pure representation of information. This means that generally speaking an image (at least if it does not show any technical features) obtained by an image processing method, or in particular an image compression method, is excluded from patentability under Art. 52 (2) (d) EPC.

Let's have a look at the following claim set:

1. A method of processing an image by applying the steps...

2. An image obtained by the method of claim 1.

In other words, claim 2 would not be allowed in the above claim set. However, Art. 64 (2) EPC stipulates that if the subject-matter of the European patent is a process, the protection conferred by the patent shall extend to the products directly obtained by such process. The author of this article is of the opinion that Art. 64 (2) EPC may be regarded as a backdoor to circumvent Art. 52 (2) (d) EPC (before national courts in infringement proceedings).

Typically, Art. 64 (2) EPC is applied to pharmaceutical products obtained by a novel and inventive process. However, Art. 64 (2) EPC is formulated so broadly that it can also be read - without much discussion - on images obtained by an image processing method. Visser writes: "Art. 64 (2) EPC extends the protection conferred by a process patent to the products obtained directly by the process, even if the products are not patentable per se. Hence, the protection may extend to a known product (not patentable under Art. 52 (1) EPC)... The non-patentability of the product does not affect the examination of the process claim, because Art. 64 (2) EPC does not belong to the requirements of the EPC on patentability to be examined (Art. 52-57 EPC) but to the provisions concerning the effects of patents and patent applications to be applied by national courts deciding on infringement. (G1/98 r.4)"[2]

This means that although claim 2 above would not be allowed, someone who has a European patent for claim 1, could sue someone else for infringement of his patent if the person merely shows/displays images obtained by the method of claim 1. Without any doubts a far reaching effect.

Inventive Step

In order to demonstrate the presence of an inventive step, a technical effect based on the distinguishing feature(s) has to be identified. Typical technical effects in the field of image processing are to increase sharpness of an image, increase contrast of an image, improve contours, improve saturation of colour, improve quantitative measurement of defects in a manufacturing process, improve accuracy of measurement of defects, allow for faster object tracking, etc. In T 2124/08, the problem of "improving user-friendliness" was also accepted. In the field of image compression, the technical effects are typically to reduce the amount of storage, reduce the processing time or reduce the transmission time. Please note that these technical effects are only acceptable since image compression is considered to be technical.

However, T 1954/08 referred to decision T 1227/05, point 3.2.5 (supra) which held that (the sole) processing speed was not a suitable criterion for distinguishing between technical and non-technical method steps since it was always possible to conceive of a slower algorithm than the one claimed. Similarly, the sole amount of memory a computer-implemented algorithm requires is equally unsuitable for determining whether or not a method step contributes to the solution of a technical problem since it is always possible to imagine an algorithm demanding more memory. As mentioned above, the innovative potential of the algorithmic scheme can be left aside since it does not serve any technical purpose and, thus, does not contribute to the technical character of the claimed method and cannot enter into the examination for an inventive step.

Also in T 42/10 it was held that processing speed is not a suitable criterion. "The appellant's third argument is that factor graphs, and the associated message passing algorithm, are technical. They address the technical problem of speeding up computation.

In its full generality, speed of computation is a mathematical problem. It may be the case that a computer has a particular processor that is particularly good, or particularly poor, at some (class of) operations. Recasting a mathematical method so as to take advantage of what the processor does quickly, or to avoid what it does slowly, might involve technical considerations. In such a case, the recast method, when performed on that particular processor, might not be "just" mathematical but also be technical. However, not all recasting of mathematical methods in order to increase speed are technical. In the days when people looked up trigonometric functions in tables, recasting a method so as to reduce the number of times the tables had to be consulted might speed up computation, but nothing technical was happening.

The appellant has not provided any evidence that there really is an increase in speed of computation. There is no analysis of the complexity of any prior art method of performing the same sort of calculation, and there is none of the complexity using factor graphs. Nor has the appellant provided any evidence for its assertion that the increase in speed would only be obtained on a computer, whereas calculations by hand would be slower. However, the Board also considers that, even if the increase in speed were established, it would not be an increase which solved a technical problem, and that is enough to reject the argument. According to the application (paragraph [0068] as published), it does not matter what sort of technical apparatus is used to perform the calculations. What matters is only the ability to carry out the necessary steps. It follows from that, that any improvement in speed is inherent in the method of calculation. It does not result from exploiting ability or avoiding some lack in the computer. At best, if the appellant is correct, and there is an increase in speed which only occurs on computers, it is a matter of abstract computer science."

T 42/10 also relates to a case (Gale`s application) from the UK for an algorithm for faster calculation of square roots which was not deemed patentable since it did not solve a technical problem. The increase in computation speed was not deemed a technical effect.

"The Board's approach to assessing questions of what is and what is not technical about a computer-implemented method, in this case, asks the same questions as Nicholls LJ in Re Gale's Application. The first is: what does the method as a whole do, and does it produce an overall technical result? The second is: if there is no overall technical result, does the method at least have a technical effect within the computer? If both questions are answered in the negative, no technical problem has been solved and there can be no inventive step."

A. Enforceability

On the one hand, patents relating to image processing methods which define a particular image operator applied on an image may be difficult to enforce due to problems that relate to the detection of the infringement of the claimed method. Since the image processing method is implemented in software, it is hard to detect whether the software infringes the claimed method or not. Normally, some form of decompilation or step-by-step analysis is required. (An interesting decision is T 2440/12 which does not deal with infringement but with a prior use of software. The prior use was in the form of sales of a software product that embodied the claimed invention. This was undisputed between the parties. In the Board's view, it can be rather convincingly argued that the mere fact that any interested (and skilled) person who acquired the software product would be able to see how the input data was processed and understand how the method implemented by the software was carried out. By executing the program line-by-line, a skilled person would be able to see how the input data was processed and understand how the method implemented by the software product was carried out step-by-step. The information provided by the stepwise execution of the software product represented a form of disclosure of a specific embodiment of this method.)

On the other hand, the field of image/video compression is dominated by many standards which are covered by patents (MPEG, H.264, JPEG, GIF, etc.). An essential patent or standard-essential patent (SEP) is a patent that claims an invention that must be used to comply with a technical standard. Companies involved in the development of image compression methods often file their patent applications before they share their knowledge with other members in the standards organization. Should the patent be granted and also become part of a compression standard, then the patent proprietor could ask for royalties from all members who wish to use the standard. (As the patent has become part of a standard, there are presumably many who wish to use it.) Standards organizations, therefore, often require members to disclose and grant licenses to their patents and pending patent applications that cover a standard that the organization is developing. If a standards organization fails to get licenses to all patents that are essential to complying with a standard, owners of the unlicensed patents may demand or sue for royalties from companies that adopt the standard. This happened to the GIF and JPEG standards, for example.

Determining which patents are essential to a particular standard can be complex. Standardisation organizations require licences of essential patents to be on fair, reasonable, and non-discriminatory (FRAND) terms.

B. Conclusion

The field of image processing and generation is susceptible to patent protection in Europe (and many other jurisdictions). Whether a patent will be granted depends on the fact whether the claimed invention is novel and inventive vis-à-vis the prior art. However, since the field is closely related to mathematics, care has to be taken not to direct the claims to the mathematical method per se (which should not be a problem since images are not considered to be abstract). An interesting (and unanswered) question arises whether an image processed by an image processing method is protected under Art. 64 (2) EPC. At least in the field of image/video compression, enforceability (i.e. detection of infringement) of a patent is often not a problem at least if it relates to a standard. In such cases, it can often be determined that someone uses a standard and thereby automatically infringes the patent.

Please send any feedback on the article to:
michaelfischer1978@web.de

1. The interrelation between image processing and image compression can be seen from the fact that Board 3.5.04 deals with appeals from both G06T and H04N.
2. D. Visser, "The Annotated European Patent Convention", 21^st edition, H. Tel Publisher, 2013, p.134