Constructor Summary
Constructors Constructor and Description TesseractOCRParser()
Method Summary
All MethodsInstance MethodsConcrete MethodsDeprecated Methods Modifier and Type Method and Description void
checkInitialization(InitializableProblemHandler problemHandler)
java.util.Set<MediaType>
getSupportedTypes(ParseContext context)
Returns the set of media types supported by this parser when used with the given parse context.boolean
hasTesseract(TesseractOCRConfig config)
void
initialize(java.util.Map<java.lang.String,Param> params)
void
parse(java.awt.Image image, org.xml.sax.ContentHandler handler, Metadata metadata, ParseContext context)
void
parse(java.io.InputStream stream, org.xml.sax.ContentHandler handler, Metadata metadata, ParseContext parseContext)
Parses a document stream into a sequence of XHTML SAX events.void
parseInline(java.io.InputStream stream, XHTMLContentHandler xhtml, ParseContext parseContext, TesseractOCRConfig config)
Use this to parse content without starting a new document.void
parseInline(java.io.InputStream stream, XHTMLContentHandler xhtml, TesseractOCRConfig config)
Deprecated.useparseInline(InputStream, XHTMLContentHandler, ParseContext, TesseractOCRConfig)
Methods inherited from class org.apache.tika.parser.AbstractParser
parse
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Developer(s) | Apache Software Foundation |
---|---|
Stable release | |
Repository | |
Written in | Java |
Operating system | Cross-platform |
Type | Search and indexAPI |
License | Apache License 2.0 |
Website | tika.apache.org |
Apache Tika is a content detection and analysis framework, written in Java, stewarded at the Apache Software Foundation.[1] It detects and extracts metadata and text from over a thousand different file types, and as well as providing a Java library, has server and command-line editions suitable for use from other programming languages.
History[edit]
The project originated as part of the Apache Nutch codebase, to provide content identification and extraction when crawling. In 2007, it was separated out, to make it more extensible and usable by content management systems, other Web crawlers, and information retrieval systems. The standalone Tika was founded by Jérôme Charron, Chris Mattmann and Jukka Zitting.[2] In 2011 Chris Mattmann and Jukka Zitting released the Manning book 'Tika in Action', and the project released version 1.0.
Features[edit]
Tika provides capabilities for identification of more than 1400 file types from the Internet Assigned Numbers Authority taxonomy of MIME types. For most of the more common and popular formats,[3] Tika then provides content extraction, metadata extraction and language identification capabilities.
While Tika is written in Java, it is widely used from other languages.[4] The RESTful server and CLI Tool permit non-Java programs to access the Tika functionality.
Notable uses[edit]
Tika is used by financial institutions including the Fair Isaac Corporation (FICO),[5] Goldman Sachs,[6]NASA and academic researchers[7] and by major content management systems including Drupal,[8] and Alfresco (software)[9] to analyze large amounts of content, and to make it available in common formats using information retrieval techniques.
On April 4, 2016[10]Forbes published an article identifying Tika as one of the key technologies used by more than 400 journalists to analyze 11.5 million leaked documents that expose an international scandal involving world leaders storing money in offshore shell corporations. The leaked documents and the project to analyze them is referred to as the Panama Papers.
See also[edit]
References[edit]
- ^'Apache Tika'. Retrieved 2016-04-15.
- ^'Tika Proposal'. Retrieved 2016-04-15.
- ^'The Apache Software Foundation'. Apache Tika formats page. Retrieved 16 April 2016.
- ^'API Bindings for Tika'. Apache Tika. Retrieved 2016-04-17.
- ^'FICO to Engage Kaggle's Community of 180,000 Data Scientists to Drive Innovation in the FICO Analytic Cloud | FICO®'. FICO® | Decisions. Archived from the original on 2016-06-03. Retrieved 2016-04-15.
- ^'Goldman Sachs Puts Elasticsearch To Work - InformationWeek'. InformationWeek. Retrieved 2017-06-21.
- ^'Studying polar data with the help of Apache Tika'. Opensource.com. Retrieved 2016-04-15.
- ^'Text Extract for Drupal using Tika | Drupal.org'. www.drupal.org. Retrieved 2016-04-15.
- ^'Content Transformation and Metadata Extraction with Apache Tika - alfrescowiki'. wiki.alfresco.com. Retrieved 2016-04-15.
- ^Fox-Brewster, Thomas. 'From Encrypted Drives To Amazon's Cloud -- The Amazing Flight Of The Panama Papers'. Forbes. Retrieved 2016-04-15.
Retrieved from 'https://en.wikipedia.org/w/index.php?title=Apache_Tika&oldid=901978408'
What is Apache Tika?
- Apache Tika is a library that is used for document type detection and content extraction from various file formats.
- Internally, Tika uses existing various document parsers and document type detection techniques to detect and extract data.
- Using Tika, one can develop a universal type detector and content extractor to extract both structured text as well as metadata from different types of documents such as spreadsheets, text documents, images, PDFs and even multimedia input formats to a certain extent.
- Tika provides a single generic API for parsing different file formats. It uses existing specialized parser libraries for each document type.
- All these parser libraries are encapsulated under a single interface called the Parser interface.
Why Tika?
According to filext.com, there are about 15k to 51k content types, and this number is growing day by day. Data is being stored in various formats such as text documents, excel spreadsheet, PDFs, images, and multimedia files, to name a few. Therefore, applications such as search engines and content management systems need additional support for easy extraction of data from these document types. Apache Tika serves this purpose by providing a generic API to locate and extract data from multiple file formats.
Apache Tika Applications
There are various applications that make use of Apache Tika. Here we will discuss a few prominent applications that depend heavily on Apache Tika.
Search Engines
Tika is widely used while developing search engines to index the text contents of digital documents.
- Search engines are information processing systems designed to search information and indexed documents from the Web.
- Crawler is an important component of a search engine that crawls through the Web to fetch the documents that are to be indexed using some indexing technique. Thereafter, the crawler transfers these indexed documents to an extraction component.
- The duty of extraction component is to extract the text and metadata from the document. Such extracted content and metadata are very useful for a search engine. This extraction component contains Tika.
- The extracted content is then passed to the indexer of the search engine that uses it to build a search index. Apart from this, the search engine uses the extracted content in many other ways as well.
Document Analysis
- In the field of artificial intelligence, there are certain tools to analyze documents automatically at semantic level and extract all kinds of data from them.
- In such applications, the documents are classified based on the prominent terms in the extracted content of the document.
- These tools make use of Tika for content extraction to analyze documents varying from plain text to digital documents.
Digital Asset Management
- Some organizations manage their digital assets such as photographs, ebooks, drawings, music and video using a special application known as digital asset management (DAM).
- Such applications take the help of document type detectors and metadata extractor to classify the various documents.
Content Analysis
- Websites like Amazon recommend newly released contents of their website to individual users according to their interests. To do so, these websites follow machine learning techniques, or take the help of social media websites like Facebook to extract required information such as likes and interests of the users. This gathered information will be in the form of html tags or other formats that require further content type detection and extraction.
- For content analysis of a document, we have technologies that implement machine learning techniques such as UIMA and Mahout. These technologies are useful in clustering and analyzing the data in the documents.
- Apache Mahout is a framework which provides ML algorithms on Apache Hadoop – a cloud computing platform. Mahout provides an architecture by following certain clustering and filtering techniques. By following this architecture, programmers can write their own ML algorithms to produce recommendations by taking various text and metadata combinations. To provide inputs to these algorithms, recent versions of Mahout use Tika to extract text and metadata from binary content.
- Apache UIMA analyzes and processes various programming languages and produces UIMA annotations. Internally it uses Tika Annotator to extract document text and metadata.
History
Year | Development |
---|---|
2006 | The idea of Tika was projected before the Lucene Project Management Committee. |
2006 | The concept of Tika and its usefulness in the Jackrabbit project was discussed. |
2007 | Tika entered into Apache incubator. |
2008 | Versions 0.1 and 0.2 were released and Tika graduated from the incubator to the Lucene sub-project. |
2009 | Versions 0.3, 0.4, and 0.5 were released. |
2010 | Version 0.6 and 0.7 were released and Tika graduated into the top-level Apache project. |
2011 | Tika 1.0 was released and the book on Tika 'Tika in Action” was also released in the same year. |
Application-Level Architecture of Tika
Application programmers can easily integrate Tika in their applications. Tika provides a Command Line Interface and a GUI to make it user friendly.
In this chapter, we will discuss the four important modules that constitute the Tika architecture. The following illustration shows the architecture of Tika along with its four modules −
- Language detection mechanism.
- MIME detection mechanism.
- Parser interface.
- Tika Facade class.
Language Detection Mechanism
Whenever a text document is passed to Tika, it will detect the language in which it was written. It accepts documents without language annotation and adds that information in the metadata of the document by detecting the language.
To support language identification, Tika has a class called Language Identifier in the package org.apache.tika.language, and a language identification repository inside which contains algorithms for language detection from a given text. Tika internally uses N-gram algorithm for language detection.
MIME Detection Mechanism
Tika can detect the document type according to the MIME standards. Default MIME type detection in Tika is done using org.apache.tika.mime.mimeTypes. It uses the org.apache.tika.detect.Detector interface for most of the content type detection.
Internally Tika uses several techniques like file globs, content-type hints, magic bytes, character encodings, and several other techniques.
Parser Interface
The parser interface of org.apache.tika.parser is the key interface for parsing documents in Tika. This Interface extracts the text and the metadata from a document and summarizes it for external users who are willing to write parser plugins.
Using different concrete parser classes, specific for individual document types, Tika supports a lot of document formats. These format specific classes provide support for different document formats, either by directly implementing the parser logic or by using external parser libraries.
Tika Facade Class
Using Tika facade class is the simplest and direct way of calling Tika from Java, and it follows the facade design pattern. You can find the Tika facade class in the org.apache.tika package of Tika API.
By implementing basic use cases, Tika acts as a broker of landscape. It abstracts the underlying complexity of the Tika library such as MIME detection mechanism, parser interface, and language detection mechanism, and provides the users a simple interface to use.
Features of Tika
- Unified parser Interface − Tika encapsulates all the third party parser libraries within a single parser interface. Due to this feature, the user escapes from the burden of selecting the suitable parser library and use it according to the file type encountered.
- Low memory usage − Tika consumes less memory resources therefore it is easily embeddable with Java applications. We can also use Tika within the application which run on platforms with less resources like mobile PDA.
- Fast processing − Quick content detection and extraction from applications can be expected.
- Flexible metadata − Tika understands all the metadata models which are used to describe files.
- Parser integration − Tika can use various parser libraries available for each document type in a single application.
- MIME type detection − Tika can detect and extract content from all the media types included in the MIME standards.
- Language detection − Tika includes language identification feature, therefore can be used in documents based on language type in a multi lingual websites.
Functionalities of Tika
Tika supports various functionalities −
- Document type detection
- Content extraction
- Metadata extraction
- Language detection
Document Type Detection
Tika uses various detection techniques and detects the type of the document given to it.
Content Extraction
Tika has a parser library that can parse the content of various document formats and extract them. After detecting the type of the document, it selects the appropriate parser from the parser repository and passes the document. Different classes of Tika have methods to parse different document formats.
Metadata Extraction
Along with the content, Tika extracts the metadata of the document with the same procedure as in content extraction. For some document types, Tika have classes to extract metadata.
Language Detection
Internally, Tika follows algorithms like n-gram to detect the language of the content in a given document. Tika depends on classes like Languageidentifier and Profiler for language identification.
This chapter takes you through the process of setting up Apache Tika on Windows and Linux. User administration is needed while installing the Apache Tika.
System Requirements
JDK | Java SE 2 JDK 1.6 or above |
Memory | 1 GB RAM (recommeneded) |
Disk Space | No minimum requirement |
Operating System Version | Windows XP or above, Linux |
Step 1: Verifying Java Installation
To verify Java installation, open the console and execute the following java command.
OS | Task | Command |
---|---|---|
Windows | Open command console | >java –version |
Linux | Open command terminal | $java –version |
If Java has been installed properly on your system, then you should get one of the following outputs, depending on the platform you are working on.
OS | Output |
---|---|
Windows | Java version '1.7.0_60' Java (TM) SE Run Time Environment (build 1.7.0_60-b19) Java Hotspot (TM) 64-bit Server VM (build 24.60-b09, mixed mode) |
Lunix | java version '1.7.0_25' Open JDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64) Open JDK 64-Bit Server VM (build 23.7-b01, mixed mode) |
- We assume the readers of this tutorial have Java 1.7.0_60 installed on their system before proceeding for this tutorial.
- In case you do not have Java SDK, download its current version from https://www.oracle.com/technetwork/java/javase/downloads/index.html and have it installed.
Step 2: Setting Java Environment
Set the JAVA_HOME environment variable to point to the base directory location where Java is installed on your machine. For example,
OS | Output |
---|---|
Windows | Set Environmental variable JAVA_HOME to C:ProgramFilesjavajdk1.7.0_60 |
Linux | export JAVA_HOME = /usr/local/java-current |
Append the full path of the Java compiler location to the System Path.
OS | Output |
---|---|
Windows | Append the String; C:Program FilesJavajdk1.7.0_60bin to the end of the system variable PATH. |
Linux | export PATH = $PATH:$JAVA_HOME/bin/ |
Verify the command java-version from command prompt as explained above.
Step 3: Setting up Apache Tika Environment
Programmers can integrate Apache Tika in their environment by using
- Command line,
- Tika API,
- Command line interface (CLI) of Tika,
- Graphical User interface (GUI) of Tika, or
- the source code.
For any of these approaches, first of all, you have to download the source code of Tika.
You will find the source code of Tika at https://Tika.apache.org/download.html, where you will find two links −
- apache-tika-1.6-src.zip − It contains the source code of Tika, and
- Tika -app-1.6.jar − It is a jar file that contains the Tika application.
Download these two files. A snapshot of the official website of Tika is shown below.
After downloading the files, set the classpath for the jar file tika-app-1.6.jar. Add the complete path of the jar file as shown in the table below.
OS | Output |
---|---|
Windows | Append the String “C:jarsTika-app-1.6.jar” to the user environment variable CLASSPATH |
Linux | Export CLASSPATH = $CLASSPATH − /usr/share/jars/Tika-app-1.6.tar − |
Apache provides Tika application, a Graphical User Interface (GUI) application using Eclipse.
Tika-Maven Build using Eclipse
- Open eclipse and create a new project.
- If you do not having Maven in your Eclipse, set it up by following the given steps.
- Open the link https://wiki.eclipse.org/M2E_updatesite_and_gittags. There you will find the m2e plugin releases in a tabular format
- Pick the latest version and save the path of the url in p2 url column.
- Now revisit eclipse, in the menu bar, click Help, and choose Install New Software from the dropdown menu
- Click the Add button, type any desired name, as it is optional. Now paste the saved url in the Location field.
- A new plugin will be added with the name you have chosen in the previous step, check the checkbox in front of it, and click Next.
- Proceed with the installation. Once completed, restart the Eclipse.
- Now right click on the project, and in the configure option, select convert to maven project.
- A new wizard for creating a new pom appears. Enter the Group Id as org.apache.tika, enter the latest version of Tika, select the packaging as jar, and click Finish.
The Maven project is successfully installed, and your project is converted into Maven. Now you have to configure the pom.xml file.
Configure the XML File
Get the Tika maven dependency from https://mvnrepository.com/artifact/org.apache.tika
Shown below is the complete Maven dependency of Apache Tika.
Users can embed Tika in their applications using the Tika facade class. It has methods to explore all the functionalities of Tika. Since it is a facade class, Tika abstracts the complexity behind its functions. In addition to this, users can also use the various classes of Tika in their applications.
Tika Class (facade)
This is the most prominent class of the Tika library and follows the facade design pattern. Therefore, it abstracts all the internal implementations and provides simple methods to access the Tika functionalities. The following table lists the constructors of this class along with their descriptions.
package − org.apache.tika
class − Tika
Sr.No. | Constructor & Description |
---|---|
1 | Tika () Uses default configuration and constructs the Tika class. |
2 | Tika (Detector detector) Creates a Tika facade by accepting the detector instance as parameter |
3 | Tika (Detector detector, Parser parser) Creates a Tika facade by accepting the detector and parser instances as parameters. |
4 | Tika (Detector detector, Parser parser, Translator translator) Creates a Tika facade by accepting the detector, the parser, and the translator instance as parameters. |
5 | Tika (TikaConfig config) Creates a Tika facade by accepting the object of the TikaConfig class as parameter. |
Methods and Description
The following are the important methods of Tika facade class −
Sr.No. | Methods & Description |
---|---|
1 | parseToString (File file) This method and all its variants parses the file passed as parameter and returns the extracted text content in the String format. By default, the length of this string parameter is limited. |
2 | int getMaxStringLength () Returns the maximum length of strings returned by the parseToString methods. |
3 | void setMaxStringLength (int maxStringLength) Sets the maximum length of strings returned by the parseToString methods. |
4 | Reader parse (File file) This method and all its variants parses the file passed as parameter and returns the extracted text content in the form of java.io.reader object. |
5 | String detect (InputStream stream, Metadata metadata) This method and all its variants accepts an InputStream object and a Metadata object as parameters, detects the type of the given document, and returns the document type name as String object. This method abstracts the detection mechanisms used by Tika. |
6 | String translate (InputStream text, String targetLanguage) This method and all its variants accepts the InputStream object and a String representing the language that we want our text to be translated, and translates the given text to the desired language, attempting to auto-detect the source language. |
Parser Interface
This is the interface that is implemented by all the parser classes of Tika package.
package − org.apache.tika.parser
Interface − Parser
Methods and Description
The following is the important method of Tika Parser interface −
Sr.No. | Methods & Description |
---|---|
1 | parse (InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) This method parses the given document into a sequence of XHTML and SAX events. After parsing, it places the extracted document content in the object of the ContentHandler class and the metadata in the object of the Metadata class. |
Metadata Class
This class implements various interfaces such as CreativeCommons, Geographic, HttpHeaders, Message, MSOffice, ClimateForcast, TIFF, TikaMetadataKeys, TikaMimeKeys, Serializable to support various data models. The following tables list the constructors and methods of this class along with their descriptions.
package − org.apache.tika.metadata
class − Metadata
Sr.No. | Constructor & Description |
---|---|
1 | Metadata() Constructs a new, empty metadata. |
Sr.No. | Methods & Description |
---|---|
1 | add (Property property, String value) Adds a metadata property/value mapping to a given document. Using this function, we can set the value to a property. |
2 | add (String name, String value) Adds a metadata property/value mapping to a given document. Using this method, we can set a new name value to the existing metadata of a document. |
3 | String get (Property property) Returns the value (if any) of the metadata property given. |
4 | String get (String name) Returns the value (if any) of the metadata name given. |
5 | Date getDate (Property property) Returns the value of Date metadata property. |
6 | String[] getValues (Property property) Returns all the values of a metadata property. |
7 | String[] getValues (String name) Returns all the values of a given metadata name. |
8 | String[] names() Returns all the names of metadata elements in a metadata object. |
9 | set (Property property, Date date) Sets the date value of the given metadata property Pathfinder core rulebook pdf download free. |
10 | set(Property property, String[] values) Sets multiple values to a metadata property. |
Language Identifier Class
This class identifies the language of the given content. The following tables list the constructors of this class along with their descriptions.
package − org.apache.tika.language
class − Language Identifier
Sr.No. | Constructor & Description |
---|---|
1 | LanguageIdentifier (LanguageProfile profile) Instantiates the language identifier. Here you have to pass a LanguageProfile object as parameter. |
2 | LanguageIdentifier (String content) This constructor can instantiate a language identifier by passing on a String from text content. |
Sr.No. | Methods & Description |
---|---|
1 | String getLanguage () Returns the language given to the current LanguageIdentifier object. |
File Formats Supported by Tika
The following table shows the file formats Tika supports.
File format | Package Library | Class in Tika |
---|---|---|
XML | org.apache.tika.parser.xml | XMLParser |
HTML | org.apache.tika.parser.html and it uses Tagsoup Library | HtmlParser |
MS-Office compound document Ole2 till 2007 ooxml 2007 onwards | org.apache.tika.parser.microsoft Star wars battlefront 2 classic dlc. org.apache.tika.parser.microsoft.ooxml and it uses Apache Poi library | OfficeParser(ole2) OOXMLParser (ooxml) |
OpenDocument Format openoffice | org.apache.tika.parser.odf | OpenOfficeParser |
portable Document Format(PDF) | org.apache.tika.parser.pdf and this package uses Apache PdfBox library | PDFParser |
Electronic Publication Format (digital books) | org.apache.tika.parser.epub | EpubParser |
Rich Text format | org.apache.tika.parser.rtf | RTFParser |
Compression and packaging formats | org.apache.tika.parser.pkg and this package uses Common compress library | PackageParser and CompressorParser and its sub-classes |
Text format | org.apache.tika.parser.txt | TXTParser |
Feed and syndication formats | org.apache.tika.parser.feed | FeedParser |
Audio formats | org.apache.tika.parser.audio and org.apache.tika.parser.mp3 | AudioParser MidiParser Mp3- for mp3parser |
Imageparsers | org.apache.tika.parser.jpeg | JpegParser-for jpeg images |
Videoformats | org.apache.tika.parser.mp4 and org.apache.tika.parser.video this parser internally uses Simple Algorithm to parse flash video formats | Mp4parser FlvParser |
java class files and jar files | org.apache.tika.parser.asm | ClassParser CompressorParser |
Mobxformat (email messages) | org.apache.tika.parser.mbox | MobXParser |
Cad formats | org.apache.tika.parser.dwg | DWGParser |
FontFormats | org.apache.tika.parser.font | TrueTypeParser |
executable programs and libraries | org.apache.tika.parser.executable | ExecutableParser |
MIME Standards
Multipurpose Internet Mail Extensions (MIME) standards are the best available standards for identifying document types. The knowledge of these standards helps the browser during internal interactions.
Whenever the browser encounters a media file, it chooses a compatible software available with it to display its contents. In case it does not have any suitable application to run a particular media file, it recommends the user to get the suitable plugin software for it.
Type Detection in Tika
Tika supports all the Internet media document types provided in MIME. Whenever a file is passed through Tika, it detects the file and its document type. To detect media types, Tika internally uses the following mechanisms.
File Extensions
Checking the file extensions is the simplest and most-widely used method to detect the format of a file. Many applications and operating systems provide support for these extensions. Shown below are the extension of a few known file types.
File name | Extention |
---|---|
image | .jpg |
audio | .mp3 |
java archive file | .jar |
java class file | .class |
Content-type Hints
Whenever you retrieve a file from a database or attach it to another document, you may lose the file’s name or extension. In such cases, the metadata supplied with the file is used to detect the file extension.
Magic Byte
Observing the raw bytes of a file, you can find some unique character patterns for each file. Some files have special byte prefixes called magic bytes that are specially made and included in a file for the purpose of identifying the file type
For example, you can find CA FE BA BE (hexadecimal format) in a java file and %PDF (ASCII format) in a pdf file. Tika uses this information to identify the media type of a file.
Character Encodings
Files with plain text are encoded using different types of character encoding. The main challenge here is to identify the type of character encoding used in the files. Tika follows character encoding techniques like Bom markers and Byte Frequencies to identify the encoding system used by the plain text content.
XML Root Characters
To detect XML documents, Tika parses the xml documents and extracts the information such as root elements, namespaces, and referenced schemas from where the true media type of the files can be found.
Type Detection using Facade Class
The detect() method of facade class is used to detect the document type. This method accepts a file as input. Shown below is an example program for document type detection with Tika facade class.
Save the above code as TypeDetection.java and run it from the command prompt using the following commands −
Tika uses various parser libraries to extract content from given parsers. It chooses the right parser for extracting the given document type.
For parsing documents, the parseToString() method of Tika facade class is generally used. Shown below are the steps involved in the parsing process and these are abstracted by the Tika ParsertoString() method.
Abstracting the parsing process −
- Initially when we pass a document to Tika, it uses a suitable type detection mechanism available with it and detects the document type.
- Once the document type is known, it chooses a suitable parser from its parser repository. The parser repository contains classes that make use of external libraries.
- Then the document is passed to choose the parser which will parse the content, extract the text, and also throw exceptions for unreadable formats.
Content Extraction using Tika
Given below is the program for extracting text from a file using Tika facade class −
Save the above code as TikaExtraction.java and run it from the command prompt −
Given below is the content of sample.txt.
It gives you the following output −
Content Extraction using Parser Interface
The parser package of Tika provides several interfaces and classes using which we can parse a text document. Given below is the block diagram of the org.apache.tika.parser package.
There are several parser classes available, e.g., pdf parser, Mp3Passer, OfficeParser, etc., to parse respective documents individually. All these classes implement the parser interface.
CompositeParser
The given diagram shows Tika’s general-purpose parser classes: CompositeParser and AutoDetectParser. Since the CompositeParser class follows composite design pattern, you can use a group of parser instances as a single parser. The CompositeParser class also allows access to all the classes that implement the parser interface.
AutoDetectParser
This is a subclass of CompositeParser and it provides automatic type detection. Using this functionality, the AutoDetectParser automatically sends the incoming documents to the appropriate parser classes using the composite methodology.
parse() method
Along with parseToString(), you can also use the parse() method of the parser Interface. The prototype of this method is shown below.
The following table lists the four objects it accepts as parameters.
Sr.No. | Object & Description |
---|---|
1 | InputStream stream Any Inputstream object that contains the content of the file |
2 | ContentHandler handler Tika passes the document as XHTML content to this handler, thereafter the document is processed using SAX API. It provides efficient postprocessing of the contents in a document. |
3 | Metadata metadata The metadata object is used both as a source and a target of document metadata. |
4 | ParseContext context This object is used in cases where the client application wants to customize the parsing process. |
Example
Given below is an example that shows how the parse() method is used.
Step 1 −
To use the parse() method of the parser interface, instantiate any of the classes providing the implementation for this interface.
There are individual parser classes such as PDFParser, OfficeParser, XMLParser, etc. You can use any of these individual document parsers. Alternatively, you can use either CompositeParser or AutoDetectParser that uses all the parser classes internally and extracts the contents of a document using a suitable parser.
Step 2 −
Create a handler class object. Given below are the three content handlers −
Sr.No. | Class & Description |
---|---|
1 | BodyContentHandler This class picks the body part of the XHTML output and writes that content to the output writer or output stream. Then it redirects the XHTML content to another content handler instance. |
2 | LinkContentHandler This class detects and picks all the H-Ref tags of the XHTML document and forwards those for the use of tools like web crawlers. |
3 | TeeContentHandler This class helps in using multiple tools simultaneously. |
Since our target is to extract the text contents from a document, instantiate BodyContentHandler as shown below −
Step 3 −
Create the Metadata object as shown below −
Step 4 −
Create any of the input stream objects, and pass your file that should be extracted to it.
FileInputstream
Instantiate a file object by passing the file path as parameter and pass this object to the FileInputStream class constructor.
Note − The path passed to the file object should not contain spaces.
The problem with these input stream classes is that they don’t support random access reads, which is required to process some file formats efficiently. To resolve this problem, Tika provides TikaInputStream.
Step 5 −
Create a parse context object as shown below −
Step 6 −
Instantiate the parser object, invoke the parse method, and pass all the objects required, as shown in the prototype below −
Given below is the program for content extraction using the parser interface −
Save the above code as ParserExtraction.java and run it from the command prompt −
Given below is the content of sample.txt
If you execute the above program, it will give you the following output −
Besides content, Tika also extracts the metadata from a file. Metadata is nothing but the additional information supplied with a file. If we consider an audio file, the artist name, album name, title comes under metadata.
XMP Standards
The Extensible Metadata Platform (XMP) is a standard for processing and storing information related to the content of a file. It was created by Adobe Systems Inc. XMP provides standards for defining, creating, and processing of metadata. You can embed this standard into several file formats such as PDF, JPEG, JPEG, GIF, jpg, HTML etc.
Property Class
Tika uses the Property class to follow XMP property definition. It provides the PropertyType and ValueType enums to capture the name and value of a metadata.
Metadata Class
This class implements various interfaces such as ClimateForcast, CativeCommons, Geographic, TIFF etc. to provide support for various metadata models. In addition, this class provides various methods to extract the content from a file.
Metadata Names
We can extract the list of all metadata names of a file from its metadata object using the method names(). It returns all the names as a string array. Using the name of the metadata, we can get the value using the get() method. It takes a metadata name and returns a value associated with it.
Extracting Metadata using Parse Method
Whenever we parse a file using parse(), we pass an empty metadata object as one of the parameters. This method extracts the metadata of the given file (if that file contains any), and places them in the metadata object. Therefore, after parsing the file using parse(), we can extract the metadata from that object.
Given below is the complete program to extract metadata from a text file.
Save the above code as GetMetadata.java and run it from the command prompt using the following commands −
Given below is the snapshot of boy.jpg
If you execute the above program, it will give you the following output −
We can also get our desired metadata values.
Adding New Metadata Values
We can add new metadata values using the add() method of the metadata class. Given below is the syntax of this method. Here we are adding the author name.
The Metadata class has predefined properties including the properties inherited from classes like ClimateForcast, CativeCommons, Geographic, etc., to support various data models. Shown below is the usage of the SOFTWARE data type inherited from the TIFF interface implemented by Tika to follow XMP metadata standards for TIFF image formats.
Given below is the complete program that demonstrates how to add metadata values to a given file. Here the list of the metadata elements is displayed in the output so that you can observe the change in the list after adding new values.
Save the above code as AddMetadata.java class and run it from the command prompt −
Given below is the content of Example.txt
If you execute the above program, it will give you the following output −
Setting Values to Existing Metadata Elements
You can set values to the existing metadata elements using the set() method. The syntax of setting the date property using the set() method is as follows −
You can also set multiple values to the properties using the set() method. The syntax of setting multiple values to the Author property using the set() method is as follows −
Given below is the complete program demonstrating the set() method.
Save the above code as SetMetadata.java and run it from the command prompt −
Given below is the content of example.txt.
If you execute the above program it will give you the following output. In the output, you can observe the newly added metadata elements.
Need for Language Detection
For classification of documents based on the language they are written in a multilingual website, a language detection tool is needed. This tool should accept documents without language annotation (metadata) and add that information in the metadata of the document by detecting the language.
Algorithms for Profiling Corpus
What is Corpus?
To detect the language of a document, a language profile is constructed and compared with the profile of the known languages. The text set of these known languages is known as a corpus.
A corpus is a collection of texts of a written language that explains how the language is used in real situations.
The corpus is developed from books, transcripts, and other data resources like the Internet. The accuracy of the corpus depends upon the profiling algorithm we use to frame the corpus.
What are Profiling Algorithms?
The common way of detecting languages is by using dictionaries. The words used in a given piece of text will be matched with those that are in the dictionaries.
A list of common words used in a language will be the most simple and effective corpus for detecting a particular language, for example, articles a, an, the in English.
Using Word Sets as Corpus
Using word sets, a simple algorithm is framed to find the distance between two corpora, which will be equal to the sum of differences between the frequencies of matching words.
Such algorithms suffer from the following problems −
- Since the frequency of matching words is very less, the algorithm cannot efficiently work with small texts having few sentences. It needs a lot of text for accurate match.
- It cannot detect word boundaries for languages having compound sentences, and those having no word dividers like spaces or punctuation marks.
Due to these difficulties in using word sets as corpus, individual characters or character groups are considered.
Using Character Sets as Corpus
Since the characters that are commonly used in a language are finite in number, it is easy to apply an algorithm based on word frequencies rather than characters. This algorithm works even better in case of certain character sets used in one or very few languages.
This algorithm suffers from the following drawbacks −
- It is difficult to differentiate two languages having similar character frequencies.
- There is no specific tool or algorithm to specifically identify a language with the help of (as corpus) the character set used by multiple languages.
N-gram Algorithm
The drawbacks stated above gave rise to a new approach of using character sequences of a given length for profiling corpus. Such sequence of characters are called as N-grams in general, where N represents the length of the character sequence.
- N-gram algorithm is an effective approach for language detection, especially in case of European languages like English.
- This algorithm works fine with short texts.
- Though there are advanced language profiling algorithms to detect multiple languages in a multilingual document having more attractive features, Tika uses the 3-grams algorithm, as it is suitable in most practical situations.
Language Detection in Tika
Among all the 184 standard languages standardized by ISO 639-1, Tika can detect 18 languages. Language detection in Tika is done using the getLanguage() method of the LanguageIdentifier class. This method returns the code name of the language in String format. Given below is the list of the 18 language-code pairs detected by Tika −
da—Danish | de—German | et—Estonian | el—Greek |
en—English | es—Spanish | fi—Finnish | fr—French |
hu—Hungarian | is—Icelandic | it—Italian | nl—Dutch |
no—Norwegian | pl—Polish | pt—Portuguese | ru—Russian |
sv—Swedish | th—Thai |
While instantiating the LanguageIdentifier class, you should pass the String format of the content to be extracted, or a LanguageProfile class object.
Given below is the example program for Language detection in Tika.
Save the above code as LanguageDetection.java and run it from the command prompt using the following commands −
If you execute the above program it gives the following outpu−
Language Detection of a Document
To detect the language of a given document, you have to parse it using the parse() method. The parse() method parses the content and stores it in the handler object, which was passed to it as one of the arguments. Pass the String format of the handler object to the constructor of the LanguageIdentifier class as shown below −
Given below is the complete program that demonstrates how to detect the language of a given document −
Save the above code as SetMetadata.java and run it from the command prompt −
Apache Tika Disable Tesseract In Minecraft
Given below is the content of Example.txt.
If you execute the above program, it will give you the following output −
Along with the Tika jar, Tika provides a Graphical User Interface application (GUI) and a Command Line Interface (CLI) application. You can execute a Tika application from the command prompt too like other Java applications.
Graphical User Interface (GUI)
- Tika provides a jar file along with its source code in the following link https://tika.apache.org/download.html.
- Download both the files, set the classpath for the jar file.
- Extract the source code zip folder, open the tika-app folder.
- In the extracted folder at “tika-1.6tika-appsrcmainjavaorgapacheTikagui” you will see two class files: ParsingTransferHandler.java and TikaGUI.java.
- Compile both the class files and execute the TikaGUI.java class file, it opens the following window.
Let us now see how to make use of the Tika GUI.
On the GUI, click open, browse and select a file that is to be extracted, or drag it onto the whitespace of the window.
Tika extracts the content of the files and displays it in five different formats, viz. metadata, formatted text, plain text, main content, and structured text. You can choose any of the format you want.
In the same way, you will also find the CLI class in the “tika-1.6tikaappsrcmainjavaorgapachetikacli” folder.
The following illustration shows what Tika can do. When we drop the image on the GUI, Tika extracts and displays its metadata.
Given below is the program to extract content and metadata from a PDF.
Save the above code as PdfParse.java, and compile it from the command prompt by using the following commands −
Below give is the snapshot of example.pdf
The PDF we are passing has the following properties −
After compiling the program, you will get the output as shown below.
Output −
Given below is the program to extract content and metadata from Open Office Document Format (ODF).
Save the above code as OpenDocumentParse.java, and compile it in the command prompt by using the following commands −
Given below is snapshot of example_open_document_presentation.odp file.
This document has the following properties −
After compiling the program, you will get the following output.
Output −
Given below is the program to extract content and metadata from a Microsoft Office Document.
Save the above code as MSExelParse.java, and compile it from the command prompt by using the following commands −
Apache Tika Server
Here we are passing the following sample Excel file.
The given Excel file has the following properties −
After executing the above program you will get the following output.
Output −
Given below is the program to extract content and metadata from a Text document −
Save the above code as TextParser.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of sample.txt file −
The text document has the following properties −
If you execute the above program it will give you the following output.
Output −
Given below is the program to extract content and metadata from an HTML document.
Save the above code as HtmlParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of example.txt file.
The HTML document has the following properties−
If you execute the above program it will give you the following output.
Output −
Given below is the program to extract content and metadata from an XML document −
Save the above code as XmlParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of example.xml file
This document has the following properties −
If you execute the above program it will give you the following output −
Output −
Given below is the program to extract content and metadata from a .class file.
Save the above code as JavaClassParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of Example.java which will generate Example.class after compilation.
Example.class file has the following properties −
After executing the above program, you will get the following output.
Output −
Given below is the program to extract content and metadata from a Java Archive (jar) file −
Save the above code as PackageParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of Example.java that resides inside the package.
The jar file has the following properties −
After executing the above program, it will give you the following output −
Output −
Given below is the program to extract content and meta data from a JPEG image.
Save the above code as JpegParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of Example.jpeg −
The JPEG file has the following properties −
After executing the program, you will get the following output.
Output −
Given below is the program to extract content and metadata from mp4 files −
Save the above code as JpegParse.java, and compile it from the command prompt by using the following commands −
Given below is the snapshot of properties of Example.mp4 file.
After executing the above program, you will get the following output −
Output −
Given below is the program to extract content and metadata from mp3 files −
Save the above code as JpegParse.java, and compile it from the command prompt by using the following commands −
Example.mp3 file has the following properties −
You will get the following output after executing the program. If the given file has any lyrics, our application will capture and display that along with the output.
Output −
PermalinkJoin GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.
Sign up Find file Copy path
Cannot retrieve contributors at this time
/* |
* Licensed to the Apache Software Foundation (ASF) under one or more |
* contributor license agreements. See the NOTICE file distributed with |
* this work for additional information regarding copyright ownership. |
* The ASF licenses this file to You under the Apache License, Version 2.0 |
* (the 'License'); you may not use this file except in compliance with |
* the License. You may obtain a copy of the License at |
* |
* http://www.apache.org/licenses/LICENSE-2.0 |
* |
* Unless required by applicable law or agreed to in writing, software |
* distributed under the License is distributed on an 'AS IS' BASIS, |
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
* See the License for the specific language governing permissions and |
* limitations under the License. |
*/ |
packageorg.apache.tika.parser.ocr; |
importorg.apache.commons.exec.CommandLine; |
importorg.apache.commons.exec.DefaultExecutor; |
importorg.apache.commons.exec.PumpStreamHandler; |
importorg.apache.commons.io.FileUtils; |
importorg.apache.commons.io.IOUtils; |
importorg.apache.commons.lang3.SystemUtils; |
importorg.apache.tika.config.Field; |
importorg.apache.tika.config.Initializable; |
importorg.apache.tika.config.InitializableProblemHandler; |
importorg.apache.tika.config.Param; |
importorg.apache.tika.exception.TikaConfigException; |
importorg.apache.tika.exception.TikaException; |
importorg.apache.tika.io.TemporaryResources; |
importorg.apache.tika.io.TikaInputStream; |
importorg.apache.tika.metadata.Metadata; |
importorg.apache.tika.mime.MediaType; |
importorg.apache.tika.mime.MediaTypeRegistry; |
importorg.apache.tika.parser.AbstractParser; |
importorg.apache.tika.parser.CompositeParser; |
importorg.apache.tika.parser.ParseContext; |
importorg.apache.tika.parser.Parser; |
importorg.apache.tika.parser.external.ExternalParser; |
importorg.apache.tika.parser.image.ImageParser; |
importorg.apache.tika.parser.image.TiffParser; |
importorg.apache.tika.parser.jpeg.JpegParser; |
importorg.apache.tika.sax.OfflineContentHandler; |
importorg.apache.tika.sax.XHTMLContentHandler; |
importorg.apache.tika.utils.XMLReaderUtils; |
importorg.slf4j.Logger; |
importorg.slf4j.LoggerFactory; |
importorg.xml.sax.Attributes; |
importorg.xml.sax.ContentHandler; |
importorg.xml.sax.SAXException; |
importorg.xml.sax.helpers.DefaultHandler; |
importjavax.imageio.ImageIO; |
importjava.awt.Image; |
importjava.awt.image.BufferedImage; |
importjava.io.ByteArrayOutputStream; |
importjava.io.File; |
importjava.io.FileInputStream; |
importjava.io.FileOutputStream; |
importjava.io.IOException; |
importjava.io.InputStream; |
importjava.io.InputStreamReader; |
importjava.io.OutputStreamWriter; |
importjava.io.Reader; |
importjava.nio.charset.Charset; |
importjava.nio.file.Files; |
importjava.nio.file.Paths; |
importjava.nio.file.StandardCopyOption; |
importjava.util.ArrayList; |
importjava.util.Arrays; |
importjava.util.Collections; |
importjava.util.HashMap; |
importjava.util.HashSet; |
importjava.util.List; |
importjava.util.Locale; |
importjava.util.Map; |
importjava.util.Set; |
importjava.util.concurrent.Callable; |
importjava.util.concurrent.ExecutionException; |
importjava.util.concurrent.FutureTask; |
importjava.util.concurrent.TimeUnit; |
importjava.util.concurrent.TimeoutException; |
import staticjava.nio.charset.StandardCharsets.UTF_8; |
/** |
* TesseractOCRParser powered by tesseract-ocr engine. To enable this parser, |
* create a {@link TesseractOCRConfig} object and pass it through a |
* ParseContext. Tesseract-ocr must be installed and on system path or the path |
* to its root folder must be provided: |
* <p> |
* TesseractOCRConfig config = new TesseractOCRConfig();<br> |
* //Needed if tesseract is not on system path<br> |
* config.setTesseractPath(tesseractFolder);<br> |
* parseContext.set(TesseractOCRConfig.class, config);<br> |
* </p> |
* |
* |
*/ |
publicclassTesseractOCRParserextendsAbstractParserimplementsInitializable { |
privatestaticfinalLoggerLOG=LoggerFactory.getLogger(TesseractOCRParser.class); |
privatestaticvolatilebooleanHAS_WARNED=false; |
privatestaticfinalObject[] LOCK=newObject[0]; |
privatestaticfinallong serialVersionUID =-8167538283213097265L; |
privatestaticfinalSet<MediaType>SUPPORTED_TYPES=Collections.unmodifiableSet( |
newHashSet<>(Arrays.asList(newMediaType[]{ |
MediaType.image('png'), MediaType.image('jpeg'), MediaType.image('tiff'), |
MediaType.image('bmp'), MediaType.image('gif'), MediaType.image('jp2'), |
MediaType.image('jpx'), MediaType.image('x-portable-pixmap') |
}))); |
privatefinalTesseractOCRConfig defaultConfig =newTesseractOCRConfig(); |
privatestaticMap<String,Boolean>TESSERACT_PRESENT=newHashMap<>(); |
privatestaticMap<String,Boolean>IMAGE_MAGICK_PRESENT=newHashMap<>(); |
@Override |
publicSet<MediaType>getSupportedTypes(ParseContextcontext) { |
// If Tesseract is installed, offer our supported image types |
TesseractOCRConfig config = context.get(TesseractOCRConfig.class, defaultConfig); |
if (hasTesseract(config)) { |
returnSUPPORTED_TYPES; |
} |
// Otherwise don't advertise anything, so the other image parsers |
// can be selected instead |
returnCollections.emptySet(); |
} |
privatevoidsetEnv(TesseractOCRConfigconfig, ProcessBuilderpb) { |
String tessdataPrefix ='TESSDATA_PREFIX'; |
Map<String, String> env = pb.environment(); |
if (!config.getTessdataPath().isEmpty()) { |
env.put(tessdataPrefix, config.getTessdataPath()); |
} |
elseif(!config.getTesseractPath().isEmpty()) { |
env.put(tessdataPrefix, config.getTesseractPath()); |
} |
} |
publicbooleanhasTesseract(TesseractOCRConfigconfig) { |
// Fetch where the config says to find Tesseract |
String tesseract = config.getTesseractPath() + getTesseractProg(); |
// Have we already checked for a copy of Tesseract there? |
if (TESSERACT_PRESENT.containsKey(tesseract)) { |
returnTESSERACT_PRESENT.get(tesseract); |
} |
//prevent memory bloat |
if (TESSERACT_PRESENT.size() >100) { |
TESSERACT_PRESENT.clear(); |
} |
//check that the parent directory exists |
if (! config.getTesseractPath().isEmpty() && |
!Files.isDirectory(Paths.get(config.getTesseractPath()))) { |
TESSERACT_PRESENT.put(tesseract, false); |
returnfalse; |
} |
// Try running Tesseract from there, and see if it exists + works |
String[] checkCmd = { tesseract }; |
boolean hasTesseract =ExternalParser.check(checkCmd); |
TESSERACT_PRESENT.put(tesseract, hasTesseract); |
return hasTesseract; |
} |
privatebooleanhasImageMagick(TesseractOCRConfigconfig) { |
// Fetch where the config says to find ImageMagick Program |
StringImageMagick= getImageMagickPath(config); |
// Have we already checked for a copy of ImageMagick Program there? |
if (IMAGE_MAGICK_PRESENT.containsKey(ImageMagick)) { |
returnIMAGE_MAGICK_PRESENT.get(ImageMagick); |
} |
//prevent memory bloat |
if (IMAGE_MAGICK_PRESENT.size() >100) { |
IMAGE_MAGICK_PRESENT.clear(); |
} |
//check that directory exists |
if (!config.getImageMagickPath().isEmpty() && |
!Files.isDirectory(Paths.get(config.getImageMagickPath()))) { |
IMAGE_MAGICK_PRESENT.put(ImageMagick, false); |
returnfalse; |
} |
if (SystemUtils.IS_OS_WINDOWS&& config.getImageMagickPath().isEmpty()) { |
LOG.warn('Must specify path for imagemagick on Windows OS to avoid accidental confusion with convert.exe'); |
IMAGE_MAGICK_PRESENT.put(ImageMagick, false); |
returnfalse; |
} |
// Try running ImageMagick program from there, and see if it exists + works |
String[] checkCmd = { ImageMagick }; |
boolean hasImageMagick =ExternalParser.check(checkCmd); |
IMAGE_MAGICK_PRESENT.put(ImageMagick, hasImageMagick); |
return hasImageMagick; |
} |
privateStringgetImageMagickPath(TesseractOCRConfigconfig) { |
return config.getImageMagickPath() + getImageMagickProg(); |
} |
staticbooleanhasPython() { |
// check if python is installed and it has the required dependencies for the rotation program to run |
boolean hasPython =false; |
TemporaryResources tmp =null; |
try { |
tmp =newTemporaryResources(); |
File importCheck = tmp.createTemporaryFile(); |
String prg ='import numpy, matplotlib, skimage, _tkinter'; |
OutputStreamWriter out =newOutputStreamWriter(newFileOutputStream(importCheck), Charset.forName('UTF-8')); |
out.write(prg); |
out.close(); |
Process p =Runtime.getRuntime().exec('python '+ importCheck.getAbsolutePath()); |
if (p.waitFor() 0) { |
hasPython =true; |
} |
} catch (Exception e) { |
} finally { |
IOUtils.closeQuietly(tmp); |
} |
return hasPython; |
} |
publicvoidparse(Imageimage, ContentHandlerhandler, Metadatametadata, ParseContextcontext) throwsIOException, |
SAXException, TikaException { |
TemporaryResources tmp =newTemporaryResources(); |
FileOutputStream fos =null; |
TikaInputStream tis =null; |
try { |
int w = image.getWidth(null); |
int h = image.getHeight(null); |
BufferedImage bImage =newBufferedImage(w, h, BufferedImage.TYPE_INT_RGB); |
File file = tmp.createTemporaryFile(); |
fos =newFileOutputStream(file); |
ImageIO.write(bImage, 'png', fos); |
tis =TikaInputStream.get(file); |
parse(tis, handler, metadata, context); |
} finally { |
tmp.dispose(); |
if (tis !=null) |
tis.close(); |
if (fos !=null) |
fos.close(); |
} |
} |
@Override |
publicvoidparse(InputStreamstream, ContentHandlerhandler, Metadatametadata, ParseContextparseContext) |
throwsIOException, SAXException, TikaException { |
TesseractOCRConfig config = parseContext.get(TesseractOCRConfig.class, defaultConfig); |
// If Tesseract is not on the path with the current config, do not try to run OCR |
// getSupportedTypes shouldn't have listed us as handling it, so this should only |
// occur if someone directly calls this parser, not via DefaultParser or similar |
if (! hasTesseract(config)) |
return; |
TemporaryResources tmp =newTemporaryResources(); |
try { |
TikaInputStream tikaStream =TikaInputStream.get(stream, tmp); |
//trigger the spooling to a tmp file if the stream wasn't |
//already a TikaInputStream that contained a file |
tikaStream.getPath(); |
//this is the text output file name specified on the tesseract |
//commandline. The actual output file name will have a suffix added. |
File tmpOCROutputFile = tmp.createTemporaryFile(); |
// Temporary workaround for TIKA-1445 - until we can specify |
// composite parsers with strategies (eg Composite, Try In Turn), |
// always send the image onwards to the regular parser to have |
// the metadata for them extracted as well |
_TMP_IMAGE_METADATA_PARSER.parse(tikaStream, newDefaultHandler(), metadata, parseContext); |
XHTMLContentHandler xhtml =newXHTMLContentHandler(handler, metadata); |
xhtml.startDocument(); |
parse(tikaStream, tmpOCROutputFile, parseContext, xhtml, config); |
xhtml.endDocument(); |
} finally { |
tmp.dispose(); |
} |
} |
/** |
* Use this to parse content without starting a new document. |
* This appends SAX events to xhtml without re-adding the metadata, body start, etc. |
* |
* @param stream inputstream |
* @param xhtml handler |
* @param config TesseractOCRConfig to use for this parse |
* @throws IOException |
* @throws SAXException |
* @throws TikaException |
* |
* @deprecated use {@link #parseInline(InputStream, XHTMLContentHandler, ParseContext, TesseractOCRConfig)} |
*/ |
publicvoidparseInline(InputStreamstream, XHTMLContentHandlerxhtml, TesseractOCRConfigconfig) |
throwsIOException, SAXException, TikaException { |
parseInline(stream, xhtml, newParseContext(), config); |
} |
/** |
* Use this to parse content without starting a new document. |
* This appends SAX events to xhtml without re-adding the metadata, body start, etc. |
* |
* @param stream inputstream |
* @param xhtml handler |
* @param config TesseractOCRConfig to use for this parse |
* @throws IOException |
* @throws SAXException |
* @throws TikaException |
* |
*/ |
publicvoidparseInline(InputStreamstream, XHTMLContentHandlerxhtml, ParseContextparseContext, |
TesseractOCRConfigconfig) |
throwsIOException, SAXException, TikaException { |
// If Tesseract is not on the path with the current config, do not try to run OCR |
// getSupportedTypes shouldn't have listed us as handling it, so this should only |
// occur if someone directly calls this parser, not via DefaultParser or similar |
if (! hasTesseract(config)) |
return; |
TemporaryResources tmp =newTemporaryResources(); |
try { |
TikaInputStream tikaStream =TikaInputStream.get(stream, tmp); |
File tmpImgFile = tmp.createTemporaryFile(); |
parse(tikaStream, tmpImgFile, parseContext, xhtml, config); |
} finally { |
tmp.dispose(); |
} |
} |
/** |
* This method is used to process the image to an OCR-friendly format. |
* @param scratchFile input image to be processed |
* @param config TesseractOCRconfig class to get ImageMagick properties |
* @throws IOException if an input error occurred |
* @throws TikaException if an exception timed out |
*/ |
privatevoidprocessImage(FilescratchFile, TesseractOCRConfigconfig) throwsIOException, TikaException { |
// fetch rotation script from resources |
InputStream in = getClass().getResourceAsStream('rotation.py'); |
TemporaryResources tmp =newTemporaryResources(); |
File rotationScript = tmp.createTemporaryFile(); |
Files.copy(in, rotationScript.toPath(), StandardCopyOption.REPLACE_EXISTING); |
CommandLine commandLine =newCommandLine('python'); |
String[] args = {'-W', |
'ignore', |
rotationScript.getAbsolutePath(), |
'-f', |
scratchFile.getAbsolutePath()}; |
commandLine.addArguments(args, true); |
String angle ='0'; |
DefaultExecutor executor =newDefaultExecutor(); |
ByteArrayOutputStream outputStream =newByteArrayOutputStream(); |
PumpStreamHandler streamHandler =newPumpStreamHandler(outputStream); |
executor.setStreamHandler(streamHandler); |
// determine the angle of rotation required to make the text horizontal |
if(config.getApplyRotation() && hasPython()) { |
try { |
executor.execute(commandLine); |
String tmpAngle = outputStream.toString('UTF-8').trim(); |
//verify that you've gotten a numeric value out |
Double.parseDouble(tmpAngle); |
angle = tmpAngle; |
} catch(Exception e) { |
} |
} |
// process the image - parameter values can be set in TesseractOCRConfig.properties |
commandLine =newCommandLine(getImageMagickPath(config)); |
args =newString[]{ |
'-density', Integer.toString(config.getDensity()), |
'-depth ', Integer.toString(config.getDepth()), |
'-colorspace', config.getColorspace(), |
'-filter', config.getFilter(), |
'-resize', config.getResize() +'%', |
'-rotate', angle, |
scratchFile.getAbsolutePath(), |
scratchFile.getAbsolutePath() |
}; |
commandLine.addArguments(args, true); |
try { |
executor.execute(commandLine); |
} catch(Exception e) { |
} |
tmp.close(); |
} |
privatevoidparse(TikaInputStreamtikaInputStream, FiletmpOCROutputFile, ParseContextparseContext, |
XHTMLContentHandlerxhtml, TesseractOCRConfigconfig) |
throwsIOException, SAXException, TikaException { |
File tmpTxtOutput =null; |
try { |
File input = tikaInputStream.getFile(); |
long size = tikaInputStream.getLength(); |
if (size >= config.getMinFileSizeToOcr() && size <= config.getMaxFileSizeToOcr()) { |
// Process image if ImageMagick Tool is present |
if(config.isEnableImageProcessing() 1&& hasImageMagick(config)) { |
// copy the contents of the original input file into a temporary file |
// which will be preprocessed for OCR |
TemporaryResources tmp =newTemporaryResources(); |
try { |
File tmpFile = tmp.createTemporaryFile(); |
FileUtils.copyFile(input, tmpFile); |
processImage(tmpFile, config); |
doOCR(tmpFile, tmpOCROutputFile, config); |
} finally { |
if (tmp !=null) { |
tmp.dispose(); |
} |
} |
} else { |
doOCR(input, tmpOCROutputFile, config); |
} |
// Tesseract appends the output type (.txt or .hocr) to output file name |
tmpTxtOutput =newFile(tmpOCROutputFile.getAbsolutePath() +'.'+ |
config.getOutputType().toString().toLowerCase(Locale.US)); |
if (tmpTxtOutput.exists()) { |
try (InputStream is =newFileInputStream(tmpTxtOutput)) { |
if (config.getOutputType().equals(TesseractOCRConfig.OUTPUT_TYPE.HOCR)) { |
extractHOCROutput(is, parseContext, xhtml); |
} else { |
extractOutput(is, xhtml); |
} |
} |
} |
} |
} finally { |
if (tmpTxtOutput !=null) { |
tmpTxtOutput.delete(); |
} |
} |
} |
/** |
* no-op |
* @param params params to use for initialization |
* @throws TikaConfigException |
*/ |
@Override |
publicvoidinitialize(Map<String, Param>params) throwsTikaConfigException { |
} |
@Override |
publicvoidcheckInitialization(InitializableProblemHandlerproblemHandler) |
throwsTikaConfigException { |
//this will incorrectly trigger for people who turn off Tesseract |
//by sending in a bogus tesseract path via a custom TesseractOCRConfig. |
//TODO: figure out how to solve that. |
if (! hasWarned()) { |
if (hasTesseract(defaultConfig)) { |
problemHandler.handleInitializableProblem(this.getClass().getName(), |
'Tesseract OCR is installed and will be automatically applied to image files unlessn'+ |
'you've excluded the TesseractOCRParser from the default parser.n'+ |
'Tesseract may dramatically slow down content extraction (TIKA-2359).n'+ |
'As of Tika 1.15 (and prior versions), Tesseract is automatically called.n'+ |
'In future versions of Tika, users may need to turn the TesseractOCRParser on via TikaConfig.'); |
warn(); |
} |
} |
} |
// TIKA-1445 workaround parser |
privatestaticParser_TMP_IMAGE_METADATA_PARSER=newCompositeImageParser(); |
privatestaticclassCompositeImageParserextendsCompositeParser { |
privatestaticfinallong serialVersionUID =-2398203346206381382L; |
privatestaticList<Parser> imageParsers =Arrays.asList(newParser[]{ |
newImageParser(), newJpegParser(), newTiffParser() |
}); |
CompositeImageParser() { |
super(newMediaTypeRegistry(), imageParsers); |
} |
} |
/** |
* Run external tesseract-ocr process. |
* |
* @param input |
* File to be ocred |
* @param output |
* File to collect ocr result |
* @param config |
* Configuration of tesseract-ocr engine |
* @throws TikaException |
* if the extraction timed out |
* @throws IOException |
* if an input error occurred |
*/ |
privatevoiddoOCR(Fileinput, Fileoutput, TesseractOCRConfigconfig) throwsIOException, TikaException { |
ArrayList<String> cmd =newArrayList<>(Arrays.asList( |
config.getTesseractPath() + getTesseractProg(), input.getPath(), output.getPath(), '-l', |
config.getLanguage(), '--psm', config.getPageSegMode() |
)); |
for (Map.Entry<String, String> entry : config.getOtherTesseractConfig().entrySet()) { |
cmd.add('-c'); |
cmd.add(entry.getKey() +'='+ entry.getValue()); |
} |
cmd.addAll(Arrays.asList( |
'-c', 'page_separator='+ config.getPageSeparator(), |
'-c', |
(config.getPreserveInterwordSpacing())?'preserve_interword_spaces=1':'preserve_interword_spaces=0', |
config.getOutputType().name().toLowerCase(Locale.US) |
)); |
ProcessBuilder pb =newProcessBuilder(cmd); |
setEnv(config, pb); |
finalProcess process = pb.start(); |
process.getOutputStream().close(); |
InputStream out = process.getInputStream(); |
InputStream err = process.getErrorStream(); |
logStream('OCR MSG', out, input); |
logStream('OCR ERROR', err, input); |
FutureTask<Integer> waitTask =newFutureTask<>(newCallable<Integer>() { |
publicIntegercall() throwsException { |
return process.waitFor(); |
} |
}); |
Thread waitThread =newThread(waitTask); |
waitThread.start(); |
try { |
waitTask.get(config.getTimeout(), TimeUnit.SECONDS); |
} catch (InterruptedException e) { |
waitThread.interrupt(); |
process.destroy(); |
Thread.currentThread().interrupt(); |
thrownewTikaException('TesseractOCRParser interrupted', e); |
} catch (ExecutionException e) { |
// should not be thrown |
} catch (TimeoutException e) { |
waitThread.interrupt(); |
process.destroy(); |
thrownewTikaException('TesseractOCRParser timeout', e); |
} |
} |
/** |
* Reads the contents of the given stream and write it to the given XHTML |
* content handler. The stream is closed once fully processed. |
* |
* @param stream |
* Stream where is the result of ocr |
* @param xhtml |
* XHTML content handler |
* @throws SAXException |
* if the XHTML SAX events could not be handled |
* @throws IOException |
* if an input error occurred |
*/ |
privatevoidextractOutput(InputStreamstream, XHTMLContentHandlerxhtml) throwsSAXException, IOException { |
xhtml.startElement('div', 'class', 'ocr'); |
try (Reader reader =newInputStreamReader(stream, UTF_8)) { |
char[] buffer =newchar[1024]; |
for (int n = reader.read(buffer); n !=-1; n = reader.read(buffer)) { |
if (n >0) { |
xhtml.characters(buffer, 0, n); |
} |
} |
} |
xhtml.endElement('div'); |
} |
privatevoidextractHOCROutput(InputStreamis, ParseContextparseContext, |
XHTMLContentHandlerxhtml) throwsTikaException, IOException, SAXException { |
if (parseContext null) { |
parseContext =newParseContext(); |
} |
xhtml.startElement('div', 'class', 'ocr'); |
XMLReaderUtils.parseSAX(is, newOfflineContentHandler(newHOCRPassThroughHandler(xhtml)), parseContext); |
xhtml.endElement('div'); |
} |
/** |
* Starts a thread that reads the contents of the standard output or error |
* stream of the given process to not block the process. The stream is closed |
* once fully processed. |
*/ |
privatevoidlogStream(finalStringlogType, finalInputStreamstream, finalFilefile) { |
newThread() { |
publicvoidrun() { |
Reader reader =newInputStreamReader(stream, UTF_8); |
StringBuilder out =newStringBuilder(); |
char[] buffer =newchar[1024]; |
try { |
for (int n = reader.read(buffer); n !=-1; n = reader.read(buffer)) |
out.append(buffer, 0, n); |
} catch (IOException e) { |
} finally { |
IOUtils.closeQuietly(stream); |
} |
LOG.debug('{}', out); |
} |
}.start(); |
} |
staticStringgetTesseractProg() { |
returnSystem.getProperty('os.name').startsWith('Windows') ?'tesseract.exe':'tesseract'; |
} |
staticStringgetImageMagickProg() { |
returnSystem.getProperty('os.name').startsWith('Windows') ?'convert.exe':'convert'; |
} |
privatestaticclassHOCRPassThroughHandlerextendsDefaultHandler { |
privatefinalContentHandler xhtml; |
publicstaticfinalSet<String>IGNORE= unmodifiableSet( |
'html', 'head', 'title', 'meta', 'body'); |
publicHOCRPassThroughHandler(ContentHandlerxhtml) { |
this.xhtml = xhtml; |
} |
/** |
* Starts the given element. Table cells and list items are automatically |
* indented by emitting a tab character as ignorable whitespace. |
*/ |
@Override |
publicvoidstartElement( |
Stringuri, Stringlocal, Stringname, Attributesattributes) |
throwsSAXException { |
if (!IGNORE.contains(name)) { |
xhtml.startElement(uri, local, name, attributes); |
} |
} |
/** |
* Ends the given element. Block elements are automatically followed |
* by a newline character. |
*/ |
@Override |
publicvoidendElement(Stringuri, Stringlocal, Stringname) throwsSAXException { |
if (!IGNORE.contains(name)) { |
xhtml.endElement(uri, local, name); |
} |
} |
/** |
* @see <ahref='https://issues.apache.org/jira/browse/TIKA-210'>TIKA-210</a> |
*/ |
@Override |
publicvoidcharacters(char[] ch, intstart, intlength) throwsSAXException { |
xhtml.characters(ch, start, length); |
} |
privatestaticSet<String>unmodifiableSet(String.. elements) { |
returnCollections.unmodifiableSet( |
newHashSet<>(Arrays.asList(elements))); |
} |
} |
protectedbooleanhasWarned() { |
if (HAS_WARNED) { |
returntrue; |
} |
synchronized (LOCK) { |
if (HAS_WARNED) { |
returntrue; |
} |
returnfalse; |
} |
} |
protectedvoidwarn() { |
HAS_WARNED=true; |
} |
@Field |
publicvoidsetTesseractPath(StringtesseractPath) { |
defaultConfig.setTesseractPath(tesseractPath); |
} |
@Field |
publicvoidsetTessdataPath(StringtessdataPath) { |
defaultConfig.setTessdataPath(tessdataPath); |
} |
@Field |
publicvoidsetLanguage(Stringlanguage) { |
defaultConfig.setLanguage(language); |
} |
@Field |
publicvoidsetPageSegMode(StringpageSegMode) { |
defaultConfig.setPageSegMode(pageSegMode); |
} |
@Field |
publicvoidsetMinFileSizeToOcr(longminFileSizeToOcr) { |
defaultConfig.setMinFileSizeToOcr(minFileSizeToOcr); |
} |
@Field |
publicvoidsetTimeout(inttimeout) { |
defaultConfig.setTimeout(timeout); |
} |
@Field |
publicvoidsetOutputType(StringoutputType) { |
defaultConfig.setOutputType(outputType); |
} |
@Field |
publicvoidsetPreserveInterwordSpacing(booleanpreserveInterwordSpacing) { |
defaultConfig.setPreserveInterwordSpacing(preserveInterwordSpacing); |
} |
@Field |
publicvoidsetEnableImageProcessing(intenableImageProcessing) { |
defaultConfig.setEnableImageProcessing(enableImageProcessing); |
} |
@Field |
publicvoidsetImageMagickPath(StringimageMagickPath) { |
defaultConfig.setImageMagickPath(imageMagickPath); |
} |
@Field |
publicvoidsetDensity(intdensity) { |
defaultConfig.setDensity(density); |
} |
@Field |
publicvoidsetDepth(intdepth) { |
defaultConfig.setDepth(depth); |
} |
@Field |
publicvoidsetColorspace(Stringcolorspace) { |
defaultConfig.setColorspace(colorspace); |
} |
@Field |
publicvoidsetFilter(Stringfilter) { |
defaultConfig.setFilter(filter); |
} |
@Field |
publicvoidsetResize(intresize) { |
defaultConfig.setResize(resize); |
} |
@Field |
publicvoidsetApplyRotation(booleanapplyRotation) { |
defaultConfig.setApplyRotation(applyRotation); |
} |
publicTesseractOCRConfiggetDefaultConfig() { |
return defaultConfig; |
} |
} |
Copy lines Copy permalink