Azure AI Search Solution: Understand Indexing Process

 

Understand The Indexing Process

For every indexed entity, a document is created as part of the indexing process. An enrichment pipeline produces the documents repeatedly during indexing, combining enriched fields retrieved by cognitive skills with metadata from the data source. You can think of each indexed document as a JSON structure, which initially consists of a document with the index fields you have mapped to fields extracted directly from the source data.

You can set up the indexer to extract image data from documents in the data source and add each image to a collection called normalized_images.

You can use the collection of photographs as an input for skills that extract information from image data by normalizing the image data in this manner. Every talent adds fields to the document. For instance, a skill that determines the language used in a document may store its result in a language field.

The document is structured hierarchically, and the skills are applied to a specific context within the hierarchy, enabling you to run the skill for each item at a particular level of the document.

Later in the pipeline, other skills can use the output fields from each skill as inputs to store their outputs in the document structure. The fields in the final document structure at the end of the pipeline are mapped to index fields by the indexer in one of two ways:

• All fields that are taken straight out of the source data are mapped to index fields. These mappings can be explicit (a mapping is established to match a source field to an index field, frequently to rename the field to something more useful or to apply a function to the data value when it is mapped) or implicit (fields are automatically mapped to in fields with the same name in the index).

• The target field in the index and the hierarchical placement of the output fields from the skills in the skillset are clearly mapped.

Search an Index

You can query an index to find information in the indexed document content after it has been built and filled. The majority of search solutions employ full text search semantics to query an index, while you could obtain index entries using basic field value matching.

Full Text Search

Full text search refers to search methods that identify query phrases by parsing text-based document contents. The Lucene query syntax, which offers a wide range of query operations for searching, filtering, and sorting data in indexes, is the foundation for full text search queries in Azure AI Search. Azure AI Search supports two variants of the Lucene syntax:

• Simple - An intuitive syntax that makes it easy to perform basic searches that match literal query terms submitted by a user.

• Full - An extended syntax that supports complex filtering, regular expressions, and other more sophisticated queries.

By providing a search phrase and additional parameters that dictate how the expression is assessed and the results are returned, client applications send queries to Azure AI Search. Some common parameters submitted with a query include:

1. search - A search expression that includes the terms to be found.
2. queryType - The Lucene syntax to be evaluated (simple or full).
3. searchFields - The index fields to be searched.
4. select - The fields to be included in the results.
5. searchMode - Criteria for including results based on multiple search terms

Query processing consists of four stages:

• Query parsing- A tree of suitable subqueries is created by evaluating and reconstructing the search query. Term queries (finding particular individual terms in the search expression, like hotel) and phrase queries (finding multi-term phrases given in quote marks in the search expression) are examples of subqueries.

• Lexical analysis - Linguistic rules are used to assess and improve the query terms. For example, unnecessary stopwords like "the," "a," "is," and so forth are eliminated and the text is changed to lower case. Next, words are broken down into their component phrases and reduced to their base form (for instance, "comfortable" may be reduced to "comfort").

• Document retrieval - The query terms are matched against the indexed terms, and the set of matching documents is identified.

• Scoring - A relevance score is assigned to each result based on a term frequency/inverse document frequency (TF/IDF) calculation.

Conclusion

We have successfully learnt about indexing process.