Azure AI Search Solution: Optimize Your Index Size And Schema

By Ashwin Venugopal -

 




Optimize Your Index Size and Schema

The effectiveness of your search queries is closely linked to the size and intricacy of your indexes. The quicker Azure AI Search can answer queries, the more compact and streamlined your indexes are.

If you don't pay attention, indexes can grow over time. You should review that all the documents in your index are still relevant and need to be searchable. Consider reviewing all the attributes you've enabled on each field. For example, adding support for filters, facets, and sorting can quadruple the storage needed to support your index.

Improve Performance Of Your Queries

If you're familiar with the workings of the search service, you can optimize your queries to significantly enhance performance. Use this checklist for writing better queries:

  • Only specify the fields you need to search using the searchFields parameter. As more fields require extra processing.

  • Return the smallest number of fields you need to render on your search results page. Returning more data takes more time.

  • Try to avoid partial search terms like prefix search or regular expressions. These kinds of searches are more computationally expensive.

  • Avoid using high skip values. This forces the search engine to retrieve and rank larger volumes of data.

  • Limit using facetable and filterable fields to low cardinality data.

  • Use search functions instead of individual values in filter criteria.

If you've applied all of the above and still have individual queries that don't perform, you can scale out your index.

Based on the service tier utilized for your search solution, you have the option to add as many as 12 partitions. Partitions represent the physical storage locations for your index. By default, all newly created search indexes come with a single partition. When you add more partitions, your index is distributed across them. For instance, if your index size is 200 GB and you have four partitions, each one will hold 50 GB of your index.

Because the search engine can operate in parallel in each segment, adding more partitions can improve speed. The queries that use facets to provide counts over large numbers of documents and those that return a big number of documents show the greatest benefits. This contributes to the computational cost of determining a document's significance.

Conclusion

We have successfully learnt about optimizing index size and schema.







Comments

Post a Comment

Popular posts from this blog

Connect Data to Azure Sentinel Using Data Connectors

By Ashwin Venugopal -
Image
  Ingest Log Data with Data Connectors You should connect your data sources with Azure Sentinel Connectors to collect the log data whose page displays a growing list of the connectors provided by Azure Sentinel, after selecting the Open connector page (the detailed connector page has a left and right blade).  The information about the connector, the connector's status, and the last time a log was received if connected is provided by the left blade. Whereas the right tab has two tabs- Instructions and Next Steps. The instructions tab can be different based on the connector having Prerequisites and Configuration which can be followed to connect to the data source. The Next Steps tab offers a quick reference to workbooks, query samples, and analytical templates. Data connectors can only be disconnected/deactivated, not deleted. Note-  The connector does not install Workbooks and Analytical Templates as they are already available in the Sentinel environment for out of the box...
Read more

Information Protection Scanner: Resolve Issues with Information Protection Scanner Deployment

By Ashwin Venugopal -
Image
  About If you are experiencing problems with the Microsoft Purview Information Protection scanner, check the health of your deployment by utilizing the Start-ScannerDiagnosticsPowerShell cmdlet to start the scanner diagnostic tool. The diagnostics tool checks the following details and then creates a log files with the results: Whether the database is up to date. Whether network URLs are accessible. Whether there's a valid authentication token and policy can be acquired. Whether the profile is defined in the Azure portal. Whether offline/online configuration exists and can be acquired. Whether the rules configured are valid.  The Start-ScannerDiagnostics command does not perform a comprehensive check of prerequisites. If you encounter problems with the scanner, you need to verify that your system meets the scanner's requirements, and that your scanner's configuration and installation are finished.  Verify Scanning Details Per Scanner Node and Repository Run the Get-ScanSt...
Read more

Azure AI Search plugin in Microsoft Security Copilot (Preview)

By Ashwin Venugopal -
Image
  Introduction Azure AI Search is a service that allows to reliably extract content search, extract, and understand at scale. It is typically used to search for documents, data surveys, and chat-style Copilot apps over proprietary data. Indexing and querying are Azure AI Search's two main workloads. While querying allows to search the content from index, indexing loads the content into the search engine and makes it searchable. By connecting to one of the Azure AI Search indexes from within Copilot, the Microsoft Security Copilot integration for Azure enables to use the content in Copilot to produce contextualized, pertinent, and targeted responses.  Requirement Before using the Azure AI Search plugin in Copilot, the desired Azure AI Search index to connect to Copilot is configured as follows: The text field must be searchable.  The title field must be filterable. The vector-field must use text-embedding-ada-002. Integrated vectorization can be used to setup an index in t...
Read more