Context
Popular software is typically accompanied by well-written documentation. This documentation can be easily aquired. However, it’s utility is severely restricted because searching a collection of documentation is difficult. The current solution of this problem is to rely on a search engine like Google, which resolves queries - often structured as questions without a direct link to the keywords expected in the solution. The goal of this system is to provide a modular framework for local search operations of this nature. Such a system can be extended to index and search a website crawled and saved locally, acting as a search engine.
Existing Solutions
Dash
Dash is a proprietary browser for MacOS that searches documentation sets generated primarily from various developer API docs and manuals. Zeal - inspired by Dash - is an alternative for Linux and Windows machines that uses the same docsets. Devdocs is an open source API documentation browser that fills a similar role, but does not use the same docset format.
One short coming of these existing systems is that semantic search is not supported - the indexing is limited in this regard (although this could be justified for a documentation browser). The “docset” feature meets the requirement of modular data sources very well.
Hister
Hister is a search engine that consists of a browser extension, indexer, web app and terminal interface 1. It was developed by Adam Tauber, who also developed SearX - a metasearch engine that dispatches queries to other search engines. The indexer and user interface should serve as a good reference, but the overall application goals are not the same; the crawling mechanism does not meet the required scale and the index is likely not modular. Some technical aspects are well presented at https://hister.org/posts.
Prototype 1
More research needs to be done on the systems that are suitable for this application. Currently, a language model for query transformation and a fragmented meilisearch index for modular sources is being explored.