Fine-Hybrid: Integration of BM25 And Finetuned SBERT to Enhance Search Relevance
DOI:
https://doi.org/10.34148/teknika.v14i2.1229Keywords:
Hybrid Search, BM25, SBERT, RRF, Generative AIAbstract
Legal information retrieval, particularly for tax law documents, faces significant challenges due to specialized terminology, complex hierarchical structures, and formal language patterns that existing search approaches inadequately address. Current methods either rely on lexical matching or use general semantic models, creating a critical gap in effectively retrieving relevant tax law information. This research develops a novel hybrid search system to enhance search result relevance for the General Provisions and Tax Procedures (KUP) dataset by integrating a lexical-based search method (BM25) with semantic search using Sentence-BERT (SBERT) that has been fine-tuned using a taxation corpus. Our methodology encompasses several innovative components: development of synthetic data using a two-stage LLM prompting approach for SBERT fine-tuning, implementation of a comprehensive query normalization system with taxation-specific terminology mapping, and integration of lexical and semantic results through Reciprocal Rank Fusion (RRF). We evaluate system performance with inputs from tax domain experts, demonstrating that the Fine-hybrid model consistently outperforms individual search methods, achieving a Precision@N of 66.021% and Average Recall of 76.51%. Our approach addresses the specific challenges of tax document retrieval while providing a generalizable framework applicable to other specialized domains with similar characteristics. This research contributes both theoretical advancements in hybrid search methodologies for legal documents and practical solutions for improving tax information accessibility, with implications for enhancing administrative efficiency and taxpayer compliance.
Downloads
References
[1] H. K. Azad and A. Deepak, ‘Query expansion techniques for information retrieval: A survey’, Inf Process Manag, vol. 56, no. 5, pp. 1698–1735, Sep. 2019, doi: 10.1016/j.ipm.2019.05.009.
[2] I. Chalkidis et al., ‘LexGLUE: A Benchmark Dataset for Legal Language Understanding in English’, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 4310–4330, May 2022, doi: 10.18653/v1/2022.acl-long.297.
[3] J. P. Salgado-Guerrero, D. F. Quisi-Peralta, M. Lopez-Nores, L. D. Paguay-Palaguachi, J. F. Murillo-Valarezo, and G. Cajamarca-Morquecho, ‘A New Hybrid Search Approach to Optimize the Retrieval of Information from the Website at the Universidad Politécnica Salesiana’, in Lecture Notes in Networks and Systems, Springer Science and Business Media Deutschland GmbH, 2024, pp. 247–257. doi: 10.1007/978-3-031-54235-0_23.
[4] S. Sharma and S. P. Panda, ‘Efficient information retrieval model: overcoming challenges in search engines-an overview’, Indonesian Journal of Electrical Engineering and Computer Science, vol. 32, no. 2, pp. 925–932, Nov. 2023, doi: 10.11591/ijeecs.v32.i2.pp925-932.
[5] S. Robertson and H. Zaragoza, ‘The probabilistic relevance framework: BM25 and beyond’, Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009, doi: 10.1561/1500000019.
[6] A. Esteva et al., ‘COVID-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization’, NPJ Digit Med, vol. 4, no. 1, Dec. 2021, doi: 10.1038/s41746-021-00437-0.
[7] N. Reimers and I. Gurevych, ‘Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks’, Aug. 2019, [Online]. Available: http://arxiv.org/abs/1908.10084
[8] H. S. Walsh, ‘Semantic Search With Sentence-BERT For Design Information Retrieval’, in Proceedings of the ASME 2022 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, Aug. 2022.
[9] Miguel A. Silva-Fuentes, Hugo D. Calderon-Vilca, Edwin F. Calderon-Vilca, and Flor C. Cardenas-Marino, ‘Semantic Search System using Word Embeddings for query expansion’, 2019 IEEE PES Innovative Smart Grid Technologies Conference - Latin America (ISGT Latin America), 2019, doi: 10.1109/ISGT-LA.2019.8894992.
[10] N. Yusuf et al., ‘Query Expansion Method For Quran Search Using Semantic Search And Lucene Ranking’, 2020.
[11] Andrew Wen, Yanshan Wang, Vinod C. Kaggal, Sijia Liu, Hongfang Liu, and Jungwei Fan, ‘Enhancing Clinical Information Retrieval through Context-Aware Queries and Indices’, IEEE International Conference on Big Data (Big Data), pp. 2800–2807, 2019, doi: 10.1109/BigData47090.2019.9006241.
[12] A. Pandey, A. Gupta, and V. Pudi, ‘CitRet: A Hybrid Model for Cited Text Span Retrieval’, Proceedings of the 29th International Conference on Computational Linguistics, pp. 4528–4536, 2022, [Online]. Available: http://www.nist.gov/tac/2014/BiomedSumm/
[13] M.-Y. Kim, J. Rabelo, K. Okeke, and R. Goebel, ‘Legal Information Retrieval and Entailment Based on BM25, Transformer and Semantic Thesaurus Methods’, The Review of Socionetwork Strategies, vol. 16, no. 1, pp. 157–174, Apr. 2022, doi: 10.1007/s12626-022-00103-1.
[14] G. V. Cormack, C. L. A. Clarke, and Stefan Buttcher, ‘Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods’, SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pp. 758–759, 2019, doi: 10.1145/1571941.1572114.
[15] Y. Yuan, Y. Liu, and L. Cheng, ‘A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models’, Apr. 2024, doi: 10.48550/arXiv.2404.14445.
[16] N. Fujishiro, Y. Otaki, and S. Kawachi, ‘Accuracy of the Sentence-BERT Semantic Search System for a Japanese Database of Closed Medical Malpractice Claims’, Applied Sciences (Switzerland), vol. 13, no. 6, Mar. 2023, doi: 10.3390/app13064051.
[17] C. M. Garcia, A. L. Koerich, A. de S. Britto, and J. P. Barddal, ‘Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams’, Mar. 2024, [Online]. Available: http://arxiv.org/abs/2403.15455
[18] J. Coelho, A. Neto, M. Tavares, C. Coutinho, R. Ribeiro, and F. Batista, ‘Semantic search of mobile applications using word embeddings’, in OpenAccess Series in Informatics, Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, Aug. 2021. doi: 10.4230/OASIcs.SLATE.2021.12.
[19] E. H. Mohamed and E. M. Shokry, ‘QSST: A Quranic Semantic Search Tool based on word embedding’, Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 3, pp. 934–945, Mar. 2022, doi: 10.1016/j.jksuci.2020.01.004.
[20] S. Gururangan et al., ‘Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks’, Apr. 2020, [Online]. Available: http://arxiv.org/abs/2004.10964
[21] A. Sihombing, A. Indrawati, A. Yaman, C. Trianggoro, L. P. Manik, and Z. Akbar, ‘A scientific expertise classification model based on experts’ self-claims using the semantic and the TF-IDF approach’, in ACM International Conference Proceeding Series, Association for Computing Machinery, Nov. 2022, pp. 301–305. doi: 10.1145/3575882.3575940.
[22] F. Sufi, ‘Addressing Data Scarcity in the Medical Domain: A GPT-Based Approach for Synthetic Data Generation and Feature Extraction’, Information (Switzerland), vol. 15, no. 5, May 2024, doi: 10.3390/info15050264.
[23] J. Opitz and A. Frank, ‘SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features’, in Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, Long Papers, Nov. 2022, pp. 625–638. doi: 10.18653/v1/2022.aacl-main.48.
[24] L. Gao, Z. Dai, and J. Callan, ‘COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List’, Apr. 2021, [Online]. Available: http://arxiv.org/abs/2104.07186
[25] V. Karpukhin et al., ‘Dense Passage Retrieval for Open-Domain Question Answering’, Apr. 2020, [Online]. Available: http://arxiv.org/abs/2004.04906
[26] H. Silalahi, ‘Taxation obligations of government institutions in Indonesia: Is it mandatory for government institutions in Indonesia to engage with taxable entrepreneurs?’, International Journal of Multidisciplinary Research and Growth Evaluation, vol. 4, no. 5, pp. 132–138, 2023, doi: 10.54660/.ijmrge.2023.4.5.132-138.
[27] A. P. Darma, ‘Implementation Of Legal Provisions In Indonesia Regarding Tax Imposition’, Lex Prospicit, vol. 2, Jul. 2024, doi: 10.1966/l.
[28] W. A. G. Kodri and M. Haris, ‘Fine-hybrid’, GitHub repository. [Online]. Available: https://github.com/ WanAhmadGK/Fine-hybrid
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Teknika
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
