The robots.txt file is applied to limit online search engine crawlers from accessing sections of your website. Even though the file is very useful, It is also a simple way to inadvertently block crawlers. Presently, there is not any precise approach for searching for inclusion in choices like Google's https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD