who we are

We are a project in Germany sponsored by netluchs.de and SuMa-eV (an association supporting the free information access). Because of our limited ressources we concentrate on our national audience. But international partners are also welcome, even if our work is only partially translated.

important note: We are no multi-billion-company! Here are humans, doing their work with much enthusiasm in their spare time. So please respect the netiquette!

our vision

the whole is more than the sum of its parts

We want to offer a platform to allow users to build their own customized searchengines. We call them "minisearchers". In contrast to our competitors, there won't be any precompiled index when you start (including all the spam). If the user enters a new URL, we start to crawl the site. Starting up a new index takes slightly longer - but you really get just the sites you want.

You can integrate the minisearcher in your website or use it just for yourself. There are some constraints concerning the load of our servers which need to be discussed individually. By now we scaled some special solutions up to several million searches per month.

We aggregate the minisearchers to a global index, which is of very hight quality. The index is offered to other SuMa-eV-projects like metager2 or special strategic partners like metager.de.

what we offer

Opencrawl is a platform for several "minisearchers". You get a (simple) admin-interface where you add your URLs. Each URL gets a "search-scope" like "subtree", "all the domain" or "domain and all subdomains". Then our crawler builds step by step an index only with theese pages. You can can query this index using a http-get-query. The results are returned in the xml/opensearch format.

the limitations

Please note, that the service is run by a NGO. With building your index and offering it to you without charges, there are serveral constraints:
  1. we need the permission to use this index for our searchenginges too.
  2. we need to know something about the "load" caused by your minisearcher. One very critical parameter is the number of search-queries per month, you are going to expect.
  3. we have to know something about you. We have to obey to the German laws. We won't index sites covering the themes "sex, porn, drugs, gambling, nazi, anti democratic, terrorism, copyrighted material" and any other illegal content.

i want to be a part of it

If you have read so far and still want to be a part of our project, the next step is pretty simple: just send an email to info@opencrawl.de and tell us
  • Who are you?
  • What do you want to crawl?
  • How do you want to use the data?
  • How many search requests per month do you expect?
Because of the high volume of mailspam: use as subject of your mail "Opencrawl - international"

We are looking forward for your mail!


info@opencrawl.de - Impressum