Skip to main content

It worked โ€” and it followed a link

Congratulations! If you're reading this from your target repository โ€” a committed document, a search index, wherever complex-config.xml pointed it โ€” you successfully ran the Norconex Web Crawler's complex example.

Reading this from a committed document rather than a browser? Then complex-config.xml did what it was built to demonstrate: variables resolved, the shared importer fragment got pulled in and interpreted, and โ€” if you ran it with -variables=./examples/complex/production.variables โ€” the conditional sections switched on too. Check the committed document's site field: it should read whichever siteName your variables file set.

This page is also where the crawl's depth setting earns its keep. The example allows following one level of links, and this page links to one companion page โ€” deliberately just the one, so what a small, contained crawl actually discovers stays predictable regardless of which profile you ran.

Pointing it at your own siteโ€‹

Once you understand what the example demonstrates, retarget it. Change the start reference in complex-config.xml (or override it from a variables file, the way production.variables overrides startUrl):

<startReferences>
<ref>https://www.your-own-site.com/</ref>
</startReferences>

And keep the crawl inside the section you actually want, rather than letting it wander the whole site:

<referenceFilters>
<referenceFilter class="GenericReferenceFilter">
<valueMatcher method="regex"
pattern="https://www\.your-own-site\.com/only-this-section/.*"/>
</referenceFilter>
</referenceFilters>

Nextโ€‹

  • Compare complex-config.xml against the rendered output of both profiles with configrender, to see exactly what the variables and conditionals produced.
  • Open the Configurator if you'd rather build a configuration visually than edit XML by hand.
  • Browse the configuration reference for everything used in this example โ€” ConstantTransformer, DeleteTransformer, GenericDelayResolver, and the rest.
  • See the source on GitHub for the full example, including its comments on Velocity templating gotchas.