It worked โ and it followed a link
Reading this from a committed document rather than a browser? Then
complex-config.xml did what it was built to demonstrate: variables
resolved, the shared importer fragment got pulled in and interpreted, and โ
if you ran it with -variables=./examples/complex/production.variables โ
the conditional sections switched on too. Check the committed document's
site field: it should read whichever siteName your variables file set.
This page is also where the crawl's depth setting earns its keep. The example allows following one level of links, and this page links to one companion page โ deliberately just the one, so what a small, contained crawl actually discovers stays predictable regardless of which profile you ran.
Pointing it at your own siteโ
Once you understand what the example demonstrates, retarget it. Change the
start reference in complex-config.xml (or override it from a variables
file, the way production.variables overrides startUrl):
<startReferences>
<ref>https://www.your-own-site.com/</ref>
</startReferences>
And keep the crawl inside the section you actually want, rather than letting it wander the whole site:
<referenceFilters>
<referenceFilter class="GenericReferenceFilter">
<valueMatcher method="regex"
pattern="https://www\.your-own-site\.com/only-this-section/.*"/>
</referenceFilter>
</referenceFilters>
Nextโ
- Compare
complex-config.xmlagainst the rendered output of both profiles withconfigrender, to see exactly what the variables and conditionals produced. - Open the Configurator if you'd rather build a configuration visually than edit XML by hand.
- Browse the configuration reference for everything
used in this example โ
ConstantTransformer,DeleteTransformer,GenericDelayResolver, and the rest. - See the source on GitHub for the full example, including its comments on Velocity templating gotchas.