Repository navigation
Conversation
github-actions
Bot
added
the
t-tooling
|
|
||
| A crawler without an `id` gets its creation index, such as `'0'`, and crawlers with an explicit `id` now count towards that index too. If you resume a v1 run with `purge_on_start=False` and mix both kinds of crawlers, a generated `id` can differ from the one v1 assigned, and the crawler won't find the state that `use_state` stored under the old one. | ||
|
|
||
| ## Only the first crawler uses the default request queue |
There was a problem hiding this comment.
Should there be some warning about multiprocessing cases breaking some assumptions and thus not behaving the same way out of the box?
There was a problem hiding this comment.
I don't think so. Our documentation doesn't contain any recommendations regarding multiprocessing.
| else: | ||
| self._id = id | ||
| self._id = str(BasicCrawler.__next_id) if id is None else id | ||
| self._default_rq_alias = None if BasicCrawler.__next_id == 0 else f'__default_{self._id}__' |
There was a problem hiding this comment.
self._default_rq_alias = None
Just double-checking since now is a good time for whatever breaking changes, are we sure that the default RQ should not have any alias?
There was a problem hiding this comment.
I'd keep it. Many users run a single crawler on the default queue, and this way nothing changes for them. Also, on the Apify platform, the default queue is the run's default storage, so with an alias the first crawler's requests would go to a separate unnamed queue instead of the one the Console shows for the run.
27ba605 to
f3a1043
Compare
vdusek
changed the title
vdusek
left a comment
vdusek
left a comment
There was a problem hiding this comment.
This is great.
My only concern is the internal aliases we use for these RQs, specifically the first one.
Take this example:
crawler_1 = BasicCrawler() # default RQ
crawler_2 = BasicCrawler() # alias `__default_1__`
crawler_3 = BasicCrawler(id='blah') # alias `__default_blah__`With the FS storage backend, this ends up on disk as:
storage/request_queues/
├── default/
├── __default_1__/
└── __default_blah__/
Isn't this weird? The first queue breaks the pattern: it has no __ and no counter.
I'd find one consistent pattern clearer:
__default_{default_storage_counter}__for crawlers without explicitid__default_{crawler_id}__for crawlers with explicitid
WDYT? Or is there any reason not do to it?
cc @janbuchar because of JS implementation
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
acf91ba to
2591a47
Compare
This branch has not been deployed
4 participants
Description
Crawlers created without a
request_managerno longer share the default request queue. The first crawler in the process still uses it, and every later one opens its own queue under the alias__default_{id}__, so crawlers don't share requests or their deduplication. The upgrade guide has a section for the change.Issues
✍️ Drafted by Claude Code