Need to think about how best to implement this - utilities/file_location_listing#16 introduced multi-threading so it's not as simple as just maintaining a counter.
Options:
run a counter per thread and then apply the limit again once the results have been concatenated
Divide the limit by CONCURRENCY and pass that into the threads
Neither is perfect. If we split the counter between threads we might end up (artificially) getting lower than the desired number (if thread1 could have returned 1500 items and thread2 10, giving them a limit of 500 each isn't going to work very well).
If we have each thread collect up to the limit and then truncate down, we're going to be doing more work than necessary: with a limit of 1000 and 4 threads, we might collect 4000 items and then only actually return 1000 to the browser.
I think, for simplicities sake, though, we'll have each thread observe the provided counter. It's not particularly efficient, but it's still more efficient than chucking 4K items into the browser unnecessarily
Activity
30-Dec-23 15:45
assigned to @btasker
30-Dec-23 15:45
mentioned in issue #16
30-Dec-23 15:50
Need to think about how best to implement this - utilities/file_location_listing#16 introduced multi-threading so it's not as simple as just maintaining a counter.
Options:
CONCURRENCYand pass that into the threadsNeither is perfect. If we split the counter between threads we might end up (artificially) getting lower than the desired number (if
thread1could have returned 1500 items and thread210, giving them a limit of 500 each isn't going to work very well).If we have each thread collect up to the limit and then truncate down, we're going to be doing more work than necessary: with a limit of 1000 and 4 threads, we might collect 4000 items and then only actually return 1000 to the browser.
30-Dec-23 15:51
I think, for simplicities sake, though, we'll have each thread observe the provided counter. It's not particularly efficient, but it's still more efficient than chucking 4K items into the browser unnecessarily
30-Dec-23 16:11
mentioned in commit 0b94f28f9cc2769727e74745f1091296a31c74db
Commit: 0b94f28f9cc2769727e74745f1091296a31c74db Author: B Tasker Date: 2023-12-30T16:10:45.000+00:00Message
feat: implement and honour
limitattribute in javascript payload (utilities/file_location_listing#20)30-Dec-23 16:12
This is implemented.
The search portal will, by default, cap to 300 results - the search result count indicates whether this has been hit
Clicking
Cappedwill remove the cap and resubmit the search (any further searches will also be uncapped, until the browser tab is next refreshed)06-Jan-24 19:32
mentioned in issue #35
03-Mar-24 11:41
mentioned in issue #49