Best Practices for Mosaicking Millions of Sparse COGs

[tariqksoliman]

Hi and I'm new to both STAC, titiler and titiler-pgstac,

I'm working on investigating the viability and performance of using titiler-pgstac instead of my homegrown solution to mosaicking. The specific site I am working on is: https://earth.jpl.nasa.gov/emit-mmgis-lb/?s=hw38j (what https://earth.jpl.nasa.gov/emit/data/data-portal/coverage-and-forecasts/ iframes).

VISION EMIT - NON-titiler-pgstac mosaicking

Right now, that site's mosaicking works by:

• Splitting gdal2tiles.py TMS tilesets into timestamped folders for each day (temporal aspect is a requirement)
• In a given day, using a workflow and a modified gdal2tiles.py script to tile individual TIFs on top of the existing tileset for that day (effectively accumulating the TMS tileset (if tiled in the right order))
• When the client/browser makes a query, it goes through a node/expressjs endpoint that crawls the relevant day folders within the requested time range, grabs the pre-tiled {z}/{x}/{y} image (if any) and draws the tiles on top of each other with node sharp before returning the tile to the user.

End-user performance is not an issue at all with the method. The problem, and why titiler-pgstac is being investigated, is that creating, moving and managing tiles for petabytes of data like this eats up a lot more space and is excruciatingly time consuming.

---

I've already setup titiler-pgstac and have a collection of ~100 COGs and have registered searches on it and viewed them with the /map endpoint. I do have a few questions (and perhaps more later):

• The docs begin with "TiTiler-PgSTAC is a TiTiler extension". I was initially confused because I thought titiler-pgstac should include all the titiler endpoints as well but it does not seem to. Am I right in thinking they would both need to be stood up side-by-side if I want the endpoints from both? (I'm more interested in using the docker images)

• What's the best practice for registering a new search query? In my case would I just make one global bounding-box query for all clients to use? Or would I register a new query each time a user pans or zooms the map or plays with the time range? Is there a way to unregister searches? I don't fully understand how they work. Whenever I tried making and using a global search ("bbox":-180,-80,180,80]), titiler-pgstac/uvicorn would frequently mention that child processes have died and restart the application and lock up all subsequent requests. (Edit: I realized that none of my "cogs" had overview tiles and once I converted them I no longer see such crashing.) Are there types of searches that should not be made and that I should prevent?

• I see in Missing / incomplete tiles where rasters' valid data borders each other #184 that setting exitwhenfull=false, skipcovered=false helps greatly when mosaicking nodata values. Is there any way to keep them true? They feels like they'd be very useful. Also what is scan_limit and item_limit more specifically? If I have 100 COG bboxes intersecting the tile I want mosaicked, item_limit=50 would only mosaic 50 of them together? If I have a millions COGs in my search and I have 100 COG bboxes intersecting the tile I want mosaicked, would scan_limit=50 potentially only give me 0 to 50 COGs being mosaicked here?

Thank you!

[vincentsarago]

Hi @tariqksoliman

Cool to see that JPL is looking at titiler-pgstac.

End-user performance is not an issue at all with the method. The problem, and why titiler-pgstac is being investigated, is that creating, moving and managing tiles for petabytes of data like this eats up a lot more space and is excruciatingly time consuming.

Yeah titiler-pgstac will not be as fast as serving pre-generated tiles but it will gives you more flexibility and less up front processing.

The docs begin with "TiTiler-PgSTAC is a TiTiler extension". I was initially confused because I thought titiler-pgstac should include all the titiler endpoints as well but it does not seem to. Am I right in thinking they would both need to be stood up side-by-side if I want the endpoints from both? (I'm more interested in using the docker images)

TiTiler-PgSTAC is not really an extension but re-use TiTiler base classes, meaning that if you create a custom application you can add all the endpoints you find in titiler in titiler-pgstac. We do not extent the default titiler.application because we would have duplicated (useless?) endpoint.

Note: in 1.4.0, we've added the possibility to have /extenal single COG endpoint https://github.com/stac-utils/titiler-pgstac/blob/main/CHANGES.md#140-2024-09-06

What's the best practice for registering a new search query? In my case would I just make one global bounding-box query for all clients to use? Or would I register a new query each time a user pans or zooms the map or plays with the time range?

There are 2 scenarios:

• Single collection: by default we have /collections/{collection_id}endpoints which can be used directly withoutregistering` anything. Those endpoints support few options like Ids, Datetime https://stac-utils.github.io/titiler-pgstac/latest/intro/#stac-collection-collectionscollection_id https://stac-utils.github.io/titiler-pgstac/latest/endpoints/collections_endpoints/

• Complex Search: For more complex searches, users first need to register a global search and use the SearchID in the endpoints

Or would I register a new query each time a user pans or zooms the map or plays with the time range?

The way titiler-pgstac works is by happening bbox for each tile request to the search (directly in pgstac) which is why we use one global Search (usually with Collection, datetime and Regional BBOX).

Is there a way to unregister searches?

No

I don't fully understand how they work. Whenever I tried making and using a global search ("bbox":-180,-80,180,80]), titiler-pgstac/uvicorn would frequently mention that child processes have died and restart the application and lock up all subsequent requests. (Edit: I realized that none of my "cogs" had overview tiles and once I converted them I no longer see such crashing.) Are there types of searches that should not be made and that I should prevent?

It's hard for the application to know what/how are the dataset, it's up to the Dataprovider to take care of this. You could work with COGs without overviews but would need to set the Minzoom to the zoom level corresponding the dataset resolution developmentseed/titiler#291

I see in #184 that setting exitwhenfull=false, skipcovered=false helps greatly when mosaicking nodata values. Is there any way to keep them true?

Yes with 1.4.0 you can use env variables

titiler-pgstac/titiler/pgstac/settings.py

Lines 123 to 145 in d212969

	class PgstacSettings(BaseSettings):
	"""Pgstac settings"""
	# Return as soon as we scan N items
	scan_limit: int = 10000
	# Return as soon as we have N items per geometru
	items_limit: int = 100
	# Return after N seconds to avoid long requests
	time_limit: int = 5
	# Return as soon as the geometry is fully covered
	exitwhenfull: bool = True
	# Skip any items that would show up completely under the previous items
	skipcovered: bool = True
	model_config = {
	"env_prefix": "TITILER_PGSTAC_SEARCH_",
	"env_file": ".env",
	"extra": "ignore",
	}

titiler-pgstac/.env.example

Line 27 in d212969

TITILER_PGSTAC_SEARCH_ITEMS_LIMIT=200

They feels like they'd be very useful. Also what is scan_limit and item_limit more specifically? If I have 100 COG bboxes intersecting the tile I want mosaicked, item_limit=50 would only mosaic 50 of them together?

No the item_limit is PER TILE bbox, in case where you would need more than 100 COG to create a single tiles, I would suggest to either create bigger COGs or re-work the search query.

If I have a millions COGs in my search and I have 100 COG bboxes intersecting the tile I want mosaicked, would scan_limit=50 potentially only give me 0 to 50 COGs being mosaicked here?

I feel you're trying to mosaic files a low zoom level, in this case you'll try to create a single tile with a lot of individual files. This is not ideal and the performance will not be great (GDAL will have to open a lot of files and maybe use only 2 or 3 pixels 🤷‍♂️ )

I hope that answer most of your question 👍