diff --git a/cardinal-android/app/src/main/java/earth/maps/cardinal/transit/TransitousService.kt b/cardinal-android/app/src/main/java/earth/maps/cardinal/transit/TransitousService.kt index 66626cde97171498b2a79a58f666f79b9d9f6efd..ee24f5b0db96b0bdde895a8390b6a8e28ae2dd87 100644 --- a/cardinal-android/app/src/main/java/earth/maps/cardinal/transit/TransitousService.kt +++ b/cardinal-android/app/src/main/java/earth/maps/cardinal/transit/TransitousService.kt @@ -43,7 +43,7 @@ class TransitousService @Inject constructor(private val appPreferenceRepository: private val client = HttpClient(Android) { install(UserAgent) { - agent = "Cardinal Maps https://github.com/ellenhp/cardinal ellen.h.poe@gmail.com" + agent = "Cardinal Maps https://gitlab.e.foundation/e/os/cardinal maps@murena.com" } install(ContentNegotiation) { json(Json { diff --git a/cardinal-geocoder/README.md b/cardinal-geocoder/README.md new file mode 100644 index 0000000000000000000000000000000000000000..b187701c176d7a1718ba31a86badb9663949f41b --- /dev/null +++ b/cardinal-geocoder/README.md @@ -0,0 +1,5 @@ +This code is derived from [Airmail](https://github.com/ellenhp/airmail). Its job is to let users search for POIs even if they don't have access to the internet. Like Airmail, this geocoder is built on top of the tantivy inverted index crate, and uses libpostal dictionaries for synonym substitution. + +The geocoder is written in Rust and is called by the Kotlin project with the help of [uniffi](https://mozilla.github.io/uniffi-rs/latest/). + +Most of the search code is in [index.rs](./src/index.rs) and the substitutions are in [substitutions.rs](./src/substitutions.rs). diff --git a/cardinal-geocoder/bin/uniffi-bindgen.rs b/cardinal-geocoder/bin/uniffi-bindgen.rs index d96eac70f9ee9cadca29d6f07833a5166d74799d..5f9345305d45fc7b5ecf4930a1c57f72e9edc3e2 100644 --- a/cardinal-geocoder/bin/uniffi-bindgen.rs +++ b/cardinal-geocoder/bin/uniffi-bindgen.rs @@ -1,3 +1,4 @@ +// This is run by the CI/CD job to create a Kotlin file definining the FFI interface of this crate, allowing it to be used from Kotlin. fn main() { uniffi::uniffi_bindgen_main() -} \ No newline at end of file +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000000000000000000000000000000000000..95f659be9eff485429ce8d49911eff2a3b5f79d3 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,13 @@ +## Cardinal Maps Architecture + +This project is composed of a frontend and a backend. The frontend is located in this repository. The backend is a bit more nebulous, and it can be broken up into a few main pieces. + +1. Tiles. Serves the data required for the client to render the map and perform offline area downloads This piece can be implemented without any code whatsoever using modern serverless tools. Data processing steps requried to run the backend are described in [BASEMAP_GENERATION.md](./BASEMAP_GENERATION.md) and [ROUTING_TILE_GEN.md](./ROUTING_TILE_GEN.md). +2. Searching. The search and autocomplete endpoints need to be compatible with the Pelias API. geocode.earth and stadia maps (v1 only) provide such API endpoints. +3. Routing. The routing endpoint must be valhalla or valhalla-compatible. Many providers offer such APIs like Mapbox, Stadia Maps, and others. +4. Trip planning (public transportation). Cardinal Maps requires a MOTIS-compatible API, like the one provided by Transitous. Cardinal maps has explicit permission to call the Transitous API. If you fork the app, you must get permission to call this API, self-host MOTIS yourself, or remove the public transit feature. Additionally, please change [the user agent](../cardinal-android/app/src/main/java/earth/maps/cardinal/transit/TransitousService.kt) providing your own contact information. + +### Frontend Architecture + +The frontend is composed of two main pieces of code. There's [an offline geocoder](../cardinal-geocoder/) responsible for allowing the user to search for POIs offline, and [the main app](../cardinal-android/) which is a Kotlin codebase using Jetpack/Android Compose, Hilt, and Room. The app is built using the Material 3 Expressive design system. + diff --git a/docs/BASEMAP_GENERATION.md b/docs/BASEMAP_GENERATION.md new file mode 100644 index 0000000000000000000000000000000000000000..6d6fd3f31f70b8636879217affa060b538602847 --- /dev/null +++ b/docs/BASEMAP_GENERATION.md @@ -0,0 +1,122 @@ +# Basemap + +## Goal and Motivation + +This document covers the backend requirements of the Cardinal Maps basemap tile server. This system serves map data to Cardinal for display on the screen and other critical functions. + +Deployment strategies, suggested production environments and data artifact generation steps are given. + +### Overview of Generation + +To generate basemap tiles for Cardinal Maps, we use a fork of _planetiler-openmaptiles_ that includes extra information about each point of interest and address on the map. This extra information is used by Cardinal to enable users to tap on icons on the map and get information about a place without relying on a round-trip to the server. We learned from Atlas that users want quick performance for these UI transitions even while offline or in degraded cellular conditions. + +This extra POI information is also used by the app's offline mode to let the user search for points of interest without access to the internet. + +## System Components + +The recommended serving strategy covered in this document involves putting a large map tile archive in object storage and deploying horizontally-scalable tileservers either co-located with the object storage system or at the edge close to users. Tileservers don't have copies of the entire planet archive, instead they make HTTP range requests to object storage to retrieve bits and pieces of the world as necessary. + +This approach is covered in detail by [the PMTiles documentation](https://docs.protomaps.com/). + +Map tiles are served by a Go server embedded into the PMTiles project. This serving strategy is described by [the official documentation](https://docs.protomaps.com/deploy/) as `pmtiles serve`. + +In order to minimize or eliminate downtime during data updates, Blue-Green deploys or similar techniques are recommended. Deployment is covered in more detail later in this document. + +## Hardware Requirements + +Planetiler is extremely memory-efficient compared to other tile generation software, but due to the scale of the data we're working with, it still requires quite a bit of memory to generate tiles for the whole planet. Official documentation lists the following requirements: + +> 10x as much disk space and at least 0.5x as much RAM as the planet.osm.pbf file you start from + +Empirically, the fork seems to require about 1.5x-2x as much storage and around 3x as much physical memory as mainline for the same input data. + +As of December 2025, `planet-latest.osm.pbf` is about 84 GB, so to be on the safe side I'd recommend a machine with about 2 TB of fast SSD storage and about 192 GB of RAM. OOM errors have occurred with as much as 128 GB of RAM. An additional thing to keep in mind is that the OpenStreetMap database is constantly growing, so these hardware requirements will increase slowly over time. + +## Generation Steps + +### Step 1: Obtain OpenStreetMap Data + +In a web browser, navigate to https://planet.openstreetmap.org/ and download the latest **planet.osm.pbf**, either via an HTTP download or BitTorrent. In either case, verify the hash to ensure the download completed successfully. + +It is recommended to note the date/version of the OpenStreetMap extract used, and to incorporate it into the file paths of downstream artifacts to avoid accidentally overwriting previous artifacts or losing track of which database version generated which tile data. Because of this recommendation, it is a good idea to understand and carefully modify the generation commands provided in this document. + +### Step 2: Preparation + +Clone our fork of _planetiler-openmaptiles_: + +```bash +git clone https://gitlab.e.foundation/ellenhp/planetiler-openmaptiles.git +``` + +Install Java 21 or later. + +### Step 3: Compilation + +In the top-level directory of the _planetiler-openmaptiles_ repo, run `./mvnw clean package` to perform a clean build of the project. When the build completes, artifacts will be placed in the `build/` directory. + +### Step 4: Verify Planetiler Functionality (Optional) + +To ensure that Planetiler is functioning correctly you can run the following command. This will download Natural Earth data, a global map of bodies of water and an OpenStreetMap extract of Monaco, then it will run the tile generation operation. + +```bash +java -jar target/*with-deps.jar --download --area=monaco +``` + +Planetiler refuses to overwrite `.mbtiles` files by default, so you will need to remove the `data/output.mbtiles` file if you run this more than once. + +### Step 5: Generate Tiles for the Planet + +Using the OpenStreetMap file downloaded in step 1, run planetiler on the entire planet. + +You can refer to the [Planetiler documentation on full-planet builds](https://github.com/onthegomap/planetiler/blob/main/PLANET.md) for tips and tricks, but keep in mind that the hardware recommendations will not apply to our fork because we are including quite a bit more information in the `poi` and `house_number` layers of our generated tiles, and this information takes up extra space in physical memory. + +As of late 2025, the following command is recommended in the documentation for full-planet builds, and should work unmodified on sufficiently powerful hardware: + +```bash +java -Xmx20g \ + -jar target/*with-deps.jar \ + `# Use the planet.osm.pbf file downloaded earlier here.` \ + --osm_path=$OSM_PATH --download \ + `# Accelerate necessary downloads by fetching the 10 1GB chunks at a time in parallel` \ + --download-threads=10 --download-chunk-size-mb=1000 \ + `# Also download name translations from wikidata` \ + --fetch-wikidata \ + --output=./data/planet.mbtiles \ + `# Store temporary node locations at fixed positions in a memory-mapped file` \ + --nodemap-type=array --storage=mmap +``` + +This command will take several hours to complete depending on hardware specifications. Plan on 1-8 hours depending on CPU count. You can refer to the [benchmarks on mainline planetiler's README](https://github.com/onthegomap/planetiler/tree/main?tab=readme-ov-file#benchmarks) for estimates, keeping in mind that our fork is performing additional work compared to upstream. + +The generated MBTiles will be located at `data/planet.mbtiles`. + +### Step 6: Convert the Generated MBTiles to PMTiles + +PMTiles is a cloud-native format for collections of map tiles that allows a tile serving system to scale horizontally without requiring large SSD volumes to be attached to each tileserver instance. The essence of the technology is to move data from local storage to an object store and require each tileserver instance to keep an in-memory index of which sections of the PMTiles contain various parts of the world. + +Latency between the tileserver instances and the Object Store should be minimized for good performance, and the Object Store must support HTTP range requests. + +To perform the conversion, ensure a sufficiently modern version of **Go** is installed, and clone the PMTiles repository: https://github.com/protomaps/PMTiles + +In the root of the PMTiles repo, run the following: `pmtiles convert $MBTILES_PATH planet.pmtiles` + +This is generally considered to be a quick operation, but in the author's experience it may take considerable time to complete, or appear to hang. + +### Step 7: Upload PMTiles to Object Storage + +The commands to execute in this step depend on the production environment used, but the goal is to make the PMTiles archive available to all tileservers. + +### Step 8: Point Tileservers to New Artifact + +It is recommended to use a Blue-Green deployement strategy for tileserver data updates, so previous versions should not be overwritten in this step. Instead, use a new key for the new PMTiles archive, ideally incorporating the original date/version code of the `planet.osm.pbf` used for generation. + +Tileserver configuration should be updated on a rolling basis to point to the new artifact. Health checks can be implemented to detect configuration issues such as typos in the path/key of the PMTiles artifact. + +Once it has been determined that deployment has succeded and a rollback is not necessary, the old artifact can be removed. It would be prudent to wait for some time to do this, or implement a retention policy of several days or more. + +## Additional Considerations + +### CDNs and Caching + +A CDN can be put in front of the tileservers to reduce latency. As the tileservers are serving mostly static data, a TTL of hours not minutes may be appropriate. There's a tradeoff here though, as an excessive cache TTL increases the risk of multiple tile versions existing in a Cardinal Maps client's own cache. This can lead to things like visually misaligned roads or other artifacts. + diff --git a/docs/ROUTING_TILE_GEN.md b/docs/ROUTING_TILE_GEN.md new file mode 100644 index 0000000000000000000000000000000000000000..3266b1b51f8b50a5d00969e56130d12931650bcd --- /dev/null +++ b/docs/ROUTING_TILE_GEN.md @@ -0,0 +1,58 @@ +# Routing + +This document covers the generation of [Valhalla tiles](https://valhalla.github.io/valhalla/tiles/) that Cardinal Maps uses for its offline routing features and potentially more functionality in the future. + +In comparison to the generation and preparation of [basemap tiles](./basemap.md), routing tiles are more straightforward to generate and serve. We do not require a fork of Valhalla, nor do we need to do any conversion of its output artifacts beyond extracting them from the tarball artifact. + +Some of the setup steps are shared across both documents, and these steps are covered more explicitly in the basemap docs, so if anything is unclear here it may be helpful to follow that document first. + +#### Warning! + +Executing the following steps prior to the introduction of a tile versioning feature in the client may cause offline features in deployed clients to break. + +## Generation + +The easiest way to generate Valhalla tiles is by using the [Valhalla docker image](https://github.com/valhalla/valhalla/pkgs/container/valhalla). + +[Documentation for tile generation](https://valhalla.github.io/valhalla/mjolnir/getting_started_guide/) is available from upstream and can be consulted should the steps in this guide ever become out-of-date. + +## Step 1: Set Up the Artifact Directory + +Set the environment variable `VALHALLA_DATA_PATH` to the directory into which you would like to put the Valhalla tiles. As in the [basemap documentation](./basemap.md), the author would recommend naming this directory based on the version/date code of the `planet.osm.pbf` extract that the tiles would be based on. + +Put the OpenStreetMap extract into the artifact directory and note its path. + +## Step 2: Start the Valhalla Container + +Until instructed otherwise, run the following steps inside of the Valhalla OCI container. This is the podman command to start it, but you can use Docker by changing the command name to `docker`. + +`podman run --rm -it -v $VALHALLA_DATA_PATH:/data:Z ghcr.io/valhalla/valhalla` + +## Step 3: Generate the Valhalla Configuration + +This step can be omitted if you re-run the tile generation process. Moreover, the configuration file it generates can be modified to include things like time zones, administrative boundaries (to avoid crossing borders unnecessarily) and more accurate road speeds. If you do modify the config, be careful not to overwrite your changes by executing this command a second time. + +`valhalla_build_config > /data/valhalla.json` + +## Step 4: Generate the Valhalla Tiles + +Remembering to substitute in the name of your OpenStreetMap extract, run the following command: + +`cd /data && valhalla_build_tiles --config /data/valhalla.json /data/planet.osm.pbf` + +This will take several hours to complete, but should only require 128 GB of RAM. Possibly less. + +## Step 5: Un-tar the Tiles + +The Valhalla tiles will be placed in the `/data` directory as one `.tar` archive and must be extracted before they can be copied to object storage. + +## Step 6: Copy the Tiles To Object Storage + +Use `rclone` or similar to copy the entire extracted directory hierarchy to object storage incorporating the OSM extract's date code into the key prefix. + +## Step 7: Point a Reverse Proxy To the New Tiles + +Set up a reverse proxy to point to the new tiles. The proxy can be re-deployed with a different destination prefix to update the underlying routing data. + +Be careful not to typo the destination prefix, and make sure to test the app's offline download after a deploy to ensure that tiles are still downloaded correctly. +