diff --git a/docs/ui-ux-improvements.md b/docs/ui-ux-improvements.md new file mode 100644 index 0000000..5c8bcea --- /dev/null +++ b/docs/ui-ux-improvements.md @@ -0,0 +1,79 @@ +# UI/UX improvement backlog + +Suggestions collected from users. Items marked **done** ship in `feat/map-ux-improvements`; the rest are open. + +Effort scale: **S** = small, **M** = medium, **L** = large. + +## Done + +| Source | Item | +|---|---| +| user1 #1 | Easier line selection on map (canvas renderer, 8 px click tolerance) | +| user1 #6 | Map scale bar | +| user1 #3 (map part), user2 | Length, frequency and polarization in the map popup | +| user2 | Default Grafana refresh set to 1 min | +| user1 #2 (partly) | Choosing a CML in the Grafana dropdown opens its map popup (user and admin map) and highlights the line; the map pans only if the line is out of view | +| user1 #3, user2 | Metadata in the admin map popup (replaces the hover tooltip) | + +## Open + +### 1. Always zoom to the CML chosen in the dropdown (user1 #2) + +- **Remaining:** the map only pans when the selected line is out of view; optionally zoom with `map.fitBounds()` instead. +- **Effort:** S. + +### 2. Show length and frequency band above the graph (user1 #3) + +- **To do:** + - Option A: add a Grafana text/stat panel or change the panel title using template variables backed by a `cml_metadata` query (`length`, `frequency`, `polarization` for `$cml_id`). + - Option B: render a small header bar in `realtime.html` above the iframe and fill it from `cmlMeta` when a CML is selected (no Grafana change; pairs with item 1). + - The iframe uses `viewPanel=2&kiosk`, so a Grafana-side change only shows up if it is part of panel 2's title. Option B is simpler. +- **Effort:** S (option B) / M (option A). + +### 3. Compare two CMLs (user1 #4) + +- **To do:** + - Make the Grafana `cml_id` variable multi-value and let the panel query return one series per CML (colour-coded), or add a second iframe stacked below for a split view. + - Add map UI to pick a second CML (e.g. shift+click) and a toggle for merged vs. split. + - Make sure the time sync and the grey coloring-window annotation (`/api/coloring-annotation`) apply to both panels. + - Update the dashboard provisioning JSON and the SQL used by the panel (`cml_data_secure`). +- **Effort:** L. Decide first on merged vs. split layout. + +### 4. Valid-record counts for 1 day and 3 days in the popup (user1 #5) + +- **To do:** + - Add `total_records_1d`, `valid_records_1d`, `total_records_3d`, `valid_records_3d` to `cml_stats` and `cml_stats_history` (new migration in `database/migrations/`, plus `init.sql`). + - Extend `update_cml_stats()` and `get_cml_stats_at()`. Keep RLS and `user_id` handling consistent. + - Add the fields to `/api/cml-stats` (live and historical column order) and to `buildPopupText()`. + - Update `parser/backfill_stats_history.py` and the tests for the stats API. + - Check the cost of the 3 d window on `cml_data` (compressed chunks); consider computing from `cml_data_1h` instead. +- **Effort:** M. Database and API changes with a migration. + +### 5. Search and display by link number (user2) + +- **Request:** search by link number, i.e. IDs with `55` at the 3rd and 4th digit (e.g. `30655 1234`); IDs with `25` or `27` at that position are unwanted. +- **Needs clarification:** is the `55` ID a separate field from the current `cml_id`, or a transformation of it? Which IDs should be hidden or mapped? +- **To do:** + - Add a search box to the map (filter or jump to a CML, zoom via item 1). + - If the 55-number is a different identifier, add a column to `cml_metadata` (migration, parser mapping, `db_writer.py`) and show it in the popup and the search. + - If only a filter is needed, filter or flag the list server-side in `get_available_cmls()` and `/api/cml-map`. +- **Effort:** S (search by existing ID) / M (new identifier column through the pipeline). + +### 6. Sublink numbers (network element numbers with 55) (user2) + +- **Needs clarification:** the source of these numbers. `cml_metadata.sublink_id` currently holds names such as `sublink_1`. +- **To do:** add a network-element ID per sublink in the metadata (migration, parser mappings, `db_writer.py`), then show it in the popup next to frequency and polarization and make it searchable (item 5). +- **Effort:** M. Depends on what the provider data contains. + +### 7. Grey area for periods with possible rain-related outage (user2) + +- **Needs clarification:** what defines such a period (a wet/dry flag from the processing, rain gauge or radar data, or a RSL-drop threshold)? Is it per CML or for the whole area? +- **To do:** + - Define and compute the rain-event periods (processor service or SQL), and store them (table or Grafana annotations). + - Draw them as shaded regions in the Grafana panel, e.g. via an annotation query on the dashboard, similar to the existing coloring-window annotation. + - Optionally mark affected CMLs on the map. +- **Effort:** L. Mostly a data and algorithm question; the display part is S-M. + +## Follow-ups on done items + +- Confirm units in `cml_metadata`: the popup assumes `length` in metres and `frequency` in MHz (as in the archive CSV). The `other_mno_csv` parser maps `length_km` straight into `length`, so that source may be stored in km. Normalize in the parser if so. Effort: S. diff --git a/grafana/admin_dashboards/cml-admin-realtime.json b/grafana/admin_dashboards/cml-admin-realtime.json index 4e597f3..d4e63cc 100644 --- a/grafana/admin_dashboards/cml-admin-realtime.json +++ b/grafana/admin_dashboards/cml-admin-realtime.json @@ -4,7 +4,7 @@ "graphTooltip": 0, "links": [], "liveNow": false, - "refresh": "10s", + "refresh": "1m", "schemaVersion": 38, "style": "dark", "tags": [ diff --git a/grafana/provisioning/dashboards/definitions/cml-realtime.json b/grafana/provisioning/dashboards/definitions/cml-realtime.json index bea7728..71450de 100644 --- a/grafana/provisioning/dashboards/definitions/cml-realtime.json +++ b/grafana/provisioning/dashboards/definitions/cml-realtime.json @@ -874,7 +874,7 @@ "type": "geomap" } ], - "refresh": "10s", + "refresh": "1m", "schemaVersion": 38, "style": "dark", "tags": [ diff --git a/webserver/main.py b/webserver/main.py index f03addb..63d30d7 100644 --- a/webserver/main.py +++ b/webserver/main.py @@ -537,7 +537,17 @@ def api_cml_map(): with user_db_scope(current_user.id) as conn: cur = conn.cursor() cur.execute( - "SELECT DISTINCT ON (cml_id) cml_id::text, site_0_lon, site_0_lat, site_1_lon, site_1_lat FROM cml_metadata ORDER BY cml_id" + """ + SELECT cml_id::text, + MIN(site_0_lon), MIN(site_0_lat), MIN(site_1_lon), MIN(site_1_lat), + MIN(length), + json_agg(json_build_object('frequency', frequency, + 'polarization', polarization) + ORDER BY sublink_id) + FROM cml_metadata + GROUP BY cml_id + ORDER BY cml_id + """ ) data = cur.fetchall() cur.close() @@ -547,6 +557,9 @@ def api_cml_map(): "cml_id": str(row[0]), "site_0": {"lon": float(row[1]), "lat": float(row[2])}, "site_1": {"lon": float(row[3]), "lat": float(row[4])}, + # length in metres, sublink frequency in MHz + "length": safe_float(row[5]), + "sublinks": row[6] or [], } for row in data ] @@ -911,11 +924,16 @@ def api_admin_cml_map(): with admin_db_scope() as conn: cur = conn.cursor() cur.execute( - """SELECT DISTINCT ON (user_id, cml_id) - user_id, cml_id::text, site_0_lon, site_0_lat, site_1_lon, site_1_lat + """SELECT user_id, cml_id::text, + MIN(site_0_lon), MIN(site_0_lat), MIN(site_1_lon), MIN(site_1_lat), + MIN(length), + json_agg(json_build_object('frequency', frequency, + 'polarization', polarization) + ORDER BY sublink_id) FROM cml_metadata WHERE site_0_lon IS NOT NULL AND site_0_lat IS NOT NULL AND site_1_lon IS NOT NULL AND site_1_lat IS NOT NULL + GROUP BY user_id, cml_id ORDER BY user_id, cml_id""" ) rows = cur.fetchall() @@ -931,6 +949,9 @@ def api_admin_cml_map(): "cml_id": str(row[1]), "site_0": {"lon": float(row[2]), "lat": float(row[3])}, "site_1": {"lon": float(row[4]), "lat": float(row[5])}, + # length in metres, sublink frequency in MHz + "length": safe_float(row[6]), + "sublinks": row[7] or [], } for row in rows ] diff --git a/webserver/templates/admin_realtime.html b/webserver/templates/admin_realtime.html index 8b71dc8..21f4932 100644 --- a/webserver/templates/admin_realtime.html +++ b/webserver/templates/admin_realtime.html @@ -51,6 +51,12 @@ height: 100%; } + /* Place the scale bar beside the zoom buttons instead of below them */ + .leaflet-top.leaflet-left .leaflet-control-scale { + clear: none; + margin-left: 8px; + } + .legend-panel { position: absolute; top: 10px; @@ -156,7 +162,7 @@
{% endif %} @@ -176,6 +182,7 @@ }); const map = L.map('admin-map', { preferCanvas: true }).setView([50, 10], 5); + L.control.scale({ position: 'topleft', metric: true, imperial: false }).addTo(map); // Esri World Light Gray Canvas - keyless light basemap (tile order is z/y/x) L.tileLayer('https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}', { attribution: 'Tiles © Esri', maxZoom: 16 @@ -223,9 +230,16 @@ function tooltipText(c) { const s = stats[c.user_id + '|' + c.cml_id]; - let t = esc(providerNames[c.user_id] || c.user_id) + ' · CML ' + esc(c.cml_id); - if (!s) return t + '
no data'; - t += '
Completeness (6h): ' + s.completeness_percent + '%'; + let t = '' + esc(providerNames[c.user_id] || c.user_id) + ' · CML ' + esc(c.cml_id) + '
'; + if (c.length !== null) t += 'Length: ' + (c.length / 1000).toFixed(2) + ' km
'; + c.sublinks.forEach(sl => { + const parts = []; + if (sl.frequency !== null) parts.push((sl.frequency / 1000).toFixed(1) + ' GHz'); + if (sl.polarization) parts.push(esc(sl.polarization.toUpperCase())); + if (parts.length) t += 'Sublink: ' + parts.join(' ') + '
'; + }); + if (!s) return t + 'no data'; + t += 'Completeness (6h): ' + s.completeness_percent + '%'; if (s.last_rsl !== null) t += '
Last RSL: ' + s.last_rsl + ' dBm'; if (s.stddev_last_60min !== null) t += '
Std Dev (1h): ' + s.stddev_last_60min; return t; @@ -235,7 +249,7 @@ Object.keys(lines).forEach(key => { const line = lines[key]; line.setStyle({ color: lineColor(line._provider, key) }); - line.setTooltipContent(tooltipText(line._cml)); + line.setPopupContent(tooltipText(line._cml)); }); } @@ -278,12 +292,17 @@ }); } - // Select a CML and point the Grafana iframe at it - function selectCml(providerId, cmlId) { - const key = providerId + '|' + cmlId; + // Highlight a CML on the map + function highlightCml(key) { if (selectedKey && lines[selectedKey]) lines[selectedKey].setStyle({ weight: 2.5 }); selectedKey = key; lines[key].setStyle({ weight: 6 }).bringToFront(); + } + + // Select a CML and point the Grafana iframe at it + function selectCml(providerId, cmlId) { + const key = providerId + '|' + cmlId; + highlightCml(key); const panel = document.getElementById('grafana-panel'); if (!panel) return; @@ -295,6 +314,49 @@ win.dispatchEvent(new PopStateEvent('popstate', { state: null })); } + // Open the popup of a CML, first bringing it into the part of the map not covered by the chart + function showCmlPopup(key) { + const line = lines[key]; + if (!line || !line._map) return; + + const chart = document.querySelector('.chart-container'); + const chartHeight = chart ? chart.offsetHeight : 0; + const size = map.getSize(); + const p = map.latLngToContainerPoint(line.getCenter()); + const visible = p.x > 0 && p.x < size.x && p.y > 120 && p.y < size.y - chartHeight - 40; + if (!visible) { + map.fitBounds(line.getBounds(), { + paddingTopLeft: [60, 120], + paddingBottomRight: [60, Math.max(60, chartHeight + 40)], + maxZoom: map.getZoom(), + animate: false + }); + } + line.openPopup(); + } + + // The provider/CML dropdowns live inside the Grafana iframe; poll its URL to react to changes + function watchGrafanaSelection() { + const panel = document.getElementById('grafana-panel'); + if (!panel) return; + let primed = false; + setInterval(() => { + let params; + try { + params = new URL(panel.contentWindow.location.href).searchParams; + } catch (e) { + return; + } + const key = params.get('var-provider') + '|' + params.get('var-cml_id'); + // Skip until both variables match a CML on the map (cml_id lags behind a provider change) + if (key === selectedKey || !lines[key]) return; + highlightCml(key); + // The first match is the dashboard's initial selection: highlight only, no popup + if (primed) showCmlPopup(key); + primed = true; + }, 500); + } + fetch('/api/admin/cml-map') .then(r => { if (!r.ok) throw new Error('HTTP ' + r.status); return r.json(); }) .then(cmls => { @@ -305,7 +367,7 @@ if (!layers[c.user_id]) layers[c.user_id] = L.featureGroup().addTo(map); counts[c.user_id] = (counts[c.user_id] || 0) + 1; const line = L.polyline([a, b], { color: NO_DATA, weight: 2.5, opacity: 0.85 }) - .bindTooltip(tooltipText(c), { sticky: true }) + .bindPopup(tooltipText(c), { autoPan: false }) .on('click', () => selectCml(c.user_id, c.cml_id)) .addTo(layers[c.user_id]); line._provider = c.user_id; @@ -319,6 +381,7 @@ renderLegend(); loadStats(); setInterval(loadStats, STATS_REFRESH_MS); + watchGrafanaSelection(); }) .catch(err => { document.getElementById('map-status').textContent = 'Failed to load CMLs (' + err.message + ')'; diff --git a/webserver/templates/realtime.html b/webserver/templates/realtime.html index 929ab6c..796a6ce 100644 --- a/webserver/templates/realtime.html +++ b/webserver/templates/realtime.html @@ -51,6 +51,12 @@ height: 100%; } + /* Place the scale bar beside the zoom buttons instead of below them */ + .leaflet-top.leaflet-left .leaflet-control-scale { + clear: none; + margin-left: 8px; + } + .chart-container { position: absolute; bottom: 0; @@ -104,7 +110,7 @@
@@ -116,7 +122,11 @@ let map; let cmlLayers = {}; let cmlStats = {}; + let cmlMeta = {}; let selectedCmlId = '10001'; + let highlightedCmlId = null; + const DEFAULT_STYLE = { weight: 2, opacity: 0.7 }; + const SELECTED_STYLE = { weight: 5, opacity: 1 }; let currentColoringOption = 'std_dev'; // Time slider state @@ -181,8 +191,26 @@ }; } + function escapeHtml(s) { + var d = document.createElement('div'); + d.textContent = String(s); + return d.innerHTML; + } + function buildPopupText(cmlId, stats) { - var popupText = 'CML ' + cmlId + '
'; + var popupText = 'CML ' + escapeHtml(cmlId) + '
'; + var meta = cmlMeta[cmlId]; + if (meta) { + if (meta.length !== null) { + popupText += 'Length: ' + (meta.length / 1000).toFixed(2) + ' km
'; + } + meta.sublinks.forEach(function (s) { + var parts = []; + if (s.frequency !== null) parts.push((s.frequency / 1000).toFixed(1) + ' GHz'); + if (s.polarization) parts.push(escapeHtml(s.polarization.toUpperCase())); + if (parts.length) popupText += 'Sublink: ' + parts.join(' ') + '
'; + }); + } if (stats) { popupText += 'Completeness (6h): ' + stats.completeness_percent + '%
'; popupText += 'Valid records (6h): ' + stats.valid_records + '/' + stats.total_records + '
'; @@ -215,7 +243,10 @@ function initializeMap() { console.log('Initializing map...'); try { - map = L.map('cml-map').setView([57.705, 11.97], 11); + // Canvas renderer with tolerance makes thin CML lines easier to hit + map = L.map('cml-map', { renderer: L.canvas({ tolerance: 8 }) }).setView([57.705, 11.97], 11); + // Top-left: the default bottom-left spot is hidden behind the chart panel + L.control.scale({ position: 'topleft', metric: true, imperial: false }).addTo(map); // Use Carto Positron - clean, minimal light basemap L.tileLayer('https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png', { attribution: '© OpenStreetMap contributors © CARTO', @@ -492,6 +523,7 @@ var cmlId = cml.cml_id; var stats = cmlStats[cmlId]; var color = getColor(stats, currentColoringOption); + cmlMeta[cmlId] = { length: cml.length, sublinks: cml.sublinks || [] }; var latlngs = [ [cml.site_0.lat, cml.site_0.lon], @@ -517,15 +549,31 @@ selectCml(cmlId); }); - // Add hover effects + // Add hover effects (the selected line keeps its highlight) polyline.on('mouseover', function () { - this.setStyle({ weight: 4, opacity: 1 }); + this.setStyle({ weight: this._selected ? 6 : 4, opacity: 1 }); }); polyline.on('mouseout', function () { - this.setStyle({ weight: 2, opacity: 0.7 }); + this.setStyle(this._selected ? SELECTED_STYLE : DEFAULT_STYLE); }); cmlLayers[cmlId] = polyline; + if (cmlId === selectedCmlId) highlightCml(cmlId); + } + + // Thicker, opaque line for the selected CML + function highlightCml(cmlId) { + var prev = cmlLayers[highlightedCmlId]; + if (prev) { + prev._selected = false; + prev.setStyle(DEFAULT_STYLE); + } + var line = cmlLayers[cmlId]; + if (!line) return; + line._selected = true; + line.setStyle(SELECTED_STYLE); + line.bringToFront(); + highlightedCmlId = cmlId; } // Update all map line colors based on current coloring option @@ -729,6 +777,7 @@ // Select CML and update Grafana dashboard function selectCml(cmlId) { selectedCmlId = cmlId; + highlightCml(cmlId); var grafanaPanel = document.getElementById('grafana-panel'); var iframeWindow = grafanaPanel.contentWindow; @@ -739,11 +788,50 @@ iframeWindow.dispatchEvent(new PopStateEvent('popstate', { state: null })); } + // Open the popup of a CML, first bringing it into the part of the map not covered by the chart + function showCmlPopup(cmlId) { + var line = cmlLayers[cmlId]; + if (!line) return; + + highlightCml(cmlId); + var pad = getMapViewportPadding(); + var size = map.getSize(); + var p = map.latLngToContainerPoint(line.getCenter()); + var visible = p.x > 0 && p.x < size.x && p.y > 120 && p.y < size.y - pad.bottomRight[1]; + if (!visible) { + map.fitBounds(line.getBounds(), { + paddingTopLeft: [60, 120], + paddingBottomRight: pad.bottomRight, + maxZoom: map.getZoom(), + animate: false + }); + } + line.openPopup(); + } + + // The CML dropdown lives inside the Grafana iframe; poll its URL to react to changes + function watchGrafanaSelection() { + setInterval(function () { + var id; + try { + var panel = document.getElementById('grafana-panel'); + id = new URL(panel.contentWindow.location.href).searchParams.get('var-cml_id'); + } catch (e) { + return; + } + if (id && id !== selectedCmlId) { + selectedCmlId = id; + showCmlPopup(id); + } + }, 500); + } + // Initialize on page load document.addEventListener('DOMContentLoaded', function () { initializeMap(); initializeResizer(); cleanupColoringAnnotations(); + watchGrafanaSelection(); }); // Initialize resizer for adjustable dashboard height diff --git a/webserver/tests/test_admin.py b/webserver/tests/test_admin.py index cc00172..89d55f6 100644 --- a/webserver/tests/test_admin.py +++ b/webserver/tests/test_admin.py @@ -120,15 +120,20 @@ def test_admin_realtime_without_grafana_org_has_map_only(users): def test_admin_cml_map_returns_all_providers(users, cursor): + sublinks = [{"frequency": 28000.0, "polarization": "v"}] cursor.fetchall.return_value = [ - ("tenant_a", "C1", 10.0, 50.0, 10.1, 50.1), - ("tenant_b", "C1", 11.0, 51.0, 11.1, 51.1), + ("tenant_a", "C1", 10.0, 50.0, 10.1, 50.1, 691.4, sublinks), + ("tenant_b", "C1", 11.0, 51.0, 11.1, 51.1, None, None), ] resp = _login("boss").get("/api/admin/cml-map") assert resp.status_code == 200 data = resp.get_json() assert [d["user_id"] for d in data] == ["tenant_a", "tenant_b"] assert data[0]["site_0"] == {"lon": 10.0, "lat": 50.0} + assert data[0]["length"] == 691.4 + assert data[0]["sublinks"] == sublinks + assert data[1]["length"] is None + assert data[1]["sublinks"] == [] def test_admin_cml_stats_returns_all_providers(users, cursor): diff --git a/webserver/tests/test_api_routes.py b/webserver/tests/test_api_routes.py index 40485b0..c826399 100644 --- a/webserver/tests/test_api_routes.py +++ b/webserver/tests/test_api_routes.py @@ -56,13 +56,16 @@ def test_api_cml_metadata_empty(auth_client): def test_api_cml_map_returns_list(auth_client): client, cursor = auth_client - cursor.fetchall.return_value = [("CML01", 10.0, 5.0, 10.5, 5.5)] + sublinks = [{"frequency": 28000.0, "polarization": "v"}] + cursor.fetchall.return_value = [("CML01", 10.0, 5.0, 10.5, 5.5, 691.4, sublinks)] resp = client.get("/api/cml-map") assert resp.status_code == 200 data = resp.get_json() assert data[0]["cml_id"] == "CML01" assert data[0]["site_0"] == {"lon": 10.0, "lat": 5.0} assert data[0]["site_1"] == {"lon": 10.5, "lat": 5.5} + assert data[0]["length"] == 691.4 + assert data[0]["sublinks"] == sublinks def test_api_data_time_range_with_data(auth_client):