incubator-sdap-nexus.git
5 days agoSDAP-400 Added NCAR insitu api to matchup (#198) master
Nga Chung [Tue, 20 Sep 2022 16:44:40 +0000 (09:44 -0700)] 
SDAP-400 Added NCAR insitu api to matchup (#198)

3 weeks agoMatchup domsresults bugfix (#197)
Stepheny Perez [Fri, 2 Sep 2022 23:45:10 +0000 (16:45 -0700)] 
Matchup domsresults bugfix (#197)

* Added environment variable to Dockerfile to fix cert error

* Updated openapi spec to link to cdmsschema endpoint

* Fixed matchup db insert. Use new names

* renamed 'details' field for matchup doms to match cdms changes

* Added provider table to openapi spec

* fixed typo

* Updated changelog

4 weeks agoAddress matchup errors and integrate SDAP Proxy changes (#196)
Stepheny Perez [Mon, 22 Aug 2022 18:47:01 +0000 (11:47 -0700)] 
Address matchup errors and integrate SDAP Proxy changes (#196)

* Fixed apidocs bug

* Addressed bug introduced by proxy

* Updated matchup output fields. Validate matchup 'parameter' param against insitu schema

* Updated openapi spec to support domsresults endpoint

* Updated changelog

* Fixed changelog merge error

* remove unwanted poetry file

6 weeks agoSDAP-390 Updated CDMS reader tool requirements.txt (#194)
Nga Chung [Wed, 10 Aug 2022 22:20:05 +0000 (15:20 -0700)] 
SDAP-390 Updated CDMS reader tool requirements.txt (#194)

6 weeks agoFix changelog (#195)
Riley Kuttruff [Wed, 10 Aug 2022 15:55:09 +0000 (08:55 -0700)] 
Fix changelog (#195)

Minor formatting error + entry moved to correct section.

6 weeks agoSDAP-398 - CDMS Regression Test Script (#183)
Riley Kuttruff [Tue, 9 Aug 2022 23:52:13 +0000 (16:52 -0700)] 
SDAP-398 - CDMS Regression Test Script (#183)

* CDMS-151: Added script for regression tests.

+ Changelog update

* Proper attribution for CDMSReader script file

* CDMS-151: Added script for regression tests.

* CDMS-151: Added test cases for SwaggerUI for both SDAP and in-situ

* CDMS-151: Update regression test script README for new environment variables

* CDMS-151: test_cdms.py lint

* CDMS-151: test_cdms.py: More dynamic approach to checking yaml files for swaggerui docs.

Falls back to default assumed value if extraction fails. Should that succeed, test case passes with warning issued. Otherwise the test case will fail.

Also fixed naming mistake

* CDMS-151: Test script: Altered test skipping functionality

Removed skip by environment variable functionality

domslist, insitu and swagger tests can no longer be manually skipped.

matchup_spark (and by extension domsresults) can be manually skipped (just for testing purposes, this will also be removed later)

cdmssubset is skipped by default until it can utilize Zarr. This can be overridden by passing --force-subset at the command line

Updated readme to reflect this

* CDMS-151: Test script: Mark test cases as integration

* CDMS-151: Replaced cdms_reader.py copy with soft link

* SDAP-398: Updated ticket tag in changelog

Set to public SDAP ticket rather than private CDMS ticket

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
6 weeks agoSDAP-390 Update NetCDF reader tool for data match-up (#178)
JordanGethers [Tue, 9 Aug 2022 18:48:36 +0000 (14:48 -0400)] 
SDAP-390 Update NetCDF reader tool for data match-up (#178)

* SDAP -390 Update NetCDF reader tool for data match-up

* Update CHANGELOG.md

* Update cdms_reader.py

* Update README.md

* Update cdms_reader.py

* Updated README.md.

Co-authored-by: Jordan Gethers <jgethers@mdc-dev-proc.coaps.fsu.edu>
Co-authored-by: nchung <nga.t.chung@jpl.nasa.gov>
7 weeks agoFixed error causing timeSeriesSpark to fail (#186)
Riley Kuttruff [Fri, 5 Aug 2022 01:27:46 +0000 (18:27 -0700)] 
Fixed error causing timeSeriesSpark to fail (#186)

* Fixed error in timeSeriesSpark

It appeared to have been caused by an import issue in RedirectHandler.py

* Updated changelog

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
7 weeks agoFix matchup bugs (#165)
Stepheny Perez [Tue, 2 Aug 2022 21:25:20 +0000 (14:25 -0700)] 
Fix matchup bugs (#165)

8 weeks agoPreserve insitu schema (#181)
Riley Kuttruff [Thu, 28 Jul 2022 17:16:15 +0000 (10:16 -0700)] 
Preserve insitu schema (#181)

* Preserve insitu schema

Prevent massive number of unneeded api hits

* Update changelog

* Changed method of preserving schema

* Revert changes to insitu api

Shouldn't be changed in pr

* Revert changes to insitu api

Shouldn't be changed in pr

* Update changelog

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
8 weeks agoFix NetCDF output (#182)
Riley Kuttruff [Thu, 28 Jul 2022 17:15:11 +0000 (10:15 -0700)] 
Fix NetCDF output (#182)

* Fixed mixup of lat & lon

* Update changelog

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
8 weeks agoAdd metadata to remote coll (#173)
thomas loubrieu [Wed, 27 Jul 2022 22:03:29 +0000 (15:03 -0700)] 
Add metadata to remote coll (#173)

* fix bugs in helm chart after update for proxy feature

* update CHANGELOG for ticket 388 (proxy)

* makes --collection-path optional, add remote collection cache for list metadata

* add unit tests for remote collection cache

* add versions to avoid build failure in docker

* make code more robust

Co-authored-by: tloubrieu-jpl <loubrieu@jpl.nasa.gov>
Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
2 months agoSDAP-396: saildrone insitu (#179)
Stepheny Perez [Wed, 20 Jul 2022 18:58:45 +0000 (11:58 -0700)] 
SDAP-396: saildrone insitu (#179)

* Added saildrone endpoint to insitu api call conf

* Updated changelog

2 months agoFix CSV/NetCDF bug (#175)
Stepheny Perez [Wed, 20 Jul 2022 18:47:08 +0000 (11:47 -0700)] 
Fix CSV/NetCDF bug (#175)

* Fix bug where metadata links fail for AWS insitu datasets

* Fix NetCDF endpoint

* Updated matchup to ensure both time and space are account for when 'matchOnce' is true

* Update changelog

2 months agoSDAP-397: Requested matchup updates (#180)
Stepheny Perez [Wed, 20 Jul 2022 18:45:09 +0000 (11:45 -0700)] 
SDAP-397: Requested matchup updates (#180)

* Added depth the insitu output

* Pull ID from insitu api response

* Added unit field to matchup response. Unit field is currently null

* updated changelog

* Query insitu schema endpoint to populate unit field

2 months agoUpdated matchup to ensure both time and space are account for when 'matchOnce' is...
Stepheny Perez [Wed, 20 Jul 2022 18:37:15 +0000 (11:37 -0700)] 
Updated matchup to ensure both time and space are account for when 'matchOnce' is true (#176)

2 months agoSDAP-371: Fix domssubset API (#172)
Stepheny Perez [Fri, 8 Jul 2022 02:26:44 +0000 (19:26 -0700)] 
SDAP-371: Fix domssubset API (#172)

* Fixed DOMS subsetter for satellite data

* Updated changelog

* Rename CSV file within ZIP based on dataset name

* Updated doms subsetter to support insitu subsetting

* Updated matchup algorithm to use insitu module

* Removed temp code

2 months agoFix for null count field in return for matchup_spark request (#171)
Riley Kuttruff [Wed, 6 Jul 2022 23:59:22 +0000 (16:59 -0700)] 
Fix for null count field in return for matchup_spark request (#171)

* Fix for null count field in return for matchup_spark request

* Update CHANGELOG.md

Co-authored-by: rileykk <rileykk@jpl.nasa.gov>
2 months agofix bugs in helm chart after update for proxy feature (#170)
thomas loubrieu [Wed, 6 Jul 2022 23:35:44 +0000 (16:35 -0700)] 
fix bugs in helm chart after update for proxy feature (#170)

* fix bugs in helm chart after update for proxy feature

* update CHANGELOG for ticket 388 (proxy)

Co-authored-by: tloubrieu-jpl <loubrieu@jpl.nasa.gov>
3 months agochore: add insitu application to ingress template (#167)
wphyojpl [Tue, 21 Jun 2022 21:16:38 +0000 (14:16 -0700)] 
chore: add insitu application to ingress template (#167)

* Add insitu ingress

* add condition to include insitu ingress

* add default value for insituAPI.enabled for ingress

* chore: update changelog

* chore: add SDAP issue number

3 months agoSdap 388 proxy (#162)
thomas loubrieu [Tue, 14 Jun 2022 17:04:05 +0000 (19:04 +0200)] 
Sdap 388 proxy (#162)

* add configmap to the webapp, not tested

* first version of the code ready, but still not tested

* proxy with redirect feature tested on laptop

* redirect validated on AQACF prod server, /list end point returns the remote collections

* add configmap to the webapp, not tested

* first version of the code ready, but still not tested

* proxy with redirect feature tested on laptop

* redirect validated on AQACF prod server, /list end point returns the remote collections

* add case to resolve merge conflict

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
3 months agoFixed failing matchup test. Matchup nowuses 'lon' instead of 'x' (#161)
Stepheny Perez [Thu, 9 Jun 2022 00:38:19 +0000 (17:38 -0700)] 
Fixed failing matchup test. Matchup nowuses 'lon' instead of 'x' (#161)

3 months agoSDAP-372 Updated match_spark_doms to interface with samos_cdms endpoint (#160)
Nga Chung [Tue, 7 Jun 2022 20:13:56 +0000 (13:13 -0700)] 
SDAP-372 Updated match_spark_doms to interface with samos_cdms endpoint (#160)

3 months agoSDAP-372: Integrate AWS insitu API (#158)
Stepheny Perez [Thu, 2 Jun 2022 05:16:15 +0000 (22:16 -0700)] 
SDAP-372: Integrate AWS insitu API (#158)

* Updated doms conf

* Updated matchup to work with aws insitu providers

* Fix platform retrieval code for insitu. New schema is dict

* Added MatchupDoms which queries the DOMS insitu endpoints instead of AWS endpoints

* Get provider name function works for both DOMS and CDMS insitu endpoints

* Fixed platform validation for new CDMS insitu platforms. Expanded CDMS insitu data fields

* Removed unused and unnecessary code

* Removed unnecessary exception

* Added changelogs

* Moved to a single top level changelog

3 months agoSDAP-387: Update NetCDF matchup output format (#159)
Stepheny Perez [Wed, 1 Jun 2022 18:53:16 +0000 (11:53 -0700)] 
SDAP-387: Update NetCDF matchup output format (#159)

* NetCDF matchup output

* Updated netCDF builder code to look at new param names

* Integrated Vardis feedback

* Remove todos, add newline to end of file

* Fixed failing test

4 months agoHotfix: Handle mismatched dataset listing and specification example (#152)
joshgarde [Tue, 24 May 2022 21:56:52 +0000 (14:56 -0700)] 
Hotfix: Handle mismatched dataset listing and specification example (#152)

* Fix rogue query parameter

* Handle example mismatch in plugin instead of spec

5 months agoSDAP-318 Updated timeSeriesSpark algorithm to find tiles by bounding box instead... distributed.0.4.5a40
Kevin [Wed, 6 Apr 2022 22:18:27 +0000 (15:18 -0700)] 
SDAP-318 Updated timeSeriesSpark algorithm to find tiles by bounding box instead of polygon (#156)

5 months agoUpdated ICOADS url (#157)
Stepheny Perez [Tue, 5 Apr 2022 22:28:44 +0000 (15:28 -0700)] 
Updated ICOADS url (#157)

* Updated ICOADS url

* import importlib metadata

* Updated FSU doms insitu url

* Updated samos url everywhere in config

* Remove inadvertently committed file

6 months agoSDAP-366: Fix CSV output for matchup (#153)
Stepheny Perez [Fri, 11 Mar 2022 03:35:30 +0000 (19:35 -0800)] 
SDAP-366: Fix CSV output for matchup (#153)

* Fix CSV output for matchup

* Update DOMS CSV format to only include values that are included in the result

* Use cf variable name if present

* CSV works when insitu results contain varying variables

* Updated CSV format with suggested changes

* Updated CSV based on feedback

6 months agoAdd toJson in data-in-bounds api (#154) distributed.0.4.5a35
Jason Min-Liang Kang [Thu, 10 Mar 2022 18:46:22 +0000 (10:46 -0800)] 
Add toJson in data-in-bounds api (#154)

8 months agoSDAP-363: Update SwaggerUI to display list of available datasets dynamically (#149)
joshgarde [Tue, 18 Jan 2022 18:38:00 +0000 (10:38 -0800)] 
SDAP-363: Update SwaggerUI to display list of available datasets dynamically (#149)

* Initial work

* Switch JsonSchema_string to original JsonSchemaForm

* Populate insitu fields

* Fix unpopulated dsMap issue

* Fix /subsubset spec

* Fix totaling

9 months agoSDAP-364: Upgrade Solr to 8.11.1 (#150)
Stepheny Perez [Fri, 17 Dec 2021 18:02:51 +0000 (10:02 -0800)] 
SDAP-364: Upgrade Solr to 8.11.1 (#150)

* Bump solr bitnami chart to 2.1.7

* Upgrade solr image tag to 8.11.1

9 months agoSDAP-361 Time series spark algorithm unit test (#148)
Kevin [Thu, 9 Dec 2021 00:49:42 +0000 (16:49 -0800)] 
SDAP-361 Time series spark algorithm unit test (#148)

* first pass at time series unittest

* Created timeseriesspark unit test for the calc function

* first pass at time series unittest

* Created timeseriesspark unit test for the calc function

* Added mock patch to avoid saving mean time series netCDF file

* Added tests for calc_average_on_day, calc with seasonal filter for monthly data and daily data, and calc with seasonal and lowpass filters

* commented out lowpass filter assertions for the time being

* Fixed bug in test_calc_average_on_day

9 months agoReplace data_val with data_vals (#147)
Jason Min-Liang Kang [Tue, 7 Dec 2021 18:22:52 +0000 (10:22 -0800)] 
Replace data_val with data_vals (#147)

9 months agoSDAP-332 Changed to use SDAP instead of DOMS for title (#146)
Nga Chung [Mon, 6 Dec 2021 18:44:43 +0000 (10:44 -0800)] 
SDAP-332 Changed to use SDAP instead of DOMS for title (#146)

9 months agochange solr helm chart dependency to stable (#145)
thomas loubrieu [Mon, 6 Dec 2021 15:40:22 +0000 (10:40 -0500)] 
change solr helm chart dependency to stable (#145)

* change solr helm chart dependency to stable

* disable solr authentication by default

* update specific sdap solr docker image for JTS support

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
10 months agoremove hyphen to avoid carriage return from being removed (#144)
thomas loubrieu [Mon, 22 Nov 2021 16:14:43 +0000 (11:14 -0500)] 
remove hyphen to avoid carriage return from being removed (#144)

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
10 months agoupgrade helm template and README for latest nginx-ingress and python 3.8 and spark...
thomas loubrieu [Tue, 16 Nov 2021 17:52:37 +0000 (09:52 -0800)] 
upgrade helm template and README for latest nginx-ingress and python 3.8 and spark 3.1.1 (#143)

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
10 months agoSDAP-355: Update matchup parameter and output to use primary and secondary (#142)
Stepheny Perez [Fri, 29 Oct 2021 23:55:57 +0000 (16:55 -0700)] 
SDAP-355: Update matchup parameter and output to use primary and secondary (#142)

* renamed 'matchup' terminology to 'secondary'

* Updated tests to work with 'matchup' -> 'secondary' terminology change

* Updated OpenAPI spec

* Remove circular reference in openapi.yml

11 months agoSDAP-353: Update matchup output to only return variables that are relevant to the...
Stepheny Perez [Thu, 14 Oct 2021 22:33:54 +0000 (15:33 -0700)] 
SDAP-353: Update matchup output to only return variables that are relevant to the given dataset (#138)

11 months agoSDAP-303 Added Elasticsearch as a metadatastore (#133)
WicketWarrick [Thu, 14 Oct 2021 21:46:58 +0000 (23:46 +0200)] 
SDAP-303 Added Elasticsearch as a metadatastore (#133)

* SDAP-303 Added Elasticsearch as a metadatastore

* SDAP-303 Updated queries + separated aggregations from queries + finalized unit tests class for ES + some fixes for solr tests class

* SDAP-303 Fixed typo in method name

* SDAP-303 Fixing methods related to day_of_year

* SDAP-338: Update match up implementation to support multi-variable tiles (#132)

* Added support for multi-variable swath tiles

* Converted Tile to dataclass

* Added matchup support for grid_multi_variable_tile

* Updated matchup to work with existing unit tests

* Update data point dict names to variable_name and variable_value

* Fixed lint warnings

* Added test case for multi-var sat to multi-var sat

* Added cf_variable_name to matchup response

* get_indices will combine masks of all variables

* Only add data point if valid

* Updated matchup to work with new multi-variable solr doc layout

* Backwards compatability for loading solr doc into tile

* Improved backwards compatability when loading solr doc into Tile. Works when standard name field isn't present

* SDAP-303 Added Elasticsearch as a metadatastore

* SDAP-303 Updated queries + separated aggregations from queries + finalized unit tests class for ES + some fixes for solr tests class

* SDAP-303 Fixed typo in method name

* SDAP-303 Fixing methods related to day_of_year

* SDAP-303 : fixed left references to solr_doc and _solr instance owned by NexusTile class

Co-authored-by: Dorian FOUQUIER <dfouquie@ifremer.fr>
Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
Co-authored-by: QUERIC <antoine.queric@ifremer.fr>
11 months agoSDAP-338: Update match up implementation to support multi-variable tiles (#132)
Stepheny Perez [Wed, 29 Sep 2021 16:19:19 +0000 (09:19 -0700)] 
SDAP-338: Update match up implementation to support multi-variable tiles (#132)

* Added support for multi-variable swath tiles

* Converted Tile to dataclass

* Added matchup support for grid_multi_variable_tile

* Updated matchup to work with existing unit tests

* Update data point dict names to variable_name and variable_value

* Fixed lint warnings

* Added test case for multi-var sat to multi-var sat

* Added cf_variable_name to matchup response

* get_indices will combine masks of all variables

* Only add data point if valid

* Updated matchup to work with new multi-variable solr doc layout

* Backwards compatability for loading solr doc into tile

* Improved backwards compatability when loading solr doc into Tile. Works when standard name field isn't present

12 months agoapidocs path hotfix (#136)
joshgarde [Fri, 17 Sep 2021 16:33:52 +0000 (09:33 -0700)] 
apidocs path hotfix (#136)

Co-authored-by: Josh Garde <jgarde@jpl.nasa.gov>
12 months agoSDAP-332: OpenAPI Documentation + SwaggerUI for Matchup Endpoints (#134)
joshgarde [Wed, 8 Sep 2021 22:34:31 +0000 (15:34 -0700)] 
SDAP-332: OpenAPI Documentation + SwaggerUI for Matchup Endpoints (#134)

* Initial OpenAPI docs

* Update absolute path

Co-authored-by: Josh Garde <jgarde@jpl.nasa.gov>
13 months agoSDAP-315: Updated matchup algorithm to support satellite to satellite (#131)
Stepheny Perez [Thu, 5 Aug 2021 00:21:12 +0000 (17:21 -0700)] 
SDAP-315: Updated matchup algorithm to support satellite to satellite (#131)

* SDAP-315: Updated matchup algorithm to support satellite to satellite

* SDAP-315: Updated unit tests to a passing state. Moved integration tests to a new test directory

15 months agomake chart work with pre-existing data-volume (pvc) (#127)
thomas loubrieu [Fri, 28 May 2021 17:52:07 +0000 (10:52 -0700)] 
make chart work with pre-existing data-volume (pvc) (#127)

Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
16 months agoAdded missing tile_service_factory param to NexusCalcHandlers endpoints (#130)
Stepheny Perez [Mon, 24 May 2021 21:43:23 +0000 (14:43 -0700)] 
Added missing tile_service_factory param to NexusCalcHandlers endpoints (#130)

16 months agoSDAP-312: Improved UTM reprojection speed (#129)
Stepheny Perez [Thu, 20 May 2021 21:17:21 +0000 (14:17 -0700)] 
SDAP-312: Improved UTM reprojection speed (#129)

16 months agoSDAP-308 Fixed deletebyquery.py to be compatible with Python 3 (#128)
Nga Chung [Wed, 19 May 2021 19:13:24 +0000 (12:13 -0700)] 
SDAP-308 Fixed deletebyquery.py to be compatible with Python 3 (#128)

17 months agoCDMS-50: Updated matchup algorithm to remove parameter restriction (#126)
Stepheny Perez [Fri, 16 Apr 2021 23:45:57 +0000 (16:45 -0700)] 
CDMS-50: Updated matchup algorithm to remove parameter restriction (#126)

* CDMS-50: Updated matchup algorithm to remove parameter restriction

* CDMS-50: Doms point uses sat var name

* CDMS-50: Cleanup

17 months agoSDAP-304 : Added optional domsconfig "create_keyspace_granted" (#125)
Antoine QUÉRIC [Fri, 16 Apr 2021 23:34:49 +0000 (01:34 +0200)] 
SDAP-304 : Added optional domsconfig "create_keyspace_granted"  (#125)

* SDAP-304 : Added optional domsconfig "create_keyspace_granted" (True [default] / False) to check before creating keyspace (in case of user permissions being not sufficient for such operation)

* SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster (#124)

* SDAP-305: Updated webapp dockerfile to work with Python 3 and new version of pyspark

* SDAP-305: Pass kubernetes config into doms/matchup endpoints and overwrite domsconfig.ini

* SDAP-305: Fixes for /domsplot endpoint

* SDAP-305: Updated /domssubset /datainbounds and /domsinsitusubset to a runnable state

* SDAP-305: Pass k8s config to /domsplot endpoint so config file is overwritten

* SDAP-304 : fixed the NoOptionError exception which lacked the module named

* SDAP-304 : Added optional domsconfig "create_keyspace_granted" (True [default] / False) to check before creating keyspace (in case of user permissions being not sufficient for such operation)

* SDAP-304 : fixed the NoOptionError exception which lacked the module named

Co-authored-by: QUERIC <antoine.queric@ifremer.fr>
Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
17 months agoSDAP-305: Get current match up working with AVHRR OI data that is currently ingested...
Stepheny Perez [Thu, 1 Apr 2021 21:47:40 +0000 (14:47 -0700)] 
SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster (#124)

* SDAP-305: Updated webapp dockerfile to work with Python 3 and new version of pyspark

* SDAP-305: Pass kubernetes config into doms/matchup endpoints and overwrite domsconfig.ini

* SDAP-305: Fixes for /domsplot endpoint

* SDAP-305: Updated /domssubset /datainbounds and /domsinsitusubset to a runnable state

* SDAP-305: Pass k8s config to /domsplot endpoint so config file is overwritten

18 months agoSDAP-305: Get current match up working with AVHRR OI data that is currently ingested...
Stepheny Perez [Mon, 15 Mar 2021 19:40:54 +0000 (12:40 -0700)] 
SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster (#123)

* CDMS-47: Only use TokenAwarePolicy when also using DCAwareRoundRobinPolicy when setting up Cassandra connection pool

* CDMS-47: Always use AuthProvider when connecting to Cassandra

* CDMS-47: Fixed bugs found while testing matchup

* CDMS-47: Update python version in analysis setup.py to 3.7

* SDAP-305: Check for Solr 'tile_max/min_lat/lon' possibly being a list

* SDAP-305: Clean up matchup

* SDAP-305: Return instead of raising StopIteration exception in spark Matchup

* SDAP-305: Return instead of raising StopIteration exception in spark Matchup

19 months agoSDAP-154: Python 3 (#122)
Stepheny Perez [Fri, 19 Feb 2021 23:33:48 +0000 (15:33 -0800)] 
SDAP-154: Python 3 (#122)

* Code fixes after Python3 upgrade

* Upgraded cassandra and pysolr to work with Python 3.8. Upgraded project webapp version to 0.4.0

19 months agoSDAP-154 : Python 3 (#121)
Antoine QUÉRIC [Fri, 19 Feb 2021 23:06:20 +0000 (00:06 +0100)] 
SDAP-154 : Python 3 (#121)

* SDAP-154 Upgraded to python 3.8 with 2to3

* SDAP-154 Updated Dockerfile & conda-requirements to comply with python v3.8

* SDAP-154 : minor fixes to complyt with upgrade to python 3.8

* SDAP-154: fixed reference to SolrProxy module

* SDAP-154 : Fixed encoding issues

Co-authored-by: QUERIC <antoine.queric@ifremer.fr>
20 months agoUpdate Docker documentation (#120)
Eamon Ford [Fri, 22 Jan 2021 22:38:33 +0000 (14:38 -0800)] 
Update Docker documentation (#120)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
20 months agoBump ingester versions (#119)
Eamon Ford [Fri, 8 Jan 2021 23:18:09 +0000 (15:18 -0800)] 
Bump ingester versions (#119)

20 months agoBump ingester versions (#118)
Eamon Ford [Tue, 5 Jan 2021 22:16:41 +0000 (14:16 -0800)] 
Bump ingester versions (#118)

20 months agoBump granule-ingester aned collection-manager versions from 0.1.2 -> 0.1.3 (#117)
Eamon Ford [Tue, 5 Jan 2021 18:21:54 +0000 (10:21 -0800)] 
Bump granule-ingester aned collection-manager versions from 0.1.2 -> 0.1.3 (#117)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
20 months agoUpdate Helm chart README file to include missing property (#116)
Eamon Ford [Tue, 5 Jan 2021 00:38:46 +0000 (16:38 -0800)] 
Update Helm chart README file to include missing property (#116)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
20 months agoSDAP-299: Update obsolete helm repo urls (fix for breaking change) (#115)
Eamon Ford [Mon, 4 Jan 2021 19:34:15 +0000 (11:34 -0800)] 
SDAP-299: Update obsolete helm repo urls (fix for breaking change) (#115)

20 months agoSDAP-294: Add optional normalizeDates parameter for time series and hofmueller (... distributed.0.3.0
Eamon Ford [Mon, 4 Jan 2021 18:06:01 +0000 (10:06 -0800)] 
SDAP-294: Add optional normalizeDates parameter for time series and hofmueller (#114)

21 months agoSDAP-298: Helm Chart 1.0.0 (#113)
Eamon Ford [Mon, 21 Dec 2020 18:06:43 +0000 (10:06 -0800)] 
SDAP-298: Helm Chart 1.0.0 (#113)

* updated helm chart for zookeeper

use solr and zk helm charts

change .Release.Namespace to .Release.Name

add rabbitmq storageclass

fix rbac

add  max_concurrency

add solr_host arg

add solr port

always deploy solr

add solr-host option

read cli args for cass and solr hosts

pass cassandra host

add support for cassandra username and password

cassandra helm chart included

fix arguments sent to spark driver, add logging in cassandraproxy

pass factory method to nexuscalchandlers to create tile service in spark nodes

fix namespace

fix bad argument order

fix cass url for granule ingester

change solr-create-collection to a deployment

make solr history default

pr

enable external solr/zk/cass hosts

rabbitmq.enabled

revert doms

revert

update images

only deploy config operator if it is enabled

remove http:// from solr hardcoded endpoint

turn off configmap by default

* revert doms

* upgrade images

* use new nexusjpl/solr image, update solr-create-collection image tag

* Add flag to disable crd creation

* Fixed typo

* Only create gitcfg if if configMap not set

* Only create rbac if configmap disabled

* update nexus version

* Add S3 support to helm chart

* Update Rabbitmq version

* Update versions

* version bumps

* Cleanup

* Configurable cassandra parameters

* wip: helm readme

* Helm readme changes

* Typo

* Typo

* Readme updates

* Readme

* Readme

* Readme updates

* Readme

* Readme

* Fix typos

* Readme

* Readme

* Readme

* Readme

* Readme

* Readme

* More docs

* Cleanup

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
22 months agoSDAP-296: Make deletebyquery script work out-of-the-box in webapp Docker image (... distributed.0.2.2
Eamon Ford [Wed, 25 Nov 2020 18:56:15 +0000 (10:56 -0800)] 
SDAP-296: Make deletebyquery script work out-of-the-box in webapp Docker image (#112)

22 months agoSDAP-292: Add cassandra auth support to delete_by_query.py script, and include the... distributed.0.2.1
Eamon Ford [Fri, 30 Oct 2020 21:09:12 +0000 (14:09 -0700)] 
SDAP-292: Add cassandra auth support to delete_by_query.py script, and include the script in nexus-webapp docker image (#111)

23 months agoSDAP-258: Use pre-computed climatologies for deseason algorithm (#109) distributed.0.2.0
Eamon Ford [Tue, 13 Oct 2020 22:01:10 +0000 (15:01 -0700)] 
SDAP-258: Use pre-computed climatologies for deseason algorithm (#109)

2 years agoSDAP-285: Upgrade custom Solr image to include JTS, and update solr-create-collection...
Eamon Ford [Mon, 14 Sep 2020 20:47:41 +0000 (13:47 -0700)] 
SDAP-285: Upgrade custom Solr image to include JTS, and update solr-create-collection image to create geo field  (#108)

2 years agoSDAP-278: Add Cassandra authentication support, remove hardcoded http:// prefix in... distributed.0.1.6
Eamon Ford [Mon, 10 Aug 2020 18:28:06 +0000 (11:28 -0700)] 
SDAP-278: Add Cassandra authentication support, remove hardcoded  prefix in Solr connector (#106)

2 years agoSDAP-263: Support for querying tiles with date 01-01-1970 (#103)
Eamon Ford [Thu, 16 Jul 2020 17:34:23 +0000 (10:34 -0700)] 
SDAP-263: Support for querying tiles with date 01-01-1970 (#103)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
2 years agoSdap 252 (#104)
thomas loubrieu [Fri, 10 Jul 2020 19:46:50 +0000 (12:46 -0700)] 
Sdap 252 (#104)

Remove singleton pattern for algorthim handlers which caused a bug on concurrent request (SDAP-252), simplifies the handler hierarchy:

remove the singleton pattern for the nexusHandlers, works for 3 spark algorithm so far

remove singleton pattern for list handler (maybe not a good idea...)

simplify class handler hierarchy

split files per objects

simplify class handler hierarchy

add details for developers deployment in README

delete moved package

correct typo in Spark handler class name

add conda dependencies versions, make docker work

update unit test

correct impact of current ticket on unit tests, some still don t work since they require specific data

resolve conflicts between SDAP-252 and SDAP-249

Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
2 years agoSDAP-249 : add solr time out client argument to webapp, use it in the data acces...
thomas loubrieu [Thu, 9 Jul 2020 01:27:59 +0000 (18:27 -0700)] 
SDAP-249 : add solr time out client argument to webapp, use it in the data acces… (#101)

* add solr time out client argument to webapp, use it in the data access configuration

* add corrections after review (mostly new unit test)

Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
2 years agoSDAP-238, SDAP-239, SDAP-246: Add Collection Manager, Granule Ingester, and Config...
Eamon Ford [Tue, 7 Jul 2020 22:47:28 +0000 (17:47 -0500)] 
SDAP-238, SDAP-239, SDAP-246: Add Collection Manager, Granule Ingester, and Config Operator to NEXUS Helm chart

2 years agoSDAP-253 MaximaMinima Algorithm (#102) distributed.0.1.5.1
Maya DeBellis [Mon, 22 Jun 2020 23:27:59 +0000 (16:27 -0700)] 
SDAP-253 MaximaMinima Algorithm (#102)

* add max min algorithm
* update init file

2 years agoSDAP-232: Add performance metrics logging. (#98)
Eamon Ford [Mon, 22 Jun 2020 23:13:08 +0000 (16:13 -0700)] 
SDAP-232: Add performance metrics logging. (#98)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
2 years agoSDAP-242: Add updates to ESRI/SDAP Integration (#99)
yepremyana [Mon, 1 Jun 2020 17:11:24 +0000 (10:11 -0700)] 
SDAP-242: Add updates to ESRI/SDAP Integration (#99)

2 years agoSDAP-223: add esri integration (#94) 100/head
yepremyana [Thu, 7 May 2020 21:01:04 +0000 (14:01 -0700)] 
SDAP-223: add esri integration (#94)

* Fix SDAP-223 add esri integration

* remove spark string from time_average_map and time_series

* updated embedded toolbox with hot url and input polygon

* updated toolbox with host url and input polygon

* updated readme for input polygon usage and importing toolboxes

2 years agoSDAP-220: temporal variance algorithm (#93)
Maya DeBellis [Mon, 6 Apr 2020 21:54:32 +0000 (14:54 -0700)] 
SDAP-220: temporal variance algorithm (#93)

* add data anomaly algorithm

* add temporal variance algorithm

* remove incorrect data_anomaly

Co-authored-by: Maya Debellis <mdebelli@deepdata-1.jpl.nasa.gov>
2 years agoSDAP-228 Add option to enable/disable root webpage in helm chart (#97)
Eamon Ford [Fri, 20 Mar 2020 00:30:21 +0000 (17:30 -0700)] 
SDAP-228 Add option to enable/disable root webpage in helm chart (#97)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
2 years agoSDAP-226: pass in the configured number of zookeeper replicas to zookeeper container...
Eamon Ford [Tue, 17 Mar 2020 20:09:20 +0000 (13:09 -0700)] 
SDAP-226: pass in the configured number of zookeeper replicas to zookeeper container args (#96)

Co-authored-by: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
2 years agoSDAP-219: Add README for Helm chart (#95)
Eamon Ford [Thu, 12 Mar 2020 17:48:27 +0000 (10:48 -0700)] 
SDAP-219: Add README for Helm chart (#95)

2 years agoSDAP-222: Add support for nodeSelectors and tolerations in the Helm chart (#92)
Eamon Ford [Mon, 2 Mar 2020 19:11:21 +0000 (11:11 -0800)] 
SDAP-222: Add support for nodeSelectors and tolerations in the Helm chart (#92)

2 years agoSDAP-221: Add Helm configuration value to enable/disable nginx-ingress subchart insta...
Eamon Ford [Mon, 2 Mar 2020 19:06:29 +0000 (11:06 -0800)] 
SDAP-221: Add Helm configuration value to enable/disable nginx-ingress subchart installation (#91)

2 years agoSDAP 218: bug correction on start/end date from nexcusli.dataset_list() (#90)
tloubrieu-jpl [Fri, 21 Feb 2020 01:08:52 +0000 (17:08 -0800)] 
SDAP 218: bug correction on start/end date from nexcusli.dataset_list()  (#90)

* Update doc after standalone deployment on my laptop

* Update layout after rst preview test

* SDAP-216 Update the quickstart guide after deployment test on my laptop

* For SDAP-218, get the iso start/end date directly from the nexus '/list' request.

* remove change from pull request #89

2 years agoSDAP-216: Update the quickstart guide (#89)
tloubrieu-jpl [Fri, 21 Feb 2020 01:08:36 +0000 (17:08 -0800)] 
SDAP-216: Update the quickstart guide (#89)

* Update doc after standalone deployment on my laptop

* Update layout after rst preview test

* SDAP-216 Update the quickstart guide after deployment test on my laptop

2 years agoSDAP-215 Use k8s as Spark scheduler and create helm chart. Fix HTTP connection timeou...
Eamon Ford [Thu, 23 Jan 2020 00:52:01 +0000 (16:52 -0800)] 
SDAP-215 Use k8s as Spark scheduler and create helm chart. Fix HTTP connection timeout bug. (#88)

2 years agoSDAP-192 Create DOMS netCDF reader tool (#83) 84/head
Jocelyn Elya [Thu, 10 Oct 2019 20:37:50 +0000 (16:37 -0400)] 
SDAP-192 Create DOMS netCDF reader tool (#83)

* Created DOMS netCDF reader tool.

* SDAP-192 DOMS netCDF reader updated to use num2date function to get datetime object from timestamp.

* SDAP-192 Added logging to DOMS netCDF reader tool.

* SDAP-192 Miscellaneous variable updates with DOMS netCDF reader.

* SDAP-192 'try/except' blocks around 'with' statements for netCDF and CSV file handling. Changed name of CSV file output from main. Moved print statement.

* SDAP-192 Changed print to LOGGER.info in DOMS netCDF reader.

* SDAP-192 Updates to DOMS netCDF reader documentation, including a README.md

3 years agoMerge pull request #79 from lewismc/SDAP-206 81/head
Lewis John McGibbney [Wed, 21 Aug 2019 18:35:12 +0000 (11:35 -0700)] 
Merge pull request #79 from lewismc/SDAP-206

SDAP-206 remove pegged numpy version in nexuscli requirements.txt

3 years agoSDAP-206 remove pegged numpy version in nexuscli requirements.txt 79/head
Lewis John McGibbney [Wed, 21 Aug 2019 17:29:49 +0000 (10:29 -0700)] 
SDAP-206 remove pegged numpy version in nexuscli requirements.txt

3 years agoSDAP-206 remove pegged numpy version in nexuscli requirements.txt
Lewis John McGibbney [Wed, 21 Aug 2019 05:48:20 +0000 (22:48 -0700)] 
SDAP-206 remove pegged numpy version in nexuscli requirements.txt

3 years agoSDAP-208 remove pegged numpy version in nexuscli requirements.txt 78/head
Lewis John McGibbney [Wed, 21 Aug 2019 05:16:15 +0000 (22:16 -0700)] 
SDAP-208 remove pegged numpy version in nexuscli requirements.txt

3 years agoSDAP-192 Created DOMS netCDF reader tool. (#68) 69/head
Jocelyn Elya [Wed, 3 Apr 2019 17:55:08 +0000 (13:55 -0400)] 
SDAP-192 Created DOMS netCDF reader tool. (#68)

* SDAP-192 Created DOMS netCDF reader tool.

* SDAP-192 DOMS netCDF reader updated to use num2date function to get datetime object from timestamp.

* SDAP-192 Added logging to DOMS netCDF reader tool.

* SDAP-192 Miscellaneous variable updates with DOMS netCDF reader.

* SDAP-192 'try/except' blocks around 'with' statements for netCDF and CSV file handling. Changed name of CSV file output from main. Moved print statement.

* SDAP-192 Changed print to LOGGER.info in DOMS netCDF reader.

3 years agoSDAP-187 Unable to change cassandra port (#66)
twnone [Wed, 20 Mar 2019 22:55:35 +0000 (23:55 +0100)] 
SDAP-187 Unable to change cassandra port (#66)

* SDAP-187 webapp : Impossible to change Cassandra cluster CQL port

* SDAP-187 : fix typo

3 years agoSDAP-191 Remove Access-Control-Allow-Origin :* (#67)
fgreg [Wed, 20 Mar 2019 22:54:27 +0000 (15:54 -0700)] 
SDAP-191 Remove Access-Control-Allow-Origin :* (#67)

3 years agoSDAP-166 SolrCloud Docker Image (#63) v1.0.0-rc1 62/head
fgreg [Wed, 23 Jan 2019 17:27:12 +0000 (09:27 -0800)] 
SDAP-166 SolrCloud Docker Image (#63)

* Now using INIT_SOLR_HOME="yes" option for creating the nexustiles core in singlenode.

* New Solr Cloud docker image

* Overhauled Solr images. Includes new Solr Cloud image and init container.

3 years agoSDAP-173 Fix Hovmoller code reporting missing get_spark_cfg attribute (#65)
Joseph Jacob [Tue, 15 Jan 2019 18:07:18 +0000 (10:07 -0800)] 
SDAP-173 Fix Hovmoller code reporting missing get_spark_cfg attribute (#65)

* SDAP-173 Fix Hovmoller code reporting missing get_spark_cfg attribute

3 years agoSDAP-105 DOMS matchup netcdf and csv generation (#61)
Maya DeBellis [Fri, 11 Jan 2019 23:44:47 +0000 (15:44 -0800)] 
SDAP-105 DOMS matchup netcdf and csv generation (#61)

* adding netcdf generation, modified csv generation to line up with netcdf

* adding all recorded variables, calc max and min, refactored

* things work! testing locally with AVHRR using /domsresults endpoint

* removing json files from git tracking

* refactor

* removing all files used for testing purposes

* Consolidate depth to a single field, fix valid_min bug

* minor fixes, typos

* fix the way depth is handled, add comments, get rid of unecessary string comparison

* update metadata links to be pulled from config.py

* add keywords

3 years agoSDAP-151 Determine parallelism automatically for Spark analytics (#50) 60/head
Joseph Jacob [Tue, 20 Nov 2018 22:00:25 +0000 (14:00 -0800)] 
SDAP-151 Determine parallelism automatically for Spark analytics (#50)

* Removed spark configuration, added nparts configuration, and autocompute parallelism for spark-based time series.

* SDAP-151 Determine parallelism automatically for Spark analytics

3 years agoSDAP-137 Webapp should have a mesos and a standalone image (#51)
fgreg [Wed, 7 Nov 2018 16:45:04 +0000 (08:45 -0800)] 
SDAP-137 Webapp should have a mesos and a standalone image (#51)

* Updating images.

* Updating images.

* ignore .nc files

* removed unused files.

* removed unused files.

* Updating images.

* ignore .nc files

* removed unused files.

* removed unused files.

* Updated bulid scripts and quickstart.

* workaround for basemap. change workdir of agent

* moved files to make more sense

* fixed error with conda home env var.

* don't remove spark package

* Need PROJ_LIB defined as agent env var.

3 years agoSDAP-157 Bump nexusproto to 1.0.0 (#46)
fgreg [Thu, 18 Oct 2018 00:21:40 +0000 (17:21 -0700)] 
SDAP-157 Bump nexusproto to 1.0.0 (#46)

* updated version dependencies.

* update dockerfiles

4 years agoSDAP-152 Error in nexus client (#43)
fgreg [Sat, 22 Sep 2018 01:38:00 +0000 (18:38 -0700)] 
SDAP-152 Error in nexus client (#43)

* Updated client to parse the new 'iso_time' returned from the time series function.

* handle both cases for backwards compatibility.

* also need to cast now.

* using dtype appropriately