Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2018-12-19T00:59:35+01:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/382KPI group by Parameter properties2018-12-19T00:59:35+01:00Matija ObrezaKPI group by Parameter propertiesI wish to add a KPI Execution to count the "Number of accessions last updated by year" (and then by institute, by crop).
The current implementation requires me to define a `Dimension` that will provide the `Set<Long>` of the **years** t...I wish to add a KPI Execution to count the "Number of accessions last updated by year" (and then by institute, by crop).
The current implementation requires me to define a `Dimension` that will provide the `Set<Long>` of the **years** that can then be passed to ExecutionDimension.
1. That requires us to either maintain a `NumericListDimension` and add all year options manually.
1. Generates a number of observations for years we *know* no updates have happened.
## Idea
Allow for **GROUP BY** clauses in Execution definition:
- Add group by `lastModifiedDate` modifier `YEAR` alias `lastModifiedYear`
Generates: `select ..., year(_base.lastModifiedDate) lastModifiedYear ... group by lastModifiedYear where ...`
- Add group by `lastModifiedDate` modifier `MONTH` alias `lastModifiedMonth`
Generates: `select ..., year(_base.lastModifiedDate) lastModifiedYear, month(_base.lastModifiedDate) lastModifiedMonth ... group by lastModifiedYear, lastModifiedDate where ...`
The resulting Observation will therefore go from being a single row with the `value` (and `stddev` for AVERAGE) to multiple rows for existing combinations of `lastModifiedYear` and `lastModifiedDate`.
The method `private Observation getSingleObservation(Query query, Object... params) {}` should therefore return `List<Observation>` and the `Observation` needs to store the group-by's somewhere.
Could use the "DimensionKey"?2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/381KPI counting accessions2018-12-19T00:59:35+01:00Matija ObrezaKPI counting accessionsI'm trying to define a KPI Execution to count the "Number of accessions documented in published Datasets".
- Parameter: `Dataset` with `state = 'PUBLISHED'`
- No dimensions.
Execution fails with *illegal attempt to dereference collecti...I'm trying to define a KPI Execution to count the "Number of accessions documented in published Datasets".
- Parameter: `Dataset` with `state = 'PUBLISHED'`
- No dimensions.
Execution fails with *illegal attempt to dereference collection [dataset0_.id.accessionRefs] with element property reference [accession]*.
This is because `Dataset#accessionRefs` is a collection and needs to be joined (as documented https://stackoverflow.com/questions/24750754/org-hibernate-queryexception-illegal-attempt-to-dereference-collection/24751098#24751098).
## Fix
We should introduce something similar to the `link` property in `ExecutionDimension` to the `Execution` to declare the join.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/379KPI API v1: Observed value on specific date2018-12-24T07:20:23+01:00Matija ObrezaKPI API v1: Observed value on specific dateCreate a new `KPIReadController` V1 (for any authenticated user) with API endpoint `/kpi/observations/{executionName}` that will:
- Locate the last execution **before** the **date** specified in query string (`?date=12-18-2018`)
- It re...Create a new `KPIReadController` V1 (for any authenticated user) with API endpoint `/kpi/observations/{executionName}` that will:
- Locate the last execution **before** the **date** specified in query string (`?date=12-18-2018`)
- It returns `List<Observation>` for the KPI `ExecutionRun` on that date:
```json
[{
"id":337605,
"value":1.0,
"stdDev":null,
"dimensions":[
{"name":"instCode","value":"POL010"},
{"name":"boolean.yesno","value":"false"}
]
},...
]
```
## Bonus points
Massage the `List<Observation>` so that the JSON returned to the client for example above looks like:
```json
{
"historic": {
"false": { "value":1.0, "stdDev":null },
"true": { "value":4.0, "stdDev":null },
}
}
```
Filter by `historic=true`:
```json
{
"institute.code": {
"POL001": { "value":112312.0, "stdDev":null },
}
}
```
## Filtering observations
The user can also specify dimensions that must match:
- `?instCode=POL010` would filter `executionRun.observations.any().observationKeys` by `key=instCode` and `value=POL010`
- Can we use `execution.dimensions#field` instead of `dimension.name`: example `institute.code=POL010`?
- Dimension "filters" provided by user are excluded from the results:
```json
[{
"id":337605,
"value":1.0,
"stdDev":null,
"dimensions":[
{"name":"boolean.yesno","value":"false"}
]
}, {
"id":337609,
"value":4.0,
"stdDev":null,
"dimensions":[
{"name":"boolean.yesno","value":"true"}
]
}, ...
]
```2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/378JSP: Download PDCI from /explore2018-12-17T15:33:30+01:00Matija ObrezaJSP: Download PDCI from /explorePlease enable downloading PDCI Excel file from JSP /explore page.
- Add "Download PDCI" under "Download MCPD" for logged-in usersPlease enable downloading PDCI Excel file from JSP /explore page.
- Add "Download PDCI" under "Download MCPD" for logged-in users2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/377Account email validation2018-12-17T16:09:28+01:00Matija ObrezaAccount email validationAfter user creates an account, they receive an email message with a link to validate their account. The current implementation is somewhat buggy and users continue to re-try the link and by then the key is invalidated.
1. Please convert...After user creates an account, they receive an email message with a link to validate their account. The current implementation is somewhat buggy and users continue to re-try the link and by then the key is invalidated.
1. Please convert the current email validation to require a POST
1. The link (GET) can pre-fill the form, but it should not automatically validate the account email
1. Add API v1 endpoint (UserController?) so that validation can be triggered from the React app2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/375Remaining API v1 with validation2018-12-09T18:25:48+01:00Matija ObrezaRemaining API v1 with validationUpdate remaining models and services referenced by any API **v1** as per #365 and #374.Update remaining models and services referenced by any API **v1** as per #365 and #374.2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/372Fix GLIS integration2018-11-19T20:50:28+01:00Matija ObrezaFix GLIS integrationGLIS integration is not working at the moment and DOIs are not reported.GLIS integration is not working at the moment and DOIs are not reported.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/371BrAPI: calls call2019-03-05T10:53:20+01:00Matija ObrezaBrAPI: calls callImplement `/calls` endpoint as per https://brapi.docs.apiary.io/#reference/callsImplement `/calls` endpoint as per https://brapi.docs.apiary.io/#reference/calls2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/369WIEWS Vocabulary2018-11-19T07:13:42+01:00Matija ObrezaWIEWS VocabularyGenesys has two WIEWS vocabulary implementations: as a `Vocabulary` and as `FaoInstitute`.
Remove WIEWS Vocabulary and updaters and update API endpoints for this vocabulary to use data from `InstituteService`. Convert `FaoInstitute` dat...Genesys has two WIEWS vocabulary implementations: as a `Vocabulary` and as `FaoInstitute`.
Remove WIEWS Vocabulary and updaters and update API endpoints for this vocabulary to use data from `InstituteService`. Convert `FaoInstitute` data to `VocabularyTerm` as required.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/368ISO3166 Vocabulary2018-11-19T07:13:50+01:00Matija ObrezaISO3166 VocabularyGenesys now has two ISO3166 Country code implementations: as `Vocabulary` and as `Country` entity.
Delete the Vocabularies (ISO-3alpha, ISO-2alpha, ISO-numeric) and replace the API endpoints so that data from `CountryService` is returne...Genesys now has two ISO3166 Country code implementations: as `Vocabulary` and as `Country` entity.
Delete the Vocabularies (ISO-3alpha, ISO-2alpha, ISO-numeric) and replace the API endpoints so that data from `CountryService` is returned instead of the Vocabulary. Convert `Country` data to `VocabularyTerm` as required.
Remove vocabulary updaters.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/367@Scheduled KPI execution2019-02-04T10:13:32+01:00Matija Obreza@Scheduled KPI executionImplement a method in KPIServiceImpl that executes all existing Executions once every day at **03:00** using `@Scheduled`.
## Details
Production servers run in a cluster and this method must only execute on one node every day. Use htt...Implement a method in KPIServiceImpl that executes all existing Executions once every day at **03:00** using `@Scheduled`.
## Details
Production servers run in a cluster and this method must only execute on one node every day. Use https://github.com/lukas-krecan/ShedLock with a **Hazelcast** lock.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/366FaoInstitute/Partner folders2018-10-30T14:57:47+01:00Matija ObrezaFaoInstitute/Partner foldersPermissions for Folders in the repository must be manually configured and granted to the same users that have permissions on the institute. This is because the `aclParentObject()` of the Folder does not have any reference back to `Instit...Permissions for Folders in the repository must be manually configured and granted to the same users that have permissions on the institute. This is because the `aclParentObject()` of the Folder does not have any reference back to `Institute` or `Partner` entities.
The system should somehow automatically link the Folder to it's "owner":
1. **/wiews/{instCode}** should have ACL parent object set to OID of the Institute with code `instCode`.
1. **/datasets/{uuid}** should have ACL parent object set to OID of the Dataset with `uuid`.
Setting these ACL parents will allow user management on the "owner" level while respecting these same permissions on the Folder and File entities.
## Implementation
1. Folder for institute should be "ensured" and linked when the institute record is created/updated.
1. Dataset folder created when the Dataset is registered.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/365Using javax.validation2019-02-18T16:20:45+01:00Matija ObrezaUsing javax.validationThe code has reference to `net.sf.oval` validators that are not used by the services.
64a8e983 demonstrates use of Bean Validation 1.1.0 API in services:
```
import javax.validation.Valid;
import org.springframework.validation.annotati...The code has reference to `net.sf.oval` validators that are not used by the services.
64a8e983 demonstrates use of Bean Validation 1.1.0 API in services:
```
import javax.validation.Valid;
import org.springframework.validation.annotation.Validated;
...
@Validated
public <T extends Dimension<?>> T save(@Valid T dimension) {
LOG.debug("Persisting dimension {}", dimension);
return dimensionRepository.save(dimension);
}
```
When implementing new service methods please use `@Valid` on model arguments where applicable.
Models use
```
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;
...
@NotNull
@Size(max = 100)
@Column(length = 100, unique = true, nullable = false)
private String name;
```
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/364KPI Execution2018-10-25T00:51:39+02:00Matija ObrezaKPI ExecutionKPIExecution only supports the `count distinct` operation that is useful for most (basic) indicators.
- Extend it to allow calculating the `sum` and `average`.
~~JPA does not define `stddev` which makes `min` and `max` useless. This re...KPIExecution only supports the `count distinct` operation that is useful for most (basic) indicators.
- Extend it to allow calculating the `sum` and `average`.
~~JPA does not define `stddev` which makes `min` and `max` useless. This requires updating the `Observation` entity to add new fields for min, max, stddev (sum and average can be stored as `value`).~~
The type of operation should be an enum.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/363Extra country updater2018-11-12T20:08:32+01:00Matija ObrezaExtra country updaterWe have existing updaters for country codes and names that update the `Country` model. Implement a custom updater (can be based on `DavrosCountrySource`) that will ensure the following **custom** ISO3316-3 codes are imported as `impl.Cou...We have existing updaters for country codes and names that update the `Country` model. Implement a custom updater (can be based on `DavrosCountrySource`) that will ensure the following **custom** ISO3316-3 codes are imported as `impl.Country` to Genesys:
```
NAME, CODE3, URL
AfricaRice, XAB, http://www.africarice.org/
Bioversity, XAC, https://www.bioversityinternational.org/
CIAT, XAD, https://ciat.cgiar.org/
CIMMYT, XAE, https://www.cimmyt.org/
CIP, XAF, https://cipotato.org/
ICARDA, XAG, http://www.icarda.org/
ICRAF, XAH, http://www.worldagroforestry.org/
ICRISAT, XAI, https://www.icrisat.org/
IITA, XAJ, http://www.iita.org/
ILRI, XAK, https://www.ilri.org/
IRRI, XAM, http://irri.org/
```
Perhaps best to put this data to `src/main/resources/iso3316/custom.csv`, other codes may follow.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/360Improve AccessionRefs2018-10-16T13:38:12+02:00Matija ObrezaImprove AccessionRefsRun the unit test `AccessionControllerTest#getAccessionsTest` and observe that the performance of `AccessionRefAspect` is horribly slow:
```
WARN AccessionRefAspect:112 - Re-referencing AccessionRefs for 2000 accessions took 25291ms
```...Run the unit test `AccessionControllerTest#getAccessionsTest` and observe that the performance of `AccessionRefAspect` is horribly slow:
```
WARN AccessionRefAspect:112 - Re-referencing AccessionRefs for 2000 accessions took 25291ms
```
`AccessionRef` is now embedded and therefore the code must check every single Dataset and Subset individually, load all data, stream and process it for each individual accession every time an accession record is persisted.
`AccessionRefAspect` must execute in a few milliseconds. It is a very important aspect and it affects the speed of uploading data to Genesys.
Ideally the aspect would only run
```sql
update accession_ref set accession_id = ? where instCode = ? and ((accenumb = ? and genus = ?) or (doi = ?))
```
## `@Entity`
We need direct access to accession references so that we can do a direct query and update. The code should allow for single and List<Accession> batch updating of references.
I see two options and am looking for ideas.
### Option 1
Remove `@Embeddable` from `AccessionRef` and make it abstract and convert `subset_accessions` and `dataset_accessions` to two separate `@Entity` classes: `DatasetAccessionRef` and `SubsetAccessionRef` that extend AccessionRef. This way the primary key can remain on instcode, genus, species and dataset/subset ID.
No data migration is required, only code changes.
### Option 2
Convert `AccessionRef` to `@Entity`, add `private Subset subset` and `private Dataset dataset` (only one can be set). Liquibase needs to move data to the new table.
## Subset
SubsetController now accepts a list of accession UUIDs, but it should accept a list of AccessionRefs (same as Dataset).2.4Andrey GarnitskiyAndrey Garnitskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/358Linking subsets and datasets with accessions2018-10-09T15:48:06+02:00Matija ObrezaLinking subsets and datasets with accessionsDataset currently uses a list of `AccessionIdentifier` records to specify which accessions (by instcode, accenumb, genus **or** doi) are referenced in the dataset.
Subset, on the other hand, uses a list of `AccessionId` so that even if ...Dataset currently uses a list of `AccessionIdentifier` records to specify which accessions (by instcode, accenumb, genus **or** doi) are referenced in the dataset.
Subset, on the other hand, uses a list of `AccessionId` so that even if the accession is deleted from Genesys (and is removed from `Accession` and added to `AccessionHistoric`) the subset still maintains the reference to the original record. This was a good idea originally, but it fails in the case where an accession is **reuploaded** to Genesys -- Subset will not link to the correct record anymore.
## Properly referencing accessions
- Rename and move `org.genesys.catalog.model.dataset.AccessionIdentifier` to `org.genesys2.server.model.genesys.AccessionRef`.
- Change `private AccessionId accession` to `private Accession accession`
- We can now use `AccessionFilter` in `DatasetFilter`!
### Serialization
Serialize property `accession` of `AccessionRef` as `@JsonUnwrapped`. This will embed all key data from `Accession` into this object (hope that works).
### AspectJ
- When an accession is **deleted** from `Accession` (table `accession`) the code should update all `AccessionRef#accession` to `null`.
- When an accession is **saved** to `Accession` (table `accession`) the code should update `AccessionRef#accession` where a match is found.
## Subset update
Replace `private List<AccessionId> accessions` with `private List<AccessionRef> accessions`.
This allows us to have Subsets that "reference" an accession that does not have passport data in Genesys, but it will in the future. It also allows users to delete an accession (rarely) and re-upload it to the system.
## Liquibase
- ~~Rename `inverseJoinColumns = @JoinColumn(name = "acce_id")` to `a_id`.~~
- Migrate all existing data2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/357Dataset model, service, controller updates2018-10-04T17:04:26+02:00Matija ObrezaDataset model, service, controller updates## Model
We need the country ISO3 code in Dataset `Location` model. Add `countryCode` field (String length 3, nullable).
## Services and controllers
Merge `DatasetCreatorServiceImpl` to `DatasetService`.
Merge `DatasetCreatorControll...## Model
We need the country ISO3 code in Dataset `Location` model. Add `countryCode` field (String length 3, nullable).
## Services and controllers
Merge `DatasetCreatorServiceImpl` to `DatasetService`.
Merge `DatasetCreatorController`, `DatasetFilesController` to `DatasetController` (catalog API v0).2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/356Dataset export details2018-10-04T17:04:26+02:00Matija ObrezaDataset export detailsDataset metadata export as Excel:
- Add url to the Catalog to sheet **metadata* (row **Dataset_URL**).
- Don't display emails. phones, fax numbers in Sheet 'creators'
Dataset metadata export as Excel:
- Add url to the Catalog to sheet **metadata* (row **Dataset_URL**).
- Don't display emails. phones, fax numbers in Sheet 'creators'
2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/355Users API v12018-10-10T19:12:31+02:00Matija ObrezaUsers API v1Copy existing User management API v0 to v1 and update for React.Copy existing User management API v0 to v1 and update for React.2.4Maxym BorodenkoMaxym Borodenko