Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2021-09-20T18:46:10+02:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/595Relaying requests for other systems2021-09-20T18:46:10+02:00Matija ObrezaRelaying requests for other systemsGenesys is a gateway for users to select (based on data in Genesys) and make requests for material (to multiple genebanks). This ticket describes making the existing Genesys request mechanisms and workflows available for use by other sys...Genesys is a gateway for users to select (based on data in Genesys) and make requests for material (to multiple genebanks). This ticket describes making the existing Genesys request mechanisms and workflows available for use by other systems (e.g. snp-seek.irri.org).
## Basic workflow
1. User makes the selection of material on Site A (shopping cart on Site A)
2. User clicks "Request for material" button on Site A
- Site A sends `requestData` to Genesys
- Genesys responds with a request identifier `reqId`
3. Site A redirects the user to Genesys website to complete the request for selected material
Step **2** is executed by the server of Site A (not in the browser).
## Initiating the Request
Request data sent to Genesys API from the external system includes only the list of selected accessions (preferably by `doi`, alternatively with FAO `instituteCode` and `accessionNumber`). No personally identifiable information is sent in the request.
```json
[
{ "doi": "10.xxxx/1231", "instituteCode": "PHL001", "accessionNumber": "IRGC 1000001" },
{ ... }
]
```
Genesys may have a limit on the number of items accepted for a single request.
## Genesys response
Genesys checks that all requested items exist in the database and that they are flagged as *available for distribution*.
**200 OK** response: all items in the list pass the checks, a request record is persisted and waiting for the user to resume the ordering process on the Genesys website.
```json
{
"id": "some-generated-id",
"url": "https://the-url-to-continue-the-order",
// maybe more detail?
}
```
**4xx Client error** response: one or more items are not available in Genesys. The request is not persisted.
```json
{
"error": "Error message",
"detail": { // TBD }
}
```
**5xx Server error** response: something went horribly wrong, response body contains a standard Genesys API error message in JSON.
## Redirecting the user
Site A has the **request identifier** and the `url` that is used to resume the ordering process on the Genesys website. Site A may store this data for further processing.
The user navigates to the `url` provided in the Genesys response either using an HTTP redirect or a regular link. Upon accessing the `url`, Genesys checks that request data exists (has not expired) and renders the forms for the user to complete the request information.
Changes to the list of items requested are not allowed.
If the user aborts the request, it is deleted from the database. If the request expires due to a timeout set by Genesys, it is deleted from the database.
## Completing the request
After the user provides the necessary request information, the existing Genesys request validation process kicks in: email validations may be sent, Easy-SMTA is checked, and eventually the request is relayed to the genebank holding the material.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/594Overviews2021-05-21T13:58:01+02:00Matija ObrezaOverviewsThe Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.The Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/593Downloading images2021-05-11T15:12:46+02:00Matija ObrezaDownloading images@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to down...@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to download and/or the total size of bytes to compress and stream to the user must be limited.
Since images are not stored on cluster nodes, they must be streamed to the resulting Zip file.2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/592Merge AccessionGeo to AccessionId2021-08-02T11:33:03+02:00Matija ObrezaMerge AccessionGeo to AccessionIdWe have a 1-to-1 relationship between `AccessionId` and `AccessionGeo`. This results in complex queries by `latitude` and `longitude`.
Please move the fields from `AccessionGeo` to `AccessionId` and drop `AccessionGeo` from the database...We have a 1-to-1 relationship between `AccessionId` and `AccessionGeo`. This results in complex queries by `latitude` and `longitude`.
Please move the fields from `AccessionGeo` to `AccessionId` and drop `AccessionGeo` from the database.
We still need the `AccessionGeo` class to keep API responses compatible. It can be created in `AccessionId#getAccessionGeo()`.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/591Database connection pool2021-04-16T11:14:53+02:00Matija ObrezaDatabase connection poolCheck why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by...Check why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
19:05:32,767 genesys-background-27 ERROR o.g.s.s.i.ElasticsearchServiceImpl:602 - org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.update(ElasticsearchServiceImpl.java:627)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.lambda$asyncUpdate$3(ElasticsearchServiceImpl.java:600)
...
Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
...
Caused by: java.sql.SQLException: Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
on HostAddress{host='mariadb2', port=3306, type='replica'},master=false. Driver has reconnect connection
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:563)
at org.mariadb.jdbc.internal.failover.FailoverProxy.handleFailOver(FailoverProxy.java:391)
...
Caused by: java.sql.SQLNonTransientConnectionException: Connection reset by peer (Write failed)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2076)
...
Caused by: java.net.SocketException: Connection reset by peer (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
```2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/590ES full reindex with slow-down2023-05-26T11:02:38+02:00Matija ObrezaES full reindex with slow-downThe `scanDatabase` can sometimes be waaaaaay faster than our ES queue processor and ES service itself.
I'm getting `Update queue size: 1559000`: 1.5M elements in the reindex queue and that indicates that the indexing is falling behind o...The `scanDatabase` can sometimes be waaaaaay faster than our ES queue processor and ES service itself.
I'm getting `Update queue size: 1559000`: 1.5M elements in the reindex queue and that indicates that the indexing is falling behind of scanning.
The queue size should not be larger than 10,000 entries at any moment.
The queue grows that large because of
1. Slow indexing
1. Fast scheduling
Where is the bottleneck?Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/589DownloadLog and incomplete downloads2021-04-16T14:28:06+02:00Matija ObrezaDownloadLog and incomplete downloadsWhen I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-...When I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-13 WARN o.g.s.s.i.DownloadServiceImpl:219 - Error generating: org.eclipse.jetty.io.EofException
18:10:42,203 qtp1586845078-13 WARN o.g.s.a.v.AccessionController:663 - Download was aborted: Closed
```
This should not write a new download log message.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/588Session cookies on /api2021-05-19T08:14:19+02:00Matija ObrezaSession cookies on /apiA normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESS...A normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESSIONID` and `hz-session-id` in the request:
```
Cookie: GENESYS_sandbox.genesys-pgr.org=eyJhbGciOixxxx.eyJleHAixxx.e9piZqWpIxxxx; G_ENABLED_IDPS=google; G_AUTHUSER_H=1; JSESSIONID=node063vg9xhbqpul1z20x2tr6ov683.node0; hz-session-id=HZ72872DD6A59743BCBF6E66EAD733E702
```
Both relevant cookies `JSESSIONID` and `hz-session-id` are set to `SameSite` (i.e. api.sandbox.genesys-pgr.org).
They should be completely ignored for requests to */api*. It's even stranger that there are **two `hz-session-id` cookies** in the response?2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/587Download log with @CreatedBy2021-04-16T09:25:10+02:00Matija ObrezaDownload log with @CreatedByThe logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|...The logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|----|----|
| 1 | DatasetController | downloadDatasetFile | /api/v1/dataset/b9c8e4db-d6b1-44a8-b1a3-307c1f496daf/download/131077 | 2021-04-09 02:48:33.000000 | 181.xxxx |
| 2 | DatasetController | downloadDatasetFile | /api/v1/dataset/f6cd4e85-7e44-4429-b00d-9d9eb99ea0bb/download/3244033 | 2021-04-09 06:46:56.000000 | 154.xxxx |
| 3 | DatasetController | downloadDatasetFile | /api/v1/dataset/929a273d-7882-43eb-8b1a-86032cbeb892/download/3112961 | 2021-04-09 08:46:07.000000 | 154.xxxx |
It would be very useful if `createdBy` was also included in the database.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/586Performance with Accessions /list as CSV2021-09-20T10:49:38+02:00Matija ObrezaPerformance with Accessions /list as CSVUsing the CSV format to retrieve large selections (pages) using the /list endpoint appear to be very slow (see https://gitlab.croptrust.org/genesys-pgr/genesysr/-/issues/2#note_53485).
We should explore two things:
1. Adding support to...Using the CSV format to retrieve large selections (pages) using the /list endpoint appear to be very slow (see https://gitlab.croptrust.org/genesys-pgr/genesysr/-/issues/2#note_53485).
We should explore two things:
1. Adding support to `exclude` fields (we now have `select` to include fields)
1. Not serializing things with `flatMap` that are not selected.
The response is now very broad and includes very deeply nested entities. We should explore what can be excluded.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/585Cleanup: API v22021-10-04T11:19:01+02:00Matija ObrezaCleanup: API v2Most of the code for API **v2** and the tests of it are a copy of API **v1**.
Can we make all API **v2** `Controllers` extend from API **v1** controllers? -- that way we can reduce duplicated code?
The same applies to unit tests of **v...Most of the code for API **v2** and the tests of it are a copy of API **v1**.
Can we make all API **v2** `Controllers` extend from API **v1** controllers? -- that way we can reduce duplicated code?
The same applies to unit tests of **v2** controllers.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/584Cleanup: Drop API v02023-09-04T12:30:44+02:00Matija ObrezaCleanup: Drop API v0I don't think we use API **/api/v0** calls anywhere, the only place could be our admin MVC.
I believe that can be dropped.I don't think we use API **/api/v0** calls anywhere, the only place could be our admin MVC.
I believe that can be dropped.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/583Release 2021.12021-04-09T07:39:41+02:00Matija ObrezaRelease 2021.12021.1Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/582JaCoCo and SpotBugs2021-04-12T10:41:57+02:00Matija ObrezaJaCoCo and SpotBugsAdd JaCoCo and SpotBugs to `pom.xml` and update the pipeline. Use GG-CE server and app-blocks projects as examples.Add JaCoCo and SpotBugs to `pom.xml` and update the pipeline. Use GG-CE server and app-blocks projects as examples.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/581Logging downloads2021-03-22T11:23:02+01:00Matija ObrezaLogging downloadsI'd like to add a KPI for the number of downloads from Genesys. Since KPI's are based on JPA, we need something that will allow us to track these:
```java
class DownloadLog implements AclAwareModel {
Date date;
String url;
// These m...I'd like to add a KPI for the number of downloads from Genesys. Since KPI's are based on JPA, we need something that will allow us to track these:
```java
class DownloadLog implements AclAwareModel {
Date date;
String url;
// These may be useful
String entity;
String filterCode;
String clientIp;
}
```
The KPI would count the downloads by `url`, `entity` and `filterCode`.
If the download is canceled, the `DownloadLog` record should not be added.2021.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/580Descriptor-Descriptor mapping2021-10-04T11:20:16+02:00Matija ObrezaDescriptor-Descriptor mappingGenesys has a large database of descriptor definitions and many of them refer to the same trait, but are categorized under a different crop or there are slight differences in the description, but not the *meaning*.
For example, we have ...Genesys has a large database of descriptor definitions and many of them refer to the same trait, but are categorized under a different crop or there are slight differences in the description, but not the *meaning*.
For example, we have 64 published "Plant height" descriptors. Similar descriptors can be linked/combined to form a "super descriptor". A super descriptor may be crop-specific. Individual descriptors are linked to the super descriptor and each of their `terms` is associated with one term in the super descriptor.
#579 helps in finding similar `Descriptor`s and is instrumental for building the database of `SuperDescriptor`s.
# Super descriptor
|IITA|NPGRL|WorldVeg|
|--|--|--|
|![image](/uploads/fddce512f92f55fd68d73fe746a74d9f/image.png)|![image](/uploads/00779c8088176cae95487b001316602b/image.png)|![image](/uploads/ba5e5de56ebb59761c9f918459b853e0/image.png)|
These could be combined into a super descriptor with:
1. `title` Hilum color
1. `description` Seed eye color
1. `lang` en
1. `crop` `null`
1. `category` `CHARACTERIZATION`
## Coding table
Each term of a descriptor is mapped to exactly one term in the super descriptor. The super has terms that are more broadly defined and applicable to many crops:
| Super | D1 | D2 | D3 | D4 |
|----------|----------------------------------------------|---------|-----------|------------------------|
| Absent | Eye absent | | COLORLESS | |
| Brown | Brown splash or grey, Tan brown | Brown | | Brown |
| Black | Blue to black, Blue to black spots or mottle | Black | BLACK | Black, imperfect Black |
| White | | White | | |
| Red | Red | | | |
| Green | Green | | | Green |
| Grey | | | GRAY | Grey |
| Speckled | Speckled | | | |
| Mottled | Mottled, Mottled and speckled | | | Buff |
| Mixture | | Mixture | MIXED | |
| Other | Other | | OTHER | Others |
| Yellow | | | | Yellow |Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/579Find similar descriptors2021-03-18T13:42:35+01:00Matija ObrezaFind similar descriptorsAdd a function that for selected input `Descriptor` (in `@RequestBody`) will return a ranked list of similar existing `Descriptor`s in the database.
Use a similar approach as for accession similarity, but:
1. `title`, `description` str...Add a function that for selected input `Descriptor` (in `@RequestBody`) will return a ranked list of similar existing `Descriptor`s in the database.
Use a similar approach as for accession similarity, but:
1. `title`, `description` string similarity
1. `columnName`, `uom`
1. `category`
1. `crop`
1. `minValue`, `maxValue` can be ignored
1. compare `descriptor.terms` by `code`, `title`, `description`
1. find the codes of descriptor **A** that match a code in descriptor **B**
1. each match adds points to the score2021.1Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/578Dataset analyzer2021-03-31T11:12:13+02:00Matija ObrezaDataset analyzerOur work in genesys-pgr/genesys-ui#338 assumes that datasets are clean and ready for final mapping to descriptors. That's not usually the case and data commonly needs more cleaning.
In this ticket, we add API endpoints to `AmphibianCont...Our work in genesys-pgr/genesys-ui#338 assumes that datasets are clean and ready for final mapping to descriptors. That's not usually the case and data commonly needs more cleaning.
In this ticket, we add API endpoints to `AmphibianController` to upload and analyze any dataset.
*/api/v1/amphibian/upload* (requires VETTEDUSER, ADMINISTRATOR): Upload a file to file repository to path */amphibian/* and initiate Amphibian preview.
Add */api/v1/amphibian/stats/{uuid}/{sheet}* to return (for selected columns `fields` and offset `startRow`):
- count of values
- count of distinct values
- list of max 100 distinct values
## The UI
The UI will allow the user to upload a data file and will then render a UI similar to:
![image](/uploads/9b730c815f71828a58fd2271f41d11ab/image.png)2021.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/577Release versioning2021-03-12T08:07:24+01:00Matija ObrezaRelease versioningAdopt the same versioning as GG-CE.Adopt the same versioning as GG-CE.2021.0Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/576Updates from GG-CE Server2021-03-11T16:27:15+01:00Matija ObrezaUpdates from GG-CE ServerBring the following updates from gg-ce-server:
1. Prevent reindexing 18814608 Maxym Borodenko 10 Mar 2021 at 09:40
1. ShortFilterService: throw proper exception a7fff53c 5 Mar 2021 at 14:10Bring the following updates from gg-ce-server:
1. Prevent reindexing 18814608 Maxym Borodenko 10 Mar 2021 at 09:40
1. ShortFilterService: throw proper exception a7fff53c 5 Mar 2021 at 14:102021.1Maxym BorodenkoMaxym Borodenko