Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2021-03-05T13:40:47+01:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/575KPI: group by storage2021-03-05T13:40:47+01:00Matija ObrezaKPI: group by storageI need to add an execution that will summarize the number of active accessions by genebank by storage type.
|institute.code|storage|count|
|--|--|--|
|ALB001|10|300|
|ALB001|20|310|
|ALB002|30|30|
The issue I'm facing is how to make a ...I need to add an execution that will summarize the number of active accessions by genebank by storage type.
|institute.code|storage|count|
|--|--|--|
|ALB001|10|300|
|ALB001|20|310|
|ALB002|30|30|
The issue I'm facing is how to make a HQL select that groups by elements of a collection:
```sql
select ?, count(acce) from accession a inner join a.acce acce
group by a.institute.code, acce.storage.elements() ??
```2.5Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/268Add picture available column in accessions browser2021-03-11T13:26:17+01:00Nora Castaneda-AlvarezAdd picture available column in accessions browserPlease add a "Picture available" column when browsing for accessions (https://sandbox.genesys-pgr.org/explore).
Accessions with pictures should be flagged with a "yes" or a small icon that indicates that pictures are available. Use this...Please add a "Picture available" column when browsing for accessions (https://sandbox.genesys-pgr.org/explore).
Accessions with pictures should be flagged with a "yes" or a small icon that indicates that pictures are available. Use this as example: https://sandbox.genesys-pgr.org/10.18730/9X2HE2.5Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/522Number of subsets and datasets2021-03-11T13:26:17+01:00Matija ObrezaNumber of subsets and datasetsFor each `Accession` it would be useful to have quick access to the number of associated **published** subsets and datasets.
Extend `AccessionId` with two new properties `long subsets = 0, datasets = 0`.
When a subset/dataset is publis...For each `Accession` it would be useful to have quick access to the number of associated **published** subsets and datasets.
Extend `AccessionId` with two new properties `long subsets = 0, datasets = 0`.
When a subset/dataset is published, the count for accessions in the list is increased by 1. When it is unpublished, it is decreased by 1.
This must be a very fast SQL update.
Updating these counters should not touch the lastModifiedDate or version fields, but it should schedule a reindex of accessions affected.
Implement an admin function that will:
1. Reset the counters to 0.
1. Scan for subsets and increase the counts
1. Scan for datasets and increase the counts.2.5Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/563Upgrade app-blocks, file-repository to 3.02021-03-11T13:26:17+01:00Matija ObrezaUpgrade app-blocks, file-repository to 3.0Releases 3.0-SNAPSHOT are built with Java 11. Please update app-blocks and file-repo and test.Releases 3.0-SNAPSHOT are built with Java 11. Please update app-blocks and file-repo and test.2.5Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/566ES aggregation by multiple terms2021-03-11T13:26:17+01:00Matija ObrezaES aggregation by multiple termsPlease add a Junit test that will query the ES for `count(species)`, aggregated by `genus`, `species` -- an aggregation of `species` within `genus`. See https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-aggregations-buck...Please add a Junit test that will query the ES for `count(species)`, aggregated by `genus`, `species` -- an aggregation of `species` within `genus`. See https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-aggregations-bucket-terms-aggregation.html
Use JUnit to print ES query and then apply it to your indexed ES.
```json
GET /_search
{
"aggs" : {
"genus" : {
"terms" : {
"field" : "genus",
// "order" : { "playback_stats.max" : "desc" }
},
"aggs" : {
// "playback_stats" : { "stats" : { "field" : "play_count" } }
"species": { "terms": { "field": "species" } },
"aggs" : {
// "playback_stats" : { "stats" : { "field" : "play_count" } }
"origin.iso3166": { "terms": { "field": "origin.iso3166" } }
}
}
}
}
}
```
The result should be something like:
|`genus`|`species`|count|
|-----|--|-----|
|Arabidopsis| Arabidopsis thaliana| 686,602 |
|Hordeum | Hordeum vulgare | 235,395|
|Hordeum | Hordeum Hordeum vulgare | 17,788|
|Hordeum | ... | . |
|Triticum | Triticum aestivum | 332,060 |
|Triticum | Triticum turgidum | 73,364 |
|Triticum | Triticum triticum aestivum |29,196|
How can we control the number of results?
For genesys-pgr/genesys-ui#7822.5Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/574API Search2021-03-11T13:26:17+01:00Matija ObrezaAPI SearchPlease implement a */api/v1/search* endpoint that will do a full-text ES search for top 5 hits for each of the indexed collections (accession, activitypost, faoinstitute).
The response should be a `Map<String, ?>` with the classname as ...Please implement a */api/v1/search* endpoint that will do a full-text ES search for top 5 hits for each of the indexed collections (accession, activitypost, faoinstitute).
The response should be a `Map<String, ?>` with the classname as key and the value as a `Page` with the top 5 search results (`totalElements`, etc.)
Do not use or generate a `filterCode`.
Extra points if the search can be made with a single call to ES.2.5Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/576Updates from GG-CE Server2021-03-11T16:27:15+01:00Matija ObrezaUpdates from GG-CE ServerBring the following updates from gg-ce-server:
1. Prevent reindexing 18814608 Maxym Borodenko 10 Mar 2021 at 09:40
1. ShortFilterService: throw proper exception a7fff53c 5 Mar 2021 at 14:10Bring the following updates from gg-ce-server:
1. Prevent reindexing 18814608 Maxym Borodenko 10 Mar 2021 at 09:40
1. ShortFilterService: throw proper exception a7fff53c 5 Mar 2021 at 14:102021.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/577Release versioning2021-03-12T08:07:24+01:00Matija ObrezaRelease versioningAdopt the same versioning as GG-CE.Adopt the same versioning as GG-CE.2021.0Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/579Find similar descriptors2021-03-18T13:42:35+01:00Matija ObrezaFind similar descriptorsAdd a function that for selected input `Descriptor` (in `@RequestBody`) will return a ranked list of similar existing `Descriptor`s in the database.
Use a similar approach as for accession similarity, but:
1. `title`, `description` str...Add a function that for selected input `Descriptor` (in `@RequestBody`) will return a ranked list of similar existing `Descriptor`s in the database.
Use a similar approach as for accession similarity, but:
1. `title`, `description` string similarity
1. `columnName`, `uom`
1. `category`
1. `crop`
1. `minValue`, `maxValue` can be ignored
1. compare `descriptor.terms` by `code`, `title`, `description`
1. find the codes of descriptor **A** that match a code in descriptor **B**
1. each match adds points to the score2021.1Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/581Logging downloads2021-03-22T11:23:02+01:00Matija ObrezaLogging downloadsI'd like to add a KPI for the number of downloads from Genesys. Since KPI's are based on JPA, we need something that will allow us to track these:
```java
class DownloadLog implements AclAwareModel {
Date date;
String url;
// These m...I'd like to add a KPI for the number of downloads from Genesys. Since KPI's are based on JPA, we need something that will allow us to track these:
```java
class DownloadLog implements AclAwareModel {
Date date;
String url;
// These may be useful
String entity;
String filterCode;
String clientIp;
}
```
The KPI would count the downloads by `url`, `entity` and `filterCode`.
If the download is canceled, the `DownloadLog` record should not be added.2021.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/578Dataset analyzer2021-03-31T11:12:13+02:00Matija ObrezaDataset analyzerOur work in genesys-pgr/genesys-ui#338 assumes that datasets are clean and ready for final mapping to descriptors. That's not usually the case and data commonly needs more cleaning.
In this ticket, we add API endpoints to `AmphibianCont...Our work in genesys-pgr/genesys-ui#338 assumes that datasets are clean and ready for final mapping to descriptors. That's not usually the case and data commonly needs more cleaning.
In this ticket, we add API endpoints to `AmphibianController` to upload and analyze any dataset.
*/api/v1/amphibian/upload* (requires VETTEDUSER, ADMINISTRATOR): Upload a file to file repository to path */amphibian/* and initiate Amphibian preview.
Add */api/v1/amphibian/stats/{uuid}/{sheet}* to return (for selected columns `fields` and offset `startRow`):
- count of values
- count of distinct values
- list of max 100 distinct values
## The UI
The UI will allow the user to upload a data file and will then render a UI similar to:
![image](/uploads/9b730c815f71828a58fd2271f41d11ab/image.png)2021.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/583Release 2021.12021-04-09T07:39:41+02:00Matija ObrezaRelease 2021.12021.1Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/582JaCoCo and SpotBugs2021-04-12T10:41:57+02:00Matija ObrezaJaCoCo and SpotBugsAdd JaCoCo and SpotBugs to `pom.xml` and update the pipeline. Use GG-CE server and app-blocks projects as examples.Add JaCoCo and SpotBugs to `pom.xml` and update the pipeline. Use GG-CE server and app-blocks projects as examples.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/587Download log with @CreatedBy2021-04-16T09:25:10+02:00Matija ObrezaDownload log with @CreatedByThe logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|...The logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|----|----|
| 1 | DatasetController | downloadDatasetFile | /api/v1/dataset/b9c8e4db-d6b1-44a8-b1a3-307c1f496daf/download/131077 | 2021-04-09 02:48:33.000000 | 181.xxxx |
| 2 | DatasetController | downloadDatasetFile | /api/v1/dataset/f6cd4e85-7e44-4429-b00d-9d9eb99ea0bb/download/3244033 | 2021-04-09 06:46:56.000000 | 154.xxxx |
| 3 | DatasetController | downloadDatasetFile | /api/v1/dataset/929a273d-7882-43eb-8b1a-86032cbeb892/download/3112961 | 2021-04-09 08:46:07.000000 | 154.xxxx |
It would be very useful if `createdBy` was also included in the database.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/591Database connection pool2021-04-16T11:14:53+02:00Matija ObrezaDatabase connection poolCheck why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by...Check why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
19:05:32,767 genesys-background-27 ERROR o.g.s.s.i.ElasticsearchServiceImpl:602 - org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.update(ElasticsearchServiceImpl.java:627)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.lambda$asyncUpdate$3(ElasticsearchServiceImpl.java:600)
...
Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
...
Caused by: java.sql.SQLException: Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
on HostAddress{host='mariadb2', port=3306, type='replica'},master=false. Driver has reconnect connection
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:563)
at org.mariadb.jdbc.internal.failover.FailoverProxy.handleFailOver(FailoverProxy.java:391)
...
Caused by: java.sql.SQLNonTransientConnectionException: Connection reset by peer (Write failed)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2076)
...
Caused by: java.net.SocketException: Connection reset by peer (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
```2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/589DownloadLog and incomplete downloads2021-04-16T14:28:06+02:00Matija ObrezaDownloadLog and incomplete downloadsWhen I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-...When I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-13 WARN o.g.s.s.i.DownloadServiceImpl:219 - Error generating: org.eclipse.jetty.io.EofException
18:10:42,203 qtp1586845078-13 WARN o.g.s.a.v.AccessionController:663 - Download was aborted: Closed
```
This should not write a new download log message.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/593Downloading images2021-05-11T15:12:46+02:00Matija ObrezaDownloading images@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to down...@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to download and/or the total size of bytes to compress and stream to the user must be limited.
Since images are not stored on cluster nodes, they must be streamed to the resulting Zip file.2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/588Session cookies on /api2021-05-19T08:14:19+02:00Matija ObrezaSession cookies on /apiA normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESS...A normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESSIONID` and `hz-session-id` in the request:
```
Cookie: GENESYS_sandbox.genesys-pgr.org=eyJhbGciOixxxx.eyJleHAixxx.e9piZqWpIxxxx; G_ENABLED_IDPS=google; G_AUTHUSER_H=1; JSESSIONID=node063vg9xhbqpul1z20x2tr6ov683.node0; hz-session-id=HZ72872DD6A59743BCBF6E66EAD733E702
```
Both relevant cookies `JSESSIONID` and `hz-session-id` are set to `SameSite` (i.e. api.sandbox.genesys-pgr.org).
They should be completely ignored for requests to */api*. It's even stranger that there are **two `hz-session-id` cookies** in the response?2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/594Overviews2021-05-21T13:58:01+02:00Matija ObrezaOverviewsThe Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.The Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/597Finding similar accessions2021-06-29T13:39:09+02:00Matija ObrezaFinding similar accessionsFinding "Similar accessions" of https://sandbox.genesys-pgr.org/a/4c669455-02b0-421b-a5be-80fa3111fa94 throws
```
java.lang.NullPointerException
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(Accessio...Finding "Similar accessions" of https://sandbox.genesys-pgr.org/a/4c669455-02b0-421b-a5be-80fa3111fa94 throws
```
java.lang.NullPointerException
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(AccessionDuplicateFinder.java:219)
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(AccessionDuplicateFinder.java:44)
at org.genesys2.server.service.worker.dupe.DuplicateFinder.findSimilar(DuplicateFinder.java:124)
```2021.2Maxym BorodenkoMaxym Borodenko