Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2021-09-22T09:28:31+02:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/606Subset types2021-09-22T09:28:31+02:00Matija ObrezaSubset typesWe would like to classify the Subsets with a set of tags applied to each subset. The set of allowed values is
- `CORE` *Core collection*
- `REFERENCE_SET` *Reference set*
- `SUSCEPTIBILITY` *Stress susceptibility*
- `TOLERANCE` *Stress ...We would like to classify the Subsets with a set of tags applied to each subset. The set of allowed values is
- `CORE` *Core collection*
- `REFERENCE_SET` *Reference set*
- `SUSCEPTIBILITY` *Stress susceptibility*
- `TOLERANCE` *Stress tolerance*
- `GENOMIC` *Genomic data*
- `TRAIT` *Trait selection*
- `LIST` *Germplasm list*
Add a `String tags` to `Subset`.
```
@Column(length = 20, nullable = true)
private SubsetType subsetType;
@Column(length = 200, nullable = true)
private String primaryTrait;
```
We should introduce an `enum SubsetType` to control the incoming values. Only one option is possible.2022.1Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/605Issue with CSVMessageConverter2021-09-20T18:23:50+02:00Matija ObrezaIssue with CSVMessageConverterOur `CSVMessageConverter` relies on Jackson serialization to JSON before producing the `text/csv` response.
The current implementation does not properly handle cases where Jackson serializes just the `id` of the referenced object it has...Our `CSVMessageConverter` relies on Jackson serialization to JSON before producing the `text/csv` response.
The current implementation does not properly handle cases where Jackson serializes just the `id` of the referenced object it has serialized before:
![image](/uploads/834ca916dbfb78671dc2bf4734b4eb40/image.png)
Please update the unit tests to ensure referenced entities (`countryOfOrigin`, `taxonomy`, `taxonomy.grinTaxonomy`, ...) are fully serialized in CSV.
Genesys website is not affected by this, it is able to handle these references.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/604HTTP CSP headers2021-09-20T18:23:49+02:00Matija ObrezaHTTP CSP headersPlease update Genesys to CSP headers, especially:
- `Content-Security-Policy: frame-src 'none'`Please update Genesys to CSP headers, especially:
- `Content-Security-Policy: frame-src 'none'`2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/603Embedded ES node2021-09-28T06:42:57+02:00Matija ObrezaEmbedded ES nodePlease bring the changes for Embedded ES node from GG-CE to Genesys.Please bring the changes for Embedded ES node from GG-CE to Genesys.2022.1Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/602Uploader: Clearing accessionId.coll2021-09-20T18:23:49+02:00Matija ObrezaUploader: Clearing accessionId.collWhile testing integration with GG-CE I stumbled upon an error where
the existing record in the database has some collecting data, including one value in `accession.accessionId.coll.collName` list and a date in `accession.accessionId.coll...While testing integration with GG-CE I stumbled upon an error where
the existing record in the database has some collecting data, including one value in `accession.accessionId.coll.collName` list and a date in `accession.accessionId.coll.collDate`. The new upload sets:
```json
...
"coll": {
"collDate": null,
"collSite": null,
"collNumb": null,
"collSrc": null,
"collMissId": null,
"collCode": [],
"collName": [],
"collInstAddress": []
},
...
```
This should effectively remove the `AccessionCollect` record from the database in `AccessionUploader` at:
![image](/uploads/a3da6ef56c801996594b9a7754d2a608/image.png)
It results in **Cannot delete or update a parent row:** *(`sandbox`.`accession_collect_name`, CONSTRAINT `FK_g6h5uuvq16105ljx8r1jjxojr` FOREIGN KEY (`collectId`) REFERENCES `accession_collect` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION)*:
```
Upload resulted in: ERROR instCode=MEX002 doi=10.18730/G8G4B acceNumb=CIMMYTMA 71 genus=Zea
error=could not execute batch; SQL [delete from accession_collect where id=? and version=?]; constraint [null];
nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch
could not execute batch
(conn=266448) Cannot delete or update a parent row:
a foreign key constraint fails
(`sandbox`.`accession_collect_name`, CONSTRAINT `FK_g6h5uuvq16105ljx8r1jjxojr` FOREIGN KEY (`collectId`) REFERENCES `accession_collect` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION)
```
Please add a unit test and figure this out.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/601Google Identity Services2023-01-23T20:13:06+01:00Matija ObrezaGoogle Identity ServicesGoogle sign-in needs upgrading.
> One or more of your web applications uses the legacy Google Sign-In JavaScript library. Please migrate your project(s) to the new Google Identity Services SDK before March 31, 2023.
>
> At your earlies...Google sign-in needs upgrading.
> One or more of your web applications uses the legacy Google Sign-In JavaScript library. Please migrate your project(s) to the new Google Identity Services SDK before March 31, 2023.
>
> At your earliest convenience, migrate to the new Google Identity Services by following the migration guide https://notifications.google.com/g/p/AD-FnEyIyeoYKdydXaPAMcuuzaawjP3y0gM7TwzpXChLgx3X2Xz4SKTR-4wEhry1iZ1DxJ4TaD504ntuXzcMkE50nh3Y20YQr1X23dvUHS_LznssfohzZm3M_eP9gsCLv_MJaQ0V6Q.
>
> Be sure to read through the Sign-In With Google resources guide.Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/600Extra localizations of country names2021-09-20T18:23:49+02:00Matija ObrezaExtra localizations of country namesGenesys sources country names in different languages from geonames. It happens that a country name is not available for a specific language.
E.g. `zh-TW` translation for `USA` is not available and I need to be able to register it.
We n...Genesys sources country names in different languages from geonames. It happens that a country name is not available for a specific language.
E.g. `zh-TW` translation for `USA` is not available and I need to be able to register it.
We need to be able to store a JSON of customized translations in `Country.nameJCustom` field and this JSON is then merged to the incoming `nameJ` string. `nameJCustom` adds or overwrites values for `nameJ`.
We will also need a `CRUDController<Country>` for ADMINISTRATORS. I don't think we have one.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/599Set the secret for an OAuthClient2021-09-20T18:23:49+02:00Maxym BorodenkoSet the secret for an OAuthClientAdd an endpoint to **set** the secret for an `OAuthClient`. Can be applied to GG-CE as well.Add an endpoint to **set** the secret for an `OAuthClient`. Can be applied to GG-CE as well.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/598Admin: Audit logs API2021-09-20T18:23:49+02:00Maxym BorodenkoAdmin: Audit logs APIhttps://gitlab.croptrust.org/genesys-pgr/genesys-ui/-/issues/822 requires CRUD API for audit logs.https://gitlab.croptrust.org/genesys-pgr/genesys-ui/-/issues/822 requires CRUD API for audit logs.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/597Finding similar accessions2021-06-29T13:39:09+02:00Matija ObrezaFinding similar accessionsFinding "Similar accessions" of https://sandbox.genesys-pgr.org/a/4c669455-02b0-421b-a5be-80fa3111fa94 throws
```
java.lang.NullPointerException
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(Accessio...Finding "Similar accessions" of https://sandbox.genesys-pgr.org/a/4c669455-02b0-421b-a5be-80fa3111fa94 throws
```
java.lang.NullPointerException
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(AccessionDuplicateFinder.java:219)
at org.genesys2.server.service.worker.dupe.AccessionDuplicateFinder.getCandidates(AccessionDuplicateFinder.java:44)
at org.genesys2.server.service.worker.dupe.DuplicateFinder.findSimilar(DuplicateFinder.java:124)
```2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/596Upgrade dependencies2021-08-03T18:01:01+02:00Matija ObrezaUpgrade dependenciesFollowing genesys-pgr/file-repository#38 and genesys-pgr/application-blocks#44 please upgrade Genesys Server dependencies.Following genesys-pgr/file-repository#38 and genesys-pgr/application-blocks#44 please upgrade Genesys Server dependencies.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/594Overviews2021-05-21T13:58:01+02:00Matija ObrezaOverviewsThe Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.The Overviews load very very slowly in Production (www.genesys-pgr.org). And also Partner list and the Subsets lists.
We should (at least) enable caching of the results of that service method.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/593Downloading images2021-05-11T15:12:46+02:00Matija ObrezaDownloading images@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to down...@jguerrero describes a use case for downloading in bulk the images of filtered accessions:
1. Filter accessions
2. Browse images
3. Click "Download" to receive a Zip archive of images of selected accessions
The number of images to download and/or the total size of bytes to compress and stream to the user must be limited.
Since images are not stored on cluster nodes, they must be streamed to the resulting Zip file.2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/592Merge AccessionGeo to AccessionId2021-08-02T11:33:03+02:00Matija ObrezaMerge AccessionGeo to AccessionIdWe have a 1-to-1 relationship between `AccessionId` and `AccessionGeo`. This results in complex queries by `latitude` and `longitude`.
Please move the fields from `AccessionGeo` to `AccessionId` and drop `AccessionGeo` from the database...We have a 1-to-1 relationship between `AccessionId` and `AccessionGeo`. This results in complex queries by `latitude` and `longitude`.
Please move the fields from `AccessionGeo` to `AccessionId` and drop `AccessionGeo` from the database.
We still need the `AccessionGeo` class to keep API responses compatible. It can be created in `AccessionId#getAccessionGeo()`.2021.3Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/591Database connection pool2021-04-16T11:14:53+02:00Matija ObrezaDatabase connection poolCheck why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by...Check why this happens during ES reindexing:
```
19:05:32.766 [genesys-background-27] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
19:05:32,767 genesys-background-27 ERROR o.g.s.s.i.ElasticsearchServiceImpl:602 - org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.update(ElasticsearchServiceImpl.java:627)
...
at org.genesys2.server.service.impl.ElasticsearchServiceImpl.lambda$asyncUpdate$3(ElasticsearchServiceImpl.java:600)
...
Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=203) Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
...
Caused by: java.sql.SQLException: Communications link failure with secondary host mariadb2:3306. Connection reset by peer (Write failed)
on HostAddress{host='mariadb2', port=3306, type='replica'},master=false. Driver has reconnect connection
at org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:563)
at org.mariadb.jdbc.internal.failover.FailoverProxy.handleFailOver(FailoverProxy.java:391)
...
Caused by: java.sql.SQLNonTransientConnectionException: Connection reset by peer (Write failed)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2076)
...
Caused by: java.net.SocketException: Connection reset by peer (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
```2021.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/590ES full reindex with slow-down2023-05-26T11:02:38+02:00Matija ObrezaES full reindex with slow-downThe `scanDatabase` can sometimes be waaaaaay faster than our ES queue processor and ES service itself.
I'm getting `Update queue size: 1559000`: 1.5M elements in the reindex queue and that indicates that the indexing is falling behind o...The `scanDatabase` can sometimes be waaaaaay faster than our ES queue processor and ES service itself.
I'm getting `Update queue size: 1559000`: 1.5M elements in the reindex queue and that indicates that the indexing is falling behind of scanning.
The queue size should not be larger than 10,000 entries at any moment.
The queue grows that large because of
1. Slow indexing
1. Fast scheduling
Where is the bottleneck?Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/589DownloadLog and incomplete downloads2021-04-16T14:28:06+02:00Matija ObrezaDownloadLog and incomplete downloadsWhen I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-...When I start a MCPD download from the UI and then cancel the download, the server records a `Download was aborted: Closed` log message.
But a new `DownloadLog` entry is still registered in the database.
```
18:10:42,195 qtp1586845078-13 WARN o.g.s.s.i.DownloadServiceImpl:219 - Error generating: org.eclipse.jetty.io.EofException
18:10:42,203 qtp1586845078-13 WARN o.g.s.a.v.AccessionController:663 - Download was aborted: Closed
```
This should not write a new download log message.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/588Session cookies on /api2021-05-19T08:14:19+02:00Matija ObrezaSession cookies on /apiA normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESS...A normal HTTP GET request to */api* should not generate new cookies:
![image](/uploads/7eefa65cf4300f6ba907059f3a55eb48/image.png)
A request to */api/v1/geo/iso3166/decode?locale=en* is quite frequent from the UI and it includes `JSESSIONID` and `hz-session-id` in the request:
```
Cookie: GENESYS_sandbox.genesys-pgr.org=eyJhbGciOixxxx.eyJleHAixxx.e9piZqWpIxxxx; G_ENABLED_IDPS=google; G_AUTHUSER_H=1; JSESSIONID=node063vg9xhbqpul1z20x2tr6ov683.node0; hz-session-id=HZ72872DD6A59743BCBF6E66EAD733E702
```
Both relevant cookies `JSESSIONID` and `hz-session-id` are set to `SameSite` (i.e. api.sandbox.genesys-pgr.org).
They should be completely ignored for requests to */api*. It's even stranger that there are **two `hz-session-id` cookies** in the response?2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/587Download log with @CreatedBy2021-04-16T09:25:10+02:00Matija ObrezaDownload log with @CreatedByThe logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|...The logging works nicely in production:
| id | controller | method | url | date | clientIp |
|----|----|----|----|----|----|
| 1 | DatasetController | downloadDatasetFile | /api/v1/dataset/b9c8e4db-d6b1-44a8-b1a3-307c1f496daf/download/131077 | 2021-04-09 02:48:33.000000 | 181.xxxx |
| 2 | DatasetController | downloadDatasetFile | /api/v1/dataset/f6cd4e85-7e44-4429-b00d-9d9eb99ea0bb/download/3244033 | 2021-04-09 06:46:56.000000 | 154.xxxx |
| 3 | DatasetController | downloadDatasetFile | /api/v1/dataset/929a273d-7882-43eb-8b1a-86032cbeb892/download/3112961 | 2021-04-09 08:46:07.000000 | 154.xxxx |
It would be very useful if `createdBy` was also included in the database.2021.2Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/586Performance with Accessions /list as CSV2021-09-20T10:49:38+02:00Matija ObrezaPerformance with Accessions /list as CSVUsing the CSV format to retrieve large selections (pages) using the /list endpoint appear to be very slow (see https://gitlab.croptrust.org/genesys-pgr/genesysr/-/issues/2#note_53485).
We should explore two things:
1. Adding support to...Using the CSV format to retrieve large selections (pages) using the /list endpoint appear to be very slow (see https://gitlab.croptrust.org/genesys-pgr/genesysr/-/issues/2#note_53485).
We should explore two things:
1. Adding support to `exclude` fields (we now have `select` to include fields)
1. Not serializing things with `flatMap` that are not selected.
The response is now very broad and includes very deeply nested entities. We should explore what can be excluded.2021.2Maxym BorodenkoMaxym Borodenko