Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2023-08-19T09:06:10+02:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/679Trait data of selected accessions2023-08-19T09:06:10+02:00Matija ObrezaTrait data of selected accessionsEmbedded Genesys uses `AccessionFilter` to limit its scope, this is usually by `{ institute: { code: [ .. ] } }`, but can also be by `{ crop: [ .. ] }`.
## Listing datasets of accessions
There are two aspects to this issue: the first i...Embedded Genesys uses `AccessionFilter` to limit its scope, this is usually by `{ institute: { code: [ .. ] } }`, but can also be by `{ crop: [ .. ] }`.
## Listing datasets of accessions
There are two aspects to this issue: the first is to just find the `Set<UUID> datasets` for the accession filter. The list of published datasets where the accession is documented is stored in `accession.datasets`, but it should be further filtered to only return Amphibian datasets.
The service method needs to use caching for the one argument: `filter`.
The second is an API endpoint in `DatasetController` that returns `Page<Datasets>` using the `Set<UUID> datasets`.
## ~~Listing descriptors of accessions~~
~~Add an API endpoint in `DatasetController` that returns `Page<Descriptor>` of all datasets from `Set<UUID> datasets` for the `filter`.~~
## New endpoint to query Amphibian
We need to be able to query Amphibian to return observations for selected descriptors in selected datasets for selected accessions, while applying the filters for the selected traits.
- This is very similar to returning observations for one accession, just that we now need to return more rows
- Pagination will be an interesting challenge
- Observations from different datasets need to be merged under the same descriptor
This will allow us to find all datasets and all traits for the selected accessions.
From here, we can allow the user to:
1. (Optionally) De-select some of the datasets
1. Select the descriptors of interest
- This is a mandatory step
1. (Optionally) specify filters for the selected trait descriptors
2. Query Amphibian to return observations for selected descriptors in selected datasets for selected accessions, while applying the filters for the selected traits
3. With this we are able to data from across datasets and display it2023.5Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/678ITPGRFA GLIS configuration2023-08-03T19:40:26+02:00Matija ObrezaITPGRFA GLIS configurationUpdate `application.properties` and documentation for the change of ITPGRFA GLIS settings according to genesys-pgr/glis-client#35.
It may be sensible to use:
```
# Obsolete, please use itpgrfa.glis.url directly
itpgrfa.glis.basepath=ht...Update `application.properties` and documentation for the change of ITPGRFA GLIS settings according to genesys-pgr/glis-client#35.
It may be sensible to use:
```
# Obsolete, please use itpgrfa.glis.url directly
itpgrfa.glis.basepath=https:....
# ITPGRFA GLIS URL
itpgrfa.glis.url=${itpgrfa.glis.basepath}/glis/
```2023.5Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/677Serialized OAuth no longer readable2023-08-03T20:59:16+02:00Matija ObrezaSerialized OAuth no longer readableSince !794 the data in `oauthauthorization` table is no longer compatible with Genesys as it refers to old class names in its JSON.
```
2023-08-02 09:30:24,590 qtp1459794865-12 ERROR JpaOAuth2AuthorizationService:376 - Exception in des...Since !794 the data in `oauthauthorization` table is no longer compatible with Genesys as it refers to old class names in its JSON.
```
2023-08-02 09:30:24,590 qtp1459794865-12 ERROR JpaOAuth2AuthorizationService:376 - Exception in deserializing map
com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'org.genesys2.spring.security.service.JpaOAuth2AuthorizationService$CustomPrincipal' as a subtype of `java.lang.Object`: no such class found
at [Source: (String)"{"@class":"java.util.HashMap","org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest":{"@class":"org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest","authorizationUri":"https://api.sandbox.genesys-pgr.org/oauth/authorize","authorizationGrantType":{"value":"authorization_code"},"responseType":{"value":"code"},"clientId":"9rFBj.jctRgeoYRouR9ALfp22i@sandbox.genesys-pgr.org","redirectUri":"https://sandbox.genesys-pgr.org","scopes":["java.util.Collecti"[truncated 6904 chars]; line: 1, column: 1233] (through reference chain: java.util.HashMap["java.security.Principal"])
at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43) ~[jackson-databind-2.14.2.jar:2.14.2]
...
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3660) ~[jackson-databind-2.14.2.jar:2.14.2]
at org.genesys.spring.security.service.JpaOAuth2AuthorizationService.parseMap(JpaOAuth2AuthorizationService.java:373) ~[classes/:?]
at org.genesys.spring.security.service.JpaOAuth2AuthorizationService.lambda$toObject$0(JpaOAuth2AuthorizationService.java:190) ~[classes/:?]
at org.springframework.security.oauth2.server.authorization.OAuth2Authorization$Builder.attributes(OAuth2Authorization.java:524) ~[spring-security-oauth2-authorization-server-0.4.1.jar:0.4.1]
at org.genesys.spring.security.service.JpaOAuth2AuthorizationService.toObject(JpaOAuth2AuthorizationService.java:190) ~[classes/:?]
at java.util.Optional.map(Optional.java:265) ~[?:?]
at org.genesys.spring.security.service.JpaOAuth2AuthorizationService.findByToken(JpaOAuth2AuthorizationService.java:167) ~[classes/:?]
at org.genesys.spring.security.InvalidatedAccessTokenFilter.doFilterInternal(InvalidatedAccessTokenFilter.java:53) ~[classes/:?]
```2023.5Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/676Accession with `curationType`2023-07-28T15:03:25+02:00Matija ObrezaAccession with `curationType`Upgrade `Accession` and add a new `@Column(nullable = true, length = x) private CurationType curationType;` where
```java
public static final enum CurationType {
FULL,
PARTIAL,
ARCHIVED,
HISTORICAL,
}
```
This should be saved t...Upgrade `Accession` and add a new `@Column(nullable = true, length = x) private CurationType curationType;` where
```java
public static final enum CurationType {
FULL,
PARTIAL,
ARCHIVED,
HISTORICAL,
}
```
This should be saved to the database as string and it must be included in ES index as a `keyword`.2023.5Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/675DownloadLog: entity update2023-07-27T14:58:18+02:00Matija ObrezaDownloadLog: entity updatePlease increase `DownloadLog.clientIp` field size to accommodate IPv6 addresses.
`DownloadLog` is now `AclAwareModel` but it doesn't need to be -- we're not checking permissions on these records. Please remove and add a programmable liq...Please increase `DownloadLog.clientIp` field size to accommodate IPv6 addresses.
`DownloadLog` is now `AclAwareModel` but it doesn't need to be -- we're not checking permissions on these records. Please remove and add a programmable liquibase change to remove ACL entries for this type.2023.5Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/674OAuth client: Contact data2023-07-24T11:50:05+02:00Matija ObrezaOAuth client: Contact dataI need to store contact information of the person using the Client. Please add `contactEmail` and `notes` to `OAuthClient` model.
Since this is a change in app-blocks, please create MRs for both Genesys and GGCE server.
@vmokliak @mtis...I need to store contact information of the person using the Client. Please add `contactEmail` and `notes` to `OAuthClient` model.
Since this is a change in app-blocks, please create MRs for both Genesys and GGCE server.
@vmokliak @mtishchenko @ypykhov This will require UI updates to the OAuth client form + display.2023.4Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/673Account lockout updates2023-07-24T15:51:10+02:00Matija ObrezaAccount lockout updatesI'm seeing many logs saying that the `AccountLockoutManager` could update last login:
```
2023-07-07 23:16:10,031 qtp1459794865-130 WARN AccountLockoutManager:104 - Could not update last login for matija.obreza@croptrust.org: Batch upd...I'm seeing many logs saying that the `AccountLockoutManager` could update last login:
```
2023-07-07 23:16:10,031 qtp1459794865-130 WARN AccountLockoutManager:104 - Could not update last login for matija.obreza@croptrust.org: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update acl_sid set active=?, version=?, lastModifiedBy=?, lastModifiedDate=?, principal=?, sid=? where id=? and version=?; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update acl_sid set active=?, version=?, lastModifiedBy=?, lastModifiedDate=?, principal=?, sid=? where id=? and version=?
2023-07-07 23:16:10,411 qtp1459794865-132 WARN AccountLockoutManager:104 - Could not update last login for matija.obreza@croptrust.org: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update acl_sid set active=?, version=?, lastModifiedBy=?, lastModifiedDate=?, principal=?, sid=? where id=? and version=?; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update acl_sid set active=?, version=?, lastModifiedBy=?, lastModifiedDate=?, principal=?, sid=? where id=? and version=?
```
Please check and update the updating logic.2023.4Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/672QueryDSL accessors2023-07-07T17:07:54+02:00Matija ObrezaQueryDSL accessorsUpdate the configuration QueryDSL to use accessor methods as in grin-global/grin-global-server!556.Update the configuration QueryDSL to use accessor methods as in grin-global/grin-global-server!556.2023.4Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/671API Permissons: Access denied on Datasets2023-07-07T06:31:59+02:00Matija ObrezaAPI Permissons: Access denied on Datasets@achan is reporting that when trying to edit the exist list or dataset in **production**, they show 'Access is denied':
![image](/uploads/6cc012724b35b6c5e49b9675dc01cb98/image.png)
Similarly:
![image](/uploads/f53a8100abab96bc8beae73...@achan is reporting that when trying to edit the exist list or dataset in **production**, they show 'Access is denied':
![image](/uploads/6cc012724b35b6c5e49b9675dc01cb98/image.png)
Similarly:
![image](/uploads/f53a8100abab96bc8beae73e9d1ac368/image.png)
`AclSid` for `partner_c6798b2e-252b-446b-a432-fe604edd7481` has all permissions on these records, and @achan has all permisions on `Partner` [c6798b2e-252b-446b-a432-fe604edd7481](https://www.genesys-pgr.org/partners/c6798b2e-252b-446b-a432-fe604edd7481)Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/670STORAGE column blank from the downloaded MCPD file2023-06-12T12:53:54+02:00Christelle RabilSTORAGE column blank from the downloaded MCPD fileThe storage information is is some cases present in Genesys, but once we download MCPD file from Genesys, the column is blank:
for example this accession: https://www.genesys-pgr.org/10.18730/9MXQY
Storage in Genesys:
![Screenshot_202...The storage information is is some cases present in Genesys, but once we download MCPD file from Genesys, the column is blank:
for example this accession: https://www.genesys-pgr.org/10.18730/9MXQY
Storage in Genesys:
![Screenshot_2023-05-29_at_14-05-23_ITC1483](/uploads/564c3ee62b8e71463f06e51db0978fcb/Screenshot_2023-05-29_at_14-05-23_ITC1483.png)
Storage in the MCPD Excel file:
![image](/uploads/6f62e12bfa600d526385bed6a92fb988/image.png)Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/669Download Amphibian dataset2023-07-22T17:58:15+02:00Matija ObrezaDownload Amphibian datasetThe UI currently has an option to generate an "Excel" from the data loaded in the UI.
https://www.genesys-pgr.org/datasets/81bab6b6-6592-4a9d-968b-2eb1058f18cd/view shows
![image](/uploads/d7ecda31dc6aaade540fc9a471051cbf/image.png)
I...The UI currently has an option to generate an "Excel" from the data loaded in the UI.
https://www.genesys-pgr.org/datasets/81bab6b6-6592-4a9d-968b-2eb1058f18cd/view shows
![image](/uploads/d7ecda31dc6aaade540fc9a471051cbf/image.png)
Ideally, the Excel file is generated by the API for the selected dataset and selected filters. The download be based on the metadata Excel template we already use (and the metadata should be populated in this download). We add two new sheets: one with raw data from Amphibian (`DATA`) and one where coded traits use their text labels (`CODED`).2023.4Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/668Get Amphibian observations for accession2023-05-02T18:07:05+02:00Matija ObrezaGet Amphibian observations for accessionWe would like to display trait observations from all (Amphibian) datasets on the accession details page.
Do we have a method to filter by accession UUID in Amphibian?We would like to display trait observations from all (Amphibian) datasets on the accession details page.
Do we have a method to filter by accession UUID in Amphibian?2023.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/667Update dependencies2023-05-05T04:43:34+02:00Matija ObrezaUpdate dependenciesPlease update maven dependencies.
https://sbom.lift.sonatype.com/report/T1-118f0f57da8c6b3097cc-378c01d9c77d3-1682668347-d23f77a16846401dbf24436fc0fa2322Please update maven dependencies.
https://sbom.lift.sonatype.com/report/T1-118f0f57da8c6b3097cc-378c01d9c77d3-1682668347-d23f77a16846401dbf24436fc0fa23222023.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/666MaterialRequest: My requests2023-05-02T15:01:55+02:00Artem HrybeniukMaterialRequest: My requests## Anonymous access to request status
Request information is accessible on a fixed URL `.../requests/status/{uuid}` that will be included in emails. This page must not display personal information (no email, notes or PID data, except th...## Anonymous access to request status
Request information is accessible on a fixed URL `.../requests/status/{uuid}` that will be included in emails. This page must not display personal information (no email, notes or PID data, except the recipient's country). Actions are not available, except "Recheck PID" and "Resend validation email".
We need a special API call for this, with all personal data is redacted from the response.
## User's dashboard
Authenticated users have access to the requests they own in the Dashboard under "Requests for material". The browse page shows the table of requests of this user (same as /admin/requests). The filters are a new component that do not have the "Text search" section.
We need a special API call for this, working only on requests of the current user.Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/665AuditLog retention period2023-04-21T06:22:23+02:00Matija ObrezaAuditLog retention periodCopy grin-global/grin-global-server!531 to Genesys with default retention period of `1Y`.Copy grin-global/grin-global-server!531 to Genesys with default retention period of `1Y`.2023.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/664ATiC to ATaP migration2023-04-19T09:24:30+02:00Matija ObrezaATiC to ATaP migrationWe introduced Access Token in Cookie (ATiC) to work around limitations of using simple GET requests against the API. While this worked for the longest time, current version of Firefox does not send cookies in cross-site requests as part ...We introduced Access Token in Cookie (ATiC) to work around limitations of using simple GET requests against the API. While this worked for the longest time, current version of Firefox does not send cookies in cross-site requests as part of *Enhanced Tracking Protection* feature. Chrome sends those cookies.
This ticket is to start migrating away from ATiC approach and use **Access Token as Parameter**. The migration steps will be as follows:
1. Rename `AccessTokenInCookieFilter` to `AccessTokenAsParameterFilter`
1. Update `AccessTokenAsParameterFilter` to first check if the method is `POST`, then look for `access_token` parameter in the form body -- but not in the URL query string!
- If the the parameter name appears in query string throw `400 Bad request`
2. If found, extract the token and apply it the same way we apply it in the current filter
- If possible, remove the `access_token` parameter from the form
- Send back the cookie
3. If no such parameter is found, then proceed with the current logic, including sending back the cookie
After we upgrade the client applications, we can:
1. Stop sending back the cookie in phase 1, and
2. Stop looking for the cookie in phase 2.
## Genesys UI and Embedded Genesys
These both need updating to not rely on ATiC when making direct requests to the API (i.e. without axios and its authentication).
1. Set `withCredentials: false` for axios, that will stop sending and receiving cookies.
2. Add access token as a parameter to forms
3. Change remaining GET requests to POSTs.2023.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/663Bug: Freemarker and Instant2023-04-13T13:30:29+02:00Matija ObrezaBug: Freemarker and InstantClicking "Print version" on https://www.genesys-pgr.org/descriptorlists/06760839-1243-4858-af1a-afa4004ba42e makes an API call to https://api.genesys-pgr.org/api/v1/descriptorlist/06760839-1243-4858-af1a-afa4004ba42e/html.
This results ...Clicking "Print version" on https://www.genesys-pgr.org/descriptorlists/06760839-1243-4858-af1a-afa4004ba42e makes an API call to https://api.genesys-pgr.org/api/v1/descriptorlist/06760839-1243-4858-af1a-afa4004ba42e/html.
This results in
```json
{
"error": "freemarker.core._TemplateModelException: The string doesn't match the expected date/time/date-time format. The string to parse was: \"2022-12-27T10:14:52Z\". The expected format was: \"MMM d, y\".\nThe nested reason given follows:\nUnparseable date: \"2022-12-27T10:14:52Z\"\n\n----\nFTL stack trace (\"~\" means nesting-related):\n\t- Failed at: ${descriptorList.lastModifiedDate?date} [in template \"descriptorlist/booklet.ftl\" at line 99, column 18]\n----\nThe string doesn't match the expected date/time/date-time format. The string to parse was: \"2022-12-27T10:14:52Z\". The expected format was: \"MMM d, y\".\nThe nested reason given follows:\nUnparseable date: \"2022-12-27T10:14:52Z\"\n\n----\nFTL stack trace (\"~\" means nesting-related):\n\t- Failed at: ${descriptorList.lastModifiedDate?date} [in template \"descriptorlist/booklet.ftl\" at line 99, column 18]\n----\nUnparseable date: \"2022-12-27T10:14:52Z\"\nUnparseable date: \"2022-12-27T10:14:52Z\"",
"localizedError": "freemarker.core._TemplateModelException: The string doesn't match the expected date/time/date-time format. The string to parse was: \"2022-12-27T10:14:52Z\". The expected format was: \"MMM d, y\".\nThe nested reason given follows:\nUnparseable date: \"2022-12-27T10:14:52Z\"\n\n----\nFTL stack trace (\"~\" means nesting-related):\n\t- Failed at: ${descriptorList.lastModifiedDate?date} [in template \"descriptorlist/booklet.ftl\" at line 99, column 18]\n----"
}
```2023.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/662OAuth client save bug2023-04-04T12:30:57+02:00Maksym TishchenkoOAuth client save bugWhen going to page (or any other OAuth client) https://api.sandbox.genesys-pgr.org/admin/oauth-clients/2cj1k.jygziihobirmaq6qmes1@api.sandbox.genesys-pgr.org/edit
I'm unable to submit form to update client.
Console logs:
![image](/upl...When going to page (or any other OAuth client) https://api.sandbox.genesys-pgr.org/admin/oauth-clients/2cj1k.jygziihobirmaq6qmes1@api.sandbox.genesys-pgr.org/edit
I'm unable to submit form to update client.
Console logs:
![image](/uploads/879dda7fb760f4a624bb6f5de650603a/image.png)Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/661OAuth issues2023-05-26T11:00:51+02:00Matija ObrezaOAuth issues@ahrybeniuk @mtishchenko @oplotnykov @crabil Please use comments to add issues related to login/logout as you spot them.@ahrybeniuk @mtishchenko @oplotnykov @crabil Please use comments to add issues related to login/logout as you spot them.Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/660Amphibian: Histograms and heatmap endpoints2023-03-16T12:08:29+01:00Matija ObrezaAmphibian: Histograms and heatmap endpointsPlease expose the new endpoints for heatmaps and histograms in Amphibian for the UI.Please expose the new endpoints for heatmaps and histograms in Amphibian for the UI.2023.2Artem HrybeniukArtem Hrybeniuk