Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2018-09-19T13:57:17+02:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/330ApiInfoController v12018-09-19T13:57:17+02:00Matija ObrezaApiInfoController v1Implement a new `ApiInfoController` **v1** at `public static final String CONTROLLER_URL = ApiBaseController.APIv1_BASE + "/info";` in the **server.api.v1** package.
Copy info method from existing ApiInfoController (from Catalog)
## Ex...Implement a new `ApiInfoController` **v1** at `public static final String CONTROLLER_URL = ApiBaseController.APIv1_BASE + "/info";` in the **server.api.v1** package.
Copy info method from existing ApiInfoController (from Catalog)
## Extra data
Add additional properties to the returned object:
- `String[] cdnServers` loaded from `application.properties` (see genesys-pgr/genesys-server/issues/328)
- `long accessionCount` number of all accessions from `accessionService.count()`
- `long datasetCount` number of published datasets
- `long subsetCount` number of published subsets
- `long instituteCount` number of institutes with accessions in Genesys
- `long descriptorCount` number of published descriptors
- `long partnerCount` number of Partners
## Implementation
Please implement these "count" methods when they are missing and use this template (when possible):
```java
@Override
public long countAccessions(AccessionFilter filter) {
long total = elasticsearchService.count(Accession.class, filter);
if (total < 10000) {
// If total is below 10K, use actual count
total = accessionRepository.count(filter.buildQuery());
}
return total;
}
```
## Extras
Also expose:
- `captchaSiteKey` as `captchaSiteKey`
- `google.analytics.account` as `googleAnalyticsId`
- `itpgrfa.glis.basepath` as `glisUrl`
2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/331Accession APIv1: wrap Overview response2018-09-19T13:55:31+02:00Matija ObrezaAccession APIv1: wrap Overview responseAccession `overview` method now returns only the ES results.
Wrap the response into:
```
private static class AccessionOverview {
public String filterCode;
public AccessionFilter filter;
Map<String, TermResult> overview;
}
```Accession `overview` method now returns only the ES results.
Wrap the response into:
```
private static class AccessionOverview {
public String filterCode;
public AccessionFilter filter;
Map<String, TermResult> overview;
}
```2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/324Could not convert filters: Organization2018-09-18T21:52:52+02:00Aleksandr KirichenkoCould not convert filters: OrganizationServer returns `500 Internal server error`: **Could not convert filters**. Add support for organization to `AccessionFilter`.
**Steps:**
1. Go to the "https://sandbox.genesys-pgr.org";
2. Click on the *CGIAR* link;
3. Click on the *OVE...Server returns `500 Internal server error`: **Could not convert filters**. Add support for organization to `AccessionFilter`.
**Steps:**
1. Go to the "https://sandbox.genesys-pgr.org";
2. Click on the *CGIAR* link;
3. Click on the *OVERVIEW* button;
**Expected result:** *Statistical overview* page is opened.
**Actual result:** 500 - Internal Server Error. See attached screen:
![SCR_0077](/uploads/47ec4d7dd329353ff8b0e2f83d801c11/SCR_0077.png)
The same issue with the *BROWSE ACCESSIONS* button**
Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/333Tile server: caching2018-09-17T07:11:49+02:00Matija ObrezaTile server: cachingTile server should use longer TTL in caching headers, especially for zoom levels 0-6, as tiles on these levels scan a lot of accession records. On higher zoom levels, fewer records are required and the TTL can be lower.
Set max-age head...Tile server should use longer TTL in caching headers, especially for zoom levels 0-6, as tiles on these levels scan a lot of accession records. On higher zoom levels, fewer records are required and the TTL can be lower.
Set max-age header as follows:
1. 0-4: max-age 1day
1. 4-6: max-age 1hr
1. 7+: max-age 5min
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/332CDN issue2018-09-16T13:13:26+02:00Matija ObrezaCDN issuehttps://cdn1.sandbox.genesys-pgr.org/acn/tile/3/4/1?f=v13fc6a432ab2b4d5e8ac8064f01d1fa03
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>C4CF5C346A9FFD4A<...https://cdn1.sandbox.genesys-pgr.org/acn/tile/3/4/1?f=v13fc6a432ab2b4d5e8ac8064f01d1fa03
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>C4CF5C346A9FFD4A</RequestId>
<HostId>ZEPMCu2JmjGtRSMjmOs84x6OMmDNd+oy3QAGu46H15RIQrxX1D2ZAU37GPx6bRRuNQFf6tkyMLc=</HostId>
</Error>
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/328Accession API v1: Geo info2018-09-16T12:41:20+02:00Matija ObrezaAccession API v1: Geo infoAdd endpoint to AccessionController **v1** API `/acn/geo` as `geoInfo(String filterCode, AccessionFilter filter)` that returns
```json
{
"bounds": [
[-4.8587000, 39.8772333],
[-6.4917667, 39.0945000]
],
"accessionCount": 1...Add endpoint to AccessionController **v1** API `/acn/geo` as `geoInfo(String filterCode, AccessionFilter filter)` that returns
```json
{
"bounds": [
[-4.8587000, 39.8772333],
[-6.4917667, 39.0945000]
],
"accessionCount": 123123,
"tileServers": [ "cdn1.sandbox.genesys-pgr.org", "cdn2.sandbox.genesys-pgr.org" ]
}
```
## Bounds
Add `getBounds(AccessionFilter filter)` to **AccessionServiceImpl**.
See https://leafletjs.com/reference-1.3.4.html#latlngbounds
It should select `min(latitude), max(longitude), max(latitude), min(longitude) from accession.accessionId.geo` and return:
```js
[
[ min(latitude), max(longitude) ],
[ max(latitude), min(longitude) ]
]
```
## Accession count
Add `public long count(AccessionFilter filter)` to **AccessionServiceImpl**:
```java
@Override
public Page<Accession> list(AccessionFilter filter) {
long total = elasticsearchService.count(Accession.class, filter);
if (total < 10000) {
// If total is below 10K, use actual count
total = accessionRepository.count(filter.buildQuery());
}
return total;
}
```
## Tile servers
Tileservers info is the `cdn.servers` setting from `application.properties` and it's accessible at:
```java
@Value("${cdn.servers}")
private String[] cdnServers;
```2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/327Subsets model2018-09-15T13:40:45+02:00Matija ObrezaSubsets modelExtend `Subset` model.
## Crops
Add `crops` to Subset:
```java
/** The crops. */
@ElementCollection(fetch = FetchType.EAGER)
@CollectionTable(name = "subset_crops", joinColumns = @JoinColumn(name = "subsetId"),
// index
indexes = ...Extend `Subset` model.
## Crops
Add `crops` to Subset:
```java
/** The crops. */
@ElementCollection(fetch = FetchType.EAGER)
@CollectionTable(name = "subset_crops", joinColumns = @JoinColumn(name = "subsetId"),
// index
indexes = { @Index(columnList = "subsetId, crop") })
@Column(name = "crop", nullable = false, length = 20)
@JsonView({ JsonViews.Minimal.class })
private Set<String> crops;
```
1. Add `crop` to `SubsetFilter`
## Other properties
1. `String date`, a MCPD date (@Column(length = 8))
1. `String source`, generally a URL. (@Column(length = 200))2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/329Accession API v1: Download KML2018-09-14T17:57:49+02:00Matija ObrezaAccession API v1: Download KMLAdd API endpoint `/api/v1/acn/downloadKml` to AccessionController v1:
```java
@RequestMapping(value = "/downloadKml", produces = "application/vnd.google-earth.kml+xml", method = RequestMethod.POST)
@ResponseBody
public void kml(@Request...Add API endpoint `/api/v1/acn/downloadKml` to AccessionController v1:
```java
@RequestMapping(value = "/downloadKml", produces = "application/vnd.google-earth.kml+xml", method = RequestMethod.POST)
@ResponseBody
public void kml(@RequestParam(value="f") String filterCode, HttpServletResponse response) throws IOException {
// get AccessionFilter from filterCode
AccessionFilter filter = ...
final int countFiltered = accessionService.countAccessions(filter);
LOG.info("Attempting to download KML for {} accessions", countFiltered);
if (countFiltered > DOWNLOAD_LIMIT) {
throw new RuntimeException("Refusing to export more than " + DOWNLOAD_LIMIT + " entries");
}
response.setContentType("application/vnd.google-earth.kml+xml");
response.addHeader("Content-Disposition", String.format("attachment; filename=\"genesys-kml-" + filterCode + ".kml\""));
// Write KML to the stream.
final OutputStream outputStream = response.getOutputStream();
try {
mappingService.filteredKml(filter, outputStream);
response.flushBuffer();
} catch (EOFException e) {
LOG.warn("Download was aborted", e);
}
}
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/323Subsets: creators2018-09-14T09:45:33+02:00Matija ObrezaSubsets: creatorsExtend `Subset` model with a list of `SubsetCreator` records (similar to `Dataset#creators`):
```java
/** The creators. */
@OneToMany(cascade = { CascadeType.ALL }, mappedBy = "subset", orphanRemoval = true, fetch = FetchType.LAZY)
@Fie...Extend `Subset` model with a list of `SubsetCreator` records (similar to `Dataset#creators`):
```java
/** The creators. */
@OneToMany(cascade = { CascadeType.ALL }, mappedBy = "subset", orphanRemoval = true, fetch = FetchType.LAZY)
@Field(type = FieldType.Object)
@JsonView({ JsonViews.Public.class })
private List<SubsetCreator> creators;
```
Add API methods to `SubsetController` and `SubsetService`. You can copy most of the business logic from `DatasetCreatorServiceImpl`.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/326Tile server: Pretty dots2018-09-14T08:53:35+02:00Matija ObrezaTile server: Pretty dotsInstead of using squares as accession points use a prettier round shape.Instead of using squares as accession points use a prettier round shape.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/307Support inverse filters2018-09-13T14:45:00+02:00Matija ObrezaSupport inverse filtersThe current implementation of Filter objects and logic in Genesys allows for specifying "matching" or "positive" filters (e.g. `"crop": [ "maize", "wheat" ]`). We're aiming to support "non-matching" or "negative" filters (e.g. crop is no...The current implementation of Filter objects and logic in Genesys allows for specifying "matching" or "positive" filters (e.g. `"crop": [ "maize", "wheat" ]`). We're aiming to support "non-matching" or "negative" filters (e.g. crop is not maize or wheat).2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/316Export: Descriptor data as Excel2018-09-13T14:41:38+02:00Matija ObrezaExport: Descriptor data as ExcelExtend DescriptorController API v0 with a method that returns filtered results in Excel format using the format of the existing sheet for descriptors of the standard Catalog Excel template.
```java
@PostMapping(value = "/export", produ...Extend DescriptorController API v0 with a method that returns filtered results in Excel format using the format of the existing sheet for descriptors of the standard Catalog Excel template.
```java
@PostMapping(value = "/export", produces = { "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" } )
public void exportDescriptors(
@RequestParam(name = "f", required = false) String filterCode,
@RequestBody(required = false) DescriptorFilter filter, HttpServletResponse response) throws IOException {
if (filterCode != null) {
filter = shortFilterService.filterByCode(filterCode, DescriptorFilter.class);
} else {
filterCode = shortFilterService.getCode(filter);
}
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
response.addHeader("Content-Disposition", String.format("attachment; filename=\"genesys-descriptors-" + filterCode + ".xlsx\""));
response.flushBuffer();
// Write XLSX to the stream.
final OutputStream outputStream = response.getOutputStream();
try {
descriptorService.exportDescriptors(filter, outputStream);
response.flushBuffer();
} catch (EOFException e) {
LOG.warn("Download was aborted", e);
}
}
```
The export should include the `UUID` and `version` as the first two columns of the updated Excel sheet.2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/318Include inherited permissions2018-09-13T14:41:35+02:00Matija ObrezaInclude inherited permissionsIn Permissions API v1, add a static private class `_AclObjectIdentity` extending AclObjectIdentity with
```java
public List<AclEntry> inherited;
```
In method `lazyLoadForJson`, return `_AclObjectIdentity` and fill the `inherited` lis...In Permissions API v1, add a static private class `_AclObjectIdentity` extending AclObjectIdentity with
```java
public List<AclEntry> inherited;
```
In method `lazyLoadForJson`, return `_AclObjectIdentity` and fill the `inherited` list recursively with ACL entries for all `parentObjects`:
```java
private void inhertied(AclObjectIdentity objectIdentity, List<AclEntry> aclEntries, Set<AclObjectIdentity> handled) {
if (objectIdentity == null || handled.contains(objectIdentity)) {
return aclEntries;
}
aclEntries.addAll(objectIdentity.getAclEntries());
handled.add(objectIdentity);
if (objectIdentity.getParentObject() != null) {
inherited(objectIdentity.getParentObject(), aclEntries, handled);
}
}
```
Also add `objectIdentity.inherited.forEach(entry -> entry.getAclSid().getId());`.2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/317Subsets API and ACL2018-09-13T11:31:06+02:00Matija ObrezaSubsets API and ACLSubsets generally belong to an institute (`FaoInstitute`). ~~Extend `Subset` model by overriding method `aclParentObject()` and return `this.institute` as parent object.~~ This will grant manage permissions to all users that have permiss...Subsets generally belong to an institute (`FaoInstitute`). ~~Extend `Subset` model by overriding method `aclParentObject()` and return `this.institute` as parent object.~~ This will grant manage permissions to all users that have permission on the institute.
Update `..ServiceImpl` to test for ADMINISTRATOR role or `hasPermission(#subset, 'WRITE')` using `@PreAuthorize`.
## API
Add a method `listMySubsets` that returns user's subsets regardless of state. Base this method on existing `listMy****` methods from the Catalog.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/321SSL errors2018-09-12T14:34:06+02:00Matija ObrezaSSL errors> I am trying to load data into Genesys Production Using Uploader tool. The data I am loading is the same as what I have finished loading in Sandbox.
>
> I started to load data again this morning and getting “org.genesys2.client.oauth.G...> I am trying to load data into Genesys Production Using Uploader tool. The data I am loading is the same as what I have finished loading in Sandbox.
>
> I started to load data again this morning and getting “org.genesys2.client.oauth.GenesysApiException: Handshake message sequence violation, 11” errors. I have attached some error messages that I am getting from the uploader tool.
Production is running an older version of JRE and it seems that the new JRE you use for the uploader triggers https://bugs.openjdk.java.net/browse/JDK-8167680 on Production server.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/320SitemapXMLController returns string2018-09-12T11:30:03+02:00Matija ObrezaSitemapXMLController returns stringThe `SitemapXMLController` controller returns a `String`:
```
$ curl https://www.genesys-pgr.org/sitemap.xml
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<sitemap>
...The `SitemapXMLController` controller returns a `String`:
```
$ curl https://www.genesys-pgr.org/sitemap.xml
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-content.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-geo.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-wiews.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-crop.xml</loc>
</sitemap>
</sitemapindex>"
```
Notice the starting and ending `"` which is incorrect and the response content is not valid XML.
Options:
1. Update `@RequestMapping` with `produces = ...`, perhaps that already fixes the problem
1. Don't use `StringBuffer` and write directly to the response output stream (and flush it when done)
1. Any other idea?2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/314Accesssion API v1: details2018-09-11T14:21:39+02:00Matija ObrezaAccesssion API v1: detailsAccession API v1 `/{uuid}` or `/{doi}` return accession information.
Add new endpoints `/details/{uuid}` and `/details/{doi}` that return the following JSON:
```json
{
"details": accession,
"pdci": acession.accessionId.pdci,
"datase...Accession API v1 `/{uuid}` or `/{doi}` return accession information.
Add new endpoints `/details/{uuid}` and `/details/{doi}` that return the following JSON:
```json
{
"details": accession,
"pdci": acession.accessionId.pdci,
"datasets": ...,
"subsets": ...
}
```
- `datasets`: includes List of basic information of Datasets where accession is described (use JsonView.Minimal)
- `subsets`: includes List of basic infomration of Subsets where accession is included (use JsonView.Minimal)
Some service methods may not yet be available (check with @mborodenko).2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/322Admin bulk ops2018-09-11T13:56:14+02:00Matija ObrezaAdmin bulk opsThe PDCI calculation was updated in 82c8bf507961f2e5a3f9695829e4894c9f71d462 yet we don't have an admin method to recalculate PDCI for the entire database.
Implement a system where an action method is provided with `Accession` as argume...The PDCI calculation was updated in 82c8bf507961f2e5a3f9695829e4894c9f71d462 yet we don't have an admin method to recalculate PDCI for the entire database.
Implement a system where an action method is provided with `Accession` as argument that does "something". This method is invoked by a thread pool worker that receives a batch of accession records loaded from the database based on `AccessionFilter`.
## Use case
Update PDCI for ICARDA: provide appropriate accession filter that loads records in batches of 1000 and recalculates and updates the PDCI for each accession.
We need to be in a separate transaction so that entity manager is appropriately cleared after each batch.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/289Keep average PDCI in FaoInstitute2018-09-11T13:23:34+02:00Matija ObrezaKeep average PDCI in FaoInstituteMaintain average PDCI score in `FaoInsitute` entity.
Add `Double` fields `pdciMin`, `pdciMax`, `pdciAvg` to `FaoInstitute` (update liquibase).
These values should be updated in the `org.genesys2.server.service.worker.AccessionCounter` ...Maintain average PDCI score in `FaoInsitute` entity.
Add `Double` fields `pdciMin`, `pdciMax`, `pdciAvg` to `FaoInstitute` (update liquibase).
These values should be updated in the `org.genesys2.server.service.worker.AccessionCounter` method **process**:
```java
FaoInstitute institute = instituteService.findInstitute(forProcessing.getObj());
if (institute != null) {
LOG.info("Updating count for {}", institute.getCode());
genesysService.updateAccessionCount(institute);
// Add
genesysService.updatePDCI(institute);
}
```
Implement method `genesysService.updatePDCI(FaoInstitute)` based on `genesysService.statisticsPDCI(FaoInstitute)`.
Remove method `genesysService.statisticsPDCI(FaoInstitute)` and refactor code to use the numbers stored in `FaoInstitute` entity. This means fix the code in `genesysService.statisticsPDCI(Organization)` -- just use numbers from the entity, don't query the database. You will need to use `institute.accessionCount` instead of `(Long) overall[3]` (which contains the count).
Update `mvc.AccessionController` and remove the call to statisticsService, use values from `institute` in the JSP.
2.5Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/319Validation tokens expiry2018-09-07T12:22:50+02:00Matija ObrezaValidation tokens expiryUsers happen to return to or find Genesys emails in Junk after a week or so. Extend token validity.Users happen to return to or find Genesys emails in Junk after a week or so. Extend token validity.2.4Matija ObrezaMatija Obreza