Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2018-09-26T19:47:21+02:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/341FiltersĀ conversion2018-09-26T19:47:21+02:00Matija ObrezaFiltersĀ conversionFiltering by `DEU` as "Country of holding institute" calls: https://sandbox.genesys-pgr.org/explore/listFilterSuggestions?filter=%7B%22institute.country.iso3%22%3A%5B%22DEU%22%5D%7D
The conversion of the old filter (`AppliedFilters`) fa...Filtering by `DEU` as "Country of holding institute" calls: https://sandbox.genesys-pgr.org/explore/listFilterSuggestions?filter=%7B%22institute.country.iso3%22%3A%5B%22DEU%22%5D%7D
The conversion of the old filter (`AppliedFilters`) fails with **Unknown base institute.country**:
```
2018-09-25T08:12:20.605602345Z Caused by: java.lang.RuntimeException: Unknown base institute.country
2018-09-25T08:12:20.605609064Z at org.genesys2.server.service.filter.AppliedFiltersConverter.convert(AppliedFiltersConverter.java:168)
2018-09-25T08:12:20.605616054Z at org.genesys2.server.service.filter.AccessionFilter.convert(AccessionFilter.java:166)
2018-09-25T08:12:20.605623037Z ... 116 more
```
The same happens for original filters names `regionHoldInst` and `regionOrigin`.
## Updates
Please extend `CountryFilter` with support for `Set<Long> regionId` and update `AppliedFiltersConverter.convert` to convert:
* `regionHoldInst` to `holder.country.regionId`
* `regionOrigin` to `origin.regionId`
Please add `StringFilter alias` to `AccessionFilter`. It should check if `accession.accessionId.aliases.any().name` matches the filter.2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/340Organization: view map2018-09-26T20:29:17+02:00Matija ObrezaOrganization: view mapThe Organization MVC controller at `/org/{slug}/map` does not show a map of institutes.
## Alternatively
... it should redirect to map of accessions `/explore/map?filter=.....`
1. https://sandbox.genesys-pgr.org/org/COGENT
1. Click "...The Organization MVC controller at `/org/{slug}/map` does not show a map of institutes.
## Alternatively
... it should redirect to map of accessions `/explore/map?filter=.....`
1. https://sandbox.genesys-pgr.org/org/COGENT
1. Click "View map"
1. **Expected:** the user is redirected to the standard accession map2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/339FTP2018-09-24T08:48:29+02:00Matija ObrezaFTPUpgrade to file-repository-ftp:1.1-SNAPSHOT that introduces repository folders and ACL.Upgrade to file-repository-ftp:1.1-SNAPSHOT that introduces repository folders and ACL.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/338AsciiDocController v12018-09-29T18:04:48+02:00Matija ObrezaAsciiDocController v1Copy `org.genesys2.server.mvc.AsciiDocController` to **server.api.v1** with **`APIv1_BASE + /cms/d`** base path.
- Add `produces=` content typesCopy `org.genesys2.server.mvc.AsciiDocController` to **server.api.v1** with **`APIv1_BASE + /cms/d`** base path.
- Add `produces=` content types2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/337CMS Controller2018-09-25T16:47:43+02:00Matija ObrezaCMS ControllerImplement a new `CMSController` with base **`APIv1_BASE + /cms`** in **server.api.v1** that exposes select methods (from `ContentService`):
```java
List<ActivityPost> lastNews();
Page<ActivityPost> allNews(int page);
Article getArticl...Implement a new `CMSController` with base **`APIv1_BASE + /cms`** in **server.api.v1** that exposes select methods (from `ContentService`):
```java
List<ActivityPost> lastNews();
Page<ActivityPost> allNews(int page);
Article getArticle(Class<?> clazz, Long id, String slug, Locale locale, boolean useDefault);
Article getArticleBySlugAndLang(String slug, String lang);
Article getGlobalArticle(String slug, Locale locale, boolean useDefault);
Page<Article> listArticles(Pageable pageable);
Page<Article> listArticlesByLang(String lang, Pageable pageable);
ActivityPost createActivityPost(String title, String body);
Article updateArticle(Class<?> clazz, Long id, String slug, String title, String body, String summary, Locale locale) throws CRMException;
Article updateArticle(long id, String slug, String title, String body, String summary);
Article createGlobalArticle(String slug, Locale locale, String title, String body, String summary, boolean isTemplate) throws CRMException;
Article updateGlobalArticle(String slug, Locale locale, String title, String body, String summary) throws CRMException;
ActivityPost getActivityPost(long id);
ActivityPost updateActivityPost(long id, String title, String body);
void deleteActivityPost(long id);
Locale getDefaultLocale();
String processTemplate(String body, Map<String, Object> root);
// Menus
Menu getMenu(String key);
Menu updateMenu(String key, List<MenuItem> menuItems);
MenuItem ensureMenuItem(String menuKey, String url, String text);
```
- Please use sensible `@RequestMapping(value=` names.
## Extras
Add `ArticleFilter` and use the filter for `listArticles(ArticleFilter filter, Pageable page)`.2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/336User registration API2018-09-24T10:45:28+02:00Matija ObrezaUser registration APIAdd `UserRegistrationController` to **server.api.v1** with base at `API_BASE = ApiBaseController.APIv1_BASE + "/user";`
Copy most of the method `addUser` from `org.genesys2.server.mvc.UserRegistrationController` and expose it as API POS...Add `UserRegistrationController` to **server.api.v1** with base at `API_BASE = ApiBaseController.APIv1_BASE + "/user";`
Copy most of the method `addUser` from `org.genesys2.server.mvc.UserRegistrationController` and expose it as API POST to **/register**.
- Check captcha response!
- Add method `validateEMail(String tokenUuid, String key)` that calls `EmailValidationService` as POST to **/validate**.
2.4Maxym BorodenkoMaxym Borodenkohttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/335Requests API v1: creating requests for material2018-09-28T22:32:18+02:00Matija ObrezaRequests API v1: creating requests for materialExtend the `RequestsController` V1 with methods for creating new requests for material.
Expose methods available in `RequestService`:
- `initiateRequest(RequestInfo requestInfo, Set<UUID> accessionUuids)` as a POST to **/initiate** (t...Extend the `RequestsController` V1 with methods for creating new requests for material.
Expose methods available in `RequestService`:
- `initiateRequest(RequestInfo requestInfo, Set<UUID> accessionUuids)` as a POST to **/initiate** (this method doesn't exist, please add it).
- `validateClientRequest(String tokenUuid, String key)` as a POST to **/validate**2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/333Tile server: caching2018-09-17T07:11:49+02:00Matija ObrezaTile server: cachingTile server should use longer TTL in caching headers, especially for zoom levels 0-6, as tiles on these levels scan a lot of accession records. On higher zoom levels, fewer records are required and the TTL can be lower.
Set max-age head...Tile server should use longer TTL in caching headers, especially for zoom levels 0-6, as tiles on these levels scan a lot of accession records. On higher zoom levels, fewer records are required and the TTL can be lower.
Set max-age header as follows:
1. 0-4: max-age 1day
1. 4-6: max-age 1hr
1. 7+: max-age 5min
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/332CDN issue2018-09-16T13:13:26+02:00Matija ObrezaCDN issuehttps://cdn1.sandbox.genesys-pgr.org/acn/tile/3/4/1?f=v13fc6a432ab2b4d5e8ac8064f01d1fa03
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>C4CF5C346A9FFD4A<...https://cdn1.sandbox.genesys-pgr.org/acn/tile/3/4/1?f=v13fc6a432ab2b4d5e8ac8064f01d1fa03
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>C4CF5C346A9FFD4A</RequestId>
<HostId>ZEPMCu2JmjGtRSMjmOs84x6OMmDNd+oy3QAGu46H15RIQrxX1D2ZAU37GPx6bRRuNQFf6tkyMLc=</HostId>
</Error>
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/331Accession APIv1: wrap Overview response2018-09-19T13:55:31+02:00Matija ObrezaAccession APIv1: wrap Overview responseAccession `overview` method now returns only the ES results.
Wrap the response into:
```
private static class AccessionOverview {
public String filterCode;
public AccessionFilter filter;
Map<String, TermResult> overview;
}
```Accession `overview` method now returns only the ES results.
Wrap the response into:
```
private static class AccessionOverview {
public String filterCode;
public AccessionFilter filter;
Map<String, TermResult> overview;
}
```2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/330ApiInfoController v12018-09-19T13:57:17+02:00Matija ObrezaApiInfoController v1Implement a new `ApiInfoController` **v1** at `public static final String CONTROLLER_URL = ApiBaseController.APIv1_BASE + "/info";` in the **server.api.v1** package.
Copy info method from existing ApiInfoController (from Catalog)
## Ex...Implement a new `ApiInfoController` **v1** at `public static final String CONTROLLER_URL = ApiBaseController.APIv1_BASE + "/info";` in the **server.api.v1** package.
Copy info method from existing ApiInfoController (from Catalog)
## Extra data
Add additional properties to the returned object:
- `String[] cdnServers` loaded from `application.properties` (see genesys-pgr/genesys-server/issues/328)
- `long accessionCount` number of all accessions from `accessionService.count()`
- `long datasetCount` number of published datasets
- `long subsetCount` number of published subsets
- `long instituteCount` number of institutes with accessions in Genesys
- `long descriptorCount` number of published descriptors
- `long partnerCount` number of Partners
## Implementation
Please implement these "count" methods when they are missing and use this template (when possible):
```java
@Override
public long countAccessions(AccessionFilter filter) {
long total = elasticsearchService.count(Accession.class, filter);
if (total < 10000) {
// If total is below 10K, use actual count
total = accessionRepository.count(filter.buildQuery());
}
return total;
}
```
## Extras
Also expose:
- `captchaSiteKey` as `captchaSiteKey`
- `google.analytics.account` as `googleAnalyticsId`
- `itpgrfa.glis.basepath` as `glisUrl`
2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/329Accession API v1: Download KML2018-09-14T17:57:49+02:00Matija ObrezaAccession API v1: Download KMLAdd API endpoint `/api/v1/acn/downloadKml` to AccessionController v1:
```java
@RequestMapping(value = "/downloadKml", produces = "application/vnd.google-earth.kml+xml", method = RequestMethod.POST)
@ResponseBody
public void kml(@Request...Add API endpoint `/api/v1/acn/downloadKml` to AccessionController v1:
```java
@RequestMapping(value = "/downloadKml", produces = "application/vnd.google-earth.kml+xml", method = RequestMethod.POST)
@ResponseBody
public void kml(@RequestParam(value="f") String filterCode, HttpServletResponse response) throws IOException {
// get AccessionFilter from filterCode
AccessionFilter filter = ...
final int countFiltered = accessionService.countAccessions(filter);
LOG.info("Attempting to download KML for {} accessions", countFiltered);
if (countFiltered > DOWNLOAD_LIMIT) {
throw new RuntimeException("Refusing to export more than " + DOWNLOAD_LIMIT + " entries");
}
response.setContentType("application/vnd.google-earth.kml+xml");
response.addHeader("Content-Disposition", String.format("attachment; filename=\"genesys-kml-" + filterCode + ".kml\""));
// Write KML to the stream.
final OutputStream outputStream = response.getOutputStream();
try {
mappingService.filteredKml(filter, outputStream);
response.flushBuffer();
} catch (EOFException e) {
LOG.warn("Download was aborted", e);
}
}
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/328Accession API v1: Geo info2018-09-16T12:41:20+02:00Matija ObrezaAccession API v1: Geo infoAdd endpoint to AccessionController **v1** API `/acn/geo` as `geoInfo(String filterCode, AccessionFilter filter)` that returns
```json
{
"bounds": [
[-4.8587000, 39.8772333],
[-6.4917667, 39.0945000]
],
"accessionCount": 1...Add endpoint to AccessionController **v1** API `/acn/geo` as `geoInfo(String filterCode, AccessionFilter filter)` that returns
```json
{
"bounds": [
[-4.8587000, 39.8772333],
[-6.4917667, 39.0945000]
],
"accessionCount": 123123,
"tileServers": [ "cdn1.sandbox.genesys-pgr.org", "cdn2.sandbox.genesys-pgr.org" ]
}
```
## Bounds
Add `getBounds(AccessionFilter filter)` to **AccessionServiceImpl**.
See https://leafletjs.com/reference-1.3.4.html#latlngbounds
It should select `min(latitude), max(longitude), max(latitude), min(longitude) from accession.accessionId.geo` and return:
```js
[
[ min(latitude), max(longitude) ],
[ max(latitude), min(longitude) ]
]
```
## Accession count
Add `public long count(AccessionFilter filter)` to **AccessionServiceImpl**:
```java
@Override
public Page<Accession> list(AccessionFilter filter) {
long total = elasticsearchService.count(Accession.class, filter);
if (total < 10000) {
// If total is below 10K, use actual count
total = accessionRepository.count(filter.buildQuery());
}
return total;
}
```
## Tile servers
Tileservers info is the `cdn.servers` setting from `application.properties` and it's accessible at:
```java
@Value("${cdn.servers}")
private String[] cdnServers;
```2.4Alexander PrendetskiyAlexander Prendetskiyhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/327Subsets model2018-09-15T13:40:45+02:00Matija ObrezaSubsets modelExtend `Subset` model.
## Crops
Add `crops` to Subset:
```java
/** The crops. */
@ElementCollection(fetch = FetchType.EAGER)
@CollectionTable(name = "subset_crops", joinColumns = @JoinColumn(name = "subsetId"),
// index
indexes = ...Extend `Subset` model.
## Crops
Add `crops` to Subset:
```java
/** The crops. */
@ElementCollection(fetch = FetchType.EAGER)
@CollectionTable(name = "subset_crops", joinColumns = @JoinColumn(name = "subsetId"),
// index
indexes = { @Index(columnList = "subsetId, crop") })
@Column(name = "crop", nullable = false, length = 20)
@JsonView({ JsonViews.Minimal.class })
private Set<String> crops;
```
1. Add `crop` to `SubsetFilter`
## Other properties
1. `String date`, a MCPD date (@Column(length = 8))
1. `String source`, generally a URL. (@Column(length = 200))2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/326Tile server: Pretty dots2018-09-14T08:53:35+02:00Matija ObrezaTile server: Pretty dotsInstead of using squares as accession points use a prettier round shape.Instead of using squares as accession points use a prettier round shape.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/323Subsets: creators2018-09-14T09:45:33+02:00Matija ObrezaSubsets: creatorsExtend `Subset` model with a list of `SubsetCreator` records (similar to `Dataset#creators`):
```java
/** The creators. */
@OneToMany(cascade = { CascadeType.ALL }, mappedBy = "subset", orphanRemoval = true, fetch = FetchType.LAZY)
@Fie...Extend `Subset` model with a list of `SubsetCreator` records (similar to `Dataset#creators`):
```java
/** The creators. */
@OneToMany(cascade = { CascadeType.ALL }, mappedBy = "subset", orphanRemoval = true, fetch = FetchType.LAZY)
@Field(type = FieldType.Object)
@JsonView({ JsonViews.Public.class })
private List<SubsetCreator> creators;
```
Add API methods to `SubsetController` and `SubsetService`. You can copy most of the business logic from `DatasetCreatorServiceImpl`.2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/322Admin bulk ops2018-09-11T13:56:14+02:00Matija ObrezaAdmin bulk opsThe PDCI calculation was updated in 82c8bf507961f2e5a3f9695829e4894c9f71d462 yet we don't have an admin method to recalculate PDCI for the entire database.
Implement a system where an action method is provided with `Accession` as argume...The PDCI calculation was updated in 82c8bf507961f2e5a3f9695829e4894c9f71d462 yet we don't have an admin method to recalculate PDCI for the entire database.
Implement a system where an action method is provided with `Accession` as argument that does "something". This method is invoked by a thread pool worker that receives a batch of accession records loaded from the database based on `AccessionFilter`.
## Use case
Update PDCI for ICARDA: provide appropriate accession filter that loads records in batches of 1000 and recalculates and updates the PDCI for each accession.
We need to be in a separate transaction so that entity manager is appropriately cleared after each batch.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/320SitemapXMLController returns string2018-09-12T11:30:03+02:00Matija ObrezaSitemapXMLController returns stringThe `SitemapXMLController` controller returns a `String`:
```
$ curl https://www.genesys-pgr.org/sitemap.xml
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<sitemap>
...The `SitemapXMLController` controller returns a `String`:
```
$ curl https://www.genesys-pgr.org/sitemap.xml
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-content.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-geo.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-wiews.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.genesys-pgr.org/sitemap-crop.xml</loc>
</sitemap>
</sitemapindex>"
```
Notice the starting and ending `"` which is incorrect and the response content is not valid XML.
Options:
1. Update `@RequestMapping` with `produces = ...`, perhaps that already fixes the problem
1. Don't use `StringBuffer` and write directly to the response output stream (and flush it when done)
1. Any other idea?2.4Viacheslav PavlovViacheslav Pavlovhttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/319Validation tokens expiry2018-09-07T12:22:50+02:00Matija ObrezaValidation tokens expiryUsers happen to return to or find Genesys emails in Junk after a week or so. Extend token validity.Users happen to return to or find Genesys emails in Junk after a week or so. Extend token validity.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/318Include inherited permissions2018-09-13T14:41:35+02:00Matija ObrezaInclude inherited permissionsIn Permissions API v1, add a static private class `_AclObjectIdentity` extending AclObjectIdentity with
```java
public List<AclEntry> inherited;
```
In method `lazyLoadForJson`, return `_AclObjectIdentity` and fill the `inherited` lis...In Permissions API v1, add a static private class `_AclObjectIdentity` extending AclObjectIdentity with
```java
public List<AclEntry> inherited;
```
In method `lazyLoadForJson`, return `_AclObjectIdentity` and fill the `inherited` list recursively with ACL entries for all `parentObjects`:
```java
private void inhertied(AclObjectIdentity objectIdentity, List<AclEntry> aclEntries, Set<AclObjectIdentity> handled) {
if (objectIdentity == null || handled.contains(objectIdentity)) {
return aclEntries;
}
aclEntries.addAll(objectIdentity.getAclEntries());
handled.add(objectIdentity);
if (objectIdentity.getParentObject() != null) {
inherited(objectIdentity.getParentObject(), aclEntries, handled);
}
}
```
Also add `objectIdentity.inherited.forEach(entry -> entry.getAclSid().getId());`.2.4Alexander PrendetskiyAlexander Prendetskiy