Genesys Backend issueshttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues2018-12-19T00:59:35+01:00https://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/382KPI group by Parameter properties2018-12-19T00:59:35+01:00Matija ObrezaKPI group by Parameter propertiesI wish to add a KPI Execution to count the "Number of accessions last updated by year" (and then by institute, by crop).
The current implementation requires me to define a `Dimension` that will provide the `Set<Long>` of the **years** t...I wish to add a KPI Execution to count the "Number of accessions last updated by year" (and then by institute, by crop).
The current implementation requires me to define a `Dimension` that will provide the `Set<Long>` of the **years** that can then be passed to ExecutionDimension.
1. That requires us to either maintain a `NumericListDimension` and add all year options manually.
1. Generates a number of observations for years we *know* no updates have happened.
## Idea
Allow for **GROUP BY** clauses in Execution definition:
- Add group by `lastModifiedDate` modifier `YEAR` alias `lastModifiedYear`
Generates: `select ..., year(_base.lastModifiedDate) lastModifiedYear ... group by lastModifiedYear where ...`
- Add group by `lastModifiedDate` modifier `MONTH` alias `lastModifiedMonth`
Generates: `select ..., year(_base.lastModifiedDate) lastModifiedYear, month(_base.lastModifiedDate) lastModifiedMonth ... group by lastModifiedYear, lastModifiedDate where ...`
The resulting Observation will therefore go from being a single row with the `value` (and `stddev` for AVERAGE) to multiple rows for existing combinations of `lastModifiedYear` and `lastModifiedDate`.
The method `private Observation getSingleObservation(Query query, Object... params) {}` should therefore return `List<Observation>` and the `Observation` needs to store the group-by's somewhere.
Could use the "DimensionKey"?2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/381KPI counting accessions2018-12-19T00:59:35+01:00Matija ObrezaKPI counting accessionsI'm trying to define a KPI Execution to count the "Number of accessions documented in published Datasets".
- Parameter: `Dataset` with `state = 'PUBLISHED'`
- No dimensions.
Execution fails with *illegal attempt to dereference collecti...I'm trying to define a KPI Execution to count the "Number of accessions documented in published Datasets".
- Parameter: `Dataset` with `state = 'PUBLISHED'`
- No dimensions.
Execution fails with *illegal attempt to dereference collection [dataset0_.id.accessionRefs] with element property reference [accession]*.
This is because `Dataset#accessionRefs` is a collection and needs to be joined (as documented https://stackoverflow.com/questions/24750754/org-hibernate-queryexception-illegal-attempt-to-dereference-collection/24751098#24751098).
## Fix
We should introduce something similar to the `link` property in `ExecutionDimension` to the `Execution` to declare the join.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/387KPI group by with null2018-12-19T18:44:29+01:00Matija ObrezaKPI group by with nullGroup by in our KPI implementation does not capture `null` values. The run below does not contain the count where `crop is null`:
![image](/uploads/3c5ce8ba09518c0d0e89140a5ab72eb6/image.png)
Entities must be left joined in JPA for `nu...Group by in our KPI implementation does not capture `null` values. The run below does not contain the count where `crop is null`:
![image](/uploads/3c5ce8ba09518c0d0e89140a5ab72eb6/image.png)
Entities must be left joined in JPA for `null` to be considered as a "group". JPA defaults to inner join.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/389Export trait data in Catalog dataset template2018-12-19T22:38:39+01:00Matija ObrezaExport trait data in Catalog dataset templateGenesys has a searchable trait database of C&E data. The list of "datasets" is available at https://www.genesys-pgr.org/data/.
It is possible to download the "dataset" as a DwC-A CSV+ZIP file (DarwinCore Archive) that contains all recor...Genesys has a searchable trait database of C&E data. The list of "datasets" is available at https://www.genesys-pgr.org/data/.
It is possible to download the "dataset" as a DwC-A CSV+ZIP file (DarwinCore Archive) that contains all records relating to this dataset.
Add a controller endpoint that will generate the Excel version using the current Catalog Dataset template based on trait data in the database.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/390Multiple matching descriptors2018-12-20T17:11:59+01:00Matija ObrezaMultiple matching descriptors`Descriptor searchMatchingDescriptor()` in descriptor service uses `findOne()`. When multiple matches are found this causes an exception to be thrown.
Update to `List<> searchMatchingDescriptor()` and return all matches. Update UI accor...`Descriptor searchMatchingDescriptor()` in descriptor service uses `findOne()`. When multiple matches are found this causes an exception to be thrown.
Update to `List<> searchMatchingDescriptor()` and return all matches. Update UI accordingly.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/392KPI alias2018-12-23T22:36:42+01:00Matija ObrezaKPI aliasI'm not able to set `accession.historic` as an alias of ExecutionGroup; the query fails.I'm not able to set `accession.historic` as an alias of ExecutionGroup; the query fails.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/393Download Genesys1 descriptor lists by crop2018-12-24T10:54:49+01:00Matija ObrezaDownload Genesys1 descriptor lists by cropGenesys1 came with a trait database and descriptor definitions. We will be migrating the data from traits database to Catalog entries and we can already export the data in Catalog-compliant Excel file.
To ensure that trait descriptors f...Genesys1 came with a trait database and descriptor definitions. We will be migrating the data from traits database to Catalog entries and we can already export the data in Catalog-compliant Excel file.
To ensure that trait descriptors from Genesys **1** are properly migrated, add support to download descriptor definitions **by crop** in Catalog format for import to the Catalog.
https://www.genesys-pgr.org/c/wheat/descriptors should allow for downloading the data for importing as **new descriptor list** to Genesys. When registering and uploading the original datasets, we will be able to reuse descriptor definitions across all imports.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/398Fix: KPI List of available metrics2019-01-07T18:24:00+01:00Matija ObrezaFix: KPI List of available metricsImplementation of `KPIServiceImpl#listExecutions` returns Executions where user has direct READ permission assigned.
Change this to `List<Execution>` with `@PostFilter("hasPermission(returnObject, ...`
# List of available metrics
**GE...Implementation of `KPIServiceImpl#listExecutions` returns Executions where user has direct READ permission assigned.
Change this to `List<Execution>` with `@PostFilter("hasPermission(returnObject, ...`
# List of available metrics
**GET** _/api/v1/kpi/observations_ returns the list of **names** of available metrics:
```json
[ "accession.updates","accessions.count","accessions.doi","dataset.accessions","institute.pdci"... ]
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/402Filter by WorldClim data2019-01-17T16:44:30+01:00Matija ObrezaFilter by WorldClim dataIntroduce a new `WorldClim` model with type `Double` properties: ***tileIndex* as primary key** `+ private Double ` *alt tmin1 tmax1 tmean1 prec1 tmin2 tmax2 tmean2 prec2 tmin3 tmax3 tmean3 prec3 tmin4 tmax4 tmean4 prec4 tmin5 tmax5 tmea...Introduce a new `WorldClim` model with type `Double` properties: ***tileIndex* as primary key** `+ private Double ` *alt tmin1 tmax1 tmean1 prec1 tmin2 tmax2 tmean2 prec2 tmin3 tmax3 tmean3 prec3 tmin4 tmax4 tmean4 prec4 tmin5 tmax5 tmean5 prec5 tmin6 tmax6 tmean6 prec6 tmin7 tmax7 tmean7 prec7 tmin8 tmax8 tmean8 prec8 tmin9 tmax9 tmean9 prec9 tmin10 tmax10 tmean10 prec10 tmin11 tmax11 tmean11 prec11 tmin12 tmax12 tmean12 prec12 bio1 bio2 bio3 bio4 bio5 bio6 bio7 bio8 bio9 bio10 bio11 bio12 bio13 bio14 bio15*.
Variable names are explained at http://worldclim.org/bioclim and http://worldclim.org/formats1 -- we need this for the UI.
All properties need their own database index.
Introduce a `WorldclimService` and `*Repository` as needed.
Update the `WorldclimUpdater` to update the new entity, not the DS2 dataset.
~~http://localhost:8080/admin/ds2/ -- update WC, then migrate current `DS2` **worldclim.org** dataset to the new model -- MVC admin action only.~~
Add `private Worldclim climate` to `AccessionGeo`, it should use the `accessionGeo#tileIndex` to link to the corresponding `WorldClim` entity. This will include the climate data in the Elasticsearch index and enable "Overviews" to work.
Implement a new filter `WorldclimFilter` with `NumberFilter` for all variables in `WorldClim` model. Add `public WorldclimFilter climate` to `AccessionFilter` to allow filtering by climatic properties.
Update `WorldclimUpdater` to use WorldClim 2.0 at http://worldclim.org/version2
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/405Transaction boundaries2019-01-22T13:35:13+01:00Matija ObrezaTransaction boundariesWe're doing something wrong either in the AuditLog or in the AccessionProcessor. It may have something to do with `REQUIRES_NEW` which will conflict with inserting audit logs (at level 1):
```java
@Override
public void beforeTransacti...We're doing something wrong either in the AuditLog or in the AccessionProcessor. It may have something to do with `REQUIRES_NEW` which will conflict with inserting audit logs (at level 1):
```java
@Override
public void beforeTransactionCompletion(final Transaction tx) {
if (level == 1 && auditLogs.get().size() > 0) {
LOG.trace("We have {} auditlogs", auditLogs.get().size());
auditLogs.get().stream().forEach(auditLog -> {
LOG.debug("Audit log to save: {}", auditLog);
});
this.auditTrailService.addAuditLogs(auditLogs.get());
auditLogs.get().clear();
}
super.beforeTransactionCompletion(tx);
}
```
I guess we need to add level-specific `auditLogs` lists.
```
13:24:19,215 genesys-background-2 DEBUG o.g.s.m.a.AdminController:366 - Assigned 654 tileIndexes
13:24:20,302 qtp147403080-32 INFO o.g.s.s.w.AccessionProcessor:198 - Processing Accessions took 4088907ms
13:24:20,335 qtp147403080-32 ERROR o.h.i.SessionImpl:521 - HHH000088: Exception in interceptor beforeTransactionCompletion()
org.springframework.orm.jpa.JpaSystemException: could not execute statement; nested exception is org.hibernate.exception.GenericJDBCException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:333)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:244)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:525)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:59)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:209)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:147)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy340.save(Unknown Source)
at sun.reflect.GeneratedMethodAccessor575.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy341.save(Unknown Source)
at org.genesys.blocks.auditlog.service.impl.AuditTrailServiceImpl.addAuditLogs(AuditTrailServiceImpl.java:75)
at sun.reflect.GeneratedMethodAccessor585.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy342.addAuditLogs(Unknown Source)
at org.genesys.blocks.auditlog.component.AuditTrailInterceptor.beforeTransactionCompletion(AuditTrailInterceptor.java:719)
at org.genesys2.spring.config.DatabaseConfig$1.beforeTransactionCompletion(DatabaseConfig.java:214)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:518)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:105)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:177)
at org.hibernate.jpa.internal.TransactionImpl.commit(TransactionImpl.java:77)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:517)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:765)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:734)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:518)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:292)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at org.genesys2.server.service.worker.AccessionProcessor$$EnhancerBySpringCGLIB$$7a6edb7c.apply(<generated>)
at org.genesys2.server.mvc.admin.AdminController.updateTileIndex(AdminController.java:348)
at org.genesys2.server.mvc.admin.AdminController$$FastClassBySpringCGLIB$$958b96d7.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at org.genesys2.server.mvc.admin.AdminController$$EnhancerBySpringCGLIB$$183bd5b0.updateTileIndex(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.sitemesh.webapp.contentfilter.ContentBufferingFilter.bufferAndPostProcess(ContentBufferingFilter.java:169)
at org.sitemesh.webapp.contentfilter.ContentBufferingFilter.doFilter(ContentBufferingFilter.java:126)
at org.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:120)
at org.sitemesh.config.ConfigurableSiteMeshFilter.doFilter(ConfigurableSiteMeshFilter.java:163)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:124)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.genesys2.server.servlet.filter.LocaleURLFilter.doFilter(LocaleURLFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:293)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.GenericJDBCException: could not execute statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:211)
at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAndExtract(IdentityGenerator.java:96)
at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:58)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3032)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3558)
at org.hibernate.action.internal.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:98)
at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:492)
at org.hibernate.engine.spi.ActionQueue.addResolvedEntityInsertAction(ActionQueue.java:197)
at org.hibernate.engine.spi.ActionQueue.addInsertAction(ActionQueue.java:181)
at org.hibernate.engine.spi.ActionQueue.addAction(ActionQueue.java:216)
at org.hibernate.event.internal.AbstractSaveEventListener.addInsertAction(AbstractSaveEventListener.java:334)
at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:289)
at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:195)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:126)
at org.hibernate.jpa.event.internal.core.JpaPersistEventListener.saveWithGeneratedId(JpaPersistEventListener.java:84)
at org.hibernate.event.internal.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:206)
at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:149)
at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:75)
at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:811)
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:784)
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:789)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:1181)
at sun.reflect.GeneratedMethodAccessor582.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:300)
at com.sun.proxy.$Proxy208.persist(Unknown Source)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.save(SimpleJpaRepository.java:508)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.save(SimpleJpaRepository.java:542)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.save(SimpleJpaRepository.java:74)
at sun.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:515)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:500)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:477)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:56)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
... 171 more
Caused by: java.sql.SQLException: Connection is read-only. Queries leading to data modification are not allowed
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2044)
at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2013)
at com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5104)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1998)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:208)
... 214 more
13:24:20,345 qtp147403080-32 WARN o.g.s.m.a.AdminController:370 - Done recalculating tileIndexes
```2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/365Using javax.validation2019-02-18T16:20:45+01:00Matija ObrezaUsing javax.validationThe code has reference to `net.sf.oval` validators that are not used by the services.
64a8e983 demonstrates use of Bean Validation 1.1.0 API in services:
```
import javax.validation.Valid;
import org.springframework.validation.annotati...The code has reference to `net.sf.oval` validators that are not used by the services.
64a8e983 demonstrates use of Bean Validation 1.1.0 API in services:
```
import javax.validation.Valid;
import org.springframework.validation.annotation.Validated;
...
@Validated
public <T extends Dimension<?>> T save(@Valid T dimension) {
LOG.debug("Persisting dimension {}", dimension);
return dimensionRepository.save(dimension);
}
```
When implementing new service methods please use `@Valid` on model arguments where applicable.
Models use
```
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;
...
@NotNull
@Size(max = 100)
@Column(length = 100, unique = true, nullable = false)
private String name;
```
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/426/geo and /iso31662019-03-04T18:29:46+01:00Matija Obreza/geo and /iso3166Make custom `X??` codes return 404 and exclude them from the country list.Make custom `X??` codes return 404 and exclude them from the country list.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/371BrAPI: calls call2019-03-05T10:53:20+01:00Matija ObrezaBrAPI: calls callImplement `/calls` endpoint as per https://brapi.docs.apiary.io/#reference/callsImplement `/calls` endpoint as per https://brapi.docs.apiary.io/#reference/calls2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/434Filter by genus+species2019-04-10T16:13:47+02:00Matija ObrezaFilter by genus+speciesAllow for filtering on full species name, not just the specific epithet portion of the scientific name.
Add `genusSpecies` to `TaxonomyFilter` which will look up `Taxonomy2` records by `taxSpecies` identifier. The filter code must first...Allow for filtering on full species name, not just the specific epithet portion of the scientific name.
Add `genusSpecies` to `TaxonomyFilter` which will look up `Taxonomy2` records by `taxSpecies` identifier. The filter code must first lookup the ID and apply it directly.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/451Delete Metadata2019-06-05T04:47:54+02:00Matija ObrezaDelete MetadataAllow for removing old `org.genesys2.server.model.genesys.Metadata` (e.g. https://www.genesys-pgr.org/data/view/3673) and all related `ExperimentAccessionTrait`, `ExperimentTrait` for administrator in the JSP page.Allow for removing old `org.genesys2.server.model.genesys.Metadata` (e.g. https://www.genesys-pgr.org/data/view/3673) and all related `ExperimentAccessionTrait`, `ExperimentTrait` for administrator in the JSP page.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/457FilterCode v22019-06-22T05:49:57+02:00Matija ObrezaFilterCode v2Filter encoding **version 1** uses a randomly generated UUID with `-` removed:
```java
private String generateShortNameV1() {
for (int i = 0; i < 10; i++) {
final String code = "v1" + UUID.randomUUID().toString().replace("-", "");
...Filter encoding **version 1** uses a randomly generated UUID with `-` removed:
```java
private String generateShortNameV1() {
for (int i = 0; i < 10; i++) {
final String code = "v1" + UUID.randomUUID().toString().replace("-", "");
// return shortName if unique
if (shortFilterRepository.findByCode(code) == null) {
return code;
}
}
throw new RuntimeException("Failed to generate a unique filters short name in several attempts");
}
```
This generates filter codes like `v1a37ad370d9ca4f62b3d9840094aafae9` and they are fairly long.
## Shorter filter codes
We can make use of https://hashids.org/java/ to generate shorter codes with a custom salt!
Initial idea is to use the current timestamp (long value, 4b), resulting in drastically smaller unique short filter codes.
```java
private String generateShortNameV2() {
Hashids hashids = new Hashids("this is my salt");
for (int i = 0; i < 10; i++) {
String hash = hashids.encode(System.currentTimeMillis());
final String code = "v2" + hash;
// return shortName if unique
if (shortFilterRepository.findByCode(code) == null) {
return code;
}
}
throw new RuntimeException("Failed to generate a unique filters short name in several attempts");
}
```
We could extend this to our use of UUIDs in URLs, and see if that makes a difference in URL `code` length.2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/56GLIS DOI support2019-08-13T10:21:01+02:00Matija ObrezaGLIS DOI supportGLIS will provide DOI (https://www.doi.org) registration service for genebanks and other PGRFA (perhaps also GRFA, PGR) holders.
The DOI assigned by GLIS will be in the regexp format `10\.18730/[A-Z0-9]+` (production GLIS) and `10\.0155...GLIS will provide DOI (https://www.doi.org) registration service for genebanks and other PGRFA (perhaps also GRFA, PGR) holders.
The DOI assigned by GLIS will be in the regexp format `10\.18730/[A-Z0-9]+` (production GLIS) and `10\.0155/[A-Z0-9]+` (test environment). It is possible (and likely) that individual organizations assign their own DOI to accessions (`10\.[0-9]+/.+` format).
In this ticket we add support for DOI only and ignore other PUID types uploaded to Genesys.
# GLIS DOI extras
Checksum: the last character of the GLIS DOI is a checksum and can be one of *, ~, $, =, U and `[A-Z0-9]`.
Characters I, L, O, U are not in the "DOI" part of the DOI.
http://www.crockford.com/wrmg/base32.html
# The "big picture"
![Genebank_Genesys_GLIS](/uploads/751baf8c6a9b6e3589e91395123c3ab8/Genebank_Genesys_GLIS.png)
# Key principles
1. For genebank material (accessions) there is one DOI assigned to one accession.
1. From the moment an accession has a DOI assigned, that record can not be deleted from Genesys (REST delete should return 403 Forbidden).
1. Any other URL (like the current `/acn/id/[0-9]+`) must permanently redirect to the DOI'zed accession URL in Genesys.
# Accession URL
For accessions with a DOI registered in Genesys, we will use the following URL template: `https://www.genesys-pgr.org/[DOI]` for example `https://www.genesys-pgr.org/10.18730/A3C9`. Every URL to an accession with DOI must use the DOI'zed URL.
# Managing accession passport data
We add a **unique** DOI field (`doi varchar(255) null`) to the accession record.
Genebanks obtain DOI for their accessions and register the DOIs with their accession passport data. On upload of passport data to Genesys, the DOI is included with the passport data (next to `INSTCODE` and `ACCENUMB`).
The **DOI** takes absolute priority when identifying accession records. We now use the combination of `INSTCODE`, `ACCENUMB` and depending on the institute also `GENUS` to identify the record in our system. With the DOI provided, we only need to look up the record by the unique DOI value.
1. When DOI does not exist Genesys, record lookup is based on current approach (`INSTCODE`, `ACCENUMB`, `GENUS`?).
1. When DOI exists in Genesys we are updating that one single accession record.
**There is a possibility in current GLIS that the same DOI would be propagated between PGRFA holders and new holders opt to reuse the DOI assigned by the source genebank.** This can only be resolved by resolving the DOI against GLIS and obtaining information on the registered holder. Non-GLIS DOI resolution may not return the same information or even use the same format. This also should be explored in the future.
# Holder validation
Only on the first registration of DOI in Genesys we would need to resolve the DOI and retrieve registration data from GLIS. We need to verify that the WIEWS code of the holding institute matches the provided data.
Validation is implemented in genesys-pgr/glis-client#21
2.4Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/genesys-pgr/genesys-server/-/issues/465Document migration from API v0 to v12019-09-16T20:38:44+02:00Matija ObrezaDocument migration from API v0 to v1......2.4Matija ObrezaMatija Obreza