PDI-8731 - Cassandra SSTable Output (bulkloader) step does not work w…

[mdamour1976]

…ith Cassandra 1.1.x

Resolve cassandra transitively to get runtime dependencies
Add support for cassandra 2.x
Split writer into CQL2 and CQL3 specific ones
Try preventing JVM from exit in case issues with cassandra config
Fixes for CQL2 writer:

• read partitioner from config, use Murmur3Partitioner by default
• store date as longs

[PDI-8731][Tests] - Added IT and made some cosmetic changes

[mdamour1976]

This is a conflict resolve of #54

[wingman-pentaho]

Build Completed

❌ This pull request has errors. They will need to be addressed before it can be accepted. See below for more details. Some links are also available below for further assistance in addressing those issues.

Build Commands

ant -Dpentaho.resolve.repo=http://ivy-nexus.pentaho.org/content/groups/approved -Dtestreports.dir=bin/reports/unit-test -f build.xml clean-all resolve jacoco && ant -Dpentaho.resolve.repo=http://ivy-nexus.pentaho.org/content/groups/approved -Dtestreports.dir=bin/reports/integration-test -f build.xml jacoco-integration checkstyle publish-local

Cleanup Commands

rm -r ~/.ivy2/local || echo no publish local to remove

Changed files

  ivy.xml
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutput.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutputDialog.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutputMeta.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/messages/messages_en_US.properties
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/AbstractSSTableWriter.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/CQL2SSTableWriter.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/CQL3SSTableWriter.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/SSTableWriterBuilder.java
  src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/messages/messages_en_US.properties
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutputIT.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutputMetaTest.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/SSTableOutputTest.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/cassandra.yaml
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/AbstractSSTableWriterTest.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/CQL2SSTableWriterTest.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/CQL3SSTableWriterTest.java
  test-src/org/pentaho/di/trans/steps/cassandrasstableoutput/writer/SSTableWriterBuilderTest.java
  test-src/org/pentaho/di/trans/steps/mock/StepMockHelper.java

Newly Broken Tests:

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputIT.testCQLS2SSTableWriter:

🔴 java.lang.NoSuchMethodError

java.lang.NoSuchMethodError: org.apache.cassandra.config.YamlConfigurationLoader$MissingPropertiesChecker.setSkipMissingProperties(Z)V
    at org.apache.cassandra.config.YamlConfigurationLoader$MissingPropertiesChecker.<init>(YamlConfigurationLoader.java:114)
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:90)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.SSTableWriterBuilder.getPartitionerClass(SSTableWriterBuilder.java:146)
    at org.pe

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputIT.testCQLS3SSTableWriter:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: org/cliffc/high_scale_lib/NonBlockingHashMap
    at org.apache.cassandra.config.Schema.<init>(Schema.java:57)
    at org.apache.cassandra.config.Schema.<clinit>(Schema.java:46)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.SSTabl

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputMetaTest.testGetXMLDefault:

🔴 junit.framework.AssertionFailedError

junit.framework.AssertionFailedError: getXml() does not cover defaults 
    at org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputMetaTest.testGetXMLDefault(SSTableOutputMetaTest.java:114)

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testBuildCreateColumnFamilyCQLStatement:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.config.Schema
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testBuildCreateColumnFamilyCQLStatement(CQL3SSTableWriterTest.java:133)

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testBuildInsertCQLStatement:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.config.Schema
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testBuildInsertCQLStatement(CQL3SSTableWriterTest.java:141)

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testClose:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.config.Schema
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testClose(CQL3SSTableWriterTest.java:124)

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testInit:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: org/cliffc/high_scale_lib/NonBlockingHashMap
    at org.apache.cassandra.config.Schema.<init>(Schema.java:57)
    at org.apache.cassandra.config.Schema.<clinit>(Schema.java:46)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testProcessRow:

🔴 java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.config.Schema
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.purgeSchemaInstance(CQL3SSTableWriter.java:49)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter.init(CQL3SSTableWriter.java:44)
    at org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriterTest.testProcessRow(CQL3SSTableWriterTest.java:112)

Unit test coverage change

These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutput

• Complexity Change: + 11.54%
• Instruction Change: + 4.85%
• Line Change: + 6.67%
• Method Change: + 42.86%

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutput.JVMShutdownAttemptedException

• Complexity New Coverage: 100.00%
• Instruction New Coverage: 100.00%
• Line New Coverage: 100.00%
• Method New Coverage: 100.00%

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutput.NoSystemExitDelegatingSecurityManager

• Branch New Coverage: 25.00%
• Complexity New Coverage: 50.00%
• Instruction New Coverage: 60.61%
• Line New Coverage: 60.00%
• Method New Coverage: 75.00%

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputMeta

• Branch Change: + 31.25%
• Complexity Change: + 30.00%
• Instruction Change: + 31.56%
• Line Change: + 38.69%
• Method Change: + 32.05%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.AbstractSSTableWriter

• Complexity New Coverage: 100.00%
• Instruction New Coverage: 100.00%
• Line New Coverage: 100.00%
• Method New Coverage: 100.00%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL2SSTableWriter

• Branch New Coverage: 88.46%
• Complexity New Coverage: 81.82%
• Instruction New Coverage: 84.74%
• Line New Coverage: 89.80%
• Method New Coverage: 88.89%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter

• Complexity New Coverage: 18.18%
• Instruction New Coverage: 4.35%
• Line New Coverage: 10.34%
• Method New Coverage: 22.22%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.SSTableWriterBuilder

• Branch New Coverage: 75.00%
• Complexity New Coverage: 73.33%
• Instruction New Coverage: 86.07%
• Line New Coverage: 89.19%
• Method New Coverage: 76.92%

Integration test coverage change

These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutput

• Branch New Coverage: 34.21%
• Complexity New Coverage: 26.92%
• Instruction New Coverage: 46.94%
• Line New Coverage: 44.44%
• Method New Coverage: 71.43%

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutput.NoSystemExitDelegatingSecurityManager

• Branch New Coverage: 25.00%
• Complexity New Coverage: 33.33%
• Instruction New Coverage: 39.39%
• Line New Coverage: 50.00%
• Method New Coverage: 50.00%

org.pentaho.di.trans.steps.cassandrasstableoutput.SSTableOutputMeta

• Complexity New Coverage: 38.10%
• Instruction New Coverage: 15.88%
• Line New Coverage: 26.73%
• Method New Coverage: 61.54%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.AbstractSSTableWriter

• Complexity New Coverage: 54.55%
• Instruction New Coverage: 66.67%
• Line New Coverage: 72.22%
• Method New Coverage: 54.55%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL2SSTableWriter

• Complexity New Coverage: 9.09%
• Instruction New Coverage: 3.16%
• Line New Coverage: 4.08%
• Method New Coverage: 22.22%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.CQL3SSTableWriter

• Complexity New Coverage: 18.18%
• Instruction New Coverage: 4.35%
• Line New Coverage: 10.34%
• Method New Coverage: 22.22%

org.pentaho.di.trans.steps.cassandrasstableoutput.writer.SSTableWriterBuilder

• Branch New Coverage: 75.00%
• Complexity New Coverage: 86.67%
• Instruction New Coverage: 87.70%
• Line New Coverage: 89.19%
• Method New Coverage: 92.31%

[buildguy]

❌ Build failed in 1h 43m 55s

Build command:

mvn clean verify -B -e -Daudit -amd

---

❗ No tests found!

Errors:

Filtered log (click to expand)

/build/wingman/workspace/maintenance/wingman/builds/pentaho.pentaho-cassandra-plugin.PR-66/pom.xml does not exist.

ℹ️ This is an automatic message