A je to!

3/19/15

Downloading JDK with command line

Getting Java straight to your server - without a browser, without GUI, with command line only.

Versions:

JDK 8

wget --header "Cookie: oraclelicense=accept-securebackup-cookie" \
 http://download.oracle.com/otn-pub/java/jdk/8u40-b26/jdk-8u40-linux-x64.tar.gz

JDK 7:

wget --header "Cookie: oraclelicense=accept-securebackup-cookie" \
 http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-7u75-linux-x64.tar.gz

TIP: wget -c will continue an interrupted download.

10/20/14

ZooKeeper ACLs

Checked with ZooKeeper 3.4.6.

Useful `zkCli` snippets:

anonymous access to a node:

setAcl /test/test01 world:anyone:crdwa

username/password access to a node (test/test)

setAcl /test/test01 digest:test:V28q/NynI4JI3Rk54h0r8O5kMug=:crwda

username/password access to a node, read for everyone

setAcl /test/test01 world:anyone:r,digest:test:V28q/NynI4JI3Rk54h0r8O5kMug=:crwda

username/password access to a node, read for another username/password

setAcl /test/test01 digest:testa:yDww3vDTFt2Vn+04UEvCSRsOLbU=:r,digest:test:V28q/NynI4JI3Rk54h0r8O5kMug=:crwda

How to get the password hash

The password can be generated with one of ZooKeeper classes. The following zkPasswd.sh will help:

#!/usr/bin/env bash

ZOOBIN="${BASH_SOURCE-$0}"
ZOOBIN="$(dirname "${ZOOBIN}")"
ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"

if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
  . "$ZOOBINDIR"/../libexec/zkEnv.sh
else
  . "$ZOOBINDIR"/zkEnv.sh
fi

"$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" \
    "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
     -cp "$CLASSPATH" $CLIENT_JVMFLAGS $JVMFLAGS \
     org.apache.zookeeper.server.auth.DigestAuthenticationProvider "$@"

Usage:

> ./zkPasswd.sh test:test
test:test->test:V28q/NynI4JI3Rk54h0r8O5kMug=

Password hashes are generated for username:password pair - if you change the username and keep the password, you need to regenerate the hash. Hash for test:test will be different than for test1:test.

6/27/13

Useful JMeter snippets

Here are some helpful copy & paste JMeter examples.

Get a value (here I will use Spring Web Flow `executionId`) from URL

Regular Expression Extractor - Main Sample / URL:

Reference Name: executionId (or similar),
Regular Expression: [?&]execution=([^&]+),
Template: $1$
Match No: 1,
Default Value: any (eg. NOTFOUND),

Get Spring Web Flow `executionId` from a form

Here's our HTML snippet #1:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head><!-- (...) --></head>
<body class="tundra">
<div id="page" class="container">
    <div id="header"><!-- (...) --></div>
    <div id="content">
        <div id="main" class="span-18 last">
            <!-- (...) -->
            <div id="bookingForm">
                <div class="span-12">
                    <form id="booking" action="/hotels/booking?execution=e1s1" method="post">
                        <fieldset>
                            <legend>Book Hotel</legend>
                            <!-- (...) -->
                            <p><input type="text" id="checkinDate" name="checkinDate" /></p>
                            <p>
                                <button type="submit" id="proceed" name="_eventId_proceed">Proceed</button>
                                <button type="submit" name="_eventId_cancel">Cancel</button>
                            </p>
                </div>
                </fieldset>
                </form>
            </div>
        </div>
    </div>
</div>
<!-- (...) -->
</div>
</body>
</html>

With a Regular Expression Extractor - Main Sample / Body:

Reference Name: formExecutionId (or similar),
Regular Expression: [?&]execution=([^&"]+)
Template: $1$ ,
Match No: 1,
Default Value: any (eg. NOT_FOUND),

With an XPath and Regular Expression (this is a 2-step process):

STEP 1 - Extract form action with XPath:

Main sample only,
Use Tidy, Quiet
Reference Name: xpathAction,
XPath query: //div[@id='bookingForm']//form[@id='booking']/@action(see HTML above),
Default Value: NOT_FOUND(anything meaningful),
We have the action value of the form now,

STEP 2 - Extract the executionId from the action, using Regular Expression Extractor:

Apply to: JMeter Variable / xpathAction,
Reference Name: xpathExecutionId,
Regular Expression: execution=([^&]+),
Template: $1$ ,
Match No.: 1,
Default Value: NOT_FOUND,

With a BeanShell Post-processor script:

    text = prev.responseDataAsString;
    patternText = "(.*)execution=([^&\"]+)(.*)";
    p = java.util.regex.Pattern.compile(patternText, java.util.regex.Pattern.DOTALL);
    m = p.matcher(text);
    if (m.find()) {
        vars.put("beanShellExecutionId", m.group(2));
    } else {
        vars.put("beanShellExecutionId", "NOT_FOUND");
    }

beanShellExecutionId

And here's a BeanShell assertion example:

    text = SampleResult.responseDataAsString;
    patternText = "(.*)execution=([^&\"]+)(.*)";
    p = java.util.regex.Pattern.compile(patternText, java.util.regex.Pattern.DOTALL);
    m = p.matcher(text);
    if (m.matches()) {
        log.info("found beanShellExecutionId value: " + m.group(2));
    } else {
        Failure = true;
        FailureMessage = "This BeanShell assertion has failed";
    }

BeanShell variables are:

ctx
vars
props
prev
log
data - byte[]

6/25/13

Testing thread-safety with JUnit

Here's a scenario showing how to test if your code is thread safe, in form of a JUnit integration test. In this example, it's a bank account (just acount number and account balance) and some logic that handles it. The application is a simple Spring/Hibernate/PostgreSQL app.

1. Application code:

BankAccount.java:

    public class BankAccount {
        Integer id;
        String number;
        Integer balance;
        (...)
    }

BankAccountDao.java:

    public interface BankAccountDao {
        BankAccount get(String number);
        void update(BankAccount bankAccount);
    }

BankAccountDaoImpl.java:

    public class BankAccountDaoImpl extends HibernateDaoSupport implements BankAccountDao {
        @Override
        public BankAccount get(String number) {
            return (BankAccount) DataAccessUtils.singleResult(
                getHibernateTemplate().find("from BankAccount where number = ?", number));
        }
        @Override
        public void update(BankAccount bankAccount) {
            getHibernateTemplate().update(bankAccount);
        }
    }

BankService.java:

    public interface BankService {
        /**
         * @param accountNumber account number
         * @param amount        amount of money, positive or negative
         */
        void transfer(String accountNumber, Integer amount);
    }

BankServiceImpl.java:

    public class BankServiceImpl implements BankService {
        private BankAccountDao bankAccountDao;
        @Override
        public void transfer(String accountNumber, Integer amount) {
            BankAccount bankAccount = bankAccountDao.get(accountNumber);
            bankAccount.setBalance(bankAccount.getBalance() + amount);
            bankAccountDao.update(bankAccount);
        }
    }

There's also some XML configuration (Spring, Hibernate, transactions, etc.), but not relevant here. The transaction interceptor wraps the transfer() method.

2. The JUnit integration test

The code above can be quite easily tested with a simple Spring's JUnit test case. I initially copied over the code from this excellent blog post, and then did my own small modifications.

BankServiceTest.java

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.unitils.UnitilsJUnit4TestClassRunner;
import org.unitils.dbunit.annotation.DataSet;
import org.unitils.orm.hibernate.HibernateUnitils;
import org.unitils.reflectionassert.ReflectionAssert;
import org.unitils.spring.annotation.SpringApplicationContext;
import org.unitils.spring.annotation.SpringBeanByType;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

/**
 */
@SpringApplicationContext({"classpath:/application-dao.xml", "classpath:/application-tx.xml", "classpath:/application-test-datasource.xml"})
@DataSet("BankServiceTest.xml")
@RunWith(UnitilsJUnit4TestClassRunner.class)
public class BankServiceTest {

    @SpringBeanByType
    private BankService bankService;

    @SpringBeanByType
    private BankAccountDao bankAccountDao;

    private int threadCount = 200;
    private int amount = 1;

    @Test
    public void testUpdateBalance() throws Exception {
        Assert.assertEquals("The balance is 1000", 1000, bankAccountDao.get("10-1000").getBalance().intValue());
        ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
        List<Future<Void>> futures = new ArrayList<Future<Void>>();
        for (int x = 0; x < threadCount; x++) {
            Callable<Void> callable = new Callable<Void>() {
                @Override
                public Void call() throws Exception {
                    bankService.transfer("10-1000", amount);
                    return null;
                }
            };
            Future<Void> submit = executorService.submit(callable);
            futures.add(submit);
        }

        List<Exception> exceptions = new ArrayList<Exception>();
        for (Future<Void> future : futures) {
            try {
                future.get();
            } catch (Exception e) {
                exceptions.add(e);
                e.printStackTrace(System.err);
            }
        }

        executorService.shutdown();

        HibernateUnitils.getSession().clear();
        BankAccount bankAccount = bankAccountDao.get("10-1000");
        ReflectionAssert.assertReflectionEquals("No exceptions", new ArrayList<Exception>(), exceptions);
        Assert.assertEquals("Balance is 1000, again", 1200, bankAccount.getBalance().intValue());
    }
}

#26, #29 - Spring beans are injected into the test case - I used Unitils here,
#37 - creating a pool of 200 threads,
#43 - we invoke the transfer() method 200 times in parallel,
#56 - collecting exceptions that may have raised, (we expect no exceptions to occur),
#61 - thread pool is closed,
#63 - clear Hibernate cache manually - so that we get the new balance, not the cached 1000,
#65 - assertion - check if no exceptions occured,
#66 - check if the account balance was correctly incremented to 1200,

java.lang.AssertionError: Balance is 1000, again expected:<1200> but was:<1046>
 at org.junit.Assert.fail(Assert.java:74)
 at org.junit.Assert.failNotEquals(Assert.java:448)
 at org.junit.Assert.assertEquals(Assert.java:102)
 at org.junit.Assert.assertEquals(Assert.java:323)
 at BankServiceTest.testUpdateBalance(BankServiceTest.java:66)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    (...)

3. Code fix

We'll try to fix that with a pessmistic lock:

BankAccountDaoImpl.java

    public class BankAccountDaoImpl extends HibernateDaoSupport implements BankAccountDao {
        @Override
        public BankAccount get(String number) {
            return (BankAccount) DataAccessUtils.singleResult(getHibernateTemplate().find("from BankAccount where number = ?", number));
        }
        @Override
        public BankAccount getForUpdate(final String number) {
            return (BankAccount) DataAccessUtils.singleResult(getHibernateTemplate().executeFind(new HibernateCallback<List<BankAccount>>() {
                @Override
                public List<BankAccount> doInHibernate(Session session) throws HibernateException, SQLException {
                    return session.createQuery("from BankAccount ba where number = :number")
                        .setLockMode("ba", LockMode.PESSIMISTIC_WRITE)
                        .setString("number", number).list();
                    }
            }));
        }
        @Override
        public void update(BankAccount bankAccount) {
            getHibernateTemplate().update(bankAccount);
        }
    }

BankAccountDao.java:

    public interface BankAccountDao {
        BankAccount get(String number);
        BankAccount getForUpdate(final String number);
        void update(BankAccount bankAccount);
    }

BankServiceImpl.java:

    public class BankServiceImpl implements BankService {
        private BankAccountDao bankAccountDao;
        @Override
        public void transfer(String accountNumber, Integer amount) {
            BankAccount bankAccount = bankAccountDao.getForUpdate(accountNumber);
            bankAccount.setBalance(bankAccount.getBalance() + amount);
            bankAccountDao.update(bankAccount);
        }
    }

4. That's it!

Some reading:

3/18/13

Pentaho Kettle ETL regression with Spring

The real-life scenario is a Java application with an ETL process to load some data. For example, update the list of names in a phone directory book. With a pretty naive approach, automated regression tests can be introduced quite easily. Here's a demo.

Overview

The application itself makes use of:

Spring Framework 3.0,
Hibernate 3.6,
Pentaho Kettle ETL 4.4.

The model is

two entities: Person and PhoneNumber,
two DAOs: PersonDao, PhoneNumberDao,
ETL loads a CSV file and updates the PERSON table,
a wrapper around Kettle Transformation/Job API,

Testing dependencies

Spring-test 3.0,
DbUnit 2.2,
JUnit 4.4,
HSQLDB 2.2,
Unitils 2.4.

General ideas

underlying database is HSQLDB instead of an actual Oracle instance,

for Kettle ETL tests, the database is temporarily switched to Oracle compatibility mode - we can use MySQL, PostgreSQL and other compatibility modes,
we can also test against a real Oracle DB - in this case, any Oracle-specific functionality can be used. I prefer a compromise with HSQLDB.

db schema is created with hbm2ddl,
db test data is loaded with DbUnit XML files (tricky!)
the actual testing happens inside a simple JUnit4 / Spring Integration test case:

first, we execute the ETL transformation,
next, we test the results with our Hibernate-based API.

The starting point

We already have a piece of code which allows us to execute the ETL process programmatically:

`KettleAdapterImpl.java`

public class KettleAdapterImpl implements KettleAdapter {

    private DataSource dataSource;
    private String repositoryPath;

    public KettleAdapterImpl() throws KettleException {
        KettleEnvironment.init(false);
    }

    @Override
    public int executeTransformation(String s, Map<String, String> parameters) {
        try {
            DataSourceProviderFactory.setDataSourceProviderInterface(new DataSourceProviderInterface() {
                @Override
                public DataSource getNamedDataSource(String s) throws DataSourceNamingException {
                    return dataSource;
                }
            });

            KettleFileRepositoryMeta repositoryMeta = new KettleFileRepositoryMeta("id", "name", "repo", ResourceUtils.getFile(repositoryPath).getAbsolutePath());
            KettleFileRepository repository = new KettleFileRepository();
            repository.setRepositoryMeta(repositoryMeta);
            TransMeta transMeta = new TransMeta(ResourceUtils.getFile(repositoryPath + "/" + s).getAbsolutePath(), repository);

            for (Map.Entry<String, String> entry : parameters.entrySet()) {
                transMeta.setParameterValue(entry.getKey(), entry.getValue());
            }

            Trans trans = new Trans(transMeta);
            trans.execute(new String[]{});
            trans.waitUntilFinished();
            int result = trans.getErrors();
            return result;
        } catch (KettleException e) {
            throw new RuntimeException(e);
        } catch (FileNotFoundException e) {
            throw new RuntimeException(e);
        }
    }

    public void setDataSource(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    public void setRepositoryPath(String repositoryPath) {
        this.repositoryPath = repositoryPath;
    }
}

Test configuration, step by step

And now to the configuration details.

Spring integration tests

We'll use some neat features of Unitils, which will speed up test development and maintenance. First, a test for PersonDao:

`PersonDaoImpl.java`

public class PersonDaoImpl extends HibernateDaoSupport implements PersonDao {
    @Override
    public Integer save(Person person) {
        Serializable save = getHibernateTemplate().save(person);
        getHibernateTemplate().flush();
        return (Integer)save;
    }

    @Override
    public Person get(Integer id) {
        return getHibernateTemplate().get(Person.class, id);
    }

    @Override
    public List<Person> getAll() {
        return getHibernateTemplate().findByNamedQuery("person.loadAll");
    }
}

`PersonDaoImplTest.java`

@SpringApplicationContext({"classpath:/application-dao.xml", "classpath:/application-test-datasource.xml"})
@DataSet("PersonDaoImplTest.xml")
public class PersonDaoImplTest extends UnitilsJUnit4 {

    @SpringBeanByName
    private PersonDao personDao;

    @SpringBeanByName
    private PhoneNumberDao phoneNumberDao;

    @Test
    public void testPersonDaoImpl() throws Exception {
        Person person = new Person();
        person.setFirstName("Steve");
        PhoneNumber phoneNumber = new PhoneNumber();
        phoneNumber.setNumber("01234");
        person.setPhoneNumbers(Sets.newHashSet(phoneNumber));
        phoneNumberDao.save(phoneNumber);
        personDao.save(person);

        Assert.assertTrue("new id above 1000", person.getId().intValue() >= 1000);
        Assert.assertEquals("all 3", 3, personDao.getAll().size());
    }

}

Testing with HSQLDB

I think it's easier to maintain tests with minimum external dependencies - hence replacing Oracle with in-memory HSQLDB. At some point, you may end up with Oracle, anyway, but until you don't have to do that, HSQLDB will be easier.

Maven

<dependency>
    <groupId>org.hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>2.2.9</version>
    <scope>test</scope>
</dependency>

Spring configuration

A separate file - test datasource configuration:

`application-test-datasource.xml`

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

    <bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="driverClassName" value="org.hsqldb.jdbcDriver"/>
        <property name="url" value="jdbc:hsqldb:mem:aname;hsqldb.tx=mvcc"/>
        <property name="username" value="sa"/>
        <property name="password" value=""/>
    </bean>

</beans>

Hack #1

The MVCC parameter will be required by Kettle - the tests tend to deadlock without this parameter.

Unitils

`unitils.properties`

database.driverClassName=org.hsqldb.jdbcDriver
database.url=jdbc:hsqldb:mem:aname;hsqldb.tx=mvcc
database.userName=sa
database.password=
database.dialect=hsqldb
database.schemaNames=PUBLIC
DbUnitModule.DataSet.loadStrategy.default=pl.erizo.random.kettle.regression.util.CleanAllInsertLoadStrategy

DbUnit

Necessary for pre-loading some data for the tests. Easy and fast, but not problem-free:

Hack #2

DbUnit 2.2 has issues with deleting data from multiple tables - by default, it tries to clean only tables mentioned in the appropriate dataset. So, we're going to always clear all tables:

`CleanAllInsertLoadStrategy.java`

public class CleanAllInsertLoadStrategy extends BaseDataSetLoadStrategy {

    @Override
    public void doExecute(DbUnitDatabaseConnection dbUnitDatabaseConnection, IDataSet dataSet) throws DatabaseUnitException, SQLException {
        try {
            InputStream xmlStream = ResourceUtils.getURL("classpath:AllTables.xml").openStream();
            FlatXmlDataSet allTablesDataSet = new FlatXmlDataSet(xmlStream);
            DatabaseOperation.DELETE_ALL.execute(dbUnitDatabaseConnection, allTablesDataSet);
            DatabaseOperation.INSERT.execute(dbUnitDatabaseConnection, dataSet);
            dbUnitDatabaseConnection.getConnection().commit();
        } catch (IOException e) {
            throw new DatabaseUnitException("Could not read AllTables.xml", e);
        }
    }
}

And the list of tables:

`AllTables.xml`

<dataset>
    <PERSON />
    <PHONE_NUMBER />
</dataset>

One last thing - Hack #3

Mixing up DbUnit datasets, Hibernate, ETL and DB sequences may cause trouble: while loading the datasets, DbUnit does not increment sequences. But this can be fixed with Hibernate SessionFactory configuration:

`application-dao.xml`

    <bean id="mySessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
        <property name="dataSource" ref="myDataSource"/>
        <property name="mappingResources">
            <list>
                <value>pl/erizo/random/kettle/regression/model/Person.hbm.xml</value>
                <value>pl/erizo/random/kettle/regression/model/PhoneNumber.hbm.xml</value>
            </list>
        </property>
        <property name="hibernateProperties">
            <value>
                hibernate.dialect=org.hibernate.dialect.HSQLDialect
                hibernate.hbm2ddl.auto=create
                hibernate.show_sql=true
                hibernate.hbm2ddl.import_files=/sequences.sql
            </value>
        </property>
    </bean>

And, situated in src/test/resources:

`sequences.sql`

-- increment to 1000
alter sequence PER_ID_SEQ restart with 1000;

Let's test

`KettleAdapterTest.java`

@SpringApplicationContext({"classpath:/application-dao.xml", "classpath:/application-test-datasource.xml"})
@DataSet("KettleAdapterTest.xml")
public class KettleAdapterTest extends UnitilsJUnit4 {

    @SpringBeanByName
    private KettleAdapter kettleAdapter;

    @SpringBeanByName
    private PersonDao personDao;

    @Before
    public void setUp() throws Exception {
        SQLUnitils.executeUpdate("SET DATABASE SQL SYNTAX ORA TRUE", DatabaseUnitils.getDataSource());
    }

    @Test
    public void testExecute() throws Exception {
        List<Person> everybody = personDao.getAll();
        Assert.assertEquals("2 results", 2, everybody.size());
         HashMap<String, String> map = Maps.newHashMap();
        map.put("input.file", ResourceUtils.getFile("classpath:pl/erizo/random/kettle/regression/phonenumbers.txt").getAbsolutePath());
        int errors = kettleAdapter.executeTransformation("LoadNumbers.ktr", map);
        Assert.assertEquals("no errors", 0, errors);
        everybody = personDao.getAll();
        ReflectionAssert.assertPropertyLenientEquals("last names ordered", "lastName",
        Lists.newArrayList("Gates", "Newman", "Smith"), everybody);
        Assert.assertEquals("3 results", 3, everybody.size());
    }

    @After
    public void tearDown() throws Exception {
        SQLUnitils.executeUpdate("SET DATABASE SQL SYNTAX ORA FALSE", DatabaseUnitils.getDataSource());
    }
}

`KettleAdapterTest.xml`

<dataset>
    <PERSON PER_ID="1" PER_FIRST_NAME="John" PER_LAST_NAME="Smith" />
    <PERSON PER_ID="2" PER_FIRST_NAME="Bill" PER_LAST_NAME="Gates" />
</dataset>

`phonenumbers.txt`

John,Smith,001
Paul,Newman,002

Some more reading

A better way to run a Kettle transformation from within Java: http://tech.tejusparikh.com/post/10920230005/there-is-a-spoon-integrating-etl-into-your-application

3/30/12

PoC

More a notepad than a blog - things that improve development.