cron java and utf-8

If you ever run java application from cron job you might notice that any unicode characters that java application emits looks like ????????

In order to avoid this situation in RHEL-based distros you need to create /etc/environment file and then put here a value of LANG variable. For example: LANG=ru_RU.UTF-8  (without word export or something else)  

Solving random numbers problem

This library https://bitbucket.org/mike1452/msprandom demonstrates a technique which allows to solve the problem of getting random numbers on computers and small devices for cryptographic purposes. So, if you need encrypt and sign data on a small device or computer without hardware RNG do the following:

1. Create true random numbers vault using this library on your computer or notebook.
2. Put this vault on your device, computer or server where you need encrypt and sign data.
3. Load the vault once at the start of the program when you need encrypt or sign data.
4. Call secure-rand function to get random bytes as many times as you need.

Vault is encrypted and secured with HMAC. Random data in a vault is updated every time you load random seed with unpredictable way, so HMAC is recalculated too.

Gathering a true random data

To get a true random data a human input is used. Algorithm of collecting a random data:

1. Run separate thread where atomic counter increments every tic from 0..255 with a very high speed.
2. Wait for unbuffered key press by human and get a scan code of pressed button.
3. Take current nanoseconds value from start of Epoch and take mod 256 to convert its value to a random byte.
4. Xor values between each other: scan-code-byte ^ current-counter-value ^ nanoseconds to produce random byte.
5. Add random byte to output vector. We suppose that only 3 bits has true randomness in this random byte. So, to get true random 32 bytes we need ~ 32*3 button press from user input. 6 Repeat steps 2-5 until we get required amount of random bytes.
6. If we collected required amount of random data then do final step -> hash output vector with cryptographically strong hash function GOST 3411-94 to guarantee that probability 1 and 0 bits in output vector will be 0.5. Note, that hash function used here only to mix random bits and do not influence to the quality of random data. So hash(random data) = random data. Hash will produce vector of 32 bytes containing a random data.
7. Repeat steps 1..7 if we need more than 32 bytes of random data.

Using this algorithm we collect a true 512 random bits. Why 512? Well, every PRNG needs a true random seed. If an attacker knows a seed then you can't protect your data. 256 bit length is far enough to keep millitary grade secrets. I did 512 to close the security question of random seed. My opinion, 512 bit of true random data is enough to use in PRNG: generating keys, signatures, etc.

call clojure code from java

Since clojure 1.6 there is no more special dances inside clojure code: special declarations or wrappers or something else. it is possible call clojure code from jar file like this. Clojure code (jar file) and clojure-1.6.jar should be in a class-path.

import clojure.java.api.Clojure;
import clojure.lang.IFn;

        IFn require = Clojure.var("clojure.core", "require");
        require.invoke(Clojure.read("my-clj-project.core"));

        IFn myf0 = Clojure.var("my-clj-project.core", "my-clojure-func0");
        myf0.invoke();

or  myf0.invoke(param1, param2);

How to break JCE crypto policy limit when using bouncycastle

    //this code allows to break limit if client jdk/jre has no unlimited policy files for JCE.
    //it should be run once. So this static section is always execute during the class loading process.
//this code is useful when working with Bouncycastle library.
    static {
        try {
            Field field = Class.forName("javax.crypto.JceSecurity").getDeclaredField("isRestricted");
            field.setAccessible(true);
            field.set(null, java.lang.Boolean.FALSE);
        } catch (Exception ex) {
        }
    }

or clojure variant of this function

(defn break-jce-policy-limit
  "This function breaks JCE crypto limits. Should be run once, primarily at the begining of the program
  to avoid JCE policy limit if JDK/JRE runtime has no installed files for break crypto limit. Returns nil."
  []
  (safe (let [field (-> (Class/forName "javax.crypto.JceSecurity")
                      (.getDeclaredField "isRestricted"))]
          (.setAccessible field true)
          (.set field nil java.lang.Boolean/FALSE))))

(defmacro safe [bindings? & forms]
  "This macro is used to execute any function inside try-catch block."
  (let [bindings (if (and (even? (count bindings?)) (vector? bindings?))
                   bindings? nil)
        forms (if bindings forms (cons bindings? forms))
        except `(catch Exception e# e#
                  (println (.getMessage e#) e#))]
    (if bindings
      `(let ~bindings (try ~@forms ~except))
      `(try ~@forms ~except))))

groovy 2.1 and maven

This is pom file example how to add groovy support to java project.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.middlesphere</groupId>
    <artifactId>hello01</artifactId>
    <packaging>jar</packaging>
    <version>0.1</version>
    <name>hello01</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <gmavenVersion>1.4</gmavenVersion>
        <gmavenProviderSelection>2.0</gmavenProviderSelection>
        <groovyVersion>2.1.6</groovyVersion>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy-all</artifactId>
            <version>2.1.6</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.8.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <target>1.6</target>
                    <source>1.6</source>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.gmaven</groupId>
                <artifactId>gmaven-plugin</artifactId>
                <version>${gmavenVersion}</version>
                <configuration>
                    <providerSelection>${gmavenProviderSelection}</providerSelection>
                    <sourceEncoding>UTF-8</sourceEncoding>
                    <source/>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>generateStubs</goal>
                            <goal>compile</goal>
                            <goal>generateTestStubs</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <dependencies>
                    <dependency>
                        <groupId>org.codehaus.groovy</groupId>
                        <artifactId>groovy-all</artifactId>
                        <version>${groovyVersion}</version>
                    </dependency>
                </dependencies>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>prepare-package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>com.middlesphere.HelloApp</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

hadoop: set file permissions to a user

#Before user can write to hdfs we need to set permissions

sudo -u hdfs hadoop fs -mkdir /user/username
sudo -u hdfs hadoop fs -chown username:usergroup /user/
username

#if we need to write to hdfs from remote machine ( append avro file, for example )

       //if user exist it return its ugi
       UserGroupInformation ugi = UserGroupInformation.createRemoteUser("username");
        ugi.doAs(new PrivilegedExceptionAction() {
            public Void run() throws Exception {

                Configuration conf = new Configuration();
                conf.set("fs.defaultFS", "hdfs://hadoop-server");

                FileSystem fs = FileSystem.get(conf);

                DatumWriter xmlOrderInfoWriter = new SpecificDatumWriter(XmlOrderInfo.class);
                DataFileWriter dataFileWriter = new DataFileWriter(xmlOrderInfoWriter);

                Path filePath = new Path("/user//data/myfolder/test-xml-files.avro");
                OutputStream out = fs.append(filePath);

                dataFileWriter.appendTo(new FsInput(filePath,conf),out);
                dataFileWriter.append(xmlOrderInfo);
                dataFileWriter.close();
                out.close();
                System.out.println("Test avro file is appended in HDFS successfully");

                return null;
            }

        });

create maven project

Maven

#create maven project
mvn archetype:generate -DgroupId=my.company.project -DartifactId=modulename -DarchetypeArtifactId=maven-archetype-quickstart -Dversion=0.1 -DinteractiveMode=false


GIT

git init
git add *
git commit -a -m"Initial commit"
git remote add origin https://username@bitbucket.org/username/myrepo.git
git push -u origin --all
mvn idea:idea