[Git][java-team/java-string-similarity][master] 6 commits: New upstream version 2.0.0
Tony Mancill
gitlab at salsa.debian.org
Tue Jul 7 05:17:19 BST 2020
Tony Mancill pushed to branch master at Debian Java Maintainers / java-string-similarity
Commits:
4b340e42 by tony mancill at 2020-07-06T21:01:24-07:00
New upstream version 2.0.0
- - - - -
f2a26ab2 by tony mancill at 2020-07-06T21:01:24-07:00
Bump Standards-Version to 4.5.0
- - - - -
f29b5d4d by tony mancill at 2020-07-06T21:01:24-07:00
Use debhelper-compat 13
- - - - -
faddfe4b by tony mancill at 2020-07-06T21:09:41-07:00
Set "Rules-Requires-Root: no" in debian/control
- - - - -
e600cd3f by tony mancill at 2020-07-06T21:09:41-07:00
Freshen debian/copyright
- - - - -
8ed2a053 by tony mancill at 2020-07-06T21:09:41-07:00
prepare for upload to unstable
- - - - -
23 changed files:
- README.md
- debian/changelog
- − debian/compat
- debian/control
- debian/copyright
- pom.xml
- + src/main/java/info/debatty/java/stringsimilarity/CharacterInsDelInterface.java
- src/main/java/info/debatty/java/stringsimilarity/Cosine.java
- src/main/java/info/debatty/java/stringsimilarity/Jaccard.java
- src/main/java/info/debatty/java/stringsimilarity/Levenshtein.java
- src/main/java/info/debatty/java/stringsimilarity/NormalizedLevenshtein.java
- + src/main/java/info/debatty/java/stringsimilarity/RatcliffObershelp.java
- src/main/java/info/debatty/java/stringsimilarity/ShingleBased.java
- src/main/java/info/debatty/java/stringsimilarity/WeightedLevenshtein.java
- src/main/java/info/debatty/java/stringsimilarity/examples/PrecomputedCosine.java
- src/main/java/info/debatty/java/stringsimilarity/experimental/Sift4.java
- src/main/java/info/debatty/java/stringsimilarity/interfaces/MetricStringDistance.java
- src/main/java/info/debatty/java/stringsimilarity/interfaces/NormalizedStringDistance.java
- src/main/java/info/debatty/java/stringsimilarity/interfaces/NormalizedStringSimilarity.java
- src/main/java/info/debatty/java/stringsimilarity/interfaces/StringDistance.java
- src/test/java/info/debatty/java/stringsimilarity/LevenshteinTest.java
- + src/test/java/info/debatty/java/stringsimilarity/RatcliffObershelpTest.java
- src/test/java/info/debatty/java/stringsimilarity/WeightedLevenshteinTest.java
Changes:
=====================================
README.md
=====================================
@@ -1,6 +1,6 @@
# java-string-similarity
-[](https://maven-badges.herokuapp.com/maven-central/info.debatty/java-string-similarity) [](https://travis-ci.org/tdebatty/java-string-similarity) [](https://coveralls.io/github/tdebatty/java-string-similarity?branch=master) [](http://api123.web-d.be/api/java-string-similarity/head/index.html) [](https://www.versioneye.com/java/info.debatty:java-string-similarity/references)
+[](https://maven-badges.herokuapp.com/maven-central/info.debatty/java-string-similarity) [](https://travis-ci.org/tdebatty/java-string-similarity) [](https://coveralls.io/github/tdebatty/java-string-similarity?branch=master) [](http://www.javadoc.io/doc/info.debatty/java-string-similarity)
A library implementing different string similarity and distance measures. A dozen of algorithms (including Levenshtein edit distance and sibblings, Jaro-Winkler, Longest Common Subsequence, cosine similarity etc.) are currently implemented. Check the summary table below for the complete list...
@@ -22,13 +22,16 @@ A library implementing different string similarity and distance measures. A doze
* [Cosine similarity](#shingle-n-gram-based-algorithms)
* [Jaccard index](#shingle-n-gram-based-algorithms)
* [Sorensen-Dice coefficient](#shingle-n-gram-based-algorithms)
+* [Ratcliff-Obershelp](#ratcliff-obershelp)
* [Experimental](#experimental)
* [SIFT4](#sift4)
* [Users](#users)
## Download
+
Using maven:
+
```
<dependency>
<groupId>info.debatty</groupId>
@@ -39,25 +42,28 @@ Using maven:
Or check the [releases](https://github.com/tdebatty/java-string-similarity/releases).
+This library requires Java 8 or more recent.
+
## Overview
The main characteristics of each implemented algorithm are presented below. The "cost" column gives an estimation of the computational cost to compute the similarity between two strings of length m and n respectively.
-| | | Normalized? | Metric? | Type | Cost |
-|-------- |------- |------------- |---------- | ------ | ---- |
-| [Levenshtein](#levenshtein) |distance | No | Yes | | O(m*n) <sup>1</sup> |
-| [Normalized Levenshtein](#normalized-levenshtein) |distance<br>similarity | Yes | No | | O(m*n) <sup>1</sup> |
-| [Weighted Levenshtein](#weighted-levenshtein) |distance | No | No | | O(m*n) <sup>1</sup> |
-| [Damerau-Levenshtein](#damerau-levenshtein) <sup>3</sup> |distance | No | Yes | | O(m*n) <sup>1</sup> |
-| [Optimal String Alignment](#optimal-string-alignment) <sup>3</sup> |distance | No | No | | O(m*n) <sup>1</sup> |
-| [Jaro-Winkler](#jaro-winkler) |similarity<br>distance | Yes | No | | O(m*n) |
-| [Longest Common Subsequence](#longest-common-subsequence) |distance | No | No | | O(m*n) <sup>1,2</sup> |
-| [Metric Longest Common Subsequence](#metric-longest-common-subsequence) |distance | Yes | Yes | | O(m*n) <sup>1,2</sup> |
-| [N-Gram](#n-gram) |distance | Yes | No | | O(m*n) |
-| [Q-Gram](#q-gram) |distance | No | No | Profile | O(m+n) |
-| [Cosine similarity](#cosine-similarity) |similarity<br>distance | Yes | No | Profile | O(m+n) |
-| [Jaccard index](#jaccard-index) |similarity<br>distance | Yes | Yes | Set | O(m+n) |
-| [Sorensen-Dice coefficient](#sorensen-dice-coefficient) |similarity<br>distance | Yes | No | Set | O(m+n) |
+| | | Normalized? | Metric? | Type | Cost | Typical usage |
+| -------- |------- |------------- |-------- | ------ | ---- | --- |
+| [Levenshtein](#levenshtein) |distance | No | Yes | | O(m*n) <sup>1</sup> | |
+| [Normalized Levenshtein](#normalized-levenshtein) |distance<br>similarity | Yes | No | | O(m*n) <sup>1</sup> | |
+| [Weighted Levenshtein](#weighted-levenshtein) |distance | No | No | | O(m*n) <sup>1</sup> | OCR |
+| [Damerau-Levenshtein](#damerau-levenshtein) <sup>3</sup> |distance | No | Yes | | O(m*n) <sup>1</sup> | |
+| [Optimal String Alignment](#optimal-string-alignment) <sup>3</sup> |distance | No | No | | O(m*n) <sup>1</sup> | |
+| [Jaro-Winkler](#jaro-winkler) |similarity<br>distance | Yes | No | | O(m*n) | typo correction |
+| [Longest Common Subsequence](#longest-common-subsequence) |distance | No | No | | O(m*n) <sup>1,2</sup> | diff utility, GIT reconciliation |
+| [Metric Longest Common Subsequence](#metric-longest-common-subsequence) |distance | Yes | Yes | | O(m*n) <sup>1,2</sup> | |
+| [N-Gram](#n-gram) |distance | Yes | No | | O(m*n) | |
+| [Q-Gram](#q-gram) |distance | No | No | Profile | O(m+n) | |
+| [Cosine similarity](#cosine-similarity) |similarity<br>distance | Yes | No | Profile | O(m+n) | |
+| [Jaccard index](#jaccard-index) |similarity<br>distance | Yes | Yes | Set | O(m+n) | |
+| [Sorensen-Dice coefficient](#sorensen-dice-coefficient) |similarity<br>distance | Yes | No | Set | O(m+n) | |
+| [Ratcliff-Obershelp](#ratcliff-obershelp) |similarity<br>distance | Yes | No | | ? | |
[1] In this library, Levenshtein edit distance, LCS distance and their sibblings are computed using the **dynamic programming** method, which has a cost O(m.n). For Levenshtein distance, the algorithm is sometimes called **Wagner-Fischer algorithm** ("The string-to-string correction problem", 1974). The original algorithm uses a matrix of size m x n to store the Levenshtein distance between string prefixes.
@@ -84,12 +90,12 @@ The MetricStringDistance interface : A few of the distances are actually metric
A lot of nearest-neighbor search algorithms and indexing structures rely on the triangle inequality. You can check "Similarity Search, The Metric Space Approach" by Zezula et al. for a survey. These cannot be used with non metric similarity measures.
-[Read Javadoc for a detailed description](http://api123.web-d.be/api/java-string-similarity/head/index.html)
+[Read Javadoc for a detailed description](http://www.javadoc.io/doc/info.debatty/java-string-similarity)
## Shingles (n-gram) based similarity and distance
A few algorithms work by converting strings into sets of n-grams (sequences of n characters, also sometimes called k-shingles). The similarity or distance between the strings is then the similarity or distance between the sets.
-Some ot them, like jaccard, consider strings as sets of shingles, and don't consider the number of occurences of each shingle. Others, like cosine similarity, work using what is sometimes called the profile of the strings, which takes into account the number of occurences of each shingle.
+Some of them, like jaccard, consider strings as sets of shingles, and don't consider the number of occurences of each shingle. Others, like cosine similarity, work using what is sometimes called the profile of the strings, which takes into account the number of occurences of each shingle.
For these algorithms, another use case is possible when dealing with large datasets:
1. compute the set or profile representation of all the strings
@@ -350,7 +356,7 @@ public class MyApp {
public static void main(String[] args) {
- // produces 0.416666
+ // produces 0.583333
NGram twogram = new NGram(2);
System.out.println(twogram.distance("ABCD", "ABTUIO"));
@@ -388,34 +394,32 @@ public class MyApp {
}
```
-Or, for large datasets, pre-compute the profile or set representation of all strings. The similarity can then be computed between profiles or sets:
+Or, for large datasets, pre-compute the profile of all strings. The similarity can then be computed between profiles:
```java
import info.debatty.java.stringsimilarity.KShingling;
import info.debatty.java.stringsimilarity.StringProfile;
+/**
+ * Example of computing cosine similarity with pre-computed profiles.
+ */
public class PrecomputedCosine {
- /**
- * @param args the command line arguments
- */
public static void main(String[] args) throws Exception {
String s1 = "My first string";
String s2 = "My other string...";
-
+
// Let's work with sequences of 2 characters...
- KShingling ks = new KShingling(2);
-
- // For cosine similarity I need the profile of strings
- StringProfile profile1 = ks.getProfile(s1);
- StringProfile profile2 = ks.getProfile(s2);
-
+ Cosine cosine = new Cosine(2);
+
+ // Pre-compute the profile of strings
+ Map<String, Integer> profile1 = cosine.getProfile(s1);
+ Map<String, Integer> profile2 = cosine.getProfile(s2);
+
// Prints 0.516185
- System.out.println(profile1.cosineSimilarity(profile2));
-
+ System.out.println(cosine.similarity(profile1, profile2));
}
-
}
```
@@ -437,13 +441,45 @@ Distance is computed as 1 - cosine similarity.
### Jaccard index
Like Q-Gram distance, the input strings are first converted into sets of n-grams (sequences of n characters, also called k-shingles), but this time the cardinality of each n-gram is not taken into account. Each input string is simply a set of n-grams. The Jaccard index is then computed as |V1 inter V2| / |V1 union V2|.
-Distance is computed as 1 - cosine similarity.
+Distance is computed as 1 - similarity.
Jaccard index is a metric distance.
### Sorensen-Dice coefficient
Similar to Jaccard index, but this time the similarity is computed as 2 * |V1 inter V2| / (|V1| + |V2|).
-Distance is computed as 1 - cosine similarity.
+Distance is computed as 1 - similarity.
+
+## Ratcliff-Obershelp
+Ratcliff/Obershelp Pattern Recognition, also known as Gestalt Pattern Matching, is a string-matching algorithm for determining the similarity of two strings. It was developed in 1983 by John W. Ratcliff and John A. Obershelp and published in the Dr. Dobb's Journal in July 1988
+
+Ratcliff/Obershelp computes the similarity between 2 strings, and the returned value lies in the interval [0.0, 1.0].
+
+The distance is computed as 1 - Ratcliff/Obershelp similarity.
+
+```java
+import info.debatty.java.stringsimilarity.*;
+
+public class MyApp {
+
+
+ public static void main(String[] args) {
+ RatcliffObershelp ro = new RatcliffObershelp();
+
+ // substitution of s and t
+ System.out.println(ro.similarity("My string", "My tsring"));
+
+ // substitution of s and n
+ System.out.println(ro.similarity("My string", "My ntrisg"));
+ }
+}
+```
+
+will produce:
+
+```
+0.8888888888888888
+0.7777777777777778
+```
## Experimental
@@ -467,9 +503,12 @@ public class MyApp {
}
```
-
-
## Users
* [StringSimilarity.NET](https://github.com/feature23/StringSimilarity.NET) a .NET port of java-string-similarity
+* [OrientDB string-metrics](https://github.com/orientechnologies/extra-functions/tree/master/string-metrics) wraps java-string-similarity to provide different string similarity and distance measures as SQL functions in [OrientDB](https://github.com/orientechnologies/orientdb)
Use java-string-similarity in your project and want it to be mentioned here? Don't hesitate to drop me a line!
+
+## Security & stability
+[](https://www.meterian.io/report/gh/tdebatty/java-string-similarity)
+[](https://www.meterian.io/report/gh/tdebatty/java-string-similarity)
=====================================
debian/changelog
=====================================
@@ -1,3 +1,13 @@
+java-string-similarity (2.0.0-1) unstable; urgency=medium
+
+ * New upstream version 2.0.0
+ * Bump Standards-Version to 4.5.0
+ * Use debhelper-compat 13
+ * Set "Rules-Requires-Root: no" in debian/control
+ * Freshen debian/copyright
+
+ -- tony mancill <tmancill at debian.org> Mon, 06 Jul 2020 21:08:40 -0700
+
java-string-similarity (0.24-2) unstable; urgency=medium
* Drop get-orig-source target from debian/rules
=====================================
debian/compat deleted
=====================================
@@ -1 +0,0 @@
-11
=====================================
debian/control
=====================================
@@ -3,12 +3,13 @@ Section: java
Priority: optional
Maintainer: Debian Java Maintainers <pkg-java-maintainers at lists.alioth.debian.org>
Uploaders: tony mancill <tmancill at debian.org>
-Build-Depends: debhelper (>= 11), default-jdk, maven-debian-helper (>= 2.1)
+Build-Depends: debhelper-compat (= 13), default-jdk, maven-debian-helper (>= 2.1)
Build-Depends-Indep: libjcip-annotations-java (>= 1.0), junit4
-Standards-Version: 4.3.0
+Standards-Version: 4.5.0
Vcs-Git: https://salsa.debian.org/java-team/java-string-similarity.git
Vcs-Browser: https://salsa.debian.org/java-team/java-string-similarity
Homepage: https://github.com/tdebatty/java-string-similarity
+Rules-Requires-Root: no
Package: libjava-string-similarity-java
Architecture: all
=====================================
debian/copyright
=====================================
@@ -1,9 +1,9 @@
-Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: java-string-similarity
Source: https://github.com/tdebatty/java-string-similarity
Files: *
-Copyright: 2015, Thibault "tibo" Debatty <thibault at debatty.info>
+Copyright: 2015-2020, Thibault "tibo" Debatty <thibault at debatty.info>
License: MIT/X11
Files: src/test/resources/*txt
@@ -54,7 +54,7 @@ Copyright:
2016, Thomas Wolf <thomas.wolf at paranor.ch>
Files: debian/*
-Copyright: 2016-2019, tony mancill <tmancill at debian.org>
+Copyright: 2016-2020, tony mancill <tmancill at debian.org>
License: MIT/X11
License: MIT/X11
=====================================
pom.xml
=====================================
@@ -5,7 +5,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>info.debatty</groupId>
<artifactId>java-string-similarity</artifactId>
- <version>0.24</version>
+ <version>2.0.0</version>
<packaging>jar</packaging>
<name>${project.artifactId}</name>
@@ -36,7 +36,7 @@
<connection>scm:git:git at github.com:tdebatty/java-string-similarity.git</connection>
<developerConnection>scm:git:git at github.com:tdebatty/java-string-similarity.git</developerConnection>
<url>git at github.com:tdebatty/java-string-similarity.git</url>
- <tag>v0.24</tag>
+ <tag>v2.0.0</tag>
</scm>
<distributionManagement>
@@ -82,6 +82,9 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.4</version>
+ <configuration>
+ <source>8</source>
+ </configuration>
<executions>
<execution>
<id>attach-javadocs</id>
@@ -112,8 +115,8 @@
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
- <source>1.5</source>
- <target>1.5</target>
+ <source>8</source>
+ <target>8</target>
</configuration>
</plugin>
=====================================
src/main/java/info/debatty/java/stringsimilarity/CharacterInsDelInterface.java
=====================================
@@ -0,0 +1,23 @@
+package info.debatty.java.stringsimilarity;
+
+
+/**
+ * As an adjunct to CharacterSubstitutionInterface, this interface
+ * allows you to specify the cost of deletion or insertion of a
+ * character.
+ */
+public interface CharacterInsDelInterface {
+ /**
+ * @param c The character being deleted.
+ * @return The cost to be allocated to deleting the given character,
+ * in the range [0, 1].
+ */
+ double deletionCost(char c);
+
+ /**
+ * @param c The character being inserted.
+ * @return The cost to be allocated to inserting the given character,
+ * in the range [0, 1].
+ */
+ double insertionCost(char c);
+}
=====================================
src/main/java/info/debatty/java/stringsimilarity/Cosine.java
=====================================
@@ -70,6 +70,7 @@ public class Cosine extends ShingleBased implements
* @return The cosine similarity in the range [0, 1]
* @throws NullPointerException if s1 or s2 is null.
*/
+ @Override
public final double similarity(final String s1, final String s2) {
if (s1 == null) {
throw new NullPointerException("s1 must not be null");
@@ -142,12 +143,14 @@ public class Cosine extends ShingleBased implements
* @return 1.0 - the cosine similarity in the range [0, 1]
* @throws NullPointerException if s1 or s2 is null.
*/
+ @Override
public final double distance(final String s1, final String s2) {
return 1.0 - similarity(s1, s2);
}
/**
- * {@inheritDoc}
+ * Compute similarity between precomputed profiles.
+ *
* @param profile1
* @param profile2
* @return
=====================================
src/main/java/info/debatty/java/stringsimilarity/Jaccard.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
=====================================
src/main/java/info/debatty/java/stringsimilarity/Levenshtein.java
=====================================
@@ -13,6 +13,13 @@ import net.jcip.annotations.Immutable;
@Immutable
public class Levenshtein implements MetricStringDistance {
+ /**
+ * Equivalent to distance(s1, s2, Integer.MAX_VALUE).
+ */
+ public final double distance(final String s1, final String s2) {
+ return distance(s1, s2, Integer.MAX_VALUE);
+ }
+
/**
* The Levenshtein distance, or edit distance, between two words is the
* minimum number of single-character edits (insertions, deletions or
@@ -35,10 +42,16 @@ public class Levenshtein implements MetricStringDistance {
*
* @param s1 The first string to compare.
* @param s2 The second string to compare.
+ * @param limit The maximum result to compute before stopping. This
+ * means that the calculation can terminate early if you
+ * only care about strings with a certain similarity.
+ * Set this to Integer.MAX_VALUE if you want to run the
+ * calculation to completion in every case.
* @return The computed Levenshtein distance.
* @throws NullPointerException if s1 or s2 is null.
*/
- public final double distance(final String s1, final String s2) {
+ public final double distance(final String s1, final String s2,
+ final int limit) {
if (s1 == null) {
throw new NullPointerException("s1 must not be null");
}
@@ -77,6 +90,8 @@ public class Levenshtein implements MetricStringDistance {
// edit distance is delete (i+1) chars from s to match empty t
v1[0] = i + 1;
+ int minv1 = v1[0];
+
// use formula to fill in the rest of the row
for (int j = 0; j < s2.length(); j++) {
int cost = 1;
@@ -88,6 +103,12 @@ public class Levenshtein implements MetricStringDistance {
Math.min(
v0[j + 1] + 1, // Cost of remove
v0[j] + cost)); // Cost of substitution
+
+ minv1 = Math.min(minv1, v1[j + 1]);
+ }
+
+ if (minv1 >= limit) {
+ return limit;
}
// copy v1 (current row) to v0 (previous row) for next iteration
=====================================
src/main/java/info/debatty/java/stringsimilarity/NormalizedLevenshtein.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
=====================================
src/main/java/info/debatty/java/stringsimilarity/RatcliffObershelp.java
=====================================
@@ -0,0 +1,132 @@
+/*
+ * The MIT License
+ *
+ * Copyright 2015 Thibault Debatty.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+package info.debatty.java.stringsimilarity;
+
+import info.debatty.java.stringsimilarity.interfaces.NormalizedStringSimilarity;
+import info.debatty.java.stringsimilarity.interfaces.NormalizedStringDistance;
+import java.util.List;
+import java.util.ArrayList;
+
+import net.jcip.annotations.Immutable;
+
+/**
+ * Ratcliff/Obershelp pattern recognition
+ * The Ratcliff/Obershelp algorithm computes the similarity of two strings a
+ * the doubled number of matching characters divided by the total number of
+ * characters in the two strings. Matching characters are those in the longest
+ * common subsequence plus, recursively, matching characters in the unmatched
+ * region on either side of the longest common subsequence.
+ * The Ratcliff/Obershelp distance is computed as 1 - Ratcliff/Obershelp
+ * similarity.
+ *
+ * @author Ligi https://github.com/dxpux (as a patch for fuzzystring)
+ * Ported to java from .net by denmase
+ */
+ at Immutable
+public class RatcliffObershelp implements
+ NormalizedStringSimilarity, NormalizedStringDistance {
+
+ /**
+ * Compute the Ratcliff-Obershelp similarity between strings.
+ *
+ * @param s1 The first string to compare.
+ * @param s2 The second string to compare.
+ * @return The RatcliffObershelp similarity in the range [0, 1]
+ * @throws NullPointerException if s1 or s2 is null.
+ */
+ @Override
+ public final double similarity(final String s1, final String s2) {
+ if (s1 == null) {
+ throw new NullPointerException("s1 must not be null");
+ }
+
+ if (s2 == null) {
+ throw new NullPointerException("s2 must not be null");
+ }
+
+ if (s1.equals(s2)) {
+ return 1.0d;
+ }
+
+ List<String> matches = getMatchList(s1, s2);
+ int sum_of_matches = 0;
+
+ for (String match : matches) {
+ sum_of_matches += match.length();
+ }
+
+ return 2.0d * sum_of_matches / (s1.length() + s2.length());
+ }
+
+ /**
+ * Return 1 - similarity.
+ *
+ * @param s1 The first string to compare.
+ * @param s2 The second string to compare.
+ * @return 1 - similarity
+ * @throws NullPointerException if s1 or s2 is null.
+ */
+ @Override
+ public final double distance(final String s1, final String s2) {
+ return 1.0d - similarity(s1, s2);
+ }
+
+ private static List<String> getMatchList(final String s1, final String s2) {
+ List<String> list = new ArrayList<String>();
+ String match = frontMaxMatch(s1, s2);
+
+ if (match.length() > 0) {
+ String frontsource = s1.substring(0, s1.indexOf(match));
+ String fronttarget = s2.substring(0, s2.indexOf(match));
+ List<String> frontqueue = getMatchList(frontsource, fronttarget);
+
+ String endsource = s1.substring(s1.indexOf(match) + match.length());
+ String endtarget = s2.substring(s2.indexOf(match) + match.length());
+ List<String> endqueue = getMatchList(endsource, endtarget);
+
+ list.add(match);
+ list.addAll(frontqueue);
+ list.addAll(endqueue);
+ }
+
+ return list;
+ }
+
+ private static String frontMaxMatch(final String s1, final String s2) {
+ int longest = 0;
+ String longestsubstring = "";
+
+ for (int i = 0; i < s1.length(); ++i) {
+ for (int j = i + 1; j <= s1.length(); ++j) {
+ String substring = s1.substring(i, j);
+ if (s2.contains(substring) && substring.length() > longest) {
+ longest = substring.length();
+ longestsubstring = substring;
+ }
+ }
+ }
+
+ return longestsubstring;
+ }
+}
=====================================
src/main/java/info/debatty/java/stringsimilarity/ShingleBased.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
@@ -39,7 +39,7 @@ import java.util.regex.Pattern;
* strings or documents.
*
* Generally speaking, a k-gram is any sequence of k tokens. We use here the
- * definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive
+ * definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive
* Datasets", Cambridge University Press: Multiple subsequent spaces are
* replaced by a single space, and a k-gram is a sequence of k characters.
*
@@ -51,7 +51,7 @@ import java.util.regex.Pattern;
* @author Thibault Debatty
*/
@Immutable
-abstract class ShingleBased {
+public abstract class ShingleBased {
private static final int DEFAULT_K = 3;
@@ -65,9 +65,9 @@ abstract class ShingleBased {
/**
*
* @param k
- * @throws IllegalArgumentException if k is <= 0
+ * @throws IllegalArgumentException if k is <= 0
*/
- ShingleBased(final int k) {
+ public ShingleBased(final int k) {
if (k <= 0) {
throw new IllegalArgumentException("k should be positive!");
}
@@ -86,7 +86,7 @@ abstract class ShingleBased {
*
* @return The length of k-shingles.
*/
- public int getK() {
+ public final int getK() {
return k;
}
@@ -117,5 +117,4 @@ abstract class ShingleBased {
return Collections.unmodifiableMap(shingles);
}
-
}
=====================================
src/main/java/info/debatty/java/stringsimilarity/WeightedLevenshtein.java
=====================================
@@ -36,23 +36,50 @@ import net.jcip.annotations.Immutable;
public class WeightedLevenshtein implements StringDistance {
private final CharacterSubstitutionInterface charsub;
+ private final CharacterInsDelInterface charchange;
/**
- * Instatiate with provided character substitution.
+ * Instantiate with provided character substitution.
* @param charsub The strategy to determine character substitution weights.
*/
public WeightedLevenshtein(final CharacterSubstitutionInterface charsub) {
+ this(charsub, null);
+ }
+
+ /**
+ * Instantiate with provided character substitution, insertion, and
+ * deletion weights.
+ * @param charsub The strategy to determine character substitution weights.
+ * @param charchange The strategy to determine character insertion /
+ * deletion weights.
+ */
+ public WeightedLevenshtein(final CharacterSubstitutionInterface charsub,
+ final CharacterInsDelInterface charchange) {
this.charsub = charsub;
+ this.charchange = charchange;
+ }
+
+ /**
+ * Equivalent to distance(s1, s2, Double.MAX_VALUE).
+ */
+ public final double distance(final String s1, final String s2) {
+ return distance(s1, s2, Double.MAX_VALUE);
}
/**
* Compute Levenshtein distance using provided weights for substitution.
* @param s1 The first string to compare.
* @param s2 The second string to compare.
+ * @param limit The maximum result to compute before stopping. This
+ * means that the calculation can terminate early if you
+ * only care about strings with a certain similarity.
+ * Set this to Double.MAX_VALUE if you want to run the
+ * calculation to completion in every case.
* @return The computed weighted Levenshtein distance.
* @throws NullPointerException if s1 or s2 is null.
*/
- public final double distance(final String s1, final String s2) {
+ public final double distance(final String s1, final String s2,
+ final double limit) {
if (s1 == null) {
throw new NullPointerException("s1 must not be null");
}
@@ -73,35 +100,50 @@ public class WeightedLevenshtein implements StringDistance {
return s1.length();
}
- // create two work vectors of integer distances
+ // create two work vectors of floating point (i.e. weighted) distances
double[] v0 = new double[s2.length() + 1];
double[] v1 = new double[s2.length() + 1];
double[] vtemp;
// initialize v0 (the previous row of distances)
- // this row is A[0][i]: edit distance for an empty s
- // the distance is just the number of characters to delete from t
- for (int i = 0; i < v0.length; i++) {
- v0[i] = i;
+ // this row is A[0][i]: edit distance for an empty s1
+ // the distance is the cost of inserting each character of s2
+ v0[0] = 0;
+ for (int i = 1; i < v0.length; i++) {
+ v0[i] = v0[i - 1] + insertionCost(s2.charAt(i - 1));
}
for (int i = 0; i < s1.length(); i++) {
+ char s1i = s1.charAt(i);
+ double deletion_cost = deletionCost(s1i);
+
// calculate v1 (current row distances) from the previous row v0
// first element of v1 is A[i+1][0]
- // edit distance is delete (i+1) chars from s to match empty t
- v1[0] = i + 1;
+ // Edit distance is the cost of deleting characters from s1
+ // to match empty t.
+ v1[0] = v0[0] + deletion_cost;
+
+ double minv1 = v1[0];
// use formula to fill in the rest of the row
for (int j = 0; j < s2.length(); j++) {
+ char s2j = s2.charAt(j);
double cost = 0;
- if (s1.charAt(i) != s2.charAt(j)) {
- cost = charsub.cost(s1.charAt(i), s2.charAt(j));
+ if (s1i != s2j) {
+ cost = charsub.cost(s1i, s2j);
}
+ double insertion_cost = insertionCost(s2j);
v1[j + 1] = Math.min(
- v1[j] + 1, // Cost of insertion
+ v1[j] + insertion_cost, // Cost of insertion
Math.min(
- v0[j + 1] + 1, // Cost of remove
+ v0[j + 1] + deletion_cost, // Cost of deletion
v0[j] + cost)); // Cost of substitution
+
+ minv1 = Math.min(minv1, v1[j + 1]);
+ }
+
+ if (minv1 >= limit) {
+ return limit;
}
// copy v1 (current row) to v0 (previous row) for next iteration
@@ -115,4 +157,21 @@ public class WeightedLevenshtein implements StringDistance {
return v0[s2.length()];
}
+
+
+ private double insertionCost(final char c) {
+ if (charchange == null) {
+ return 1.0;
+ } else {
+ return charchange.insertionCost(c);
+ }
+ }
+
+ private double deletionCost(final char c) {
+ if (charchange == null) {
+ return 1.0;
+ } else {
+ return charchange.deletionCost(c);
+ }
+ }
}
=====================================
src/main/java/info/debatty/java/stringsimilarity/examples/PrecomputedCosine.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
=====================================
src/main/java/info/debatty/java/stringsimilarity/experimental/Sift4.java
=====================================
@@ -30,8 +30,8 @@ import java.util.LinkedList;
* Sift4 - a general purpose string distance algorithm inspired by JaroWinkler
* and Longest Common Subsequence.
* Original JavaScript algorithm by siderite, java port by Nathan Fischer 2016.
- * https://siderite.blogspot.com/2014/11/super-fast-and-accurate-string-
- * distance.html
+ * https://siderite.dev/blog/super-fast-and-accurate-string-distance.html
+ * https://blackdoor.github.io/blog/sift4-java/
*
* @author Thibault Debatty
*/
@@ -55,8 +55,8 @@ public class Sift4 implements StringDistance {
* JaroWinkler and Longest Common Subsequence.
* Original JavaScript algorithm by siderite, java port by Nathan Fischer
* 2016.
- * https://siderite.blogspot.com/2014/11/super-fast-and-accurate-string-
- * distance.html
+ * https://siderite.dev/blog/super-fast-and-accurate-string-distance.html
+ * https://blackdoor.github.io/blog/sift4-java/
*
* @param s1
* @param s2
=====================================
src/main/java/info/debatty/java/stringsimilarity/interfaces/MetricStringDistance.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
=====================================
src/main/java/info/debatty/java/stringsimilarity/interfaces/NormalizedStringDistance.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
=====================================
src/main/java/info/debatty/java/stringsimilarity/interfaces/NormalizedStringSimilarity.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
@@ -25,7 +25,7 @@ package info.debatty.java.stringsimilarity.interfaces;
/**
*
- * @author tibo
+ * @author Thibault Debatty
*/
public interface NormalizedStringSimilarity extends StringSimilarity {
=====================================
src/main/java/info/debatty/java/stringsimilarity/interfaces/StringDistance.java
=====================================
@@ -1,7 +1,7 @@
/*
* The MIT License
*
- * Copyright 2015 tibo.
+ * Copyright 2015 Thibault Debatty.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
@@ -33,7 +33,7 @@ public interface StringDistance extends Serializable {
/**
* Compute and return a measure of distance.
- * Must be >= 0.
+ * Must be >= 0.
* @param s1
* @param s2
* @return
=====================================
src/test/java/info/debatty/java/stringsimilarity/LevenshteinTest.java
=====================================
@@ -45,6 +45,11 @@ public class LevenshteinTest {
assertEquals(2.0, instance.distance("My string", "M string2"), 0.0);
assertEquals(1.0, instance.distance("My string", "My $tring"), 0.0);
+ // With limits.
+ assertEquals(2.0, instance.distance("My string", "M string2", 4), 0.0);
+ assertEquals(2.0, instance.distance("My string", "M string2", 2), 0.0);
+ assertEquals(1.0, instance.distance("My string", "M string2", 1), 0.0);
+
NullEmptyTests.testDistance(instance);
}
}
=====================================
src/test/java/info/debatty/java/stringsimilarity/RatcliffObershelpTest.java
=====================================
@@ -0,0 +1,124 @@
+/*
+ * The MIT License
+ *
+ * Copyright 2015 Thibault Debatty.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+package info.debatty.java.stringsimilarity;
+
+import info.debatty.java.stringsimilarity.testutil.NullEmptyTests;
+import org.junit.Test;
+import static org.junit.Assert.*;
+
+/**
+ *
+ * @author Agung Nugroho
+ */
+public class RatcliffObershelpTest {
+
+
+ /**
+ * Test of similarity method, of class RatcliffObershelp.
+ */
+ @Test
+ public final void testSimilarity() {
+ System.out.println("similarity");
+ RatcliffObershelp instance = new RatcliffObershelp();
+
+ // test data from other algorithms
+ // "My string" vs "My tsring"
+ // Substrings:
+ // "ring" ==> 4, "My s" ==> 3, "s" ==> 1
+ // Ratcliff-Obershelp = 2*(sum of substrings)/(length of s1 + length of s2)
+ // = 2*(4 + 3 + 1) / (9 + 9)
+ // = 16/18
+ // = 0.888888
+ assertEquals(
+ 0.888888,
+ instance.similarity("My string", "My tsring"),
+ 0.000001);
+
+ // test data from other algorithms
+ // "My string" vs "My tsring"
+ // Substrings:
+ // "My " ==> 3, "tri" ==> 3, "g" ==> 1
+ // Ratcliff-Obershelp = 2*(sum of substrings)/(length of s1 + length of s2)
+ // = 2*(3 + 3 + 1) / (9 + 9)
+ // = 14/18
+ // = 0.777778
+ assertEquals(
+ 0.777778,
+ instance.similarity("My string", "My ntrisg"),
+ 0.000001);
+
+ // test data from essay by Ilya Ilyankou
+ // "Comparison of Jaro-Winkler and Ratcliff/Obershelp algorithms
+ // in spell check"
+ // https://ilyankou.files.wordpress.com/2015/06/ib-extended-essay.pdf
+ // p13, expected result is 0.857
+ assertEquals(
+ 0.857,
+ instance.similarity("MATEMATICA", "MATHEMATICS"),
+ 0.001);
+
+ // test data from stringmetric
+ // https://github.com/rockymadden/stringmetric
+ // expected output is 0.7368421052631579
+ assertEquals(
+ 0.736842,
+ instance.similarity("aleksander", "alexandre"),
+ 0.000001);
+
+ // test data from stringmetric
+ // https://github.com/rockymadden/stringmetric
+ // expected output is 0.6666666666666666
+ assertEquals(
+ 0.666666,
+ instance.similarity("pennsylvania", "pencilvaneya"),
+ 0.000001);
+
+ // test data from wikipedia
+ // https://en.wikipedia.org/wiki/Gestalt_Pattern_Matching
+ // expected output is 14/18 = 0.7777777777777778
+ assertEquals(
+ 0.777778,
+ instance.similarity("WIKIMEDIA", "WIKIMANIA"),
+ 0.000001);
+
+ // test data from wikipedia
+ // https://en.wikipedia.org/wiki/Gestalt_Pattern_Matching
+ // expected output is 24/40 = 0.65
+ assertEquals(
+ 0.6,
+ instance.similarity("GESTALT PATTERN MATCHING", "GESTALT PRACTICE"),
+ 0.000001);
+
+ NullEmptyTests.testSimilarity(instance);
+ }
+
+ @Test
+ public final void testDistance() {
+ RatcliffObershelp instance = new RatcliffObershelp();
+ NullEmptyTests.testDistance(instance);
+
+ // TODO: regular (non-null/empty) distance tests
+ }
+}
=====================================
src/test/java/info/debatty/java/stringsimilarity/WeightedLevenshteinTest.java
=====================================
@@ -27,6 +27,73 @@ public class WeightedLevenshteinTest {
assertEquals(0.5, instance.distance("String1", "Srring1"), 0.1);
assertEquals(1.5, instance.distance("String1", "Srring2"), 0.1);
+ // One insert or delete.
+ assertEquals(1.0, instance.distance("Strng", "String"), 0.1);
+ assertEquals(1.0, instance.distance("String", "Strng"), 0.1);
+
+ // With limits.
+ assertEquals(0.0, instance.distance("String1", "String1", Double.MAX_VALUE), 0.1);
+ assertEquals(0.0, instance.distance("String1", "String1", 2.0), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2", Double.MAX_VALUE), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2", 2.0), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2", 1.5), 0.1);
+ assertEquals(1.0, instance.distance("String1", "Srring2", 1.0), 0.1);
+ assertEquals(4.0, instance.distance("String1", "Potato", 4.0), 0.1);
+
+ NullEmptyTests.testDistance(instance);
+ }
+
+ @Test
+ public void testDistanceCharacterInsDelInterface() {
+ WeightedLevenshtein instance = new WeightedLevenshtein(
+ new CharacterSubstitutionInterface() {
+ public double cost(char c1, char c2) {
+ if (c1 == 't' && c2 == 'r') {
+ return 0.5;
+ }
+ return 1.0;
+ }
+ },
+ new CharacterInsDelInterface() {
+ public double deletionCost(char c) {
+ if (c == 'i') {
+ return 0.8;
+ }
+ return 1.0;
+ }
+
+ public double insertionCost(char c) {
+ if (c == 'i') {
+ return 0.5;
+ }
+ return 1.0;
+ }
+ });
+
+ // Same as testDistance above.
+ assertEquals(0.0, instance.distance("String1", "String1"), 0.1);
+ assertEquals(0.5, instance.distance("String1", "Srring1"), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2"), 0.1);
+
+ // Cost of insert of 'i' is less than normal, so these scores are
+ // different than testDistance above. Note that the cost of delete
+ // has been set differently than the cost of insert, so the distance
+ // call is not symmetric in its arguments if an 'i' has changed.
+ assertEquals(0.5, instance.distance("Strng", "String"), 0.1);
+ assertEquals(0.8, instance.distance("String", "Strng"), 0.1);
+ assertEquals(1.0, instance.distance("Strig", "String"), 0.1);
+ assertEquals(1.0, instance.distance("String", "Strig"), 0.1);
+
+ // Same as above with limits.
+ assertEquals(0.0, instance.distance("String1", "String1", Double.MAX_VALUE), 0.1);
+ assertEquals(0.0, instance.distance("String1", "String1", 2.0), 0.1);
+ assertEquals(0.5, instance.distance("String1", "Srring1", Double.MAX_VALUE), 0.1);
+ assertEquals(0.5, instance.distance("String1", "Srring1", 2.0), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2", 2.0), 0.1);
+ assertEquals(1.5, instance.distance("String1", "Srring2", 1.5), 0.1);
+ assertEquals(1.0, instance.distance("String1", "Srring2", 1.0), 0.1);
+ assertEquals(4.0, instance.distance("String1", "Potato", 4.0), 0.1);
+
NullEmptyTests.testDistance(instance);
}
}
\ No newline at end of file
View it on GitLab: https://salsa.debian.org/java-team/java-string-similarity/-/compare/de0d2e47a09e3292abb0a0c34dcbe89349c73dfe...8ed2a053e9ba39439ad7d7c6d0e045b6c40f66b6
--
View it on GitLab: https://salsa.debian.org/java-team/java-string-similarity/-/compare/de0d2e47a09e3292abb0a0c34dcbe89349c73dfe...8ed2a053e9ba39439ad7d7c6d0e045b6c40f66b6
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-java-commits/attachments/20200707/84234183/attachment.html>
More information about the pkg-java-commits
mailing list