Duplicate Detection

The integration of multiple data sources is a common problem in a large variety of applications. We propose a deep Siamese neural network, capable of learning a similarity measure that is tailored to the characteristics of a particular dataset. With the properties of deep learning methods, we are able to eliminate the manual feature engineering process and thus considerably reduce the effort required for model construction.

Our system uses automatically discovered MDs, various dataset features, and known gold standards to train a model that selects MDs as duplicate detection rules. Once trained, the model can select useful MDs for duplicate detection on any new dataset.

Before any important task can take place, a certain data quality has to be guaranteed, by eliminating a number of different errors that may appear in data. Typically, most of these errors are fixed with data preparation methods, such as whitespace removal. However, the particular error of duplicate records, where multiple records refer to the same entity, is usually eliminated independently with specialized techniques. Our work is the first to bring these two areas together by applying data preparation operations under a systematic approach, prior to performing duplicate detection.

In this paper, we study the problem of matching records that contain address information, including attributes such as Street-address and City. To facilitate this matching process we propose a domain-specific procedure to first enrich each record with a more complete representation of the address information through geocoding and reverse-geocoding, and second to select the best similarity measure per each address attribute, that will finally help the classifier to achieve the best f-measure.

This paper describes a first step to solve the problem of progressive search-driven Entity Resolution: resolving all the entities described by a user through a handful of keywords, progressively (according to an order by clause)

Matching financial entities (FEs) is important for many private and governmental organizations. In this paper we describe the problem of matching such FEs across three datasets: FFIEC, LEI and SEC. We were able to achieve an f-measure of 93.78% in the first task, which is comparable to the maximum 97.44%, and 70.44% for the second task, where the maximum is 88.38%.

Collaborative recommenders rely on the assumption that similar users may exhibit similar tastes while content-based ones favour items that found to be similar with the items a user likes. Weak related entities, which are often considered to be useful, are neglected by those similarity-driven recommenders. To take advantage of this neglected information, we introduce a novel dissimilarity-based recommender that bases its estimations on degrees of dissimilarities among items’ attributes.