Partial Entity Deduper!

After loong 3h of constant designing and rethinking my previous architecture choices (xD) I managed to create direct entity deduplication system.

Hold up… Direct??

Direct meaning actual duplicates/cross references. For example:

1. entity x, aliases: [y,z,c]
2. entity y, aliases: [a,b,c]
   Current deduper setup will merge these two because name y appears in aliases for entity x.

Not so fast

This is just to ilustrate how to distinguish direct from indirect. But in reality hardest part of this implementation was to figure out how to merge these two together (How properties should behave). For example:

• How confidence for given entity should be calculated?
• How to merge unique properties?
• How to preserve history of entity merging (kinda like git diffs)

Still more to come…

I still didn’t design a solution for indirect entity matching (What I will be working on now). How to figure out that entity of name: “Alice” is the same as entity named: “Dr. Alice” when neither of those is present in subject’s aliases array. :((((

There is hope!

After entity deduper is finally finished I will be able to test this system on actual tests checking how well it handles entity resolution, memory understanding etc…

Btw I changed my model to xiaomi-mimo-v2.5 but don’t tell anyone…. shhh

That’s all for today. As always…

!! THX FOR READING AND SEE YOU NEXT TIME !!