I have a dataset with pairs (and sometimes triples) of IDs that act as links in a chain.
+------+-----+
| from | to |
+------+-----+
| id1 | id2 |
| id2 | id3 |
| id4 | id5 |
+------+-----+
I am looking to organize these links into clusters or families:
+-----+----------+
| id | familyID |
+-----+----------+
| id1 | 1 |
| id2 | 1 |
| id3 | 1 |
| id4 | 2 |
| id5 | 2 |
+-----+----------+
This involves grouping all chained links into a single family and assigning an ID to each family. In the example above, the first two rows create one family, while the last row creates another family.
Solution Approach
To accomplish this task, I plan to use node.js to process large batches of data rows efficiently and insert them into a table with assigned family IDs.
Challenge Encountered
One major challenge is dealing with the high volume of ID pairs within the dataset. Additionally, new IDs will need to be added over time, which requires updating existing families with new members.
Are there effective algorithms available for clustering pairs of data into families or clusters, taking scalability into account?